DEV Community: Datawharf Tools

Why your EU TED tender search is missing notices

Datawharf Tools — Mon, 09 Nov 2026 14:05:12 +0000

If your EU TED query returns fewer notices than the TED website shows, you may not have a bug. The maintainers have explained why in a public thread, and there are simple ways to query around it.

Disclosure: I maintain the Actor mentioned at the end. The script is free and stands on its own.

What the maintainers said

In ted-open-data discussion 17, a user's SPARQL query for publication number 00305240-2025 returned nothing, although the notice was visible on the TED website. The reply from the TED team was:

The information had not yet been propagated to Cellar, their triple store, and that normally takes about one day.
Later in the thread another user reported that notices two weeks old were still missing from SPARQL. In September 2025 a maintainer replied that up to 31 July 2025 they had only transformed competition and result notices for SDK 1.3 to 1.10, and now transform all of them up to SDK 1.13. A backlog of earlier notices was being worked through, updated daily, with coverage from 25 June 2025 onwards at that time.

So two things stack up: a short propagation delay for new notices, and a backlog for older ones. I cannot tell how much of that applies today, so treat the thread as the source and check your own data.

Two ways in, two behaviours

SPARQL / open data. This is the linked-data (RDF) view of notices. It is the right tool when you want the relationships between entities. Per the maintainers, a notice appears there only after it has been transformed into the triple store.
Search API v3. A JSON POST to https://api.ted.europa.eu/v3/notices/search with an expert-search query. In my test below it needed no key, and it returned the notice from that thread (publication number 305240-2025, published 2025-05-13), the one the SPARQL query could not find. Note the number is used without the leading zeros.

I am not claiming v3 never lags, only that it found that notice. Docs: https://docs.ted.europa.eu/api/latest/search.html

How to query so you do not miss things

Use date windows, not "since last run". Query publication-date>=START AND publication-date<=END over a range.
Overlap and re-poll. Re-run the last several days each time, so anything that appears late is picked up on the next pass.
Dedupe by notice ID. Overlapping windows return the same notices again. Key your storage on publication-number.
Page with the iteration token. Set paginationMode to ITERATION and pass back iterationNextToken until a page comes back empty.
Reconcile. Compare a window's totalNoticeCount with what you stored, and alert on gaps.

Free script

Standard library only:

import datetime, json, urllib.request

URL = "https://api.ted.europa.eu/v3/notices/search"
FIELDS = ["publication-number", "publication-date", "notice-title"]


def search(query, token=None):
    body = {"query": query, "fields": FIELDS, "limit": 100, "paginationMode": "ITERATION"}
    if token:
        body["iterationNextToken"] = token
    req = urllib.request.Request(URL, data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"}, method="POST")
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.load(r)


def window(start, end, extra=""):
    q = f"publication-date>={start:%Y%m%d} AND publication-date<={end:%Y%m%d}"
    seen, token = {}, None
    while True:
        page = search(q + (f" AND {extra}" if extra else ""), token)
        for n in page["notices"]:
            seen[n["publication-number"]] = n   # dedupe by notice ID
        token = page.get("iterationNextToken")
        if not page["notices"] or not token:
            return seen


if __name__ == "__main__":
    end = datetime.date.today()
    start = end - datetime.timedelta(days=7)
    a = window(start, end, "buyer-country=DEU AND classification-cpv=72*")
    b = window(start, end, "buyer-country=DEU AND classification-cpv=72*")  # re-poll
    print(len(a), "notices", start, "to", end, "| re-poll adds", len(set(b) - set(a)))
    by_day = {}
    for n in a.values():
        d = n["publication-date"][:10]
        by_day[d] = by_day.get(d, 0) + 1
    print(dict(sorted(by_day.items())))
    for n in list(a.values())[:2]:
        t = n["notice-title"]
        print(n["publication-number"], t.get("eng") or next(iter(t.values())))

I ran it on 2026-11-08 for German buyers and IT-services CPV codes (72*) over the previous seven days:

158 notices 2026-11-01 to 2026-11-08 | re-poll adds 0
{'2026-11-02': 38, '2026-11-03': 32, '2026-11-04': 23, '2026-11-05': 28, '2026-11-06': 37}
756775-2026 Germany – Software integration consultancy services – Rahmenvertrag Anlageneinbindung MES System - PR1242292-9940
757094-2026 Germany – IT services: consulting, software development, Internet and support – Rahmenvertrag für Dienstleistungen in den Bereichen Usability, User Experience und kontinuierliche Erfolgsmessung

The second poll added nothing because I ran both back to back. The value of re-polling shows up over days. The days with no notices (1, 7 and 8 November) fall on a weekend, which I expect but did not investigate. The thread's notice also comes back when queried by number: publication-number=305240-2025 returns publication date 2025-05-13.

If you would rather not run it yourself

I publish an Apify Actor over the TED API for scheduled searches without code. It costs $0.40 per 1,000 rows, plus standard Apify platform usage. Examples are in the examples repo. If the script covers your needs, use it.

Not affiliated with or endorsed by the EU Publications Office.

Search papers, PubMed and clinical trials in one query (no API keys)

Datawharf Tools — Mon, 09 Nov 2026 13:57:29 +0000

A literature sweep usually means three tabs: Crossref for papers, PubMed for biomedical citations, ClinicalTrials.gov for studies. All three have free public APIs, and none needs a key. This post shows a short Python script that queries all three, the traps to expect, and a no-code option that returns one normalized table.

Disclosure: I maintain the Actor mentioned in the second half. The script in the first half is free and stands on its own.

What trips people up

Three response shapes. Crossref returns message.items, PubMed needs an esearch call for IDs and then esummary for details, and ClinicalTrials.gov v2 nests everything under protocolSection.
Politeness rules. Crossref wants a mailto; NCBI asks for tool and email and allows about 3 requests per second without a key.
Dates mean different things. Publication date for papers, start date for trials.
Abstracts are a copyright grey area, so this post only collects citation metadata and links.

The DIY way (free)

Standard library only. Set CONTACT_EMAIL to your own address.

import json
import os
import sys
import urllib.parse
import urllib.request

EMAIL = os.environ["CONTACT_EMAIL"]  # Crossref and NCBI ask for a contact address


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": f"research-example (mailto:{EMAIL})"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


def crossref(q, n=5):
    url = "https://api.crossref.org/works?" + urllib.parse.urlencode({"query": q, "rows": n, "mailto": EMAIL})
    return [{"source": "crossref", "id": w.get("DOI"), "title": (w.get("title") or [""])[0]}
            for w in get(url)["message"]["items"]]


def pubmed(q, n=5):
    base = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
    common = {"tool": "research-example", "email": EMAIL}
    ids = get(base + "esearch.fcgi?" + urllib.parse.urlencode(
        {**common, "db": "pubmed", "term": q, "retmax": n, "retmode": "json"}))["esearchresult"]["idlist"]
    if not ids:
        return []
    summ = get(base + "esummary.fcgi?" + urllib.parse.urlencode(
        {**common, "db": "pubmed", "id": ",".join(ids), "retmode": "json"}))["result"]
    return [{"source": "pubmed", "id": i, "title": summ[i].get("title")} for i in ids]


def trials(q, n=5):
    url = "https://clinicaltrials.gov/api/v2/studies?" + urllib.parse.urlencode({"query.term": q, "pageSize": n})
    return [{"source": "clinicaltrials", "id": s["protocolSection"]["identificationModule"]["nctId"],
             "title": s["protocolSection"]["identificationModule"].get("briefTitle")}
            for s in get(url).get("studies", [])]


if __name__ == "__main__":
    sys.stdout.reconfigure(encoding="utf-8")
    query = " ".join(sys.argv[1:]) or "CRISPR gene therapy"
    for fn in (crossref, pubmed, trials):
        for row in fn(query):
            print(row)

I ran it with CRISPR gene therapy on 2026-11-08 and got 15 rows, five per source:

Source	Example id
crossref	10.1007/978-3-030-93333-3_5
pubmed	42999773
clinicaltrials	NCT06465537

That is enough for a quick look. For deeper searches you add paging (Crossref cursors, PubMed retstart, ClinicalTrials.gov nextPageToken), retries, and filters for trial status and phase.

The scaled way: the Datawharf research-search Actor

I publish research-search on Apify. It calls the same three official APIs, paces requests to the documented limits, keeps going if one source fails, and returns one flat table with the same columns for every source (source, id, title, authors, date, url, plus doi, pmid or nctId). No abstracts, on purpose.

Python (pip install "apify-client>=3", token in APIFY_TOKEN), asking for recruiting phase 2/3 diabetes trials started since 2024:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

trials = client.actor("datawharf-tools/research-search").call(run_input={
    "mode": "clinicalTrials",
    "condition": "type 2 diabetes",
    "trialStatus": ["RECRUITING"],
    "trialPhase": ["PHASE2", "PHASE3"],
    "dateFrom": "2024-01-01",
    "maxResultsPerSource": 50,
})
for r in client.dataset(trials.default_dataset_id).iterate_items():
    print(r["nctId"], r["phase"], r["sponsor"], r["title"])

A JavaScript version and more examples are in the examples repo. Export the dataset as CSV or Excel from the run page, or put the saved input on a weekly Schedule to watch a topic.

Gotchas

all mode limit applies per source. With maxResultsPerSource: 20 you can get up to 60 rows.
Trial dates. dateFrom and dateTo filter on study start date for trials, publication date for papers.
Check the record. ClinicalTrials.gov records are sponsor-submitted and change; open the linked record before relying on it.
Not medical advice. This is a discovery tool for citations and links.

Cost

The script is free. The Actor charges $0.40 per 1,000 result rows plus standard Apify usage, so the 60-row all-mode case is about $0.024 in result charges; you can cap the charge per run.

If something looks wrong, tell me in the comments.

Disclosure: I maintain this Actor. It is not affiliated with or endorsed by Crossref, NLM, NIH or ClinicalTrials.gov.

Federal and EU tender alerts in one daily digest: SAM.gov + EU TED

Datawharf Tools — Mon, 09 Nov 2026 13:56:13 +0000

If you sell to governments on both sides of the Atlantic, you check two portals every morning: SAM.gov for US federal contracts and TED for EU public tenders. Both publish through APIs. Here is what each needs, a standard-library Python script for each, and a short script that runs both and merges them into one CSV you can put on a schedule.

Disclosure: I maintain the two Apify Actors mentioned in the second half. The scripts in the first half are free and stand on their own.

What each API wants

SAM.gov Opportunities API: needs a free API key (renew about every 90 days) and has a small daily quota, so check the table on open.gsa.gov. postedFrom and postedTo are required, formatted MM/dd/yyyy, and span at most one year. offset is a page index from 0, limit is at most 1000, and you get one NAICS code and one notice type per request.

TED Search API v3: no key. A POST with an "expert query" such as buyer-country=DEU AND classification-cpv=72* (ISO alpha-3 countries, CPV prefixes allowed). Titles and buyer names come as {language: [text]}, and page size is capped at 250.

Script 1: SAM.gov (free, needs your key)

Open solicitations under NAICS 541512 that are total small business set-asides, posted in the last 30 days. I did not run this one while writing because it needs a personal key; the request format follows the API documentation.

import os
import csv
import json
import urllib.parse
import urllib.request
from datetime import date, timedelta

API_KEY = os.environ["SAM_API_KEY"]  # your own free key; never share or commit it
BASE = "https://api.sam.gov/prod/opportunities/v2/search"


def fmt(d):
    return d.strftime("%m/%d/%Y")  # the API wants MM/dd/yyyy


def search(**params):
    params["api_key"] = API_KEY
    url = BASE + "?" + urllib.parse.urlencode(params)
    with urllib.request.urlopen(url, timeout=30) as r:
        return json.load(r)


def fetch_all(ncode, set_aside, ptype="o", days=30, page_size=1000, max_pages=3):
    today = date.today()
    base = {
        "postedFrom": fmt(today - timedelta(days=days)),  # window must be 1 year or less
        "postedTo": fmt(today),
        "ncode": ncode,
        "typeOfSetAside": set_aside,
        "ptype": ptype,
        "limit": page_size,
    }
    rows = []
    for page in range(max_pages):  # each page is one request against your daily quota
        res = search(**base, offset=page)  # offset is the page index, starting at 0
        items = res.get("opportunitiesData") or []
        rows.extend(items)
        if len(items) < page_size or len(rows) >= res.get("totalRecords", 0):
            break
    return rows


if __name__ == "__main__":
    notices = fetch_all("541512", "SBA")
    with open("opportunities.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["noticeId", "title", "agency", "posted", "deadline", "setAside", "naics", "link"])
        for o in notices:
            w.writerow([
                o.get("noticeId"), o.get("title"), o.get("fullParentPathName"),
                o.get("postedDate"), o.get("responseDeadLine"),
                o.get("typeOfSetAsideDescription"), o.get("naicsCode"), o.get("uiLink"),
            ])
    print(len(notices), "notices written")

Script 2: EU TED (free, no key)

IT-services notices (CPV branch 72) from German buyers since 1 October 2026, newest first. I ran this on 2026-11-08. The query matched 840 notices. The newest two:

Notice	Date	Buyer	Type
774927-2026	2026-11-06	IKT-Ost AoeR	can-standard
774840-2026	2026-11-06	Muehlenkreiskliniken (AoeR)	cn-standard

import csv
import json
import sys
import urllib.request

ENDPOINT = "https://api.ted.europa.eu/v3/notices/search"
FIELDS = ["publication-number", "notice-title", "buyer-name", "buyer-country",
          "classification-cpv", "notice-type", "publication-date",
          "deadline-receipt-tender-date-lot"]


def pick(value, prefer="eng"):
    """Titles and buyer names come as {language: [text, ...]}; prefer English."""
    if not isinstance(value, dict):
        return value
    for lang in (prefer, *value):
        v = value.get(lang)
        if v:
            return v[0] if isinstance(v, list) else v
    return None


def search(query, limit=10):
    body = json.dumps({"query": query, "fields": FIELDS, "limit": limit,
                       "paginationMode": "ITERATION"}).encode()
    req = urllib.request.Request(ENDPOINT, data=body, method="POST",
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


if __name__ == "__main__":
    # German buyers (DEU), CPV branch 72 (IT services), published since 1 Oct 2026, newest first
    sys.stdout.reconfigure(encoding="utf-8")
    q = ("publication-date>=20261001 AND buyer-country=DEU AND classification-cpv=72* "
         "SORT BY publication-number DESC")
    res = search(q, limit=5)
    print("total matching:", res.get("totalNoticeCount"))
    rows = []
    for n in res.get("notices", []):
        rows.append({
            "id": n["publication-number"],
            "date": (n.get("publication-date") or "")[:10],
            "title": pick(n.get("notice-title")),
            "buyer": pick(n.get("buyer-name")),
            "country": (n.get("buyer-country") or [None])[0],
            "type": n.get("notice-type"),
            "url": f"https://ted.europa.eu/en/notice/-/detail/{n['publication-number']}",
        })
    w = csv.DictWriter(sys.stdout, fieldnames=list(rows[0]) if rows else ["id"])
    w.writeheader()
    w.writerows(rows)
    # next page: pass res["iterationNextToken"] back as "iterationNextToken" in the body

The scaled way: two Apify Actors and one digest

If you would rather not maintain two clients, I publish SAM.gov Opportunities and EU TED Tenders as Apify Actors over the same official sources.

The SAM.gov Actor works without a key by reading SAM.gov's public daily extract (up to about a day old), or with your own key for the live API.
The TED Actor needs no key, accepts 2- or 3-letter country codes and picks your title language.

Python, merging both into tenders.csv (pip install "apify-client>=3", token in APIFY_TOKEN):

import csv
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])


def run(actor, run_input):
    r = client.actor(actor).call(run_input=run_input)
    return list(client.dataset(r.default_dataset_id).iterate_items())


us = run("datawharf-tools/sam-gov-opportunities",
         {"keywords": ["software"], "noticeTypes": ["o", "k"], "maxResults": 100})
eu = run("datawharf-tools/eu-ted-tenders",
         {"countries": ["DE", "FR"], "cpvCodes": ["72"], "titleLanguage": "en", "maxResults": 100})

rows = (
    [{"market": "US", "id": r["noticeId"], "title": r["title"], "buyer": r["agency"],
      "deadline": r.get("responseDeadline"), "link": r.get("samUrl")} for r in us]
    + [{"market": "EU", "id": r["noticeId"], "title": r["title"], "buyer": r["buyerName"],
        "deadline": r.get("deadline"), "link": r.get("tedUrl")} for r in eu]
)
if rows:
    with open("tenders.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0]))
        w.writeheader()
        w.writerows(rows)
print(len(rows), "tenders written")

A JavaScript version and more examples are in the examples repo. To make it an alert, save each input as an Apify Task, add a daily Schedule, and deliver new rows by webhook or the email, Slack or Google Sheets integrations.

Gotchas

The keyless SAM.gov file is not live. It is rebuilt about once a day and holds active notices only. For archived or fresher notices use the live API with your key.
TED CPV matching. A notice matches if any lot has a matching CPV, so a row's cpvCodes can include unrelated codes.
Missing fields are empty, not guessed. Award notices have no deadline, and many notices state no value.
Licence and contacts. TED content is CC BY 4.0, so keep the attribution line. SAM.gov contact details are published for bidders; follow the laws that apply to you for outreach.

Cost

The scripts are free. Both Actors charge $0.40 per 1,000 result rows plus standard Apify usage, so the merge above (at most 200 rows) is about $0.08 in result charges; you can cap the charge per run.

Questions, or something that looks wrong? Tell me in the comments.

Disclosure: I maintain these Actors. They are not affiliated with or endorsed by GSA/SAM.gov or the European Union.

SEC EDGAR revenue data looks inconsistent? It's the XBRL concept names

Datawharf Tools — Mon, 09 Nov 2026 13:53:53 +0000

If you pull revenue from the SEC's companyfacts API and the numbers look unreliable, the data is usually fine. The problem is that "revenue" is not one field. It is a handful of different us-gaap concepts, and the same period shows up in several filings.

Disclosure: I maintain the Actor mentioned at the end. The script is free and stands on its own.

Why there is no single revenue field

Every number in a 10-K is tagged with an XBRL concept from the us-gaap taxonomy. Companies choose the concept that fits their income statement, and the accounting rules and the taxonomy have changed over time. Three concepts cover most cases:

SalesRevenueNet is the older tag many companies used for net sales.
RevenueFromContractWithCustomerExcludingAssessedTax is the tag tied to the newer revenue-recognition standard (ASC 606), and many companies moved to it.
Revenues is a general total, common among banks and insurers.

A company can also change tags between years. I ran the script below against the live API, taking the first concept in my priority order for each year. Apple's annual revenue comes back as SalesRevenueNet for fiscal 2007 to 2015, Revenues for 2016, and RevenueFromContractWithCustomerExcludingAssessedTax from 2017 on. Microsoft goes Revenues (2008 to 2010), SalesRevenueNet (2011 to 2015), then the contract-revenue tag from 2016. If you hard-code one concept you lose years of history, and it looks like missing data.

Why values change between filings

Duplicate periods. A 10-K includes prior-year comparatives, so fiscal 2023 appears in the 2023, 2024 and 2025 filings. Quarterly, year-to-date and annual durations are mixed together too. Filter on form, fiscal period and a start-to-end span of about 12 months.
Restatements and recasts. A later filing can report a different value for an old period. In my run, JPMorgan's Revenues for 2014 is 94,205,000,000 in the earliest filing and 95,112,000,000 in the latest. Apple's fiscal 2008 is 32,479,000,000 first and 37,491,000,000 later. Neither is a bug. Decide whether you want as-originally-reported (earliest filing) or restated (latest).
Dimensions are not there. companyfacts returns facts without segment dimensions, so a figure you see broken out by segment on a website may not be a single fact here. Compare against the actual filing, using the accn accession number, before concluding the API is wrong.

The API also has "frames" (/api/xbrl/frames/...), which return one value per company for a single concept and period. They are convenient for cross-company comparison, but they still depend on which concept each company used, so the fallback problem remains.

The free script

Standard library only. It tries the concepts in priority order per period (so a company that switched tags keeps full history), keeps 10-K full-year values, and dedupes by period end. Pass restated=True to keep the latest filing instead of the earliest. The SEC requires a User-Agent that identifies you, so use your own contact.

import datetime, json, urllib.request

UA = "YourName your@email.com"  # use your own contact; the SEC requires one
CONCEPTS = [  # priority order
    "RevenueFromContractWithCustomerExcludingAssessedTax",
    "Revenues",
    "SalesRevenueNet",
]


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


def cik_for(ticker):
    for row in get("https://www.sec.gov/files/company_tickers.json").values():
        if row["ticker"].upper() == ticker.upper():
            return str(row["cik_str"]).zfill(10)
    raise ValueError(ticker)


def annual_revenue(ticker, restated=False):
    gaap = get(f"https://data.sec.gov/api/xbrl/companyfacts/CIK{cik_for(ticker)}.json")["facts"]["us-gaap"]
    out = {}
    for concept in CONCEPTS:  # fallback per period, not per company
        points = gaap.get(concept, {}).get("units", {}).get("USD", [])
        for p in sorted(points, key=lambda p: p["filed"], reverse=restated):
            if p["form"] != "10-K" or p["fp"] != "FY" or not p.get("start"):
                continue
            days = (datetime.date.fromisoformat(p["end"]) - datetime.date.fromisoformat(p["start"])).days
            if 350 <= days <= 380:
                out.setdefault(p["end"], (concept, p["val"], p["accn"]))
    return dict(sorted(out.items()))


if __name__ == "__main__":
    for t in ["AAPL", "MSFT", "JPM"]:
        rows = annual_revenue(t)
        used = {c for c, _, _ in rows.values()}
        print(t, len(rows), "periods; concepts used:", sorted(used))
        for end, (c, v, accn) in list(rows.items())[-3:]:
            print("  ", end, f"{v:>15,}", c[:30], accn)

Real output

I ran exactly this on 2026-11-08:

AAPL 20 periods; concepts used: ['RevenueFromContractWithCustomerExcludingAssessedTax', 'Revenues', 'SalesRevenueNet']
   2024-09-28     391,035,000,000 RevenueFromContractWithCustome 0000320193-24-000123
   2025-09-27     416,161,000,000 RevenueFromContractWithCustome 0000320193-25-000079
   2026-09-26     478,834,000,000 RevenueFromContractWithCustome 0000320193-26-000026
MSFT 19 periods; concepts used: ['RevenueFromContractWithCustomerExcludingAssessedTax', 'Revenues', 'SalesRevenueNet']
   2024-06-30     245,122,000,000 RevenueFromContractWithCustome 0000950170-24-087843
   2025-06-30     281,724,000,000 RevenueFromContractWithCustome 0000950170-25-100235
   2026-06-30     331,839,000,000 RevenueFromContractWithCustome 0001193125-26-323660
JPM 19 periods; concepts used: ['Revenues']
   2023-12-31     158,104,000,000 Revenues 0000019617-24-000225
   2024-12-31     177,556,000,000 Revenues 0000019617-25-000270
   2025-12-31     182,447,000,000 Revenues 0001628280-26-008131

Apple and Microsoft each use all three concepts over their history, while JPMorgan uses one. A single hard-coded tag would have returned a partial series for the first two.

Checking before you trust a number

Compare one or two periods to the filing itself using the accn value.
Pick one policy, earliest or latest, and apply it everywhere.
Add a short time.sleep when looping over many tickers (the SEC asks for at most 10 requests per second) and cache the ticker map, which this demo re-downloads for each ticker.
Foreign filers file 20-F or 40-F and may use the ifrs-full taxonomy, which this script does not read.

If you would rather not maintain it

I publish an Apify Actor that reads the same SEC APIs and returns one row per reported data point, with concept, period and accession fields included. It costs $0.34 per 1,000 rows, plus standard Apify platform usage. The Python and JavaScript examples are in the examples repo. If the script above is enough for you, use it.

Not affiliated with or endorsed by the SEC. Not investment advice.

SEC EDGAR financials in 5 minutes: 10-K revenue for 50 companies, no API key

Datawharf Tools — Sun, 08 Nov 2026 20:46:09 +0000

Annual revenue for public companies is free from the SEC and needs no API key. Getting it into a spreadsheet is still fiddly. Here is what the work involves, a Python script that does it for one company, and a way to do 50 at once.

Disclosure: I maintain the Actor mentioned in the second half. The script in the first half is free and stands on its own.

The problem

data.sec.gov has everything you need, but a few things get in the way:

Ticker to CIK. The API is keyed by a 10-digit, zero-padded CIK, not a ticker. The SEC publishes a mapping file you have to fetch and search.
Nested XBRL. The companyfacts endpoint returns one JSON document per company, with every reported fact nested under taxonomy, concept and unit. For big filers that is tens of MB.
Concept names change. Apple reports revenue as RevenueFromContractWithCustomerExcludingAssessedTax. Other companies, or the same company in older years, use Revenues or SalesRevenueNet. There is no single "revenue" field.
Repeated facts. A 10-K includes prior-year comparatives, so the same period shows up in several filings.
Fair access. The SEC asks for at most 10 requests per second per address, and wants a User-Agent that identifies you (name and email).

The DIY way (free)

This uses only the standard library. It maps the ticker to a CIK, downloads the company facts, tries revenue concepts in order, keeps only full-year values reported on a 10-K, and de-duplicates by period (earliest filing wins). Put your own contact in UA.

import json
import urllib.request

UA = "YourName your@email.com"  # the SEC requires an identifying contact
REVENUE_CONCEPTS = [
    "RevenueFromContractWithCustomerExcludingAssessedTax",
    "Revenues",
    "SalesRevenueNet",
]


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


def cik_for(ticker):
    data = get("https://www.sec.gov/files/company_tickers.json")
    for row in data.values():
        if row["ticker"].upper() == ticker.upper():
            return str(row["cik_str"]).zfill(10)
    raise ValueError(f"unknown ticker {ticker}")


def annual_revenue(ticker):
    cik = cik_for(ticker)
    facts = get(f"https://data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json")
    gaap = facts["facts"]["us-gaap"]
    for concept in REVENUE_CONCEPTS:  # first concept the company reports
        if concept not in gaap:
            continue
        rows, seen = [], set()
        for p in gaap[concept]["units"]["USD"]:
            if p["form"] != "10-K" or p["fp"] != "FY":
                continue
            if not p.get("start") or not p["end"]:
                continue
            # keep ~12-month periods only (10-Ks also repeat prior years)
            days = (_d(p["end"]) - _d(p["start"])).days
            if not 350 <= days <= 380:
                continue
            if p["accn"] in seen and any(r["end"] == p["end"] for r in rows):
                continue
            seen.add(p["accn"])
            rows.append({"ticker": ticker, "concept": concept, "end": p["end"],
                         "value": p["val"], "accn": p["accn"], "filed": p["filed"]})
        # same period appears in later filings; keep the earliest accession per period
        best = {}
        for r in sorted(rows, key=lambda r: r["filed"]):
            best.setdefault(r["end"], r)
        if best:
            return sorted(best.values(), key=lambda r: r["end"])
    return []


def _d(s):
    import datetime
    return datetime.date.fromisoformat(s)


if __name__ == "__main__":
    for r in annual_revenue("AAPL")[-5:]:
        print(r)

I ran it against AAPL on 2026-11-08. Two requests, and the last five fiscal years:

Fiscal year end	Revenue (USD)
2022-09-24	394.3B
2023-09-30	383.3B
2024-09-28	391.0B
2025-09-27	416.2B
2026-09-26	478.8B

For 50 tickers, loop over it with a short time.sleep between companies (stay well under 10 requests per second) and cache the ticker map so you fetch it once, not 50 times. Write the rows with csv.DictWriter and you have your spreadsheet. If that is all you need, stop here.

The scaled way: the Datawharf SEC EDGAR Actor

If you would rather not maintain the script, handle retries, or schedule it yourself, I publish an Apify Actor that reads the same official data.sec.gov APIs. No SEC key. It does the ticker-to-CIK lookup, takes up to 500 companies per run, paces itself at 5 requests per second per run (slowing down if the SEC pushes back) and retries.

Input for annual revenue (list up to 500 tickers in companies):

{
  "mode": "companyFacts",
  "companies": ["AAPL", "MSFT", "GOOGL"],
  "concepts": ["RevenueFromContractWithCustomerExcludingAssessedTax", "Revenues", "SalesRevenueNet"],
  "forms": ["10-K"],
  "fiscalPeriod": "FY",
  "userAgent": "YourName your@email.com"
}

Download the results as CSV or Excel, push them to Google Sheets with Apify's integration, or put the run on a quarterly Schedule.

From your own Python code (pip install "apify-client>=3"):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("datawharf-tools/sec-edgar-api").call(run_input={
    "mode": "companyFacts",
    "companies": ["AAPL", "MSFT", "GOOGL"],
    "concepts": ["RevenueFromContractWithCustomerExcludingAssessedTax", "Revenues"],
    "forms": ["10-K"],
    "fiscalPeriod": "FY",
    "userAgent": "YourName your@email.com",
    "maxResultsPerCompany": 6,
})
for r in client.dataset(run.default_dataset_id).iterate_items():
    print(r["ticker"], r["periodEnd"], r["value"])

A JavaScript version and more examples are in the examples repo.

Gotchas

Use the period fields. fiscalYear labels the filing, not the period. Use periodStart and periodEnd.
Duplicates are normal. Comparatives repeat across filings. Dedupe on period end (keep the latest filing if you want restated figures).
Amendments and foreign filers. 10-K does not match 10-K/A; foreign issuers file 20-F/40-F, often under ifrs-full.
Not investment advice. Values are as published by the SEC.

Cost

The script is free. The Actor charges about $0.34 per 1,000 rows plus standard Apify usage, so a 50-company, six-year revenue table (about 300 rows) is roughly $0.10 in result charges. You can cap the charge per run.

If something looks wrong in either route, tell me in the comments.

Disclosure: I maintain this Actor. It is not affiliated with or endorsed by the SEC.