DEV Community: Steadfetch

Web data that doesn't break: five official public sources your agent can monitor

Steadfetch — Fri, 23 Oct 2026 05:02:20 +0000

Web data that doesn't break: five official public sources your agent can monitor

If you are building an agent or an alerting workflow, the least glamorous part is usually the data
layer: parsing HTML that changes, chasing rate limits, and debugging a broken selector at 2 a.m.
For a surprising share of "web data" needs, you do not need to scrape arbitrary web pages at all.
Governments publish machine-readable APIs for the records people build products on.

Here are five official sources — and ready-made Actors that deliver their records as
schema-validated JSON, pay per result.

1. SEC EDGAR filings (US)

Track filings by issuer CIK or full-text query: accession number, form type, dates and primary
document URL. Use cases: filing alerts, compliance checks, market research.
→ SEC EDGAR Filings Data Scraper

2. UK Companies House records (UK)

Official company records as clean JSON: status, type, SIC codes, incorporation date,
accounts-overdue flag and registered address. Use cases: lead lists, KYB, compliance monitoring.
Company fields only — no officer or PSC data. Bring your own free Companies House API key.
→ UK Companies House Data Scraper

3. EU procurement notices (TED)

Monitor public procurement notices from Tenders Electronic Daily: buyer organisation, country,
title, procedure, estimated value, deadline and CPV codes. Use cases: bid sourcing, public-sector
pipelines, procurement analysis.
→ EU Tenders Scraper — TED Procurement Notices

4. UK food hygiene ratings (FSA FHRS)

Food establishment hygiene ratings from the Food Standards Agency open-data API: rating, scheme,
local authority and business details, filterable by authority and minimum rating. Use cases:
food-safety due diligence, site checks, compliance dashboards.
→ UK Food Hygiene Ratings Scraper

5. US CPSC product recalls

Official recall notices from the public SaferProducts API: recall dates, products, hazards,
remedies and involved organisations. Use cases: product-safety monitoring, compliance alerts,
marketplace risk checks.
→ US CPSC Product Recalls Scraper

Why official APIs beat HTML scraping

Stability. A documented API changes less often than a page layout, and when it does, the change is announced rather than discovered by a broken selector.
Clean provenance. Every record carries the official identifiers (accession numbers, CIK, recall IDs) that let you cite and audit what your agent used.
Predictable cost. Pay per result means empty searches do not burn budget.

How we keep it from breaking

Every result is checked against the Actor's published dataset schema, and Actors are re-tested daily —
that is the whole promise behind Web data that doesn't break. Public data only: no logins, no
paywalls, no personal data.

Start with one source, assert on the schema of every record you store, and keep the official record
URL next to each claim your agent makes.

— Steadfetch, Apify Store

How to Track SEC EDGAR Filings as JSON (Without Maintaining a Scraper)

Steadfetch — Thu, 22 Oct 2026 04:59:14 +0000

How to Track SEC EDGAR Filings as JSON (Without Maintaining a Scraper)

If you are building filing alerts, compliance checks or equity-research workflows, SEC EDGAR is the
source: every filing from every US issuer, published minutes after acceptance. The official EDGAR
APIs are free and return JSON — but wiring them up yourself still means declaring a User-Agent the
SEC accepts, staying under its published limit of 10 requests per second, paginating full-text
search results, and handling fields that one endpoint returns while another does not.

Here is how to get the data in two ways, with a real sample of the output.

The official API, in a nutshell

Endpoints: the EDGAR application programming interfaces (submissions by CIK, filing metadata, full-text search).
No API key: EDGAR's endpoints are anonymous.
Politeness rules: send a declared User-Agent with your contact information and stay under the published limit of 10 requests per second; requests that ignore this are rejected.
Scope: filing metadata and issuer organisations only. There is no officer or contact data here — personal data stays out of scope, by design.

Option A — call the official API directly

curl -sS -A "your-app your@email.com" \
  "https://data.sec.gov/submissions/CIK0000320193.json" | jq '.filings.recent | .accessionNumber[0:3]'

That works fine for one issuer. The moment you watch a shortlist of companies or run a full-text
search, you are maintaining pagination, date filtering and field reconciliation for every endpoint
you touch.

Option B — run a ready-made Actor

steadfetch/sec-edgar-monitor wraps those endpoints on the Apify platform: it handles the declared
User-Agent, the 10 req/s budget, pagination, retries and an output schema check on every run. No SEC
key needed (an optional proxy of your own token keeps your run budget isolated).

Input — by issuer, or by full-text query:

{ "cik": "0000320193", "formTypes": ["10-K", "10-Q", "8-K"], "startDate": "2026-01-01", "maxResults": 10 }

{ "query": "supply agreement", "startDate": "2026-10-01", "endDate": "2026-10-17", "maxResults": 25 }

At least one of cik or query is required.

Run it with one API call:

curl -sS -X POST \
  "https://api.apify.com/v2/acts/steadfetch~sec-edgar-monitor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"cik":"0000320193","formTypes":["10-Q"],"startDate":"2026-07-01","maxResults":5}'

Real output sample (a genuine filing record, as published in the Actor's docs):

{
    "accession_no": "0000320193-26-000020",
    "cik": "0000320193",
    "company_name": "Apple Inc.",
    "form_type": "10-Q",
    "filing_date": "2026-07-31",
    "report_date": "2026-06-27",
    "acceptance_datetime": "2026-07-31T14:01:02.000Z",
    "primary_document_url": "https://www.sec.gov/Archives/edgar/data/320193/000032019326000020/aapl-20260627.htm",
    "is_amendment": false,
    "url": "https://www.sec.gov/Archives/edgar/data/320193/000032019326000020/aapl-20260627.htm",
    "scrapedAt": "2026-10-19T12:00:00.000Z"
}

One dataset item per filing, validated against the published dataset schema on every run — if a field
type changes upstream, the run fails loudly instead of writing bad rows.

JavaScript and Python

Node.js, with apify-client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const { defaultDatasetId } = await client.actor('steadfetch/sec-edgar-monitor').call({
  cik: '0000320193',
  formTypes: ['10-K', '10-Q', '8-K'],
  startDate: '2026-01-01',
  maxResults: 10,
});

const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(JSON.stringify(items, null, 2));

Python, with apify-client:

import json, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("steadfetch/sec-edgar-monitor").call(run_input={
    "cik": "0000320193", "formTypes": ["10-K", "10-Q", "8-K"],
    "startDate": "2026-01-01", "maxResults": 10,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(json.dumps(items, indent=2))

Runnable versions of all three (curl, Node, Python) plus the sample output live in
steadfetch/examples/sec-edgar-monitor.

What it costs

Pay-per-result pricing on Apify: 5.00 USD per 1,000 filings (0.005 USD per filing) plus a
0.00005 USD run-start fee. Failed requests, retries and empty searches are not charged per result,
and no proxy is billed — EDGAR answers with structured JSON, so no browser is involved.

Fields you get

Field	Meaning
`accession_no`	Official accession number of the filing
`cik` / `company_name`	Issuer identity
`form_type`	10-K, 10-Q, 8-K, and the rest of the EDGAR form taxonomy
`filing_date` / `report_date`	When it was filed and the period it covers
`acceptance_datetime`	EDGAR acceptance timestamp — `null` when the full-text endpoint omits it; never invented
`primary_document_url` / `url`	Direct link to the filing document
`is_amendment`	Whether the filing amends an earlier one
`scrapedAt`	Extraction timestamp, on every record

Getting started

Open the Actor page in the Apify Store, run it
from the console or with the API snippets above, and pull the dataset into whatever you already use.

Steadfetch builds public-data Actors that don't break: schema-validated output, official sources
only. More at apify.com/steadfetch.

How to Get UK Companies House Data as JSON (Without Maintaining a Scraper)

Steadfetch — Tue, 20 Oct 2026 23:18:31 +0000

How to Get UK Companies House Data as JSON (Without Maintaining a Scraper)

If you are building KYB checks, lead lists or compliance monitoring for UK companies, you need the
official register fields: status, type, SIC codes, incorporation date, accounts-overdue flag and
registered office address. Companies House publishes exactly that as JSON — but wiring up the official
API yourself still means a key, pagination, retries and a rate limit you must not trip.

Here is how to get the data in two ways, with a real sample of the output.

The official API, in a nutshell

Base URL: https://api.company-information.service.gov.uk/
Free API key: https://developer.company-information.service.gov.uk/
Rate limits: one request per 550 ms and 600 requests per 5 minutes per key; 429 responses carry a Retry-After header worth honouring.
Scope: company-level fields only. The officer and PSC endpoints return personal data — if you do not have a lawful basis for personal data, do not call them (GDPR). The examples below never do.

Option A — call the official API directly

curl -sS -u "$CH_API_KEY:" \
  "https://api.company-information.service.gov.uk/company/00063903" | jq .

That works fine for one company. The moment you search the register by name, you are maintaining
pagination, backoff and schema handling for every field you store.

Option B — run a ready-made Actor

steadfetch/companies-house-monitor wraps that API on the Apify platform: it handles pagination,
the 550 ms / 600-per-5-min rate limits, retries and an output schema check on every run. You need no
Companies House key of your own (an optional apiKey input lets you use your own if you want an
isolated rate-limit budget).

Input — one company, or a register search:

{ "mode": "company", "companyNumber": "00063903", "maxResults": 10 }

{ "mode": "search", "query": "tesla", "maxResults": 100, "since": "2025-01-01" }

Run it with one API call:

curl -sS -X POST \
  "https://api.apify.com/v2/acts/steadfetch~companies-house-monitor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"company","companyNumber":"00063903","maxResults":10}'

Real output sample (a genuine register record, as published in the Actor's docs):

{
    "url": "https://find-and-update.company-information.service.gov.uk/company/08209948",
    "title": "! LTD",
    "scrapedAt": "2026-10-19T23:45:44.207Z",
    "company_number": "08209948",
    "company_name": "! LTD",
    "company_status": "active",
    "company_type": "ltd",
    "sic_codes": ["99999"],
    "incorporated_on": "2012-09-11",
    "accounts_overdue": false,
    "address": {
        "address_line_1": "9 PRINCES SQUARE",
        "locality": "HARROGATE",
        "postal_code": "HG1 1ND",
        "country": "England"
    }
}

One dataset item per company, validated against the published dataset schema on every run — if a field
type changes upstream, the run fails loudly instead of writing bad rows (strictSchema defaults to
true).

JavaScript and Python

Node.js, with apify-client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const { defaultDatasetId } = await client.actor('steadfetch/companies-house-monitor').call({
  mode: 'search',
  query: 'tesla',
  since: '2025-01-01',
});

const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(JSON.stringify(items, null, 2));

Python, with apify-client:

import json, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("steadfetch/companies-house-monitor").call(run_input={
    "mode": "search", "query": "tesla", "since": "2025-01-01",
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(json.dumps(items, indent=2))

Runnable versions of all three (curl, Node, Python) plus the sample output live in
steadfetch/examples/companies-house-monitor.

What it costs

Pay-per-result pricing on Apify: 2.50 USD per 1,000 valid results (0.0025 USD per company) plus a
0.00005 USD run-start fee. Failed requests, retries and empty searches are not charged per result, and
no proxy is billed — Companies House answers with structured JSON, so no browser is involved.

Fields you get

Field	Meaning
`company_number` / `company_name` / `company_status`	Register identity and state
`company_type`, `sic_codes`	Legal form and activity classification
`incorporated_on`, `accounts_overdue`	Age and filing health signals
`address`	Registered office (business address, not personal)
`url`, `scrapedAt`	Source link and extraction timestamp, on every record

No officers, no PSCs, no personal data — by design.

Getting started

Open the Actor page in the Apify Store, run it
from the console or with the API snippets above, and pull the dataset into whatever you already use.

Steadfetch builds public-data Actors that don't break: schema-validated output, official sources only.
More at apify.com/steadfetch.

Update — 22 October 2026

Thanks to reader @veristria for the caution on SIC codes in lead lists: they are self-declared and
often stale. That is exactly why this output keeps company_status and accounts_overdue right next
to sic_codes. On a lead list, active status plus no overdue accounts is the stronger filter, and SIC
works better as a soft signal on top; dormant shells tend to reveal themselves through status and
filing health long before anyone updates their SIC codes.

Clarification from the Steadfetch team — happy to take questions on the pipeline below.