DEV Community: Stefano Seggio

How We Built a Zero-Cost Unchanged Run SEC Enforcement API

Stefano Seggio — Wed, 16 Sep 2026 14:56:37 +0000

SEC.gov publishes two feeds that matter a lot to anyone tracking enforcement
risk: litigation releases and administrative proceedings. Neither has an
API. Both are static web pages you're expected to refresh by hand, and
neither one tells you the actual EDGAR-registered company behind the
respondent's name — just whatever name the release itself used.

This post walks through how I built a small, pay-per-event Actor that turns
both feeds into a structured, delta-tracked stream — and specifically, the
three design decisions that mattered most: fingerprint-based delta tracking,
automatic EDGAR CIK resolution, and honest failure classification.

The core problem: re-scraping is wasteful, and identity resolution is manual

If you scrape SEC.gov's litigation-release page on a schedule, the naive
approach re-pulls and re-processes the entire feed every run. Most of that
feed hasn't changed since your last run. You end up paying (in compute, in
API costs, in your own attention re-reading output) for a mountain of
"nothing happened" noise around a handful of genuinely new records.

Worse: the release only ever names a respondent the way the SEC wrote it —
"Example Capital Management LLC." If your downstream system needs the
EDGAR CIK to actually join this against a company you have on file, that's
manual cross-referencing, every single time, for every single release.

Architecture

[ SEC.gov Releases ] ──┐
                       ├──> [ Canonicalize Record ] ──> [ Fingerprint Hash ]
[ SEC.gov Proceedings ] ┘                                       │
                                                                ▼
                                                       / Seen Before? \
                                                      /                \
                                                    YES                NO
                                                     │                  │
                                                     ▼                  ▼
                                            [ Drop - $0.00 Billed ]   [ EDGAR CIK Match ]
                                                                        │
                                                                        ▼
                                                              [ Parse Sanctions ]
                                                                        │
                                                                        ▼
                                                              [ Persist New Hash ]

1. Fingerprint-based delta tracking

Every record — release or proceeding — gets canonicalized (consistent field
ordering, whitespace normalization, date formatting) and hashed. That hash
is compared against a persisted state store from the previous run. If the
hash matches, the record is dropped before it ever reaches the billing
layer. Apify's Pay-Per-Event model means an all-unchanged run genuinely
costs the caller $0.00 — the fingerprinting isn't just an optimization, it's
the actual pricing mechanism.

import requests

response = requests.post(
    "https://api.apify.com/v2/acts/EDhT9Mvrdm2hzTECA/run-sync-get-dataset-items",
    params={"token": "<YOUR_API_TOKEN>"},
    json={"userAgent": "YourCompany your-email@example.com", "maxItemsPerRun": 50, "onlyNew": True},
)
records = response.json()
print(f"{len(records)} records returned")

Run this twice in a row against an unchanged feed and the second call
returns an empty array — and bills nothing.

2. Automatic EDGAR CIK resolution

This is the part that actually took the most engineering time. Each
respondent name gets cross-referenced against EDGAR's own filer database,
including known former company names (mergers, rebrands), and the match
comes back with a numeric confidence score rather than a blind first-result
match:

{
  "record_id": "LR-26636",
  "event_type": "NEW_LISTING",
  "primary_respondent": "Example Capital Management LLC",
  "linked_edgar_cik": "0001234567",
  "edgar_match_confidence": 0.94,
  "monetary_sanctions": { "sought_usd": 450000, "ordered_usd": 310000 }
}

The confidence score matters because respondent-name matching against a
public filer database is inherently fuzzy — surfacing the score instead of
hiding it behind a boolean lets a downstream consumer decide their own
threshold for "trust this match automatically" vs. "flag for manual
review."

3. Honest failure classification

This lesson came from a sibling project, not this one. A monitor I built
for the UK's Health and Safety Executive register hit a real multi-day
upstream outage — HSE's own servers were unreachable at the network level.
Without explicit handling, that outage would have surfaced as a generic
"run failed" error, indistinguishable from an actual bug in my own
scraping logic. That's a genuinely bad failure mode for anyone depending on
the feed: you can't tell "wait it out" from "file a bug report" from the
error alone.

So this Actor (and now the rest of the fleet) explicitly classifies a
network-level failure to reach the source as an upstream outage, tagged
distinctly from an internal parsing or logic error. It sounds like a small
thing until you're the one staring at a red run and trying to decide
whether to open an issue.

Pricing, for context

New enforcement release: $0.05
Correction to a previously-delivered release (rare — these are close to immutable once published): $0.02
Unchanged release: $0.00

Try it

curl -X POST "https://api.apify.com/v2/acts/EDhT9Mvrdm2hzTECA/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"userAgent": "YourCompany your-email@example.com", "maxItemsPerRun": 50, "onlyNew": true}'

Source is on GitHub:
stefanoseggio/sec-enforcement-litigation-delta-feed.
The hosted Actor is on the Apify Store
if you'd rather not run it yourself.

If you're solving a similar "no first-party API, entity resolution
required" problem against a different government data source, I'd like to
hear how you approached it — the delta-fingerprinting pattern here
generalizes pretty directly to most of these feeds.

Resilient HTTP Retry Strategies: 3 Real Bug Patterns Found Auditing Retry Logic Across 18 Scrapers

Stefano Seggio — Fri, 11 Sep 2026 13:10:15 +0000

TL;DR: Three real retry-logic bug patterns across an 18-actor fleet — 429 excluded from retry entirely (critical, fixed on one actor), permanent errors retried pointlessly (fixed on two, still present elsewhere), and Retry-After ignored (fixed on a minority). Full before/after code for all three, plus an honest accounting of how far each fix has actually propagated.

Retry logic looks simple until you actually audit it

"Retry on failure with exponential backoff" is one of those patterns everyone thinks they've implemented correctly. The happy path (a transient failure, a retry, success) is easy to write and easy to eyeball-verify. The bugs live in the edge cases nobody eyeballs: which status codes actually deserve a retry, whether the server's own Retry-After header gets honored, and whether a genuinely permanent failure gets retried anyway and wastes your retry budget.

Auditing retry logic across a fleet of 18 Apify Actors surfaced three real bug patterns — one critical, two lower-severity but real. None of the three were fixed everywhere in one pass; here's each one, with the actual before/after code, and an honest read on how far the fix has actually spread across the fleet so far.

Bug 1 (critical): HTTP 429 excluded from retry entirely

Found on this fleet's drug-safety-recalls monitor. The original code:

if (error instanceof HttpError && error.status >= 400 && error.status < 500) {
    throw error; // don't retry ANY 4xx
}

This looks reasonable at a glance — "don't retry client errors" is a defensible general rule. The problem: 429 (Too Many Requests) is a 4xx status code, and it is exactly the case retry logic exists for. A rate-limited request under this code failed immediately instead of backing off. That's the opposite of the intended behavior, and it's happening on the one status code where retrying is unambiguously correct.

The fix: carve 429 out explicitly, alongside 5xx, before the generic 4xx-skip rule applies:

function isRetryableStatus(status: number): boolean {
    return status === 429 || status >= 500;
}

// ... in the retry loop:
if (error instanceof HttpError && !isRetryableStatus(error.status)) {
    throw error; // genuinely permanent - 400, 401, 403, 404, etc.
}
// 429 and 5xx fall through to the backoff-and-retry path below

Bug 2: generic "retry any non-ok status" wastes budget on permanent errors

Two other actors' retry logic had the opposite problem: they retried on any non-ok response, including a permanent 401 (bad credentials) or 404 (not found) — errors that will never succeed on retry no matter how many times you ask. Not a correctness bug (nothing gets silently dropped), but a real inefficiency: every retry attempt against a permanent error burns time and, on a rate-limited API, actual quota.

The fix, applied to both: a typed error class that distinguishes a genuine HTTP-status failure (retryable only if 429/5xx) from a network-level failure (DNS, connection reset — always worth retrying, since those are inherently transient):

class HttpError extends Error {
    constructor(message: string, public readonly status: number) {
        super(message);
        this.name = 'HttpError';
    }
}

// in the catch block:
if (error instanceof HttpError) throw error; // already decided above whether this was retryable
// only a genuine network-level failure reaches here
if (attempt < maxRetries) await sleep(baseDelayMs * 2 ** attempt);

Bug 3: `Retry-After` never read

This is the one still genuinely in progress. Most of the fleet's hand-written retry logic uses a fixed exponential backoff on 429/503 without ever checking whether the server told you exactly how long to wait. RFC 9110 defines Retry-After for exactly this — either a delay in seconds or an HTTP-date — and ignoring it means guessing at a delay the server already told you. As of this writing it's implemented on a minority of the fleet's actors; the rest still guess.

function parseRetryAfterMs(headerValue: string | null): number | null {
    if (!headerValue) return null;
    const seconds = Number(headerValue);
    if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
    const dateMs = Date.parse(headerValue);
    if (!Number.isNaN(dateMs)) return Math.max(0, dateMs - Date.now());
    return null;
}

// Retry-After is authoritative when present - the server told you exactly
// how long to wait, so a shorter computed backoff would just get
// rate-limited again.
const delayMs = error.retryAfterMs !== null ? error.retryAfterMs : computedBackoffMs;

The real fix requires jitter too, if you might run concurrently

Pure exponential backoff with no randomization means every concurrent instance retrying the same failure retries at the exact same intervals. That's a thundering herd against the very API you're trying to be polite to. A small random jitter fraction added to the computed backoff (not to a server-provided Retry-After, which is authoritative) fixes this cheaply:

const backoffMs = baseDelayMs * 2 ** attempt;
const jitterMs = Math.random() * backoffMs * 0.25;
const delayMs = retryAfterMs ?? backoffMs + jitterMs;

How it all fits together

The three fixes above are decisions made at different points of the same retry path. Composed into one flow, the mechanism looks like this: a network-level failure (no HTTP response) is always retryable, an HTTP-level failure is retryable only per isRetryableStatus, and the actual delay prefers a server's Retry-After over a computed, jittered backoff.

graph TD
    A[Request fails] --> B{Network-level failure?<br/>DNS, connection reset}
    B -->|Yes| C[Always retryable]
    B -->|No - got an HTTP response| D{isRetryableStatus?<br/>429 or 5xx}
    D -->|No - 400/401/403/404 etc.| E[Throw - permanent, don't retry]
    D -->|Yes| C
    C --> F{Retry-After header present?}
    F -->|Yes| G[Wait exactly that long<br/>authoritative, no jitter]
    F -->|No| H[Wait computed backoff + jitter]
    G --> I[Retry the request]
    H --> I

Python: the same retry logic via `apify-client`

The bugs above are HTTP-level decisions, not framework-specific ones. The same isRetryableStatus / Retry-After / jittered-backoff logic applies whether the caller is Node.js or Python. Here's the identical decision logic translated to Python, wrapped around a real apify-client call (start an Actor run, then iterate its dataset):

import random
import time
from typing import Optional

from apify_client import ApifyClient


class HttpError(Exception):
    """Mirrors the TypeScript HttpError: a genuine HTTP-status failure."""
    def __init__(self, message: str, status: int):
        super().__init__(message)
        self.status = status


def is_retryable_status(status: int) -> bool:
    # Same rule as Bug 1's fix: 429 is retryable, alongside 5xx.
    # A bare 4xx-only exclusion (the original bug) would wrongly
    # throw on 429 instead of backing off.
    return status == 429 or status >= 500


def parse_retry_after_ms(header_value: Optional[str]) -> Optional[int]:
    if not header_value:
        return None
    try:
        seconds = float(header_value)
        return max(0, int(seconds * 1000))
    except ValueError:
        pass
    try:
        from email.utils import parsedate_to_datetime
        date = parsedate_to_datetime(header_value)
        delay_ms = (date.timestamp() * 1000) - (time.time() * 1000)
        return max(0, int(delay_ms))
    except (TypeError, ValueError):
        return None


def call_actor_with_retry(
    client: ApifyClient,
    actor_id: str,
    run_input: dict,
    max_retries: int = 5,
    base_delay_ms: int = 1000,
):
    """Same retry/backoff/jitter logic as the TypeScript version, applied
    around an apify-client Actor call instead of a raw HTTP request."""
    attempt = 0
    while True:
        try:
            return client.actor(actor_id).call(run_input=run_input)
        except HttpError as error:
            if not is_retryable_status(error.status):
                raise  # genuinely permanent - 400, 401, 403, 404, etc.

            retry_after_ms = parse_retry_after_ms(
                getattr(error, "retry_after_header", None)
            )
            if retry_after_ms is not None:
                delay_ms = retry_after_ms  # authoritative, no jitter
            else:
                backoff_ms = base_delay_ms * (2 ** attempt)
                jitter_ms = random.random() * backoff_ms * 0.25
                delay_ms = backoff_ms + jitter_ms

            if attempt >= max_retries:
                raise
            time.sleep(delay_ms / 1000)
            attempt += 1


# Usage: start the run (with the same retry/backoff/jitter protection
# as the Node.js version), then iterate the resulting dataset.
client = ApifyClient("<APIFY_API_TOKEN>")
run = call_actor_with_retry(
    client,
    actor_id="<ACTOR_ID>",
    run_input={"startUrls": [{"url": "https://example.com"}]},
)

# run.default_dataset_id, not run["defaultDatasetId"] - as of apify-client
# v3, .call() returns a typed object, not a dict. Dataset items themselves
# are unaffected - iterate_items() still yields plain dicts.
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

How this got caught

None of these three bugs were caught by a green local test suite alone — they needed either a direct code re-read against the exact failure case (bug 1, caught by tracing through what statuses the exclusion condition actually matched), or a deliberate mocked-response test asserting the retry behavior for each specific status code (429 retries, 503 retries, 401 doesn't) rather than just testing the happy path. Being candid about where this stands: finding a pattern on one actor doesn't mean it's fixed everywhere. Bug 1 was a one-actor fix. Bug 2 has now been fixed on two actors and is confirmed still present, unfixed, on several others. Bug 3 is fixed on a minority of the fleet. Auditing 18 independently-evolved codebases surfaces the same handful of mistakes repeatedly. Propagating a fix once it's found is a separate, ongoing piece of work. It doesn't happen automatically the moment the pattern is identified.