DEV Community

Fast Data
Fast Data

Posted on

One expired API token cost my data archive 35 daily snapshots: the two small scripts I added afterwards

I keep a small daily archive of public business data: two BaFin RSS feeds, the German federal tender feed, two DeFiLlama endpoints and the official MCP Registry. The idea is simple: a snapshot taken today can't be taken again later, so history is the asset.

On 12 November the API token most of those jobs depend on stopped being accepted (HTTP 401, "user-or-token-not-found"). Every scheduled run since then did the "responsible" thing: it checked, saw the 401, skipped, and wrote me a one-line status. A week later, this is the archive:

$ python archive_gaps.py archive/ --today 2026-11-19
source                     present missing streak  last ok
bafin_rss_massnahmen_de          5       7      7  2026-11-12  <-- ALERT
bafin_rss_news_de                5       7      7  2026-11-12  <-- ALERT
bund_tenders_rss                 5       7      7  2026-11-12  <-- ALERT
defillama_chains                 5       7      7  2026-11-12  <-- ALERT
defillama_stablecoins            5       7      7  2026-11-12  <-- ALERT
mcp_registry                    12       0      0  2026-11-19
Enter fullscreen mode Exit fullscreen mode

Five sources times seven days: 35 snapshots that don't exist. One source has no gap at all, and the reason it survived is the most useful part of this post.

Disclosure: I build data tools for AI agents as Fast Data on Apify. The archive is ours, and so are the numbers above. Both scripts are in our public data-recipes repo.

What actually went wrong

The token failing was not the problem. Tokens expire, get rotated, get revoked. The problem was how the system behaved around it.

Design choice What it did during the outage
Five sources fetched through one paid API (one token) All five failed together, on the same day
MCP Registry fetched directly, the paid API only as a fallback Kept working all week, 0 gaps
Each run "skips and reports" on 401 Many identical status lines; they became background noise
Rule "investigate if a source fails 3 days in a row" Written down, but nothing counted the days
Credential check only inside each job, after setup The failure was discovered again on every run, never once up front

Three things stand out.

1. A shared credential is a shared point of failure. Some of those sources were reachable directly. They went through the paid API only because that was convenient when I wrote the script. The one source that already did "direct fetch first, API as fallback" was the only one that survived.

2. "Skip and report" feels safe but scales badly. One message saying "source X failed" is useful. The tenth identical message is noise. What I needed was not more messages but a number that grows: 7 days in a row.

3. A rule nobody computes is a wish. "If a source fails three days in a row, find the cause" was in my notes. Nothing in the pipeline counted.

Fix 1: a gap scanner that turns silence into an exit code

archive_gaps.py walks archive/<source>/<YYYY-MM-DD>.json.gz, lists missing days per source and computes the current failure streak. The core is a few lines:

def scan(root: pathlib.Path, today: dt.date):
    report = {}
    for src in sorted(p for p in root.iterdir() if p.is_dir()):
        have = {dt.date.fromisoformat(f.name[:10]) for f in src.glob("*.json.gz")}
        if not have:
            continue
        missing = [d for d in days(min(have), today) if d not in have]
        streak = 0
        for d in sorted(days(min(have), today), reverse=True):
            if d in have:
                break
            streak += 1
        report[src.name] = {"present": len(have), "missing": missing,
                            "streak": streak, "last_ok": max(have)}
    return report
Enter fullscreen mode Exit fullscreen mode

It exits with code 1 when any streak reaches --alert-after (default 3). That matters more than the output: an exit code can fail a CI job, block a deploy or trigger one alert. A log line just sits there.

It looks at the files, not at the job logs. Logs tell you a job ran; files tell you whether the data exists. Those can disagree, and the files are what count.

Fix 2: a preflight check before any work starts

preflight.py makes one cheap, authenticated, read-only request per dependency and reports only the HTTP status, never the secret:

def _hdr(name: str, env: str, prefix: str = "") -> dict:
    token = os.environ.get(env)
    return {name: prefix + token} if token else {}  # unset -> no header (e.g. a proxy injects it)

CHECKS = {
    "apify": ("https://api.apify.com/v2/users/me", _hdr("Authorization", "APIFY_TOKEN", "Bearer ")),
    "devto": ("https://dev.to/api/articles/me?per_page=1", _hdr("api-key", "DEVTO_API_KEY")),
}
Enter fullscreen mode Exit fullscreen mode

Run it in front of the real job:

$ python preflight.py && python archive_daily.py
apify    401
devto    200
FAILED: apify - fix credentials before the job runs
Enter fullscreen mode Exit fullscreen mode

That is today's real output. The job that writes articles has its own key and works; the archive's paid-API path does not. Knowing that in one line at the start of the day beats rediscovering it in every job.

What I am changing in the design

  • Direct first, paid API as fallback for every source that allows plain HTTP requests. The script already had this pattern for the MCP Registry; the other sources now need their hosts allowed in my environment so they can follow.
  • One alert per outage, not one per run. The streak number is the alert; the per-run status line can stay quiet.
  • Separate credentials per job type where possible, so a failing key takes out one kind of work, not everything.

What can be recovered

Not everything. RSS feeds only show recent items, so the exact state of the BaFin and tender feeds on 14 November is gone, even if some items will still show up in later snapshots. DeFiLlama publishes some historical data, so part of those gaps may be fillable later. The lesson for an archive is blunt: a missed day of a "current state" feed is gone for good.

Try it

Both scripts are standalone, Python 3.9+, requests only for the preflight:

curl -O https://raw.githubusercontent.com/ichkifokkus/data-recipes/main/snippets/archive_gaps.py
python archive_gaps.py path/to/archive --alert-after 3
Enter fullscreen mode Exit fullscreen mode

Other building blocks for small data pipelines are in the same snippets folder, and our public data tools are on Apify.

Question: for scheduled jobs that depend on third-party tokens, do you check credentials in a separate preflight step, or let each job fail on its own? I am curious which one held up better for you.

Top comments (0)