I Monitored 15 Popular APIs for 7 Days. 73% Changed While Nobody Was Watching.

Aviad Shakargy — Thu, 21 May 2026 22:40:39 +0000

Every backend depends on a graveyard of third-party APIs. Stripe for billing, Twilio for SMS, OpenAI for inference, that internal auth service from two teams over that nobody remembers who owns.

Ask any backend engineer to list every external HTTP call their service makes and you'll get half the list. The other half lives in a forgotten cron job, a legacy integration, or a vendor your predecessor added and never documented.

I wanted to know: how often do these APIs actually change? Not the big announcements that hit Hacker News - the quiet ones. The field that becomes required. The header that gets renamed. The deprecation notice buried in page 4 of a changelog.

So I built a system to find out.

The Experiment

I pointed my monitoring engine at the changelogs and release notes of 15 widely-used APIs:

Stripe · OpenAI · Twilio · GitHub · Auth0 · SendGrid · Slack · Shopify · Cloudflare · Vercel · Supabase · Notion · Firebase · HubSpot · Datadog

Every 6 hours, a worker fetches each changelog URL, extracts the text, computes a content hash, diffs it against the previous snapshot, and runs the diff through a severity classifier that scans for keywords like breaking change, deprecated, removed, security, end of life.

I let it run for a week with zero manual intervention. No tweaking, no hand-holding. Just the pipeline doing its job.

Here's what came back.

The Results

7 days. 15 APIs. 58 change events detected.

11 out of 15 APIs - 73% - shipped at least one change to their changelog or docs during a single week.

API	Changes Detected	Risk Keywords Found
GitHub	14	security, deprecated, removed
Vercel	10	security, deprecated, beta
Cloudflare	9	security, vulnerability, removed
Shopify	5	deprecated, removed, beta
Datadog	2	shutdown, breaking change, vulnerability
Slack	2	breaking change, deprecated, renamed
Firebase	2	breaking change, vulnerability, renamed
Supabase	2	breaking change, deprecated
Auth0	1	end of life, breaking change
OpenAI	1	breaking change, deprecated
HubSpot	1	beta
Stripe	0	-
Twilio	0	-
SendGrid	0	-
Notion	0	-

Let that sink in. In one week, the classifier flagged entries containing high-risk terms like breaking change, deprecated, removed, and end of life across 11 different APIs. Cloudflare, Datadog, and Firebase had vulnerability mentions. Auth0 had an end of life notice. Datadog's changelog contained shutdown.

Not every flagged entry means something broke. But each one is the kind of change worth reviewing before your next deploy - and in a normal week, most teams probably would not have seen them in time.

And this is just what's in the public changelogs. This is vendors doing the responsible thing and documenting their changes. How many don't?

Methodology note

This experiment tracked visible changes in public changelog and release-note pages. It does not prove that every detected change was breaking, only that the monitored source changed and contained risk-related language worth reviewing.

What Surprised Me

During this week, GitHub changed almost every 12 hours

14 change events in 7 days. During this monitoring period, GitHub's blog changelog updated almost twice a day with deprecations, security patches, and feature removals. If your CI/CD pipeline, your auth flow, or your webhook integrations depend on GitHub's API - and they probably do - things are moving under your feet constantly.

The "stable" APIs really are stable

Stripe, Twilio, SendGrid, and Notion had zero detected changes all week. That's not boring - that's a signal. These teams invest heavily in API stability. When you're choosing between vendors, this kind of data matters.

Most detected changes carried high-risk keywords

A large share of the 58 events contained keywords like security, deprecated, or removed. That doesn't mean every change was production-breaking - it means these were the changes most worth reviewing. Changelog entries tend to cluster around consequential updates. Nobody updates their changelog to say "we fixed a typo in an error message." When an entry appears, it usually matters.

Auth0 dropped an "end of life" notice

One event, one week, but the keywords were: end of life, breaking change, deprecated, removed, renamed. If you integrate with Auth0 and you didn't read their changelog this week, you might have missed a deprecation that affects your auth flow.

How It Works Under The Hood

This isn't a cron job running curl and diff. It's the same detection engine that powers API Graveyard, my API dependency intelligence platform, running against real vendor data in production.

The pipeline:

Celery Beat dispatches fetch jobs every 6 hours for each monitored source
Worker fetches the changelog URL, extracts text, and computes a content hash
Diff engine compares against the previous snapshot - if the hash changed, it generates a diff
Severity classifier scans the diff for risk keywords and assigns a severity level (critical / high / medium / low)
Risk events are created and surfaced on the dashboard with severity, timestamp, and source attribution
Public API serves the data through unauthenticated read-only endpoints - no login, no paywall

The whole thing runs on a single Docker Compose stack: FastAPI backend, React frontend, PostgreSQL, Redis, Celery workers. Deployed to a GCP Compute Engine instance via GitHub Actions. Total infra cost: less than the price of one Datadog alert channel.

Some engineering choices that matter:

Hash-based change detection means we don't store raw changelog HTML (copyright-safe) - we only store diffs when something actually changes
Keyword classifier is intentionally simple (pattern matching, not ML) because false negatives are worse than false positives here - I'd rather flag something that isn't breaking than miss something that is
Each vendor is an isolated adapter (~40 lines each). When Stripe redesigns their docs site - and they will - only the Stripe adapter breaks. The other 14 keep running.
SSRF protection on the fetch layer blocks private IP ranges and localhost - because this system fetches arbitrary URLs from a database, and that's exactly the kind of thing that gets you in trouble if you're not careful

Why This Matters

Most teams find out about breaking API changes one of three ways:

A deploy breaks in production. The Slack channel lights up at 2 AM.
Someone happens to read the changelog. Usually the one person who's subscribed to the vendor's blog, and they mention it casually in standup two days later.
A quarterly dependency audit catches it. Three months late, during a sprint nobody wanted.

None of these are good. The first one costs you an incident. The second one depends on luck. The third one is always too late.

What I learned from this week of data is that the problem is worse than most people think. 73% of the APIs we commonly depend on are changing every single week. Not every change is breaking, but the ones that are? They're buried in changelogs that nobody reads, mixed in with dozens of other entries.

For teams that depend on multiple external APIs, automated monitoring is quickly becoming less of a nice-to-have and more of a safety net. It's the difference between catching a deprecation notice on Tuesday and debugging a production outage on Saturday.

See It Live

The public dashboard is running right now at www.api-graveyard.com/api-health - no login, no signup, no paywall. Real data from real APIs, updated every 6 hours.

The source code for the case study (architecture diagrams, screenshots, SDK links) is on GitHub: github.com/Shakargy/api-graveyard-case-study

This case study only monitors public changelogs. The full product goes one layer deeper: discovering the actual APIs your services call from live traffic, mapping dependency risk across your entire stack, and alerting when something changes. One-line SDK integration for Python, Go, and Node.js - check it out at www.api-graveyard.com.

Built by the solo developer behind API Graveyard.

DEV Community: Aviad Shakargy