DEV Community

Kerolos Atallah
Kerolos Atallah

Posted on

I read 172 official status feeds every 10 minutes. Here is what is quietly broken in them.

For the last few weeks I've had a cron job reading the official status feed of 172 cloud and SaaS providers every ten minutes. Not scraping their HTML, not counting tweets, just reading the machine-readable source each vendor publishes about itself.

I wrote about the first version of this in July, when it was 96 providers. This is what another 76 vendors and a month of collected data added, and most of it is stranger than the first batch.

I expected the hard part to be scale. It wasn't. The hard part is that "the official status feed" is not one thing, and several of them are quietly broken in ways their own vendors seem not to know about.

There is no standard, and it isn't close

Statuspage's /api/v2/summary.json is the closest thing the industry has to a convention, and it covers most but not all of what I track. The rest need their own parsers. Google publishes a dashboard incidents.json in its own shape, AWS has a health event feed, Azure emits RSS, Slack has a bespoke API, Oracle Cloud has a region-by-service matrix, Status.io has its own API, and Instatus serves a near-copy of Statuspage's format.

That last one is the instructive case. Instatus exposes /api/v2/summary.json at the same path as Statuspage, and it parses, but it omits the components array, page.updated_at, and the incident archive. A parser written against Statuspage will happily consume it and conclude that a vendor with a live outage has no components and nothing wrong. Compatible-looking is worse than incompatible.

AWS ships its status feed in UTF-16, big-endian

https://status.aws.amazon.com/data.json redirects to health.aws.amazon.com/public/currentevents and returns a body that starts with the bytes FE FF.

That's a UTF-16 big-endian BOM. Not UTF-8. Not even the little-endian UTF-16 you'd occasionally expect from a Windows toolchain. response.text() in a browser or Node will not give you usable JSON, and neither will a naive UTF-8 decode. You get a string of interleaved null bytes and a parse error that looks like the endpoint is down.

Everyone consuming this feed has independently discovered this and written the same three-branch decoder. It has apparently been this way for years.

Stripe's status JSON has said "all services are online" since February 2024

$ curl -s https://status.stripe.com/current
{"statuses":{"api":"up","webhooks":"up",...},
 "largestatus":"up",
 "message":"All services are online.",
 "time":"February 09, 2024 @ 06:08PM +00:00"}
Enter fullscreen mode Exit fullscreen mode

HTTP 200. Well-formed JSON. Reassuring message. And a timestamp from over two years ago that has not moved since. I re-ran that curl the morning I published this, and it still says exactly that.

Stripe's human status page is fine. They migrated, and the new one works. The old JSON endpoint was simply never turned off, so it still answers, still says everything is up, and will keep saying that through any incident you care about. If you wired a dashboard to it in 2023, it is still green today and it means nothing.

This is the single strongest argument I know of for not trusting a status endpoint you haven't checked the freshness of. It's also why I refuse to list Stripe at all rather than serve a fossil. A green light that cannot turn red is worse than no light.

Vendors forget to close incidents

At the time of writing, AWS still lists two open events in its own health feed, one in me-central-1 and one in me-south-1, opened on 1 and 2 March. That is over 3,700 hours each, about 157 days. Narrow the question to incidents that opened in the last 30 days and the longest is a Telnyx incident at 506 hours, nearly 21 days, followed by a KnowBe4 purchasing page at 463 hours.

I don't think these are month-long outages. I think they're incidents someone opened, fixed, and never went back to close. But there is no way to tell that from the outside, and the vendor's own page still says open, so any honest aggregator has to either report it as open or start silently overriding vendors about their own systems. I report it, and I label the duration "stayed open" rather than "downtime", because those are not the same claim.

The genuinely strange ones

Cisco Duo publishes 847 components with 94 distinct names. Core Authentication Service appears 80 times, once per deployment shard. If you build your lookup with a naive new Map(components.map(c => [c.name, c])), last wins, and 79 shards reporting an outage get masked by one healthy one at the end of the array. You have to take the worst status across every occurrence.

ClickHouse has a component named AWS us-west-2, with two spaces. Match on the exact string or you will never map it.

NetSuite once published its internal telemetry as public incidents. Real entries, in the real feed, titled _system_metadata:eyJ1cHRpbWUi…. Base64-decode that and you get {"uptime":"99.99","requests_served":…}. Somebody's monitoring pipeline was writing to the customer-facing incident stream.

Several status pages block non-browser clients. 8x8's WAF returns 403 to anything without a browser user-agent. Okta's status API returns 401. A status page that only a human can read is, for automation purposes, not a status page.

No consumer carrier publishes one at all. Vodafone, AT&T, Verizon, T-Mobile, BT, Orange, Telstra: I checked all of them. There is no official machine-readable status feed for any major consumer telecom operator. Every site that appears to show you one is showing you crowd reports.

What a month of this actually looks like

Across the last 30 days, the 172 providers logged 1,455 incidents between them, and 151 of the 172 had at least one. 369 were rated major or critical by the vendor themselves: 267 major, 102 critical, against 1,084 minor. Telecom and connectivity was the busiest category with 305, then cloud and infrastructure at 219, then developer tools at 196, then AI at 147.

The distribution is the part I found most useful. Most of it is minor, and most of what a status page publishes is noise relative to your stack. The question that actually matters during an incident isn't "what's broken somewhere". It's "is anything I depend on broken, right now, according to the vendor itself".

The takeaway I'd want if I were reading this

If you consume vendor status programmatically:

  1. Check the timestamp, always. A stale-but-green endpoint is the failure mode you will not notice.
  2. Assume component names repeat. Aggregate to worst-status, never last-wins.
  3. Don't trust a compatible-looking payload. Assert that the fields you need exist.
  4. Decode by bytes, not by assumption. Check for a BOM before parsing.
  5. Treat "open" as "the vendor hasn't closed it", which is not the same as "still down".

I turned all of this into OutageDeck, which reads those 172 feeds every ten minutes so you don't have to write the decoders. But honestly, the five rules above are most of the value. If you're building this in-house, that's the list I wish I'd had.

Happy to answer anything about specific feeds in the comments. If there's a vendor whose format has broken you in an interesting way, I'd like to hear it.

Top comments (0)