On July 16, 2026, a chunk of the internet started returning 504s. Hugging Face went dark "from most regions in the world." The UK National Lottery apologized on X. Here in Japan, PayPay (the country's biggest QR payment app), niconico, note, and Hatena Blog all became unreachable at almost the same moment, right in the middle of the evening.
The interesting part: AWS wasn't "down." Amazon CloudFront wasn't even down. One specific feature was — VPC Origins — and everything built on it failed together for about three and a half hours.
I dug through the AWS Health Dashboard updates while the incident was unfolding, and I think this outage deserves a closer look than "AWS had a bad day." It says something specific about a trade-off many of us made when we adopted VPC Origins: we removed a public attack surface, and in exchange we put a shared AWS-managed subsystem on our critical path. This was the day that trade-off presented its bill.
A quick refresher: what VPC Origins is
CloudFront VPC Origins launched at re:Invent 2024. Before it existed, if you wanted CloudFront in front of an ALB, that ALB had to be internet-facing. You'd then bolt on defenses to make sure nobody bypassed CloudFront and hit the ALB directly: custom secret headers, AWS-managed prefix lists, WAF rules. It worked, but the origin was still technically reachable from the public internet.
VPC Origins removed all of that. CloudFront reaches into your VPC and talks to a private ALB, NLB, or EC2 instance directly. No public IP, no secret header dance, no bypass risk. It's genuinely a better architecture, and adoption has been strong — which is exactly why this outage was so visible.
Under the hood, though, that "reaching into your VPC" is not magic. There's an AWS-managed fleet that maintains the connections from CloudFront's edge locations into customer VPCs, and a packet processing subsystem that routes each request to the right private origin. Every VPC Origins customer shares that machinery. On July 16, that machinery broke.
The timeline
All times below are PDT, taken from the AWS Health Dashboard (JST is +16 hours — the outage ran from 16:45 to 20:18 JST, prime evening hours in Japan).
| Time (PDT) | What happened |
|---|---|
| 12:45 AM | Increased 5xx errors begin for CloudFront customers using VPC Origins |
| 1:44 AM | First Health Dashboard post: "We are investigating increased 5xx errors" |
| 2:21 AM | Scope confirmed: only VPC Origins affected. Workaround offered: switch origin type |
| 2:57 AM | Root cause identified internally |
| 3:18 AM | Public update points to "a packet processing subsystem" routing edge-to-VPC requests |
| 3:52 AM | Mitigation actions applied |
| 4:16 AM | Issue scoped to "routing table capacity within the packet processing subsystem" |
| 4:18 AM | Full recovery |
| 5:21 AM | Marked resolved |
Roughly 3.5 hours of impact, with the first hour spent before AWS publicly confirmed the scope. Customers on S3, public ALB, or custom origins were never affected.
What actually broke
AWS's final summary is unusually specific, so let me quote the key sentence:
"We identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections."
Translated out of incident-report language: the fleet that connects CloudFront to private VPCs has an internal limit — the 4:16 AM update called it "routing table capacity." VPC Origins adoption grew until that limit was hit. And when it was hit, the failure mode wasn't a graceful "new configurations rejected" — the routing configuration distribution broke for existing connections too. A capacity ceiling turned into a routing failure for everyone on the feature.
This is a classic pattern with successful new infrastructure features. The feature works beautifully at launch scale. Adoption grows faster than anyone load-tested for. Somewhere there's a fixed-size table or a fleet constraint nobody has hit yet, and the day it's reached, the failure is correlated across every customer at once. Your architecture diagrams show your VPC, your ALB, your CloudFront distribution — they don't show the shared routing fleet in between, but it was always there.
The workaround that wasn't really a workaround
Throughout the incident, AWS recommended that "customers who are able to do so temporarily change their origin type to resolve the errors."
Think about what that means in practice. You adopted VPC Origins so your ALB could be private. To switch origin types mid-incident, you'd need an internet-facing ALB ready to serve traffic — which most VPC Origins adopters deliberately deleted, because not having one was the whole point. Standing up a new public ALB, wiring security groups, updating the distribution, and waiting for CloudFront to deploy the change, all during an outage, with the security hardening (secret headers, WAF) done under pressure or skipped entirely? That's not a workaround, that's an architecture migration with a fire behind you.
I don't say this to dunk on AWS — there wasn't a better short-term answer available. The point is that the workaround's impracticality is itself the lesson.
What I'm taking away from this
Origin failover doesn't cover this failure class. CloudFront origin groups fail over when your origin is unhealthy. Here, both members of an origin group would have been fine — the path between the edge and any VPC origin was what failed. If your failover origin is also a VPC origin, you had zero protection on July 16. A meaningful failover target has to use a different origin type: an S3 static fallback page, or a public origin in another path.
Decide your break-glass posture before the incident, not during it. There are only two honest positions. Either you accept that a VPC Origins outage means downtime until AWS fixes it (a legitimate choice — 3.5 hours of 504s might be cheaper than maintaining an escape hatch), or you keep a dormant internet-facing path that can be activated quickly: an internal-facing ALB you can re-create as public from IaC, with the custom-header check and WAF rules already written in code. What you can't do is decide this at 1 AM while your payment app is down.
A static fallback is embarrassingly effective. An origin group with your VPC origin as primary and an S3 bucket serving a friendly "we're having trouble" page as secondary won't keep your app working, but it converts a raw 504 into a controlled degradation. For a lot of services, the difference between those two during a 3.5-hour incident is the difference between "they had an issue" and screenshots on social media.
Newer managed features carry correlated-failure risk that mature ones have already paid down. S3 and public ALB origins have had well over a decade of scaling incidents to burn down their unknown limits. VPC Origins is a 2024 feature that just hit one of its limits in production, with all of us on board. That's not a reason to avoid new features — I use VPC Origins-style private architectures myself and I'd choose it again — but it is a reason to ask, for each new managed dependency: "when this fails for every customer simultaneously, what's my move?"
Closing thought
The security argument for VPC Origins is still correct. Removing your public origin surface eliminates a whole category of bypass attacks, permanently. What July 16 clarified is the price: you've traded a security risk you controlled for an availability risk you don't. That's often a good trade. But it's a trade, and it deserves a line item in your incident runbook — not just a checkmark in your security review.
Timeline and root cause quotes are from the AWS Health Dashboard event "Increased 5xx Errors" (Amazon CloudFront, July 16, 2026). Impact reports via The Register and ITmedia.
Top comments (0)