The bill that broke Reddit
Friday, July 17, 1:33 AM Pacific. Developers with $0.19 monthly AWS bills started refreshing their console to see this instead:
$2,500,000,000,000.
Not a typo. Two and a half trillion dollars. Other accounts saw $1.7 billion. Some saw $126,000. Nobody's actual usage had changed. One Redditor put it best:
"I nearly had a heart attack when my two S3 buckets with a few MBs of data generated a half-billion-dollar forecast."
Another didn't wait to find out if it was real:
"Needless to say, I panicked and destroyed everything on this account."
That second quote is the whole story. Not the trillion-dollar number — the fact that a display bug got someone to nuke their own infrastructure at 2 AM. Let's get into what actually broke.
What actually happened
AWS's own postmortem language, stripped of the corporate hedge:
- 1:33 AM PT — inaccurate estimated billing data starts showing up on the AWS Health Dashboard and in customer accounts.
- ~3:00 AM PT — AWS finds the root cause in about 90 minutes. Not bad, honestly.
- AWS tries a rollback. It doesn't fix it. The bad numbers keep displaying (though AWS freezes them from climbing further).
- Corrected data promised by noon Pacific, July 18.
The official explanation: "an issue with unit pricing within the estimated billing computation subsystem." Translate that out of PR-speak and it means something like a rate multiplier got corrupted or mis-joined upstream of the estimator — a bad price-per-unit fed into a usage calculation that has zero sanity bounds on the output. Multiply garbage by real usage and you get a garbage total. Nineteen cents times a broken multiplier becomes $2.5 trillion just fine, mathematically speaking.
Critically: this only hit Cost Explorer's estimated billing UI, not actual invoices. AWS was explicit — "the displayed billing estimates do not reflect actual usage and charges." Your real bill was never $2.5 trillion. But that distinction lives in a sentence buried in a status update, and almost nobody reads status updates before they read their bill.
Why this is a bigger deal than "lol Amazon glitch"
Estimated billing isn't just a number on a dashboard anymore. It's a data source other systems act on:
- AWS Budget Actions can automatically stop resources, revoke IAM permissions, or apply restrictive policies when a budget threshold is breached.
- Cost Anomaly Detection fires alerts — and in a lot of shops, those alerts are wired into Slack bots, PagerDuty, or custom Lambda functions that take action without a human in the loop.
- Plenty of teams have their own home-grown "kill switch" scripts that watch Cost Explorer's API and shut things down past a threshold.
None of that is hypothetical — it's the standard advice every AWS cost-optimization blog post gives you: automate your guardrails so a runaway bill can't happen while you're asleep. Good advice, until the guardrail's data source itself lies to it. A corrupted estimate isn't just cosmetic when there's a Lambda on the other end of it with ec2:TerminateInstances permissions.
AWS got lucky here in one specific way: the bug inflated numbers instead of deflating them. Nobody's anomaly detection stayed silent when it should have screamed. But run the tape the other direction — a unit-pricing bug that makes a real cost spike look like nothing — and you have a very different, much quieter incident that costs someone real money before anyone notices.
The actual engineering lesson
If you're building anything that consumes a metrics or billing API and takes action on it — your own cost guardrails, a usage-based alerting system, whatever — the fix isn't "trust the vendor's number more." It's: never let a single upstream value directly trigger a destructive action without a bound.
A trivial sanity clamp would have turned this into a non-event for anyone with automation downstream:
def sanity_checked_estimate(current_estimate, previous_estimate, max_multiplier=50):
"""
Reject an estimate that jumps implausibly relative to trailing spend.
A real cost spike from, say, a runaway EC2 fleet can be 10x-20x.
A jump into the billions from a $0.19 baseline is not a cost spike,
it's a bad signal.
"""
if previous_estimate <= 0:
return current_estimate # no baseline to compare against
if current_estimate > previous_estimate * max_multiplier:
raise SuspiciousEstimateError(
f"Estimate jumped {current_estimate / previous_estimate:.0f}x "
f"vs trailing baseline — flagging for human review, not auto-acting"
)
return current_estimate
That's it. That's the whole pattern: bound the delta, not just the absolute value, and fail closed into "notify a human" instead of "take the automated action." AWS didn't have this on the display layer, which is embarrassing but low-stakes. What should worry you is whether you have it on the automation layer you built on top of their API.
The part nobody's saying out loud
AWS is a company that runs some of the most sophisticated anomaly detection in the world — they sell it to you as a product. And their own internal billing pipeline produced a 13-trillion-x error that took 90 minutes to diagnose and a failed rollback to (not) fix. If Cost Anomaly Detection is a good enough product to trust with your infrastructure's kill switch, it should have been good enough to catch this before it hit a customer's dashboard.
It wasn't. So don't trust it blindly either. Put a clamp on anything upstream of TerminateInstances. Your on-call self at 2 AM will thank you for not being the guy who deleted two S3 buckets over a phantom trillion dollars.
Top comments (0)