Last year I was asked to investigate a startup's AWS bill.
It had jumped from roughly $200/month to over $3,000 in a few weeks.
Nobody knew why.
...
For further actions, you may consider blocking this person and/or reporting abuse
I feel that... I was adding features to my accounting suite, so naturally I added cloud logging. But I didnt limit the logs that cloud is meant to generate... Nor storage time... Yeah, so $50 of log files later, wont be making that mistake again.
Ouch, that one hurts! At least $50 is a lesson learned cheap — I've heard of people hitting thousands just from CloudWatch logs with no retention policy set. The scary part is how long it can go unnoticed before the billing alert fires (if you even have one set up). Did you catch it from the bill or did something else tip you off?
This is on GCP free trial credits. So I just saw my credit balance went down, had to dig into the reports, filter by SKU, disable savings and BOOM cloud run cost 2c, vertex ai $2, cloud log $20, cloud log storage $30... Not a fun 1, cuz it means I need to pay overage outta pocket when clients hit the limit a whole 5 months earlier
Was the issue related to improper handling of async operations or perhaps a misconfigured cluster autoscaler, leading to unnecessary resource utilization?
Nope. No fancy infrastructure failure—just a few lines of code enthusiastically looking for work that didn't exist. 😅
Real-world engineering lessons like this are far more valuable than theoretical best practices. The bill hurt, but the takeaway will probably save many developers from making the same mistake!
Thanks! That's exactly why I wanted to share it. The expensive lessons tend to be the ones that stick. 😅
That closing question, what does this code do when there's nothing to do, is the part worth stealing. It travels way past polling loops, into retries that never back off or a health check firing way more than it needs to. On the fix, the 5 second sleep stops the bleeding but it's still polling, so you're trading a huge bill for a few seconds of pickup delay, which is fine right up until that delay matters and you wish you'd gone event driven from the start. The scariest line in the whole thing is that nothing crashed, because a bug with no exception and no failed metric can run for weeks before the invoice is what finally tells you.
Exactly. The invoice ended up being the first real alert, which is never where you want to discover a bug. 😅
"what does this code do when there's nothing to do?" is the question i run on every background worker in code review. catches more than polling bugs — retry handlers that compound on empty error queues, health checks on intervals too short for the underlying service, scheduled jobs that lock a row already processed.
the 5 second sleep stops the bleed but it is still polling. the next question after that fix: how long can pickup delay be before it matters? if the answer is "zero, realtime pickup" then you have found the event driven argument already.
the scariest line is "nothing crashed." silent cost bugs run for weeks because every metric you watch looks fine.
what did your CloudWatch log retention look like before this? usually the second bill hiding next to the polling one.
WoW, that's crazy, ☺️
Thanks! :)
I'm curious if the issue was related to an
Good question. The main issue was that
getJobs()queried the database continuously when the queue was empty. There was no delay between checks, so the service ended up making an enormous number of requests for work that didn't exist.Have you ever run into something similar with polling loops or background workers?