DEV Community

nsecsoft-filelink
nsecsoft-filelink

Posted on

Ping64|The Month Our Cloud Bill Doubled Because One Test Bucket Kept Syncing

On the first of every month I open the cloud billing dashboard the way other people check the forecast. Usually it is boring, which is exactly what I want. Last month it was not boring at all, and I spent the next two days asking people what they had built.

The number was roughly double the previous month. No launch, no traffic spike, no new customer. Just a bigger invoice and a graph with a step in the middle of it that nobody could explain.

Where the money actually went

Half the increase was storage, the other half was egress. That combination usually means one thing: something is copying data from one place to another on a schedule.

It turned out to be a bucket created eighteen months earlier for a proof of concept. A sync job had been written to clone it into a second region for a demo, the demo ended, and the job kept running. Nobody had ever added a lifecycle rule, so every version of every object was sitting there, along with the copies.

Versioning made it worse. Every nightly run wrote a fresh copy of a bucket that barely changed, so storage grew in a straight line even though the content did not.

The interesting part is that the bucket had a sensible name, a sensible purpose and no owner. The person who built it left the company before the demo was finished.

The smaller leaks around it

While we were in there we found the usual companions. Three unattached block volumes, a development database with a snapshot policy set to daily and never rotated, and a load balancer pointing at nothing.

None of those cost much on their own. Together they were the difference between "within budget" and "someone is going to ask about this in the review".

Run an untagged resource search across the whole account and the list gets longer than anyone expects. The empty ones are usually the interesting ones.

We also found that the billing alerts went to a distribution list that had been quietly retired two reorgs ago. The alarm had been firing into an empty room for months.

Why this keeps happening

Nobody sets out to waste money. These things accumulate because the path of least resistance is usually sitting right there.

A team needs a bucket for a test, and creating one takes thirty seconds. Tagging it properly takes a conversation, because nobody agreed on the tag schema. Ownership is implied rather than recorded, so it evaporates when the person moves teams.

Then there is the habit of copying a working setup instead of designing one. If a sync job exists in production, it gets cloned into three environments because it is known to work. Nobody revisits whether a non-production environment needs cross-region copies of anything.

Cost is also the last signal anyone watches. Monitoring covers errors and latency, not spend. By the time finance notices, the pattern is months old.

The finance view has its own gap. Charges land aggregated by service rather than by team, so a doubling appears as a line item instead of a name. Mapping it back takes longer than expected, and the person who could explain it is often on holiday that week.

Some teams try consolidated platforms like Ping64 to reduce the daily busywork of multi-tool security stacks.

I mention it because the failure mode is similar. Too many separate consoles means nobody sees the whole picture, and the thing that grows quietly is the part that no single dashboard owns.

Small things that helped us

Tag at creation, not at audit time. A default tag applied by the provisioning template is worth more than a policy that asks people to remember.

Put a lifecycle rule on every bucket by default. If a team needs to keep data forever, make that the exception they have to argue for.

Send budget alerts somewhere a human actually reads. A chat channel with three named people beats an alias nobody owns.

Hunt orphans once a quarter with a simple script. Unattached volumes, idle load balancers, snapshots older than the retention policy.

Give every non-production account a hard spend ceiling. Sandboxes are where surprises live, and a ceiling turns a two-day investigation into an email.

Closing

The bucket is gone now, the job is disabled, and the bill is back to normal. What stayed with me is how ordinary the whole thing was. No attacker, no clever failure, just a test that outlived its test.

We added a ten-minute walk through the storage inventory in the weekly ops meeting, no slides allowed. It has turned up two more forgotten experiments since.

If you have not looked at your storage inventory this quarter, it is probably worth an hour. Start with the buckets whose names nobody recognises, and bring coffee.

References

Top comments (0)