The GCP account I remember most clearly had a billing problem nobody could locate
Medium-sized SaaS company. Four GCP projects. Total monthly spend: $16,500. Their internal estimate for what the infrastructure should cost: $10,000 to $12,000
Six months of "we should look into that" conversations. Nobody had time. The gap between what it should cost and what it was costing kept sitting there, not getting addressed
I started with the billing export to BigQuery, which is the right way to diagnose GCP spend. It gives you full granularity - project, service, resource label, SKU. Far more detail than the billing console alone
The anomaly appeared in the first 30 minutes
Cloud Logging. $3,800 a month
For their workload that was immediately suspicious. I pulled the log volume breakdown by project
One project was generating 2.4TB of logs a month
Their entire production environment across the other three projects generated 180GB a month combined
I found the source. A GKE application running in a non-production project that had been set up for load testing eight months ago. The load test had run. The environment had been left running. And the application was logging at a level designed for load testing - high verbosity, every request, every response, every internal state change
Running continuously for eight months
Nobody had looked at non-production billing because it was assumed to be low
We deleted the environment. Not just the logging - the whole project. It had been generating $4,200 a month in total: logging plus the GKE nodes running the application
The fix took 20 minutes once we found it
The finding took 30 minutes of looking at the right data
Eight months, $33,600
Because a load test environment got left running and nobody was watching the bill by project
Top comments (0)