DEV Community: Vlad Z

A $16,500 GCP bill nobody could explain. Logging was $3,800

Vlad Z — Fri, 16 Oct 2026 21:24:37 +0000

The GCP account I remember most clearly had a billing problem nobody could locate

Medium-sized SaaS company. Four GCP projects. Total monthly spend: $16,500. Their internal estimate for what the infrastructure should cost: $10,000 to $12,000

Six months of "we should look into that" conversations. Nobody had time. The gap between what it should cost and what it was costing kept sitting there, not getting addressed

I started with the billing export to BigQuery, which is the right way to diagnose GCP spend. It gives you full granularity - project, service, resource label, SKU. Far more detail than the billing console alone

The anomaly appeared in the first 30 minutes

Cloud Logging. $3,800 a month

For their workload that was immediately suspicious. I pulled the log volume breakdown by project

One project was generating 2.4TB of logs a month

Their entire production environment across the other three projects generated 180GB a month combined

I found the source. A GKE application running in a non-production project that had been set up for load testing eight months ago. The load test had run. The environment had been left running. And the application was logging at a level designed for load testing - high verbosity, every request, every response, every internal state change

Running continuously for eight months

Nobody had looked at non-production billing because it was assumed to be low

We deleted the environment. Not just the logging - the whole project. It had been generating $4,200 a month in total: logging plus the GKE nodes running the application

The fix took 20 minutes once we found it

The finding took 30 minutes of looking at the right data

Eight months, $33,600

Because a load test environment got left running and nobody was watching the bill by project

BigQuery is the easiest way to burn a budget by accident

Vlad Z — Wed, 14 Oct 2026 09:17:23 +0000

BigQuery is one of the most powerful tools in GCP and one of the easiest ways to accidentally generate a very large bill

The power is real. You can query terabytes of data in seconds. The cost implication of that power is also real when queries aren't written carefully

BigQuery has two pricing models. On-demand charges per TB of data processed. Capacity pricing charges for reserved slots regardless of queries run

For most teams starting out, on-demand is what they're using. And on-demand pricing punishes full table scans

A query that selects every column from a 500GB table processes 500GB of data and charges you accordingly. A query that selects only the three columns you actually need from a partitioned table that filters to the relevant date range might process 2GB of data

Same result. Very different cost

This is the BigQuery optimization that matters most: query efficiency

Use column selection instead of SELECT *. Always. The cost difference on large tables is not marginal

Partition your tables and use partition filters in your queries. BigQuery will only scan the relevant partitions. A date-partitioned table queried with a date filter processes only the data in those dates

Cluster your tables on the columns most frequently used in WHERE clauses and JOINs. Clustering reduces the data scanned for filtered queries

The second optimization: control who can run what

Uncontrolled BigQuery access means analysts running expensive exploratory queries against production data at full scale. Query costs that belong to individual analyses ending up on the company bill with no visibility into what generated them

Column-level access controls, project-level quotas on bytes processed, and requiring query review for tables above a certain size - these are organizational controls but they have direct cost implications

BigQuery's built-in INFORMATION_SCHEMA views let you see exactly which queries are processing the most data and which users are generating the most cost

Look at that data. It will tell you where to focus

GKE is where GCP bills get complicated fast

Vlad Z — Tue, 13 Oct 2026 15:54:46 +0000

GKE is where GCP accounts get complicated fast

Kubernetes is powerful. It's also genuinely possible to spend a lot of money running it inefficiently, and the complexity of the system makes it hard to see where the inefficiency is

The first question when looking at a GKE cluster: Autopilot or Standard mode?

Standard mode gives you full control over the nodes, the node pools, the machine types, everything. You're responsible for cluster infrastructure. You pay for the nodes whether they're being used or not. The optimization burden is on you

Autopilot mode manages the infrastructure for you. You pay for the resources your pods actually request and use. GCP handles node provisioning, scaling, and maintenance. You lose some customization but you gain a billing model that's much harder to waste with

For most small to medium teams that aren't running specialized workloads: Autopilot is worth serious consideration

Not because it's always cheaper. Because the cost model aligns with usage in a way Standard doesn't. In Standard mode, under-utilized nodes still bill. In Autopilot, unused capacity doesn't

The right recommendation depends on what you're running

For Standard mode clusters where you want to optimize:

Node pool right-sizing is the first thing. Look at actual CPU and memory utilization across nodes. Are you running n2-standard-8 nodes at 20 percent utilization? You have options

Cluster Autoscaler should be configured to scale down aggressively when demand drops. The default settings are conservative. Explicitly configure scale-down behavior

Pod resource requests need to reflect actual usage. If pods request 2 CPU and use 0.3 CPU, the scheduler cannot bin-pack efficiently. The node looks "full" to the scheduler while actually running at low utilization

Accurate resource requests let the cluster pack pods efficiently, which means fewer nodes, which means lower cost

GKE has a built-in resource usage dashboard. Almost nobody looks at it. The teams that do consistently find optimization opportunities the teams that don't will never see

Google Cloud Storage has the same problem as S3: Standard forever

Vlad Z — Sun, 11 Oct 2026 13:11:50 +0000

GCP Cloud Storage has the same problem S3 has

Everything goes into Standard class. It's the default. It's what people know. And it's the most expensive option for data you don't access frequently

The four classes in Cloud Storage:

Standard: for data accessed frequently. Highest price per GB, no retrieval fees

Nearline: for data accessed less than once a month. About 50 percent cheaper than Standard, small retrieval fee

Coldline: for data accessed less than once a quarter. About 75 percent cheaper than Standard, higher retrieval fee

Archive: for data accessed less than once a year. About 95 percent cheaper than Standard, highest retrieval fee, minimum storage duration of a year

The retrieval fees are the caveat everyone needs to understand before changing storage classes. Moving data to Archive class and then reading it constantly is not cheaper. You need to know your access patterns

For data you don't know the access patterns on: Object Lifecycle Management

You define rules and GCP handles the transitions automatically. After 30 days without access, move to Nearline. After 90 days, Coldline. After 365 days, Archive or delete

For data with unpredictable access patterns: Autoclass

Similar to AWS Intelligent-Tiering. Objects are automatically moved between Standard and Nearline based on access patterns. Small management fee per object. No retrieval fees from Autoclass transitions

In most GCP accounts with significant storage spend, the combination of lifecycle policies on predictable data and Autoclass on unpredictable data reduces storage costs 30 to 60 percent

The data doesn't change. The cost of keeping it does

The GCP discount AWS has no equivalent for

Vlad Z — Fri, 09 Oct 2026 22:47:26 +0000

GCP has a discount mechanism that AWS doesn't have an equivalent for

Sustained Use Discounts apply automatically when you run Compute Engine instances for more than 25 percent of a month. The longer the instance runs, the bigger the discount. Run it the entire month and you get up to 30 percent off automatically, with no commitment and no upfront payment

This is genuinely good. It means GCP rewards you for running stable workloads without requiring you to predict the future

But there's a better option on top of it for workloads you're confident about

Committed Use Discounts

One year commitment: 37 percent off. Three year commitment: 55 percent off. Applied at the vCPU and memory level rather than specific instance types, which gives you more flexibility than AWS reserved instances if you change machine types

The mistake teams make with CUDs is the same one they make with AWS reserved instances: committing before they've rightsized

If your instances are 40 percent larger than they need to be and you commit to that size for three years, you've locked in a discounted price on waste

The right sequence on GCP:

Rightsize your compute first. Use the Recommender in the GCP console - it analyzes actual utilization and suggests appropriate machine types. Take those recommendations seriously, test them, implement them

Then look at what's left. What are you running that's stable, correctly sized, and not going away? That's your commitment baseline

Commit to that baseline with CUDs. Let the variable workload stay on the automatic Sustained Use Discounts

The combination of rightsizing plus CUDs on GCP typically saves 45 to 60 percent compared to running on-demand unoptimized instances

The savings are there. The order matters

The AWS savings nobody can touch: legacy systems

Vlad Z — Thu, 08 Oct 2026 21:48:34 +0000

The second category of AWS caveats is the legacy problem

And legacy doesn't just mean old

It means: built by people who are no longer here, in ways that are no longer documented, for reasons that are no longer clear, and that everything else has since been built on top of

I've seen EC2 instances running since 2017 that nobody will touch. Not because they're critical in a way anyone can articulate. Because they're connected to enough other things that removing them feels like pulling a thread that might unravel something important

The fear is not irrational. It's based on past experience. Someone removed something once and something else broke. Now nothing gets removed

The cost of that conservatism is real. Every month. Indefinitely

Here's how I navigate legacy in AWS accounts

First: observe before acting. Look at network traffic, API calls, CloudWatch metrics. Does anything talk to this resource? Is anything depending on it in a measurable way? Often the answer is no. The fear was about unknown dependencies and the monitoring shows there are none

Second: tag it with intention. If you can't remove it yet, tag it with a date and a note. "Reviewed 2024-Q1. No observed dependencies. Candidate for removal." That tag is a commitment to revisit. It also means the next person who looks at this has context instead of starting from scratch

Third: isolate before removing. If the resource is in a VPC, you can often restrict its network access before removing it. Leave it running but disconnected. Wait 30 days. If nothing complains, the dependency was a ghost

Legacy systems in AWS are expensive partly because of what they cost to run and partly because of what they cost to think about

Every hour an engineer spends being careful around an undocumented legacy resource is an hour not spent building something

The documentation debt has a bill too. It just doesn't show up in Cost Explorer

AWS optimization always looks cleaner from the outside

Vlad Z — Wed, 07 Oct 2026 21:31:35 +0000

AWS optimization always looks cleaner from the outside than it is from the inside

Here are the things that slow down or stop work that looks straightforward on paper

IAM and permission boundaries

In organizations with mature security posture, the engineer doing the optimization work often doesn't have permission to make the changes. They can see the problem. They can define the solution. They cannot execute it without going through an access request process that takes days

In multi-account organizations this gets more complex. The permissions structure in AWS Organizations means that even with appropriate access in one account, cross-account actions require separate setup. A simple "move this workload to a different account" involves IAM roles, resource policies, and organizational SCPs that all need to align

Plan for permission lead time. It's often longer than the technical work

Reserved instance and Savings Plans complexity in organizations

If your AWS accounts are under an organization with consolidated billing, reserved instances and Savings Plans purchased in one account can be shared across the organization. This is good. It also means that the optimization decision in one team's account affects the economics of another team's account

Before purchasing any commitment-based discounts in an organizational context, understand who controls the payer account, how reservations are currently allocated, and whether there's an existing RI management process

Buying reservations without this context can create conflicts with existing commitments or miss opportunities to consolidate under better terms

Compliance and data residency constraints

Optimization often involves moving data or workloads. Moving a database to a cheaper region, for example

If the data has residency requirements - GDPR, financial regulations, healthcare data requirements - those constrain where it can go. Sometimes the cheaper option is simply not available for compliant data

Know your constraints before you design the solution. The cost of discovering a compliance blocker mid-migration is higher than the cost of asking the question upfront

A $22,000 GPU bill with a very quiet night shift

Vlad Z — Tue, 06 Oct 2026 21:12:33 +0000

The one I think about most is the account where the infrastructure was correct and the cost was still wrong

ML startup. Eight engineers. Production inference API with real traffic

Their AWS bill was $22,000 a month. Mostly GPU compute for inference

I went in expecting to find over-provisioned instances. I found the opposite. The instance types were appropriate for the workload. Utilization was reasonable. Nothing obviously wasteful

What I found instead was the scheduling

Their inference traffic followed a completely predictable pattern. 90 percent of requests came between 9am and 11pm US Eastern time. The overnight period was nearly silent. A few health checks. Background jobs. No real user traffic

The GPU instances ran 24 hours a day

At full capacity. Full price. Through eight hours of near-zero utilization every single night

We implemented a scale-down schedule. At 11pm Eastern, scale the inference cluster down to a minimal warm state - enough to handle the background jobs and respond to the health checks. At 8am Eastern, scale back up before the traffic arrives

The change took two days to implement and test safely

Monthly savings: $4,800

Not from changing the architecture. Not from changing the model. Not from changing anything about how the product worked

From matching the cost to the actual demand pattern instead of running at peak capacity all the time because that felt safer

The infrastructure was fine

The schedule was wrong

Sometimes that's all it is

An $18,000 AWS bill that should have been $12,000

Vlad Z — Mon, 05 Oct 2026 20:38:59 +0000

The second account I want to talk about was a different kind of problem

Not wasteful. Efficient, actually. A well-run engineering team who had done their homework. Good tagging. Reasonable instance sizes. Clean account structure

Their bill was still too high and they couldn't figure out why

$18,000 a month. For their workload it should have been closer to $12,000

I spent longer in the initial review on this one. The obvious things weren't obvious here

It took me two hours to find it

OpenSearch

They were running a production OpenSearch cluster for application logging and search. Reasonable. The cluster was sized at three data nodes of r6g.2xlarge each. That's a serious cluster. About $2,200 a month just for those nodes

I looked at the index patterns. They were retaining 14 months of log data in the hot tier. Fully indexed. Fully searchable. At all times

I looked at the search queries. 98 percent of all queries were against data from the last 7 days. The other 2 percent were against data from the last 30 days. Nobody had searched data older than 30 days in the history of the cluster

They were paying to keep 14 months of data in fast hot storage so that 2 percent of queries could touch data from 30 days ago

We implemented Index Lifecycle Management. Hot tier for 7 days, warm tier for 30, cold tier after that with significantly cheaper storage. Reduced the node count on the cluster after the data distribution changed

New OpenSearch cost: $800 a month

$1,400 a month saved. No change to search functionality. Not a single query that previously worked stopped working

The data was the same. The cost of keeping it was not

This is the thing about well-run accounts. The easy wins are already gone. What's left is understanding the system deeply enough to find where expensive decisions were made with good intentions and incomplete information

That's still findable. It just takes longer

The CloudWatch fix was the easy part. $9,300 was still missing

Vlad Z — Sun, 04 Oct 2026 19:57:03 +0000

After the CloudWatch fix we kept going

$11,400 a month. $340 on CloudWatch now. $9,300 still unaccounted for

RDS was next. $2,800 a month across four database instances

I looked at the instance types first. All four were db.r5.2xlarge. 8 vCPUs, 64GB memory each

Then I looked at the utilization metrics. Average CPU across all four: 12 percent. Peak CPU in the last 30 days: 34 percent. Memory utilization: consistently under 25 percent

Four databases running at a fraction of their capacity, 24 hours a day, 7 days a week

I asked the founder why they were that size. The answer: an engineer had sized them for expected growth at Series A. Series A was planned for 18 months ago. The growth hadn't come yet. The databases had been running at that size for over a year

We moved two of the four to db.r5.large - one quarter the size, which was still 4x more than current peak utilization required. We scheduled a maintenance window, ran the resize, monitored for 72 hours

No issues

RDS bill the following month: $1,100

We were 6 days into the engagement

CloudWatch plus RDS: $3,360 saved per month from two changes

The rest of the work took another three weeks. EC2 rightsizing, S3 lifecycle policies, removing a handful of completely unused services

Total reduction: $6,800 a month

From $11,400 to $4,600. The product had the same 800 users. The infrastructure had the same capability. The bill was 60 percent lower

The money was always there to be found

It just needed someone to look

An $11,400 AWS bill, and CloudWatch Logs was $2,100 of it

Vlad Z — Sat, 03 Oct 2026 18:17:24 +0000

I got access to the account on a Tuesday morning

The founder had sent over credentials the night before with a message that said roughly: I think we're overpaying but I don't know where

Total monthly spend: $11,400

Their product had about 800 active users. For 800 users that number felt wrong immediately

First 15 minutes: just looking. Not touching anything. Building a mental map of what's here and roughly why

CloudWatch Logs was $2,100 a month. For an account this size that's the first red flag

I pulled the log groups. Forty-three of them. Most of them with no retention policy set, which means logs accumulate forever. Several of them receiving data from services that no longer existed. Three of them receiving data at a rate that made no sense for their stated purpose

One log group was ingesting 180GB of data a month from a service that was logging at debug level in production

Not info level. Debug level. Every function call, every variable state, every internal operation. All of it being shipped to CloudWatch and stored indefinitely at full price

That one log group was $900 a month

The service had been deployed eight months ago. Debug logging had been turned on during initial development and never turned off. Nobody had looked at the CloudWatch bill specifically. It was just part of the total that nobody questioned because the total was growing gradually

We set retention policies on all 43 log groups. Turned off debug logging on the offending service. Deleted the log groups receiving data from services that no longer existed

CloudWatch bill the following month: $340

$1,760 saved. One afternoon of work

And we hadn't touched anything else yet

Data transfer is the AWS line item everyone underestimates

Vlad Z — Thu, 01 Oct 2026 21:17:54 +0000

Data transfer costs are the most consistently underestimated line item in AWS

They're not the biggest cost in most accounts. But they're the one that grows the fastest as systems scale, the one nobody was thinking about when the architecture was designed, and the one that's hardest to reduce without touching the architecture itself

That's what makes it hard high-impact

The charges come from several directions

Traffic leaving AWS to the internet is charged per GB. Traffic between regions is charged per GB in both directions. Traffic between availability zones within the same region is charged per GB. Only traffic within the same AZ between the same service type is free

In a microservices architecture where services are spread across AZs, every API call can generate billable transfer. Not much per call. At thousands of calls per second it adds up to a significant monthly line item that wasn't in anyone's cost model

The diagnosis starts with the data transfer line in your bill broken down by usage type. Most people have never looked at this breakdown. It usually contains at least one surprise

The fixes depend on what you find

Services talking across AZs: consolidate them or use a service mesh that routes traffic locally. Services unnecessarily calling APIs in other regions: move the data closer or cache aggressively. Internet egress higher than expected: check if data is being transferred to clients in an uncompressed format, check if assets are being served directly from S3 instead of through CloudFront

CloudFront is worth a specific mention. Serving content through CloudFront instead of directly from origin reduces egress costs because AWS prices CloudFront egress lower than direct S3 or EC2 egress, and CDN caching reduces the total volume of data transferred from origin

The architecture changes required to fix data transfer problems are real work

But the savings compound with every point of traffic growth

The team that fixes this today pays less at 2x traffic, 5x traffic, 10x traffic

The team that doesn't pays more every month, forever