DEV Community

Cover image for Cloud Logging is Silently Eating Your GCP Budget at $0.50/GB (Here's How One Team Saved $140K) 📊
Suhas Mallesh
Suhas Mallesh

Posted on

Cloud Logging is Silently Eating Your GCP Budget at $0.50/GB (Here's How One Team Saved $140K) 📊

#gcp #terraform #cloud #devops

GCP charges $0.50 per GB of log ingestion. You get 50 GB free per project. After that, every health check, every debug statement, every verbose container log costs real money. Most teams don't notice until the bill arrives and Cloud Logging is the #2 line item. Exclusion filters, log sinks, and retention policies in Terraform fix this in under 30 minutes.

Here's what makes Cloud Logging so expensive: every GCP service generates logs by default, and they all route to the _Default sink. Load balancer health checks hitting /healthz 100 times per minute? Logged. Kubernetes container stdout with DEBUG-level noise? Logged. VPC flow logs for internal traffic? Logged. You pay $0.50 for every GB of that.

One company (Harness) found that just applying exclusion filters and routing logs to cheaper storage cut their logging bill in half, saving over $140K annually.

Let's do the same with Terraform.

📊 Cloud Logging Pricing Cheat Sheet

Component Cost Notes
Log ingestion $0.50/GB The main cost driver
Free tier 50 GB/month/project Disappears fast at scale
Retention (default) Free 30 days included with ingestion
Retention (extended) $0.01/GB/month For logs kept > 30 days
_Required bucket Free Admin audit logs, can't modify
Exclusion filters Free Dropped logs = $0 ingestion
Log sinks to GCS GCS pricing only ~$0.020/GB vs $0.50/GB

The math that hurts:

Daily Ingestion Monthly Volume Billable (minus 50GB free) Monthly Cost
5 GB/day 150 GB 100 GB $50
20 GB/day 600 GB 550 GB $275
50 GB/day 1,500 GB 1,450 GB $725
150 GB/day 4,500 GB 4,450 GB $2,225

Most teams generating 50+ GB/day can cut 40-60% with the filters below. That's $290-$435/month saved just from the 50 GB/day example.

🔧 Step 1: Exclusion Filters (Biggest Win, Zero Cost)

Exclusion filters drop logs before ingestion. You never pay for excluded logs. These are the 5 filters that eliminate the most common noise:

# Filter 1: Health check logs (often the #1 volume source)
resource "google_logging_project_exclusion" "health_checks" {
  name        = "exclude-health-check-logs"
  project     = var.project_id
  description = "Exclude load balancer health check noise"

  filter = <<-EOT
    resource.type = "http_load_balancer"
    AND (httpRequest.requestUrl = "/health"
      OR httpRequest.requestUrl = "/healthz"
      OR httpRequest.requestUrl = "/readiness"
      OR httpRequest.requestUrl = "/livez")
  EOT
}

# Filter 2: Debug-level logs in non-dev environments
resource "google_logging_project_exclusion" "debug_logs" {
  name        = "exclude-debug-logs"
  project     = var.project_id
  description = "Exclude DEBUG severity in staging/prod"

  filter = "severity = \"DEBUG\""
}

# Filter 3: GKE system container noise
resource "google_logging_project_exclusion" "gke_system" {
  name        = "exclude-gke-system-noise"
  project     = var.project_id
  description = "Exclude noisy kube-system and istio logs"

  filter = <<-EOT
    resource.type = "k8s_container"
    AND (resource.labels.namespace_name = "kube-system"
      OR resource.labels.namespace_name = "istio-system"
      OR resource.labels.namespace_name = "gke-managed-system")
    AND severity <= "INFO"
  EOT
}

# Filter 4: VPC flow logs for internal traffic
resource "google_logging_project_exclusion" "internal_vpc_flows" {
  name        = "exclude-internal-vpc-flows"
  project     = var.project_id
  description = "Exclude VPC flow logs for internal-only traffic"

  filter = <<-EOT
    resource.type = "gce_subnetwork"
    AND jsonPayload.connection.src_ip =~ "^10\\."
    AND jsonPayload.connection.dest_ip =~ "^10\\."
  EOT
}

# Filter 5: Cloud SQL routine logs
resource "google_logging_project_exclusion" "cloudsql_routine" {
  name        = "exclude-cloudsql-routine"
  project     = var.project_id
  description = "Exclude Cloud SQL routine checkpoint and stats logs"

  filter = <<-EOT
    resource.type = "cloudsql_database"
    AND (textPayload =~ "checkpoint starting"
      OR textPayload =~ "checkpoint complete"
      OR textPayload =~ "automatic analyze")
  EOT
}
Enter fullscreen mode Exit fullscreen mode

⚠️ Critical: Excluded logs are gone forever. You cannot recover them. Always test your filter in the Logs Explorer first by pasting the filter string into the query box. Verify it only matches what you intend before deploying.

Typical savings from these 5 filters:

Filter Typical Volume Reduction Why
Health checks 15-30% LBs check every 5-10 seconds per backend
Debug logs 10-25% Apps default to DEBUG in many frameworks
GKE system noise 10-20% kube-system is extremely chatty
Internal VPC flows 5-15% High-traffic internal services
Cloud SQL routine 3-5% Constant checkpoint/analyze messages

Combined: 40-60% reduction in ingestion volume.

📦 Step 2: Route Logs to Cheaper Storage with Sinks

Some logs you need for compliance or debugging, but don't need in Cloud Logging's expensive $0.50/GB storage. Route them to Cloud Storage (GCS) at $0.020/GB instead:

# Create a GCS bucket for archived logs
resource "google_storage_bucket" "log_archive" {
  name     = "${var.project_id}-log-archive"
  location = "US"

  # Use lifecycle rules from previous post!
  lifecycle_rule {
    action {
      type          = "SetStorageClass"
      storage_class = "NEARLINE"
    }
    condition {
      age                   = 30
      matches_storage_class = ["STANDARD"]
    }
  }

  lifecycle_rule {
    action {
      type          = "SetStorageClass"
      storage_class = "COLDLINE"
    }
    condition {
      age                   = 90
      matches_storage_class = ["NEARLINE"]
    }
  }

  lifecycle_rule {
    action {
      type = "Delete"
    }
    condition {
      age = 365
    }
  }

  labels = local.common_labels
}

# Sink: Route data access audit logs to GCS
resource "google_logging_project_sink" "audit_to_gcs" {
  name        = "audit-logs-to-gcs"
  project     = var.project_id
  destination = "storage.googleapis.com/${google_storage_bucket.log_archive.name}"

  filter = "logName:\"cloudaudit.googleapis.com/data_access\""

  unique_writer_identity = true
}

# Grant the sink's service account write access to the bucket
resource "google_storage_bucket_iam_member" "log_writer" {
  bucket = google_storage_bucket.log_archive.name
  role   = "roles/storage.objectCreator"
  member = google_logging_project_sink.audit_to_gcs.writer_identity
}

# Exclude audit logs from _Default sink (avoid double-paying)
resource "google_logging_project_exclusion" "audit_from_default" {
  name        = "exclude-audit-from-default"
  project     = var.project_id
  description = "Audit logs go to GCS, skip Cloud Logging"

  filter = "logName:\"cloudaudit.googleapis.com/data_access\""
}
Enter fullscreen mode Exit fullscreen mode

⚠️ Gotcha: If you create a sink WITHOUT adding an exclusion on the _Default sink, you pay twice - once for Cloud Logging ingestion and once for the sink destination. Always pair sinks with exclusions to avoid double-paying.

Cost comparison for 100 GB/month of audit logs:

Destination Storage Cost Ingestion Cost Total
Cloud Logging (default) Included $50/month $50/month
GCS Standard $2/month $0 $2/month
GCS Nearline (after 30d) $1/month $0 $1/month

That's a 96% savings on audit log storage while keeping them accessible for compliance.

⏱️ Step 3: Custom Retention Policies

Not all logs need 30 days of retention. Debug logs from last week? Useless. Reduce retention to save on storage:

# Custom log bucket with 7-day retention for debug/dev logs
resource "google_logging_project_bucket_config" "short_retention" {
  project        = var.project_id
  location       = "global"
  bucket_id      = "short-retention-logs"
  retention_days = 7
  description    = "Short retention for debug and development logs"
}

# Sink to route dev namespace logs to the short-retention bucket
resource "google_logging_project_sink" "dev_to_short" {
  name        = "dev-logs-short-retention"
  project     = var.project_id
  destination = "logging.googleapis.com/projects/${var.project_id}/locations/global/buckets/short-retention-logs"

  filter = <<-EOT
    resource.type = "k8s_container"
    AND resource.labels.namespace_name =~ "^dev-"
  EOT

  unique_writer_identity = true
}
Enter fullscreen mode Exit fullscreen mode

⚠️ Warning: Reducing retention below the current value deletes existing logs that fall outside the new window. There is no undo. Plan this change during a maintenance window and communicate with your team first.

🏢 Multi-Project Exclusion Module

Deploy the same exclusion filters across every project in your org:

variable "project_ids" {
  type        = list(string)
  description = "List of project IDs to apply exclusion filters"
}

variable "exclusion_filters" {
  type = map(object({
    description = string
    filter      = string
  }))
  default = {
    health-checks = {
      description = "Exclude LB health check logs"
      filter      = "resource.type = \"http_load_balancer\" AND (httpRequest.requestUrl = \"/health\" OR httpRequest.requestUrl = \"/healthz\")"
    }
    debug-logs = {
      description = "Exclude DEBUG severity"
      filter      = "severity = \"DEBUG\""
    }
    gke-system = {
      description = "Exclude kube-system INFO and below"
      filter      = "resource.type = \"k8s_container\" AND resource.labels.namespace_name = \"kube-system\" AND severity <= \"INFO\""
    }
  }
}

# Deploy all exclusions to all projects
resource "google_logging_project_exclusion" "org_wide" {
  for_each = {
    for pair in setproduct(var.project_ids, keys(var.exclusion_filters)) :
    "${pair[0]}-${pair[1]}" => {
      project     = pair[0]
      filter_key  = pair[1]
      filter_conf = var.exclusion_filters[pair[1]]
    }
  }

  name        = each.value.filter_key
  project     = each.value.project
  description = each.value.filter_conf.description
  filter      = each.value.filter_conf.filter
}
Enter fullscreen mode Exit fullscreen mode

New project? Add it to project_ids. New filter? Add it to exclusion_filters. terraform apply. Done across every project instantly. ✅

💡 Quick Reference: What to Do First

Action Effort Savings
Add health check exclusion filter 5 min 15-30% of ingestion
Add debug log exclusion 5 min 10-25% of ingestion
Add GKE system namespace exclusion 5 min 10-20% of ingestion
Route audit logs to GCS with sink 15 min 96% on audit storage
Set 7-day retention for dev logs 10 min Reduced retention cost
Deploy org-wide exclusion module 20 min Consistent across all projects

Start with the health check exclusion. It's 5 minutes of Terraform and typically eliminates the single largest source of log waste. 🎯

📊 TL;DR 

Cloud Logging ingestion    = $0.50/GB (adds up FAST)
Free tier                  = 50 GB/month/project (not enough)
Exclusion filters          = drop logs before ingestion = $0 cost
Health check logs          = usually the #1 volume source, exclude them
Debug logs in prod         = pure waste, exclude them
Sinks to GCS               = $0.020/GB vs $0.50/GB (96% cheaper)
Sink + no exclusion        = you pay TWICE (don't forget the exclusion!)
_Required bucket           = free, can't modify (admin audit logs)
Retention < 30 days        = no impact on bill (already included)
Test filters first         = excluded logs are gone forever
Enter fullscreen mode Exit fullscreen mode

Bottom line: Cloud Logging charges $0.50 for every GB you don't explicitly exclude. Five exclusion filters deployed in Terraform can cut your logging bill by 40-60%. That's thousands per year at scale for 30 minutes of work. 📉

Open your GCP billing console right now. Filter by Cloud Logging. If it's in your top 5 services by cost, come back here and deploy those exclusion filters. Your CFO will think you're a genius. 😀

Found this helpful? Follow for more GCP cost optimization with Terraform! 💬

Top comments (0)