Your Observability Bill Is a Codebase Nobody Reviews

It started with one $120K/year config fix. By the time I stopped asking questions, it was $150K+/year in waste — and a FinOps framework I wish I'd had from day one.

Observability Spend Feels Like Insurance

I'm an Sr. SRE at a company that processes payroll for hundreds of thousands of employees. Datadog is our nervous system — monitoring, APM, logs, synthetics, the works.

Like every engineering org, our Datadog bill grew steadily. New services, new teams, new logs. Nobody questioned it because observability spend feels like insurance — you don't optimize your fire alarm budget.

That instinct is exactly why the waste piles up. Your observability bill is a codebase: it accumulates config, it drifts, it carries dead code. But unlike your actual codebase, nobody reviews it. There's no PR, no linter, no one asking "do we still need this?"

Here's a year of what we found when I finally started asking.

The $120K Question

One month, our Sensitive Data Scanner cost hit $390/day. That's $11,700/month. For scanning data for PII.

I asked one question: "What are we actually scanning?"

The answer was: everything. APM spans. RUM sessions. Data sources where PII couldn't possibly exist. The default config scanned it all, and nobody had revisited the scope since initial setup.

The fix was a scoping change — restrict the scanning group to the log sources where PII can actually appear, and exclude the rest:

# Sensitive Data Scanner — scope the group, don't scan the world.
# Before: the group had no filter, so every log (plus APM/RUM) was in scope.
# After: only route customer-facing services through the expensive scanner.
scanning_group:
  name: pii-scan-prod
  # Only logs matching this query are scanned. Everything else is skipped.
  filter:
    query: "service:(checkout-api OR onboarding OR user-api) env:prod"
  is_enabled: true

Daily cost dropped from $390 to $56. That's an 87% reduction — $120,000/year in savings from a config change that took an afternoon.

But the interesting part isn't the fix. It's why nobody found it sooner. So I started looking at our entire observability footprint the same way.

The Taxonomy of Observability Waste

I found waste in places I didn't expect. Three patterns showed up again and again.

The Ghost Service Problem

Our Datadog Service Catalog showed services that didn't actually exist. A shared library was overriding service_name in application instrumentation, creating phantom entries alongside the real service name. These ghost services had their own monitors, their own alerting gaps, and their own entries in our coverage audit — inflating our service count and muddying our incident response routing.

The root cause was a well-meaning shared logging wrapper that hard-coded the service tag:

# The shared library did this — overriding whatever the service set.
# Every app that imported it reported the SAME ghost service_name,
# in addition to its real one.
tracer.set_tags({"service": "shared-logging-lib"})  # <-- the ghost

# Fix: let each service own its identity. Set it once, at the edge,
# from config — never inside a shared dependency.
tracer.set_tags({"service": os.environ["DD_SERVICE"]})

I fixed this across 30+ production repositories spanning every business domain. Every service had the same ghost contamination from the same library.

Lesson: Your service catalog is probably lying to you. If you haven't audited how service names are set at the instrumentation layer, you have ghost services.

The Staging-in-Production Index

Our production log index was breaching its quota. My initial assessment: "organic growth." My manager disagreed and looked closer — 13M day-over-day increase on two services.

The culprit: 24.5 million staging load balancer access logs per day were being indexed in the production index. A staging ALB was routing its logs to the same index as production. That's 27% of the index's volume — pure waste, consuming quota that existed to protect production observability.

You can catch this in minutes by grouping the offending index by environment:

# Datadog log analytics — is staging leaking into prod?
index:main-prod
| group by @env
| count

# Result that started the investigation:
#   env:production   66.1M
#   env:staging      24.5M   <-- should be zero in this index

The fix was an exclusion filter on the production index so staging traffic never gets billed as production data.

Lesson: Log indexes are routing rules, and routing rules drift. If you haven't checked which environments are actually landing in each index, you're paying production prices for staging data.

The Scanning-Everything Default

Back to the scanner. The pattern is: you enable Sensitive Data Scanner for compliance. You configure it to scan "logs." A year later, your APM and RUM data has grown 5x, and it's scanning all of it because the scope was never tightened. Nobody gets an alert that says "you're scanning data that can't contain PII." The bill just grows.

Lesson: Every observability feature that charges by volume will silently expand its scope as your infrastructure grows. The defaults are designed to capture everything, not to be cost-efficient.

The Observability FinOps Framework

After a year of hunting waste, here's the framework I wish I'd had from day one. It moves from a one-time audit at the base to continuous prevention at the top.

flowchart BT
    L1["Layer 1: Know what you pay for"]
    L2["Layer 2: Measure each signal"]
    L3["Layer 3: Govern in the pipeline"]
    L4["Layer 4: Cost as an incident signal"]
    L1 --> L2 --> L3 --> L4

Layer 1: Know What You're Paying For

Most engineers can't answer these questions about their observability platform:

• How many services are in your catalog? How many are real?
• Which log indexes are ingesting data from which environments?
• What percentage of your PII scanning covers data that could actually contain PII?
• Which monitors have notification targets that don't resolve to a real on-call team?

If you can't answer these, you can't optimize. The first step is an audit — not of your bill, but of your configuration.

Action: Run a service catalog audit. Match every service to a real deployment. Delete the ghosts. Check your log index routing — make sure staging stays in staging. Review your scanning groups — remove data sources where PII can't exist.

Layer 2: Measure the Cost of Each Signal

Not all observability data has equal value. A log line from a payroll transaction is worth more than a health check response. But they cost the same to ingest.

Action: For your top 10 services by log volume, answer: what percentage of these logs have been read by a human in the last 30 days? If a log pattern has never been read, it's a candidate for exclusion or downsampling.

This is the layer where auditing by hand stops scaling — reading log samples across 50 services to judge which patterns carry diagnostic value is weeks of work. It's also where AI tooling earns its place, which is a whole story of its own. (More on that in a follow-up piece.)

Layer 3: Build Cost Governance Into the Pipeline

One-time audits decay. The real win is building cost awareness into your engineering workflow:

• Terraform modules with cost guardrails: our monitoring Terraform module standardizes alerting across all teams. Extending it with default exclusion filters means every new service starts with sane defaults.
• Log pipeline restructuring: we restructured our log pipeline into main + sub-pipelines, giving us a single place to apply org-wide filtering before logs hit indexes.
• Proactive quota monitoring: instead of reacting to quota breaches, we monitor the rate of change in log volume. A service that doubled its log output this week gets flagged before it breaches the quota next month.

Baking the exclusions into the module is what makes this stick — new services inherit the guardrails instead of relearning them:

# monitoring module — every service that adopts it gets sane defaults.
variable "health_check_paths" {
  type    = list(string)
  default = ["/health", "/healthz", "/ping", "/ready"]
}

resource "datadog_logs_index" "service" {
  name = var.service_name

  # Drop health-check noise before it's ever billed.
  exclusion_filter {
    name       = "exclude-health-checks"
    is_enabled = true
    filter {
      query       = "@http.url_details.path:(${join(" OR ", var.health_check_paths)})"
      sample_rate = 1.0  # 1.0 = exclude 100%
    }
  }
}

Layer 4: Make Cost a First-Class Incident Signal

We added a cost monitor that alerts when any dimension's cost increases more than 10% month-over-month:

# Datadog monitor — treat a cost spike like any other regression.
# Alerts when this week's est. indexed-log cost per service jumps 10%+ WoW.

Monitor type:  Change Alert
Metric:        datadog.estimated_usage.logs.ingested_events
Group by:      service
Alert when:    change (week_before) is above 10 %
Notify:        @slack-observability-finops

This monitor has caught:

• A new service that shipped with DEBUG logging in production
• A message consumer that started logging every message body
• A load balancer configuration change that doubled access log volume

Cost spikes are often the first signal of a misconfiguration. They surface faster than performance degradation or error rates because they reflect total volume, not just error ratios.

The Numbers

Here's what this approach delivered in one year:

Finding	Savings	How Found
Scanner covering unnecessary data sources	$120,000/year	Manual audit after questioning the bill
Staging logs in production index	~$30K/year (estimated)	Cost anomaly investigation
Ghost service cleanup (30+ repos)	Reduced noise, better routing	Service catalog audit

Total identified savings: $150K+/year, most of it surfaced by one engineer asking questions.

Your First Week

If you manage an observability platform and haven't done a cost audit, here's your week:

• Monday: List every service in your catalog. Flag any you can't match to a real deployment.
• Tuesday: For each log index, check which environments are routing to it. Staging in production? Fix it.
• Wednesday: Review your PII scanning scope. What data sources are being scanned? Which ones could actually contain PII?
• Thursday: Pick your top 3 services by log volume. Read 50 log lines from each. How many are useful?
• Friday: Calculate what you found. Multiply daily waste by 365. That's your pitch to leadership.

You don't need any special tooling to start. You need curiosity and a spreadsheet.

Ask the Boring Questions

The best SRE work isn't heroic incident response. It's the quiet, systematic questioning of things everyone accepts as normal.

Why are we scanning this? Why is staging in this index? Why does this service exist in our catalog?

$120K/year was sitting in plain sight. It just needed someone to ask.

If you've found waste hiding in your own observability bill, I'd love to hear about it in the comments — the taxonomy only gets more useful the more patterns we name.

---

Nishant Arora is a Senior SRE specializing in observability, incident response, and platform reliability. He builds and maintains the monitoring and payment processing infrastructure for a platform serving hundreds of thousands of employees. He's passionate about making infrastructure costs as visible as infrastructure performance.