The observability bill has become a board-level worry and the numbers back it up. In the 2026 industry predictions, more than a third of teams expect to spend over a million dollars and a meaningful slice expects to cross five million. The common responses treat the symptoms. Teams shorten retention, cut sampling blindly or reopen the vendor contract. These levers exist but they all act too late in the chain. The real control point sits earlier, before storage and billing, inside the Collector.
The idea fits in one sentence. Observability cost is not driven by what you observe but by what you decide to keep and that decision is made most effectively at the source.
Let'see together in details.
Why the bill drifts
Three mechanisms combine and the first one dwarfs the other two.
Cardinality is the cost driver that surprises people most. A metric is not billed per data point, it is billed per active time series, meaning per unique combination of labels. A latency metric carrying service, pod, instance, endpoint and status_code already produces thousands of series. Add one unbounded label, a user_id or a raw request_path and you jump from thousands to millions of series with no gain in readability. Cardinality grows multiplicatively rather than additively and it is what inflates storage and query compute.
Raw volume comes next. Traces kept at 100 percent, redundant application logs, health checks logged every second on every pod: the mass piles up and the transport and ingestion cost follows.
The reflex closes the list. The "keep everything, sort it out later" habit feels safe at instrumentation time and turns expensive six months on. Whatever is never queried is rarely deleted, so the telemetry debt grows quietly.
The Collector as a control point
A Collector pipeline, whether the OpenTelemetry Collector or vmagent on the metrics side, always follows the same shape: receivers on the way in, processors in the middle, exporters on the way out. The cost battle is won in the processors because that is the last place you control the data before it becomes a line on an invoice.
Working at the source rather than at the backend changes the nature of the problem. Filtering at the backend means you pay to ingest data that you then throw away. The same filtering at the Collector drops the data before transport, ingestion and storage. You stop paying for what you do not keep.
Four families of processors carry most of the gain:
- Filtering, to drop signals with no operational value, typically health checks and debug logs in production.
- Attribute transformation, to remove or recode high-cardinality labels before they create series.
- Aggregation, to move from per-pod to per-service granularity on metrics you never query at the pod level.
- Trace sampling, to keep only what serves diagnosis.
Cardinality, where the biggest win lives
Before you sample anything, deal with cardinality, because it is the lever with the best effort-to-gain ratio.
The base rule is simple. A label whose values are unbounded has no place on a metric. A request id, a user id or a non-normalized URL path belongs in a trace or a log, not in a time series. When you need to tie a metric back to a specific trace, use exemplars, which are built for exactly that.
Next comes aggregating away the dimensions you do not use. For each label, ask whether you actually queried it in the last quarter. Labels that never serve as a filter or a grouping in your queries are direct candidates for removal at the relabeling step. On the Prometheus and VictoriaMetrics side, vmagent stream aggregation pre-aggregates those series on the fly before write, which cuts the active series count without touching application code.
Sampling, the right trade-offs
Sampling is often framed badly. Random sampling throws away as many incidents as it does routine traffic, which is the opposite of what you need.
For traces, tail sampling changes the picture. The keep-or-drop decision is made once the trace is complete, so you can keep 100 percent of error traces, 100 percent of traces above a latency threshold and a small percentage of nominal traces. You retain everything useful for diagnosis and discard the background noise with no loss of signal.
For metrics, you do not sample, you aggregate and downsample over time. Fine granularity on the recent hours and coarser granularity beyond covers almost every use case.
For logs, filtering and routing come first. The noise goes to the bin and the rest is sorted by severity and destination.
Data contracts, to make it stick
Everything above stays fragile if it depends on each team's goodwill. The data contract is the piece that turns one-off cuts into durable policy.
A data contract defines, per signal, what is allowed: which labels, which cardinality budget, which retention and which destination. It is written in the Collector configuration and checked in continuous integration, the same way as any other infrastructure artifact. The OpenTelemetry semantic conventions give the shared vocabulary that keeps those contracts readable across teams. From there, a new high-cardinality metric no longer reaches production by accident, it is stopped at review.
An illustrative before and after
The figures below describe a representative scenario, not a specific engagement. They show the order of magnitude of the savings rather than promise a result.
Take a mid-size Kubernetes platform, roughly 120 services and 800 pods, instrumented across all three signals.
| Signal | Before | Action at the Collector | After |
|---|---|---|---|
| Metrics | 6,000,000 active series | Drop 2 unbounded labels, aggregate per service | 1,800,000 series (down 70 percent) |
| Traces | 100 percent of spans, around 800 GB per day | Tail sampling, errors and slow traces at 100 percent, nominal at 8 percent | around 140 GB per day (down 82 percent) |
| Logs | 1.2 TB per day | Filter noise, route compliance data to cold storage | 400 GB hot plus 300 GB cold archived |
On that basis, a monthly observability bill in the region of 40,000 euros typically lands in a 14,000 to 18,000 euro range, with the final figure depending on the vendor pricing model. The absolute number is not the point, the proportion is. Most of the gain comes from cardinality and tail sampling, two Collector settings, with no change to application code.
The regulated constraint
In a regulated environment, cost control cannot come at the price of compliance. Some data, audit logs and traces with evidentiary value, must be kept whatever happens, sometimes for long durations imposed by frameworks such as DORA or NIS2.
The answer is not to keep everything in hot storage, it is to route. Operational signals live in fast expensive storage with short retention. Compliance signals go to cold storage, slow and cheap but durable and exportable if an audit calls for it. The Collector is the natural place to apply that split through routing rules. You cut the hot bill without touching your obligations.
Wrapping up
The Collector is not plumbing, it is where your cost policy is enforced. Treat telemetry as a budget. Decide upfront what is worth keeping, write that decision into data contracts and apply it at the source. Cardinality first, sampling next, routing for the rest. That is the difference between observability that drifts and observability that holds over time.
Top comments (0)