Kubernetes Observability 2026: Metrics, Logs and Traces Without the Noise

TL;DR: Kubernetes observability is three signals with three different jobs — metrics tell you that something is wrong, logs tell you what happened, traces tell you where it went — plus a fourth everyone forgets: events, the cluster's own diary. The craft is not collecting more; it's discipline: golden signals per service, structured logs shipped off the pod, traces sampled honestly, strict label cardinality, and alerts that fire on symptoms users feel. Wire those in and incidents get shorter instead of dashboards getting longer.

Originally published on DevOps Enginer.

The 2 a.m. questions

Every incident, whatever the technology, comes down to two questions: what changed? and what is it doing now? I learned the audit-trail half of that lesson teaching myself AWS on my own account — my habit was to open CloudTrail and read what had actually been created and changed, instead of guessing from memory. Clusters deserve the same respect: the systems that answer those two questions in minutes are called "observable," and the ones that answer in hours are called "down."

Kubernetes makes this both easier and harder than the VM world. Easier, because everything is an API object with a state and a history. Harder, because the things you used to SSH into — containers, pods — are ephemeral by design: the pod that errored at 02:00 has been rescheduled twice by 02:15, and its logs went with it. Observability in Kubernetes is therefore not a monitoring add-on; it is the deliberate act of exporting state out of things that are designed to disappear.

This guide is the working model: what to collect, what each signal is actually for, and the disciplines that keep the signal bill — and the alert noise — from eating the team.

The three signals, and the jobs they're for

• Metrics — numbers over time (request rate, error rate, latency percentiles, CPU saturation). Cheap to store, fast to query, perfect for alerting and dashboards. Useless for "why" — a metric tells you the error rate tripled, never which error.
• Logs — timestamped event records. The detail layer: stack traces, request IDs, the specific "what happened." Expensive at volume, worthless unstructured — a log you can't query during an incident is a diary nobody reads.
• Traces — the path of one request across services, as timed spans. The only signal that answers "which hop in this chain is slow" without an afternoon of log archaeology. Costs you instrumentation effort, repaid on the first cross-service incident.

And the unofficial fourth: events — the cluster's own record of decisions (scheduled, pulled, started, OOMKilled, failed probes). kubectl get events is the first stop in the troubleshooting workflow for a reason. Know its limit: events are retained for roughly an hour by default, so they are a live-incident tool, not an archive — anything you need historically must be exported.

Metrics: golden signals, ruthless cardinality

The Prometheus model won this layer: applications and exporters expose metrics, Prometheus scrapes and stores them, Grafana draws them. Managed equivalents exist on every cloud; the concepts transfer. Two disciplines make or break it:

Per-service golden signals. For every service users touch, four numbers on one row: latency (with percentiles, not averages — the average hides the suffering), traffic (requests per second), errors (rate and ratio), saturation (how full the service is: CPU vs. request, memory headroom, queue depth). This RED/USE-style framing is what turns 400 dashboards into one glance: a service is healthy or one of its four numbers is red. For the cluster itself, the same idea one level down: node saturation, pending pods, and requests-vs-usage — the last of which doubles as your cost signal (see the cost optimization guide).

Cardinality is the budget. Every unique label combination is a separate time series, and series are what you pay for — in memory, storage, and query speed. Labels are for dimensions with few values (service, namespace, status code class). They are never for user IDs, request IDs, full URLs, or error message text — those belong in logs and traces. The classic self-inflicted wound is a path label with unbounded values quietly multiplying a metric into a million series. Cardinality discipline is to metrics what requests discipline is to cost: invisible when done right, an incident when ignored.

Alert on the golden signals, recorded at sane resolution, and let recording rules pre-compute the expensive queries your dashboards and alerts run every minute.

Logs: structured, shipped, and correlated

Three rules convert logs from storage expense into incident fuel:

1. Structured or it didn't happen. JSON (or equivalent key-value) logs with level, service, and a correlation/request ID in every line. The request ID is the thread that ties a log line to a trace to a user complaint — propagate it at ingress and through every downstream call.
2. Off the pod, always. kubectl logs reads the current container on the current node; a rescheduled pod takes its history with it — the checklist version of this rule is blunt: pod logs die with the pod. Ship to a central store (Loki, Elasticsearch/OpenSearch, a cloud logging service) with retention tiers: hot and searchable for days, archived cheap for the compliance horizon.
3. Log decisions and failures, not chatter. A line per request at INFO is a billing strategy for your logging vendor. Log the boundaries (request received/completed at DEBUG or sampled), the anomalies, and the state changes — and make error logs carry the context a stranger needs at 2 a.m.: what was attempted, with what key inputs, and what the system decided.

Traces: OpenTelemetry and honest sampling

Distributed tracing's centre of gravity is now OpenTelemetry: one instrumentation standard, vendor-neutral, with auto-instrumentation for common frameworks and manual spans where the story needs them. The practical shape in a cluster:

• Instrument the edges first — ingress to service, service to service, service to database. A trace that covers the hops covers the argument.
• Propagate context properly (W3C traceparent headers) — a broken chain is worse than none, because it looks like data.
• Sample honestly. Head-based sampling at a modest rate for normal traffic, tail-based sampling (keep the interesting ones) where your collector supports it, and one sacred rule: errors and slow requests are always kept. Tracing everything is a storage bill; tracing nothing is a hope; tracing the failures is the point.
• Connect the signals: exemplars and trace IDs in your metrics and logs so an alert can jump to the metric, the metric to example traces, the trace to the exact log lines. Observability pays out at the joins.

Alerting: symptoms, budgets, and the noise floor

The fastest way to destroy an observability practice is alerting on causes ("CPU > 80%") until the team mutes the channel. The working rules:

• Page on symptoms users feel: error ratio burning, latency SLO at risk, saturation heading for the cliff. Causes belong on dashboards and in tickets, where they can be investigated at human speed.
• Every page carries a runbook link and answers "what do I do now?" An alert without a next action is a criticism, not a tool.
• An alert budget is real. Track pages per week the way you track cost. When the budget is blown, the work is alert hygiene — deleting, demoting, or fixing the flapping few — before any new feature work. This is also the honest metric of your self-healing setup: the best alert is the one the system resolved before a human was involved, and said so quietly in the morning report.
• Multi-window burn-rate alerts (fast burn pages, slow burn tickets) beat fixed thresholds for SLOs, because they measure the thing the business actually promised.

Dashboards: three that earn their tabs

Dashboard sprawl is cardinality's visual cousin. The set that survives contact with real operations is small:

1. Service overview — the four golden signals for one service, with deploy markers overlaid. The deploy overlay is the highest-value line on any dashboard: most incidents correlate with a change, and "what changed?" should be answerable by looking, not archaeology.
2. Cluster capacity — node saturation, pending pods, requests-vs-usage by namespace, and the top offenders list. Operations and cost share this view, which is as it should be.
3. The incident view — one screen the on-call actually opens at 2 a.m.: golden signals for the affected service, its dependencies' health, recent events, and links into logs and traces pre-filtered to the service. Build it during a calm week; you will not build it during an incident.

Everything else is a report, a debugging scratchpad, or nostalgia. Archive aggressively; a dashboard nobody opens in a quarter is a liability with a URL.

Observability as code

Dashboards and alerts defined by clicking are configuration drift with a login page. The GitOps answer applies here exactly as it does to workloads: dashboards as JSON in git, alert rules as PrometheusRule objects or files in the repo, recording rules reviewed in pull requests. The benefits compound: changes are reviewed (a threshold change is a policy change), history is blame-able in the good sense, and a rebuilt monitoring stack comes back identical — our Argo CD tutorial pattern works for the observability stack itself, which can live in the same app-of-apps layout as everything else you run.

What observability costs (and how it balloons)

Observability spend surprises teams the way cluster spend does: through accumulation, not decisions. The three leaks:

• Metric cardinality — the unbounded label, priced per series (see above).
• Log volume — the chatty service at INFO, priced per gigabyte, forever. Sample, tier, and give every team's its own volume number — showback works here exactly as it does for compute.
• Trace retention — full-fidelity tracing retained for months, priced per span. Sample the boring, keep the broken, archive the rest cold.

Put the observability bill on the same monthly one-pager as the cluster bill. A monitoring stack that costs a visible fraction of the infrastructure it watches is normal; an unexamined one grows until finance asks the question in a tone nobody enjoys.

Mistakes that make observability useless

1. Collecting everything, querying nothing. Data without a dashboard, alert, or runbook attached is storage, not observability. Every signal should name the decision it feeds.
2. Averages for latency. The mean of a bimodal distribution describes nobody's experience. Percentiles (p50/p95/p99) or it didn't happen.
3. Unbounded labels. One user_id label can take down the metrics store that was supposed to tell you about the outage — during the outage.
4. Logs only on the pod. The first rescheduling during an incident deletes the evidence. Shipping is not optional.
5. Alerting on causes until nobody listens. A muted channel is worse than no channel: it looks like coverage. Symptom-based paging, budgeted and reviewed, or expect the 2 a.m. page to be the one everyone sleeps through.
6. Observability configured by clicking. Unreviewed thresholds and dashboards that vanish when the stack is rebuilt. If it matters, it lives in git.

FAQ

Prometheus/Grafana, or my cloud's managed monitoring?
Start with whichever your team will actually open — a managed stack beats an unmaintained Prometheus. The concepts in this guide (golden signals, cardinality, symptom alerts) are portable across both; move when scale, multi-cluster needs, or cost make the case, not before.

Do I need traces if I have good logs?
Logs tell you what each service did; traces tell you how the request's time and errors distribute across services. With two services, logs plus correlation IDs may carry you. Past a handful of hops, the cross-service incident without traces is an afternoon of archaeology — that's the moment tracing pays for its instrumentation.

What should retention be?
Tier it by use: metrics at full resolution for weeks, downsampled for a year or more (capacity questions are seasonal); logs hot and searchable for days to weeks, archived to cheap storage for your compliance horizon; traces sampled with short retention — their value is in the incident week, not the archive.

How do I observe the observability stack itself?
Meta, but necessary: scrape the scrapers, alert on Prometheus/Loki/collector health from a dead-simple external check (even an uptime ping on the alerting path), and rehearse the "monitoring is down" case once — the worst incidents are the ones you discover from users because the dashboards were the thing that broke.

Where does this fit the production checklist?
Section 5 of the Kubernetes Production Checklist — the four signals per service, logs shipped off-pod, alerts with runbooks, and the one-command answers to "what changed / what's running / what's it costing" — is this guide compressed into checkboxes. (The full checklist PDF is the free lead magnet on the DevOps Enginer homepage.)

Conclusion

Observability is not a stack you install; it's the property that your two 2 a.m. questions — what changed, and what is it doing now — have fast, trustworthy answers. Metrics for the that, logs for the what, traces for the where, events for the cluster's own account of itself, all of it disciplined by cardinality, structure, honest sampling, and alerts that respect the humans they wake. Build the three dashboards, wire the deploy markers, put it all in git, and review the alert budget like a bill — because it is one. Your future on-call self, mid-incident, is the customer. Build for them.