It usually starts the same way. A PagerDuty alert goes off, Slack lights up with messages, and someone posts a screenshot showing HTTP 500 errors climbing while customers begin reporting failed requests. Within minutes, everyone is staring at different dashboards. One engineer is looking at Prometheus graphs, another is digging through logs, someone else is opening distributed traces, and a fourth person is convinced Kubernetes is the problem because a few pods restarted twenty minutes ago. Nobody is wrong, but nobody has the complete picture either.
This is the hidden cost of modern observability. Most engineering teams
have invested heavily in collecting telemetry, yet incident response
still feels like detective work. The problem isn't a lack of data. Quite the opposite. We have more metrics, logs, and traces than ever before, but they often live in separate places, forcing engineers to mentally connect information that should already be linked. Every context switch steals a few more seconds, and during an outage those seconds quickly become minutes. Improving Mean Time to Resolution (MTTR) is rarely about collecting more telemetry; it's about making the telemetry work together so engineers spend less time searching and more time solving.
The Cost of the Three Panes of Glass
For years, observability platforms evolved as separate ecosystems.
Metrics were collected by Prometheus, logs flowed into systems like Loki or Elasticsearch, and distributed traces landed in Tempo, Jaeger, or Zipkin. Each tool solved a different problem extremely well, but incident response became an exercise in jumping between multiple browser tabs.
Imagine investigating an outage where your payment service suddenly
begins returning HTTP 500 responses. Your first stop is usually a
Prometheus dashboard. The graphs immediately tell you something is
wrong: request latency has increased dramatically, error rates are
climbing, and request throughput is beginning to fall. Metrics are
excellent at telling you what is happening across the system, but
they deliberately sacrifice detail for aggregation. They cannot explain
why one specific request failed or which dependency introduced the
delay.
Naturally, the next step is opening your logging platform. Now you're
searching around the same timestamp, filtering by Kubernetes namespace,
pod name, container name, or service labels. Hundreds or even thousands
of log entries appear, many of which have nothing to do with the
incident. You narrow the time window, adjust the filters, search for
"ERROR," and hope the application logged something useful. Sometimes it
did. Sometimes it didn't. Even when you find an interesting error
message, you're still guessing whether it belongs to the requests
responsible for the spike you saw on the metrics dashboard.
Distributed tracing becomes the third stop. Now you're trying to locate
the trace that corresponds to the failed request you observed earlier.
If your services process thousands of requests per second, finding the
right trace using only timestamps can feel like searching for a specific grain of sand on a beach. Every manual lookup increases cognitive load because your brain is acting as the integration layer between three independent systems.
This is what many engineers jokingly call the "three panes of glass" problem. The telemetry exists. The answers exist. What doesn't exist is a natural path between them. During a production incident, engineers shouldn't have to remember timestamps, copy request IDs between tools, or manually correlate unrelated datasets. That work should already be done before the alert even arrives.
The Glue: How OpenTelemetry and Prometheus Actually Correlate
The good news is that modern observability doesn't require replacing
your entire stack. Instead, it relies on common standards that allow
metrics, traces, and logs to reference one another automatically.
OpenTelemetry has become the foundation for this approach by providing a consistent way to generate, enrich, and transport telemetry across
distributed systems.
At the center of this architecture sits the OpenTelemetry Collector. Rather than every application exporting telemetry directly to multiple destinations, applications send metrics, traces, and logs to the Collector. From there, the Collector can enrich telemetry with
Kubernetes metadata, normalize attributes using OpenTelemetry semantic
conventions, batch data efficiently, sample traces when necessary, and
forward telemetry to systems such as Prometheus-compatible backends,
Grafana Loki, Tempo, Jaeger, or other storage platforms.
The first piece of correlation comes from W3C Trace Context.
traceparent:
00-4bf92f3577b34da6a3ce929d0e0e4736
Applications inject the active trace_id into every log entry.
{
"service.name":"payment-service",
"level":"ERROR",
"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
"message":"Database connection timeout"
}
Prometheus Exemplars bridge aggregated metrics to individual traces.
http_server_request_duration_seconds
Value: 5.8 seconds
Exemplar:
trace_id=4bf92f3577b34da6a3ce929d0e0e4736
Anatomy of a Failure: Following the Telemetry Instead of Guessing
It's a normal weekday afternoon when your payment service begins
failing. Within a minute, Prometheus fires an alert because HTTP 500
responses have increased from less than one percent to nearly thirty
percent. Customer support reports failed checkouts, and engineers start
assembling in the incident channel.
The metrics dashboard confirms the impact. Request latency has increased from around 200 milliseconds to almost six seconds, throughput is declining, and the error rate continues climbing. Instead of searching through logs, an engineer clicks a Prometheus Exemplar attached to the latency graph.
The distributed trace shows the request flowing through the API Gateway, the payment service, inventory validation, and finally spending nearly six seconds waiting on PostgreSQL.
Gateway
│
▼
Payment Service
│
▼
Inventory Service
│
▼
PostgreSQL
Database Span:
Duration: 5.8 seconds
Status: ERROR
Exception: Connection Pool Timeout
Because every application log contains the same trace_id, the
observability platform automatically filters the logs associated with
that request.
ERROR
Unable to acquire database connection
Pool Size: 100
Active Connections: 100
Waiting Requests: 243
Connection Timeout: 30 seconds
A few lines later:
WARN
Long-running transaction detected
Transaction Duration: 94 seconds
The root cause is immediately clear. A long-running transaction
exhausted the PostgreSQL connection pool, causing new requests to wait
until they timed out. Metrics revealed what changed, the trace
showed where the delay occurred, and the logs explained why it
happened---all without switching tools.
Wrap-up
Engineering teams have largely solved the problem of collecting
telemetry. The harder problem is making those signals tell a single,
coherent story during an incident.
Technologies like W3C Trace Context, Prometheus Exemplars,
trace_id log injection, OpenTelemetry semantic conventions,
and the OpenTelemetry Collector transform isolated telemetry into
connected evidence that follows every request across your system.
When the next 3 AM alert arrives, you shouldn't have to play detective
across half a dozen browser tabs. You should be able to click a graph,
open the trace responsible for the anomaly, inspect the logs attached to that request, identify the root cause, and get back to sleep.
Top comments (0)