You see an error or latency spike.
You group by deployment and one deployment stands out. Now the actual debugging starts.
Which pods belong to it? Did one restart? Was there a rollout around the same time? Did a ConfigMap change? Was a pod rescheduled onto a bad node? Is a PVC having trouble? What Kubernetes events were firing when the spike started?
Usually, answering those questions means leaving the chart.
Copy the deployment or pod name. Open the Kubernetes dashboard. Run kubectl. Check events. Find the logs for the same workload. Adjust the time range. Check the rollout history. Then try to reconstruct what was happening ten minutes ago from a collection of views that mostly show you what exists now.
The annoying part is that you already had the starting point. The series on the chart already knew the cluster, namespace, deployment, pod, and time range.
So why am I copying any of that into another tool?
We started trying to fix this as a small UI problem in Cardinal. It ended up changing how we use OpenTelemetry to understand Kubernetes.
Start with the thing that looks wrong
The first part came almost for free from OpenTelemetry.
By the time a metric, log, or trace reaches Cardinal, it often already carries the Kubernetes identity of the thing that produced it: cluster, namespace, deployment, pod, node, and so on. We realized we could use that identity for more than filtering and grouping.
In Cardinal Explore, if the labels on a series identify a Kubernetes object — a pod, deployment, node, PVC, or even a CSI volume — we add an Infra Map link directly to that series in the chart legend. Click it and Cardinal opens the Kubernetes neighborhood for exactly what you were looking at.
For a deployment, that might be:
Deployment → ReplicaSet → Pods
with the related Service, ConfigMaps, Nodes, PVCs, PVs, and storage volumes attached where applicable.
The cluster and namespace are already known. So is the time range you were investigating. There is nothing to search for and nothing to copy.
It sounds like a small UI trick. But it changes the handoff between application telemetry and Kubernetes: the thing that looks wrong becomes the starting point for exploring the infrastructure underneath it.
The useful part isn't the graph
Getting from a chart into the right Kubernetes neighborhood solved the navigation problem. But just drawing Kubernetes objects as a graph isn't particularly useful. kubectl can already give me a large collection of objects to inspect.
During an incident, the question I actually care about is: what changed here?
This is where the OpenTelemetry Collector became more than just the thing shipping our application telemetry.
We use its Kubernetes events receiver to ingest Kubernetes events as logs. Because those events now live alongside the rest of the telemetry, Cardinal can attach them directly to the Kubernetes objects they describe.
Kubernetes events attached to a pod inside the Infra Map
If a pod is repeatedly crashing, we mark that pod unhealthy and you might see:
BackOff ×312
If a deployment rolled out during the latency spike, the rollout appears on the deployment. If a PVC is reporting mount failures, that shows up on the storage side of the same neighborhood.
We group the events into a few categories that are useful while debugging: Deploy/Scale, Image, Crash/Error, and Health. Collapsed parts of the graph inherit the worst state underneath them, so a problem doesn't disappear just because the affected pod isn't currently visible.
Kubernetes events overlaid onto the chart timeline
That changes the purpose of the graph. I don't want to inspect 40 Kubernetes objects one at a time. I want the graph to tell me which branch I should investigate first.
Historical debugging is where this gets harder
There was one problem with this approach: Kubernetes is a pretty bad historical record of itself.
Events are short-lived. Pods disappear. Jobs disappear. And once an object is gone, the ownership metadata you would normally use to navigate from that object back to its workload can disappear with it.
That matters because observability is mostly about debugging something that already happened.
If a pod died 15 minutes ago, I still want to click the spike it caused, see its Kubernetes neighborhood, and answer: what happened to that pod and what did it belong to?
This is where we leaned further into OpenTelemetry.
We use the Collector's k8s_objects receiver to collect the object metadata we need, and separately ingest Kubernetes events as logs. Because those events become normal telemetry, we can retain them for as long as the rest of the investigation instead of losing them when Kubernetes' event history disappears.
We don't need to preserve the entire Kubernetes API forever. Every 15 minutes, we snapshot just enough ownership metadata for Pods, ReplicaSets, and Jobs:
uid, name, namespace, ownerReferences
That's enough to preserve relationships like:
Pod → ReplicaSet → Deployment
and:
Pod → Job → CronJob
even after the original objects are gone.
On our production cluster, those snapshots amount to roughly 125 KB every 15 minutes, or about 12 MB per day.
That tiny amount of retained state gives the telemetry something Kubernetes itself eventually loses: historical topology.
If an old event references a pod UID, we can still resolve the pod's ownership chain instead of guessing from a generated pod name.
Kubernetes event counts have another trap
While building this we found a Kubernetes behavior that is easy to get wrong.
Repeated events aren't necessarily emitted as one new event object per occurrence. Kubernetes can update an existing Event and increment its count.
For example, we saw FailedDraining updates with counts:
38696 → 38699 → 38701
There are three records there. There were not three failures. There were also not 38,701 new failures. There were five.
So Cardinal tracks the Event UID and folds the counter deltas. That's what the ×N next to an event represents.
It's a small implementation detail, but without it an event view can be confidently wrong during exactly the incidents where events repeat the most.
This is mostly a navigation problem
None of the underlying data is exotic.
Kubernetes already has object relationships. Kubernetes already emits events. OpenTelemetry already has a receiver capable of collecting them. Your metrics and logs probably already contain most of the Kubernetes labels needed to establish the starting identity.
The frustrating part is that our debugging tools tend to make us reconstruct those relationships manually. If I'm looking at a latency spike grouped by deployment, I shouldn't have to ask another system what pods were underneath that deployment.
If one of those pods was failing, I shouldn't have to open another view to find its Kubernetes events. And if that pod disappeared three hours ago, the investigation shouldn't disappear with it.
The chart already knows the k8s entity tags we are looking at. We decided that should be enough.
Top comments (0)