Chronological K8s Debugging: Merging Logs, Events, and Node Alerts on the Fly

If you've ever had to debug a failing Kubernetes microservice in real time, you know the "context-shift trap" all too well.

You find yourself with three terminal split panes open:

1. One running kubectl logs -f pod-xyz (or stern) to tail application output.
2. A second running kubectl get events -w to see if there are liveness probe failures or scheduling issues.
3. A third polling kubectl describe node to track resource pressure or disk problems.

The hard part isn't getting the information—it's stitching it together. When your app throws a database connection timeout at 14:02:15.102, did it happen before or after the Kubelet posted a MemoryPressure alert on the node? Did a ConfigMap update trigger a rolling update that caused the database pod to restart?

Answering these questions down to the millisecond requires tedious manual correlation. I wanted a way to see everything in one place, so I built KubeCorrelate.

---

What is KubeCorrelate?

KubeCorrelate (kubecorrelate) is a lightweight, zero-dependency CLI tool written in Go. It operates purely client-side using your existing kubeconfig context to multiplex container logs, Kubernetes events, configuration changes, and node pressure warnings into a single, chronological, color-coded stream.

Here is what a live debugging session looks like in action:

• [23:55:01.002] [LOG] Processing order #98213...
• [23:55:02.155] [CONFIG] ConfigMap "app-config" was modified
• [23:55:03.441] [LOG] Reloading routing rules...
• [23:55:05.801] [EVENT] [WARN] LivenessProbeFailed
• [23:55:07.100] [NODE] [WARN] MemoryPressure: System memory usage exceeded 95%
• [23:55:07.350] [EVENT] [WARN] Container restarted (Reason: OOMKilled)

Instead of switching tabs, you see the exact sequence of events that led to the crash.

---

How It Works Under the Hood

KubeCorrelate uses the client-go libraries to spin up lightweight watch goroutines for different telemetry signals:

1. Log Streams: Pulls real-time container logs with Timestamps: true enabled.
2. Kubernetes Events: Watches pod-specific warnings and lifecycle state transitions.
3. Configurations: Monitors ConfigMaps and Secrets referenced by the target pod.
4. Node Metrics: Listens for resource pressure signals (Memory, Disk, PID pressure) on the specific node hosting the pod.

Solving the Out-of-Order Logging Problem

If you simply print these streams as they arrive, they will be out of order. Network latencies, API aggregation delays, and container runtime flush intervals mean that a log line from a container might reach your client machine after a Kubernetes event that actually occurred later.

To solve this, KubeCorrelate implements a client-side Bounded Slop Sorting Buffer (defaulting to 1.5s, customizable via --buffer-delay):

Telemetry Sources [Logs] [Events] [Configs] [Node Alerts] │ │ │ │ ▼ ▼ ▼ ▼ ┌─────────────────────────────────────────┐ │ Bounded Slop Sorting Buffer │ <-- Queues and sorts client-side └─────────────────────────────────────────┘ │ ▼ Sorted Terminal Output

Incoming events are temporarily queued in the buffer. After the buffer delay passes, the events are flushed in guaranteed chronological order down to the millisecond.

Built for Production Clusters

When writing KubeCorrelate, I focused on making it safe for real-world enterprise environments:

• Graceful RBAC Degradation: If you're running in a locked-down namespace where you don't have cluster-wide permissions to watch Nodes or ConfigMaps, KubeCorrelate doesn't crash. It outputs a single warning and keeps monitoring the logs and events you do have access to.
• Dynamic Pod Discovery: In a rolling deployment, pods are constantly terminated and replaced. KubeCorrelate automatically detects when a new pod starts, attaches watchers to it, and tears down the old ones cleanly.

* Zero Dependencies: Built entirely with standard Go libraries and official Kubernetes packages. No extra agents or CRDs required in your cluster.

Getting Started

KubeCorrelate is officially accepted in the CNCF Krew index. You can install it directly as a kubectl plugin:

bash
kubectl krew install correlate
Once installed, use it as:

bash
kubectl correlate -l app=order-processor
Alternatively, you can build from source:

bash
go install github.com/Dasmat13/kubecorrelate/cmd/kubecorrelate@latest
Try it out!
Check out the repository on GitHub: Dasmat13/kubecorrelate.

Would love to hear your thoughts, feedback, and feature requests. If you find it useful, feel free to drop a star!