DEV Community

Manvitha Potluri
Manvitha Potluri

Posted on

Which Hop Is Broken? Diagnosing Kubernetes Incidents Along the Request Path

Introducing Tessera, a free, open-source desktop app that traces every request from entry point to node, pinpoints the failing layer, and does it safely across every cluster you run.

When a service running on Kubernetes starts returning errors, the fix is rarely the hard part. The hard part is locating the failure.

A single request passes through several independent layers before it reaches application code: an ingress or cloud load balancer, a Service, the workload that owns the pods, the pods themselves, and the nodes that host them. Each layer has its own configuration, its own status fields, and its own ways of failing. When users see a 503, any one of those layers may be responsible.

The usual response is a sequence of commands: kubectl get ingress, describe svc, get endpoints, get pods, describe pod, logs --previous, describe node. The engineer then assembles the results mentally into a single explanation. It works, but it is slow, it depends heavily on experience, and it is error-prone under pressure, which is exactly when it matters most.

Tessera is an attempt to automate that reasoning.

The core idea: diagnose the path, not the resource

Most Kubernetes tools, whether command-line, terminal or graphical, are organised around resources. They show lists of pods, deployments and services, often very well. What they generally leave to the operator is the question that matters during an incident: how are these resources connected, and where along that connection does traffic stop?

Tessera is organised around the request path instead. For every service in a cluster, it builds the chain

Entry point → Service → Workload → Pods → Nodes

and evaluates each link. The result is presented in two complementary ways.

A traffic map. Every path is drawn from left to right. Healthy routes show live request indicators, and broken hops are drawn as red dashed lines. A Service whose selector matches no pods appears, quite literally, connected to nothing.

Tessera traffic map showing a service disconnected from its pods

A diagnosis for each issue. Each problem is traced through the same layers, with the layer where it originates marked explicitly as the root cause.

Tessera issue view tracing a failure through each layer

Root causes, with evidence

A useful diagnosis must be specific, verifiable, and aimed at the cause rather than its symptoms. Tessera's findings are designed around those three requirements.

Specific. Findings name the exact object, value and fix. These are results from testing Tessera against a live cluster, with faults introduced deliberately:

Fault introduced Tessera's diagnosis
Service selector changed to app=wrong Service web selects app=wrong, which matches no pods. The web pods are labelled app=web. Suggested fix: change the selector to app=web.
CPU request set to 64 cores Deployment web requests 64 CPUs; the largest node offers 12. Identified as "no node is large enough", which is distinct from "the cluster is full".
Memory limit of 20Mi on a growing process Container busybox was killed for exceeding its 20Mi memory limit. Distinguished from a generic crash loop, with a suggested new limit.
Image tag that doesn't exist Deployment web can't pull its image: the tag does not exist. Distinguished from a registry authentication failure.

Verifiable. Every finding includes the evidence it is based on, and the read-only kubectl commands that let an engineer confirm it independently. The tool is never the only source of truth.

Causal. Failures cascade. When pods crash, their Service also loses its endpoints, and the ingress starts returning errors. Tessera groups pod-level failures by workload and downgrades downstream symptoms when an upstream cause explains them, so the first item on the list is the one to fix.

Coverage

The diagnosis engine currently covers twelve categories:

Category Examples
Routing Missing ingress backends, invalid ingress classes, missing TLS secrets, selector mismatches, target port errors, failed cloud load balancers
Network policy Policies that block a Service's port; egress policies that block DNS
DNS CoreDNS unavailable or degraded; node resolver configuration being truncated
Service mesh Missing Istio sidecars; VirtualService destinations or subsets that don't exist or match no pods
Images Missing tags versus registry authentication failures
Configuration and admission Missing ConfigMaps and Secrets; pods rejected by admission webhooks or Pod Security
Storage Unbound volume claims; mount and attach failures
Scheduling Requests larger than any node; taints, affinity and unbound claims
Quota Namespace ResourceQuota exhaustion
Autoscaling HPAs without metrics, pinned at maximum, or unable to scale
Crashes and probes OOM kills, crash loops with exit-code interpretation, liveness-induced restarts, misconfigured probes
Nodes and pod networking Node conditions, CNI agent and kube-proxy failures, VPC CNI IP exhaustion on EKS

The rules are deliberately conservative. Where the available data cannot prove a problem, Tessera reports nothing rather than guessing.

Beyond the Kubernetes API

Some of the most time-consuming failures can't be seen from inside the cluster at all. Tessera offers two optional capabilities for these.

Cloud load balancer health. A familiar and frustrating situation is when every pod is healthy inside the cluster while the cloud load balancer considers every target unhealthy. With cloud checks enabled, Tessera queries AWS for target health, maps each target back to its pod or node, and explains the failure. For example, it can report that the health check requests / and receives a 404, while the workload's readiness probe uses /ready.

Differential network testing. For connectivity problems, Tessera can run short-lived probe pods: one on the node hosting a target pod, and one on a different node. Each probe tests DNS resolution, the Service's cluster IP, each pod IP directly, and the pods' own health endpoints. Comparing the results isolates the failing component:

Observation Conclusion
Pod IPs respond; the Service IP does not Service routing (kube-proxy or its replacement) on that node
The same node succeeds; another node times out Pod networking between nodes (CNI or node firewall rules)
Connection times out Traffic is being dropped (NetworkPolicy or security groups)
Connection refused The application isn't listening on that port
Health endpoint returns an error The kubelet's probe will fail in the same way

Tessera network test results comparing same-node and cross-node probes

Designed to be safe in production, across many clusters

An incident tool is most valuable at precisely the moment when it is least acceptable for a tool to change anything. And most organisations don't run one cluster: they run many, across multiple accounts and regions, where the most expensive mistake is acting on the wrong environment. Tessera's design treats both as first-order requirements.

Guardrail Behaviour
Read-only by default Only get, list and log reads. A minimal read-only role is provided.
Explicit approval Network tests display the exact pod manifests before anything is created. Nothing runs without approval, and the interface can only approve plans built by the backend.
Constrained probes Probe pods run as non-root, with no service account token, no Linux capabilities and a read-only filesystem. They stop after 90 seconds and are always deleted.
Production protection Production clusters are recognised by name and marked with a red banner. Active tests are disabled there by default; where an organisation permits them, the cluster name must be typed to confirm. This is enforced in the backend, not only in the interface.
Per-cluster isolation Each cluster uses its own kubeconfig identity, AWS profile and region. Settings for one cluster never apply to another.
Organisation policy An optional policy file can disable features, restrict which clusters appear, and define what counts as production. A system-managed file takes precedence over user settings and fails safe if it can't be read.
Activity log Every network test, the pods it created and deleted, and every blocked attempt is recorded locally.
Minimal cloud access Cloud checks are opt-in and limited to five read-only AWS describe operations, enforced in code. A least-privilege IAM policy is provided.
Privacy No telemetry, no stored credentials.

A dedicated guide covers the recommended setup for multi-account, multi-region environments, including SSO profiles, RBAC, IAM and policy configuration.

Architecture

Tessera is a Tauri 2 desktop application. The Rust backend holds all cluster and cloud access, and the web-based interface has no file system, shell or network permissions of its own.

The diagnosis engine is a standalone Rust library built on kube-rs. It takes a snapshot of the cluster, converts it into a request-path graph, and applies a set of rules to it. Because both steps are pure functions over data, the rules are tested against fixtures modelled on real Kubernetes objects, and contributors can add new failure patterns without needing a live cluster.

Authentication works exactly as it does for kubectl, including exec credential plugins such as aws eks get-token. If kubectl can reach a cluster, Tessera can too.

Current status

Tessera is at version 0.2 and is published as a pre-release. The diagnosis engine is covered by automated tests, and its core scenarios have been validated against a live cluster. The installers for macOS (Apple Silicon and Intel), Windows and Linux are not yet code-signed; the README explains how to open them.

The next releases will focus on watch-based live updates, Gateway API support, and Azure load balancer health.

Try it

Installers are available on the project's releases page. The quickest way to see Tessera in action is on a local cluster with a deliberately introduced fault:

kind create cluster --name tessera-lab
kubectl create deployment web --image=nginx --port=80
kubectl expose deployment web --port=80
kubectl patch service web -p '{"spec":{"selector":{"app":"wrong"}}}'
Enter fullscreen mode Exit fullscreen mode

Open Tessera, select the kind-tessera-lab context, and review its diagnosis.

Contributing

Tessera is released under the Apache 2.0 licence, and contributions are welcome. Three kinds of input are especially valuable at this stage:

  • Bug reports from real-world clusters
  • Failure patterns encountered during incidents that Tessera should recognise (the repository includes an issue template for these)
  • Code contributions, particularly for Azure support and Gateway API routes

Repository: https://github.com/ManvithaP-hub/tessera
Multi-environment guide: https://github.com/ManvithaP-hub/tessera/blob/main/docs/multi-environment.md

Manvitha Potluri is a DevOps and cloud solutions architect with more than eleven years of experience in cloud infrastructure, Kubernetes and AI systems engineering. Manvitha contributes to CNCF ecosystem projects and writes about platform engineering and incident diagnosis.

Top comments (0)