DEV Community

aiunplugged
aiunplugged

Posted on Originally published at aiunplugged.in

Fix: kubectl exec fails with 'connection refused' — 5 real causes

#kubernetes #devops #kubectl #troubleshooting

Trying to exec into a running pod and Kubernetes refuses:

error: unable to upgrade connection: pod does not exist
Enter fullscreen mode Exit fullscreen mode

Or:

Error from server: error dialing backend: dial tcp 10.42.0.7:10250: connect: connection refused
Enter fullscreen mode Exit fullscreen mode

The pod shows Running in kubectl get pods, yet exec doesn't work. Here are the real causes and how to fix each.

Diagnose first

Before guessing, gather facts:

For RBAC-related issues, also check what permissions the current user has:

# Confirm pod is actually running
kubectl get pod <pod-name> -n <namespace>

# Check node the pod is on
kubectl get pod <pod-name> -n <namespace> -o wide

# Check the node's status
kubectl get node <node-name>

# Check kubelet logs on the node (if you have SSH access)
journalctl -u kubelet -n 100
Enter fullscreen mode Exit fullscreen mode

The exact error message narrows down which of the 5 causes below applies.

Cause 1 — kubelet not reachable from API server

kubectl exec requires the API server to open a tunnel to the kubelet on port 10250. If a firewall or security group blocks that port, exec fails while everything else works.

Symptom: error like dial tcp <node-ip>:10250: connect: connection refused or i/o timeout.

Fix:

  • Cloud clusters (EKS/GKE/AKS): check that the control plane security group can reach worker nodes on 10250/TCP. On EKS, this is the eksClusterSecurityGroup inbound rule on the node security group.
  • Self-hosted: verify iptables or ufw allows 10250 from control plane nodes. 
# On the node, verify kubelet is listening
sudo ss -tlnp | grep 10250

# From the control plane, test connectivity
nc -zv <node-internal-ip> 10250
Enter fullscreen mode Exit fullscreen mode

If the port is closed, add the firewall rule. If open but still failing, jump to cause 2.

Cause 2 — kubelet is running but crashed the streaming server

kubelet has a separate HTTPS server for exec/attach/logs. It sometimes crashes without taking down the whole kubelet.

Symptom: logs works but exec and attach fail. Or intermittent failures.

Fix — restart kubelet:

sudo systemctl restart kubelet
sudo journalctl -u kubelet -f
Enter fullscreen mode Exit fullscreen mode

Watch for errors during startup. If kubelet keeps crashing, the underlying issue is often disk pressure, memory pressure, or a stale container runtime socket.

Cause 3 — pod is Running but container isn't ready for TTY

Distroless containers, scratch images, or containers without a shell installed have no /bin/sh — exec fails immediately.

Symptom: OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Fix — use an ephemeral debug container (Kubernetes 1.23+):

kubectl debug -it <pod-name> -n <namespace> \
  --image=busybox:1.36 \
  --target=<container-name>
Enter fullscreen mode Exit fullscreen mode

This attaches a debug container to the same pod's namespace, giving you a shell that CAN inspect the target container's processes, network, and mounts. Works even for distroless.

Cause 4 — RBAC blocking your user from exec

You can list pods but not exec into them.

Symptom: Error from server (Forbidden): pods "<name>" is forbidden: User "<you>" cannot create resource "pods/exec"

Diagnose:

kubectl auth can-i create pods/exec -n <namespace>
Enter fullscreen mode Exit fullscreen mode

If it returns no, your Role or ClusterRole is missing the pods/exec sub-resource.

Fix — grant exec permission:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-exec
  namespace: YOUR_NAMESPACE
rules:
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
Enter fullscreen mode Exit fullscreen mode

Bind it to your user with a RoleBinding and retry.

Cause 5 — kubectl version mismatch with cluster

Very old kubectl talking to a much newer cluster (or vice versa) can hit exec-protocol incompatibilities.

Symptom: exec starts, prints a partial banner, then disconnects. Or error: unable to upgrade connection: unauthorized.

Diagnose:

kubectl version
Enter fullscreen mode Exit fullscreen mode

The default output shows both client and server versions. Kubernetes officially supports client/server skew of +/- 1 minor version. Beyond that, exec is unreliable.

Fix — align kubectl version:

# macOS
brew install kubernetes-cli
# or pinned to a specific minor version:
brew install kubernetes-cli@1.29

# Linux — pinned install
curl -LO "https://dl.k8s.io/release/v1.29.0/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
Enter fullscreen mode Exit fullscreen mode

Match kubectl to your cluster's minor version and retry.

The universal exec debug flow

Every stuck-exec case, run in order:

# 1. Pod actually running?
kubectl get pod <pod> -n <ns>

# 2. Node healthy?
kubectl get node -o wide

# 3. Can I even see logs? (rules out kubelet-total-failure)
kubectl logs <pod> -n <ns> --tail=5

# 4. RBAC permission for exec?
kubectl auth can-i create pods/exec -n <ns>

# 5. Try debug container (bypasses distroless issue)
kubectl debug -it <pod> --image=busybox --target=<container>
Enter fullscreen mode Exit fullscreen mode

Ninety percent of exec failures resolve after step 5.

Prevention

Bake these into every cluster setup:

  • Explicit security group rules for control-plane to node on 10250/TCP; documented in Terraform, not vibes-based
  • Ephemeral containers enabled on all managed clusters (default in K8s 1.25+; verify with kubectl api-resources | grep ephemeral)
  • Standardized debug image in a private registry — a common debug:latest with curl, bind-utils, netcat, iputils pre-installed
  • RBAC role called debugger with pods/exec permission, granted temporarily via kubectl auth reconcile for incident response
  • Regular kubectl version audits across the team — pin to cluster version in your dotfiles

Bottom line

kubectl exec failing usually points at one of five layers: network to kubelet, kubelet streaming server, missing shell in the container, RBAC, or version skew. Diagnose with kubectl get pod -o wide, kubectl auth can-i create pods/exec, and kubectl debug. The debug container is a strong Swiss army knife — reach for it early rather than fighting distroless exec issues directly.

Top comments (1)

Collapse
 
devsupport profile image
Dev Support •

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‍‌