DEV Community

Python-T Point
Python-T Point

Posted on • Originally published at pythontpoint.in

🐍 kubectl exec hangs when running Python scripts — what's going on

#kubernetes #devops #tutorial #cloud

⏱ Minute 0-2 — Stop the Bleed

kubectl exec hangs when running python scripts

If a pod that runs a data‑processing Python script becomes unresponsive, the first two minutes determine whether the incident escalates.

📑 Table of Contents

  • ⏱ Minute 0-2 — Stop the Bleed
  • 🛡 Minute 2-10 — Contain and Assess
  • 🔀 Minute 10‑X — Recovery Decision Tree
  • 🔐 Preventive Controls — Stop This From Happening Again
  • 🟩 Final Thoughts
  • ❓ Frequently Asked Questions
  • Why does adding -u to the Python command fix the hang?
  • Can I still use an interactive shell for debugging if I set stdin: false?
  • What kernel version introduced the exec websocket improvements referenced in the docs?
  • 📚 References & Further Reading

🛡 Minute 2-10 — Contain and Assess

Gather low‑level diagnostics to pinpoint why kubectl exec hangs when running python scripts inside the container.

Repeat the exec command without allocating a TTY. The Kubernetes API treats --tty as a separate subsystem; removing it eliminates the pseudo‑terminal layer that often blocks Python’s stdout buffering.

$ kubectl exec -n analytics data‑worker-7d9f9c5c-ktm2l - python -u /app/process.py



Processing batch 1/10
Processing batch 2/10
...
Processing batch 10/10
Completed in 12.3s
Enter fullscreen mode Exit fullscreen mode

The script now prints progress lines and exits. The -u flag forces unbuffered stdout, which is necessary when the exec session lacks a TTY because Python defaults to block buffering on non‑interactive streams.

Next, inspect the kubelet log for exec‑related errors. The kubelet writes a JSON line per exec request; filter by pod name.

$ journalctl -u kubelet | grep data‑worker-7d9f9c5c-ktm2l | tail -n 5



Oct 12 14:03:21 node01 kubelet[1234]: I1102 14:03:21.123456 1234 exec.go:215] Exec request for container "data‑worker" in pod "data‑worker-7d9f9c5c-ktm2l" (namespace "analytics") failed: container not ready
Enter fullscreen mode Exit fullscreen mode

The log shows that the container was not ready for exec when a TTY was requested. This race condition occurs because the pod’s entrypoint may still be initializing the Python interpreter, and the PTY allocation forces the exec request to wait for the container to expose a pseudo‑terminal device.

Kubernetes documentation states that the exec API creates a new process inside the container and attaches the client’s STDIN/STDOUT/STDERR streams via a websocket. When a TTY is requested, the container runtime must allocate a PTY before the process can start; if PTY creation is delayed, the client appears to hang.

Mode Behavior Typical Use‑Case
Exec with --tty Allocates a PTY; blocks if the container is not ready for a PTY. Interactive debugging, bash shells.
Exec without --tty Runs the command directly; returns immediately when stdout is unbuffered. Batch scripts, CI pipelines.

What this does:

  • kubectl exec -it : allocates a TTY and keeps the session open.
  • kubectl exec -: runs the command without a TTY.
  • -u : disables Python’s internal buffering, ensuring logs appear in real time.

Key point: Removing the TTY layer eliminates the deadlock that causes kubectl exec hangs when running python scripts in containers that are still initializing.

🔀 Minute 10‑X — Recovery Decision Tree

Recover from the exec hang while preserving the pod’s runtime state.

“If the exec request blocks, first remove the TTY; if that succeeds, the root cause is a PTY allocation race.”

Determine whether the container is still serving traffic.

If the pod is still serving requests: Restart only the exec subsystem by patching the pod’s stdin flag. This forces the kubelet to recreate the exec channel without a PTY.

$ kubectl patch pod data‑worker-7d9f9c5c-ktm2l -n analytics -p '{"spec":{"containers":[{"name":"data‑worker","stdin":false}]}}'



pod/data‑worker-7d9f9c5c-ktm2l patched
Enter fullscreen mode Exit fullscreen mode

Re‑enable stdin after the container reports ready: (More onPythonTPoint tutorials)

$ kubectl patch pod data‑worker-7d9f9c5c-ktm2l -n analytics -p '{"spec":{"containers":[{"name":"data‑worker","stdin":true}]}}'



pod/data‑worker-7d9f9c5c-ktm2l patched
Enter fullscreen mode Exit fullscreen mode

If the pod has stopped serving traffic: Perform a rolling restart of the Deployment to create a fresh container.

$ kubectl rollout restart deployment/data‑worker -n analytics



deployment.apps/data‑worker restarted
Enter fullscreen mode Exit fullscreen mode

If the container image lacks a proper entrypoint: Build a new image that explicitly starts the Python process with -u and disables TTY allocation in the pod spec.

# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY . .
ENV PYTHONUNBUFFERED=1
CMD ["python","-u","/app/process.py"]
Enter fullscreen mode Exit fullscreen mode

What this does:

  • ENV PYTHONUNBUFFERED=1 : forces unbuffered output for all Python processes.
  • CMD : defines the default command without a TTY requirement.

If none of the above resolves the hang: Escalate to platform engineering; the node’s kubelet may be experiencing low‑level socket exhaustion that blocks exec websockets.

Key point: The decision tree isolates the problem to PTY allocation, container readiness, or node‑level exec handling, enabling a targeted fix without full service disruption.

🔐 Preventive Controls — Stop This From Happening Again

Implement the following controls to avoid future exec hangs with Python workloads.

  1. SetPYTHONUNBUFFERED=1 in the container image. Guarantees that stdout is flushed immediately, removing the need for a TTY to see logs.
  2. Configure the pod spec withstdin: false unless an interactive shell is required. Prevents accidental PTY allocation that can block exec.
  3. Use readiness probes that return200 only after the Python interpreter has fully started. Exec requests will then wait for a ready state, avoiding race conditions.
  4. Enable the kubelet feature gateExecProbeTimeout (if available) to bound exec wait time. This forces the API server to abort a hanging exec after a configurable timeout.
  5. Monitor kubelet and container‑runtime logs for “container not ready” exec errors. Automated alerts can trigger a pod restart before operators notice a hang.

🟩 Final Thoughts

Understanding the interaction between kubectl exec , PTY allocation, and Python’s output buffering removes the mystery behind a silent hang. By default, exec creates a pseudo‑terminal; when the container is still initializing, that allocation can block indefinitely. The simplest mitigation is to run the script without a TTY and enforce unbuffered output. For production workloads, embed the unbuffered flag in the image and keep stdin disabled unless an interactive session is explicitly required. These practices keep data pipelines responsive and reduce mean time to recovery during incidents.

Applying the preventive controls creates a deterministic exec path, turning a seemingly random hang into a predictable, observable event that can be automated and monitored.

❓ Frequently Asked Questions

Why does adding -u to the Python command fix the hang?

The -u flag disables internal buffering, forcing Python to write each line to stdout immediately. When kubectl exec runs without a TTY, the client reads the stream directly; unbuffered output prevents the exec session from appearing idle and timing out.

Can I still use an interactive shell for debugging if I set stdin: false?

Yes. Override the pod spec temporarily with kubectl edit pod … to set stdin: true and tty: true. After the debugging session, revert the changes to keep the default non‑interactive configuration.

What kernel version introduced the exec websocket improvements referenced in the docs?

The exec websocket handling was refined in Kubernetes v1.24, which aligns with the underlying container runtime’s support for simultaneous STDIN/STDOUT streams without requiring a PTY.

💡 Want to practise this hands-on? DigitalOcean gives new accounts $200 free credit for 60 days — enough to spin up a full Linux/Docker/Kubernetes environment at no cost.

📚 Recommended reading: Best DevOps & cloud books on Amazon — from Linux fundamentals to Kubernetes in production, curated for working engineers.

📚 References & Further Reading

  • Kubernetes exec API documentation — official description of exec behavior and websocket handling: kubernetes.io
  • Python documentation on unbuffered binary stdout and stderr: docs.python.org
  • Dockerfile best practices for environment variables and CMD configuration: docs.docker.com

Top comments (0)