This post is only a brief overview, with references collected at the end. If you’d like to dive deeper, I recommend starting with Deep Diving Kubernetes Ephemeral Containers and kubectl debug Command and then following the related Task: Copy Files To/From a Distroless Kubernetes Pod.
Ephemeral containers
Ephemeral containers provide a powerful way to debug applications running in Kubernetes. Unlike regular containers, they are not part of the pod’s original specification but can be injected into a running pod when needed. This makes them especially valuable for interactive troubleshooting, particularly when kubectl exec is insufficient—for example, when a container has already crashed or when the original image lacks essential debugging tools.
Modern container practices often favor minimal or distroless images to improve security and performance. In many cases, base images are stripped down to the essential. sometimes even using scratch with nothing but the application binary. While this approach:
- have a smaller attack vector area.
- have faster-scanning performance.
- Reduced image size.
- have a faster build and CD/CI cycle.
- have fewer dependencies.
It also means that traditional debugging utilities (like a shell or package manager) are unavailable.
Ephemeral containers allow engineers to inject a temporary container image that includes the required debugging tools, without altering the original pod definition. This is particularly useful for inspecting the live state of an application, reproducing elusive issues, and running arbitrary commands inside a pod—giving teams the best of both worlds: lean production images with flexible on-demand debugging capabilities.
Share Process
While troubleshooting a Pod, I'd typically want to see the processes of all pods containers, as well as I'd be interested in exploring their filesystems. Can an ephemeral container be a little more unraveling?
However, if you explore the filesystem with ls, you'll notice that it's a filesystem of the ephemeral container itself (busybox in our case), and not of any other container in the Pod. Well, it makes sense - containers in a Pod (typically) share net, ipc, and uts namespaces, they can share the pid namespace, but they never share the mnt namespace. Also, filesystems of different containers always stay independent. Otherwise, all sorts of collisions would start happening.
Understanding process namespace sharing:
The container process no longer has PID 1. Some containers refuse to start without PID 1 (for example, containers using systemd) or run commands like
kill -HUP 1to signal the container process. In pods with a shared process namespace,kill -HUP 1will signal the pod sandbox (/pause).Processes are visible to other containers in the pod. This includes all information visible in
/proc, such as passwords that were passed as arguments or environment variables. These are protected only by regular Unix permissions.Container filesystems are visible to other containers. This makes debugging easier, but it also means that filesystem secrets are protected only by filesystem permissions.Thus, if you know a PID of a process from the container you want to explore, you can find its filesystem at
/proc/<PID>/rootfrom inside the debugging container.
Lets get hands dirty
Create pod with a distroless image.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: distroless-nginx
labels:
app: distroless-nginx
spec:
selector:
matchLabels:
app: distroless-nginx
replicas: 1
template:
metadata:
labels:
app: distroless-nginx
spec:
containers:
- name: distroless-nginx-ctr
image: cgr.dev/chainguard/nginx
After the pod is created, try to get shell via kubectl exec (try with sh, ash , ...):
k exec -it -n default distroless-nginx-cbbdbd698-49fjr -- bash
It wont be able to give you typical TTY shell.
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "c336401c1da1996fc382a2c8aefd82552a60b6de52da1d12aec0d3b5f9b3b65a" : OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH
kubectl debug
POD_NAME=$(kubectl get pods -l app=distroless-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl debug -it -c debugger --image=busybox ${POD_NAME}
When you inject an ephemeralContainer into a pod, it behaves like any other container, unless you explicitly grant it additional privileges or enable process and filesystem sharing with the target container.
So, unless your workloads run with shareProcessNamespace enabled by default (which is unlikely a good idea since it reduces the isolation between containers), we still need a better solution.
POD_NAME=$(kubectl get pods -l app=distroless-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl debug -it -c debugger --target=distroless-nginx-ctr --image=busybox ${POD_NAME}
Targeting container "distroless-nginx-ctr". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
If you don't see a command prompt, try pressing enter.
/ # ps aux
PID USER TIME COMMAND
1 65532 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf -e /dev/stderr -g daemon off;
7 65532 0:00 nginx: worker process
8 65532 0:00 nginx: worker process
9 65532 0:00 nginx: worker process
10 65532 0:00 nginx: worker process
17 root 0:00 sh
22 root 0:00 ps aux
/ #
In this case the debugger container can access to other processes. what does the --target do ?
k debug --help
--target='':
When using an ephemeral container, target processes in this container name.
Note:
The --target parameter must be supported by the Container Runtime. When not supported, the Ephemeral Container may not be started, or it may be started with an isolated process namespace so that ps does not reveal processes in other containers.
Now, let’s say we want to modify the distroless-nginx-ctr container’s /etc/nginx/nginx.conf. Immediately we run into a problem: we don’t even have permission to change directories with cd, nor can we list the root directory of the target container.
~ # ls /proc/1/root
ls: /proc/1/root: Permission denied
~ # cd /proc/1/root
sh: cd: can't cd to /proc/1/root: Permission denied
~ #
This highlights a key limitation—process sharing alone isn’t enough. To properly inspect and interact with the target container’s filesystem, we also need elevated privileges. That’s where the --profile flag comes in, as it can grant the necessary access for debugging and modification.
POD_NAME=$(kubectl get pods -l app=distroless-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl debug -it -c debugger --target=distroless-nginx-ctr --profile=sysadmin --image=busybox ${POD_NAME}
Note:
- kubectl debug automatically generates a container name if you don't choose one using the
--containerflag. - The
-iflag causes kubectl debug to attach to the new container by default. You can prevent this by specifying --attach=false. If your session becomes disconnected you can reattach usingkubectl attach. - The
--share-processesallows the containers in this Pod to see processes from the other containers in the Pod. For more information about how this works, see Share Process Namespace between Containers in a Pod - Don't forget to clean up the debugging Pod when you're finished with it.
- Install a Kubernetes policy engine (OPA/Gatekeeper) to prevent exposing your cluster:
- Block privileged debug containers
- Require
automountServiceAccountToken: false - Implement just-in-time access for debugging
- Use managed debug tools like Teleport instead of raw kubectl debug
- Debug Session Monitoring Now we alert on:
- Added Debug Session Monitoring
- Any ephemeral container creation
- Service account token access from unexpected pods
- Debug sessions lasting >5 minutes
Top comments (0)