DEV Community

Cover image for One missing name, a Kubernetes namespace full of restarts
Vitaliy Dmitriev
Vitaliy Dmitriev

Posted on AI-assisted

One missing name, a Kubernetes namespace full of restarts

One missing name, a Kubernetes namespace full of restarts

Compare these two commands:

kubectl delete deployment -n prod
# error: resource(s) were provided, but no name was specified

kubectl rollout restart deployment -n prod
# deployment.apps/api restarted
# deployment.apps/billing restarted
# deployment.apps/checkout restarted
# ...
Enter fullscreen mode Exit fullscreen mode

The first refuses to run. The second attempts to restart every Deployment in the namespace, without confirmation or an explicit --all. It's documented behavior.

That makes this a surprisingly dangerous bit of shell:

kubectl rollout restart deployment $SERVICE -n prod
Enter fullscreen mode Exit fullscreen mode

If SERVICE is empty, the command gets a much bigger audience than intended.

Each Deployment still follows its own update strategy. But with rolling updates, those rollouts can overlap across the namespace. Caches warm up, connections reconnect, and replacement pods compete for capacity together. Per-Deployment rollout settings don't coordinate that load across services.

We wanted to remove this particular footgun without taking away routine troubleshooting access. That led to a Jenkins job, and then to a more interesting question: what should its credential be allowed to do?

RBAC doesn't see your intention

There is no dedicated Kubernetes “restart” API. For a Deployment, kubectl patches the pod template with a timestamp:

spec:
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2026-01-01T12:00:00Z"
Enter fullscreen mode Exit fullscreen mode

The changed template triggers a rollout. Without a name, kubectl lists the Deployments and sends a separate patch for each one.

RBAC authorizes those requests individually. It cannot express “allow any one Deployment, but reject a command that targets several.” A resourceNames allowlist can restrict which Deployments are patchable, but it still permits patching every allowed name.

We removed patch on Deployments from users and provided the restart through Jenkins instead. Users kept their existing scaling and pod-deletion access.

There is an important limit here: our users also retained update on Deployments. They could still change the template with a PUT request. We were preventing an easy accident, not preventing users from modifying workloads.

Make the target explicit

The job requires a resource name, validates its format, checks that the object exists, and takes the namespace from a fixed choice list. The restart path always uses kind/name:

// After validating name and checking ns against the allowed namespaces:
withEnv(["NS=${ns}", "NAME=${name}"]) {
    sh '''
        set -eu
        : "${NS:?Namespace is required}"
        : "${NAME:?Deployment name is required}"
        kubectl get "deployment/$NAME" -n "$NS" -o name
        kubectl rollout restart "deployment/$NAME" -n "$NS"
        kubectl rollout status "deployment/$NAME" -n "$NS" --timeout=180s
    '''
}
Enter fullscreen mode Exit fullscreen mode

The shell rejects an empty name before kubectl runs. Even without that check, deployment/ fails instead of expanding the operation to the namespace.

The single-quoted Groovy script matters too: validated input reaches the shell through environment variables and stays quoted, rather than becoming part of the script text.

One easy permission to miss is watch, which rollout status needs. Otherwise the restart can succeed while the build fails to observe it. A status timeout also doesn't undo the restart; the rollout can continue after the build fails.

The job records who requested the operation, the namespace, and the target in its build history. It also supports scaling to zero and deleting a pod, but those have different failure modes. Neither a rolling update nor a controller-owned pod guarantees zero downtime.

Don't give the restart button cluster-admin

The existing Jenkins deployer credential in our setup had cluster-admin access. Reusing it would have made this a very expensive restart button.

Pipeline validation cannot compensate for that. Anyone who can modify or replay the Pipeline may be able to run different commands with credentials available to it. Jenkins permissions and credential scope matter as much as the form fields.

We used a dedicated service account, bound only in the intended namespaces through RoleBindings. The restart path needs this Deployment rule:

rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "patch"]
Enter fullscreen mode Exit fullscreen mode

The other methods need their own permissions. None needs cluster-admin, Secret reads, or pods/exec.

This credential still permits a namespace-wide Deployment restart. The job limits the normal workflow to one target; Kubernetes permissions limit where a modified workflow could act. With unrestricted Deployment patch, it can also change more than the restart annotation.

That distinction is the useful part of the exercise. We made the everyday operation harder to mistype, while keeping the automation's access narrow enough to inspect and test.

And for a runbook or local script, the smallest useful improvement is just:

kubectl rollout restart "deployment/${SERVICE:?Set SERVICE first}" -n prod
Enter fullscreen mode Exit fullscreen mode

An empty variable should produce an error, not a namespace full of restarts ;)

Top comments (0)