Build a Node Drain Agent: PDB-Aware Kubernetes Maintenance That Never Violates a Disruption Budget

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

What This Agent Does

A node drain agent takes the most tedious job in cluster maintenance, emptying nodes for kernel patches, AMI rotations, and version upgrades, and makes it safe to run unattended. Before it cordons anything, deterministic code checks every pod on the node against every PodDisruptionBudget that selects it and predicts whether the eviction will succeed. When a drain would stall, the LLM diagnoses why the budget is exhausted and picks one of a small set of remediations: scale the workload up first, wait for an in-flight rollout, ask the owning team, or skip the node. It never bypasses a budget. The flag that would let it, --disable-eviction, does not exist in its toolset, and neither does permission to edit a PDB.

The reason to build this rather than script kubectl drain in a loop is the failure mode everyone has met: the drain hangs for forty minutes printing the same line, the maintenance window closes, and whoever is on-call ends up deciding at 2 a.m. whether a single-replica service with a minAvailable: 1 budget is allowed to go down. That decision has context (who owns it, whether it is mid-rollout, whether the SLO is already burning), and gathering that context is exactly the work an agent can do at 2 p.m. instead.

Why Drains Get Stuck

Every stuck drain prints some variant of this:

evicting pod payments/ledger-7d9c4b6f8-x2k9q
error when evicting pods/"ledger-7d9c4b6f8-x2k9q" -n "payments" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.

That message comes from the Eviction API returning HTTP 429. The API server checked the PDBs matching the pod and found disruptionsAllowed: 0. The status block tells you which of four situations you are in:

kubectl get pdb -n payments ledger -o jsonpath='{.status}' | jq .

{
  "currentHealthy": 2,
  "desiredHealthy": 2,
  "disruptionsAllowed": 0,
  "expectedPods": 2,
  "conditions": [{"type": "DisruptionAllowed", "status": "False", "reason": "InsufficientPods"}]
}

Pattern in status	What it means	Right fix
currentHealthy == desiredHealthy == expectedPods, allowed 0	Budget is minAvailable equal to replicas, or maxUnavailable: 0. The budget can never admit an eviction.	Scale up by one, drain, scale back. Or fix the PDB in Git.
currentHealthy under desiredHealthy	Some replicas are unhealthy. The budget is protecting a service that is already degraded.	Wait for the rollout, or investigate the unhealthy pods. Do not drain.
expectedPods: 1	Single-replica workload with a PDB. Any eviction violates it by construction.	Owner decision. Usually a scheduled restart during their window.
conditions[].reason: SyncFailed	The PDB selector matches pods from more than one controller, or the scale subresource is missing.	Fix the PDB. The eviction API returns 500 here, not 429.

Two more blockers are not budget-related but stall a naive script. Pods with the karpenter.sh/do-not-disrupt: "true" annotation, or cluster-autoscaler.kubernetes.io/safe-to-evict: "false", are ones a human explicitly marked as not-now. And pods not managed by a controller need --force, which deletes them permanently. The agent treats both as skip-and-report, never override. A drained pod that lands nowhere becomes a Pending pod, so read the FailedScheduling guide before you drain a cluster that is already tight on capacity.

Step 1: Predict the Drain Before Cordoning

The pre-flight is pure code. It evaluates PDB selectors against the node's pods and reads the status the controller already computed, so it costs nothing and touches nothing.

# preflight.py — deterministic, read-only
from kubernetes import client, config

config.load_incluster_config()
core, policy = client.CoreV1Api(), client.PolicyV1Api()

SKIP_ANNOTATIONS = {
    "karpenter.sh/do-not-disrupt": "true",
    "cluster-autoscaler.kubernetes.io/safe-to-evict": "false",
}

def selector_matches(selector, labels):
    for k, v in (selector.match_labels or {}).items():
        if labels.get(k) != v:
            return False
    return True   # matchExpressions handled the same way in the full version

def preflight(node: str) -> list[dict]:
    pods = core.list_pod_for_all_namespaces(
        field_selector=f"spec.nodeName={node}").items
    pdbs = policy.list_pod_disruption_budget_for_all_namespaces().items
    report = []
    for pod in pods:
        owners = [o.kind for o in (pod.metadata.owner_references or [])]
        ann = pod.metadata.annotations or {}
        entry = {"pod": f"{pod.metadata.namespace}/{pod.metadata.name}",
                 "owner": owners[0] if owners else None, "blockers": []}
        if "DaemonSet" in owners:
            continue                       # ignored by drain anyway
        if not owners:
            entry["blockers"].append("unmanaged_pod")
        for k, v in SKIP_ANNOTATIONS.items():
            if ann.get(k) == v:
                entry["blockers"].append(f"annotation:{k}")
        for pdb in pdbs:
            if pdb.metadata.namespace != pod.metadata.namespace:
                continue
            if not selector_matches(pdb.spec.selector, pod.metadata.labels or {}):
                continue
            st = pdb.status
            if (st.disruptions_allowed or 0) == 0:
                entry["blockers"].append({
                    "pdb": pdb.metadata.name,
                    "healthy": st.current_healthy, "desired": st.desired_healthy,
                    "expected": st.expected_pods,
                    "reason": (st.conditions or [{}])[-1].reason
                              if st.conditions else None,
                })
        if entry["blockers"]:
            report.append(entry)
    return report

If the report is empty, the node drains without anyone thinking about it. If it is not, the entries go to the model. Only the blocked pods, never the full node inventory, get sent. Keeping the model's input to the handful of things that need judgment is what keeps the run cheap and the diagnosis focused.

One thing the status block cannot tell you is whether an eviction would succeed right now for a pod with two overlapping PDBs. For that, the Eviction API supports a server-side dry run, and kubectl drain --dry-run=server wraps it. The agent runs that as a final confirmation on nodes the preflight passed. It is a real request evaluated by the admission chain with nothing persisted.

Step 2: The Diagnosis Call

Each blocked workload gets exactly one model call with a forced tool schema. Alongside the preflight entry, the wrapper fetches the owning Deployment or StatefulSet spec, its rollout status, the service's owner from the catalog (the Backstage MCP server is the cleanest source), and the PDB manifest from Git.

DRAIN_TOOL = {
    "name": "plan_workload_disruption",
    "description": "Decide how to get one blocked workload off a node.",
    "input_schema": {
        "type": "object",
        "properties": {
            "diagnosis": {"enum": [
                "budget_equals_replicas", "unhealthy_replicas",
                "single_replica_with_pdb", "rollout_in_progress",
                "pdb_misconfigured", "explicit_do_not_disrupt",
                "unmanaged_pod"]},
            "action": {"enum": [
                "scale_up_then_drain", "wait_for_rollout",
                "ask_owner", "skip_node", "propose_pdb_fix_pr"]},
            "scale_to": {"type": "integer",
                "description": "Only for scale_up_then_drain. Current "
                               "replicas plus one, never more."},
            "wait_minutes": {"type": "integer", "maximum": 30},
            "evidence": {"type": "string",
                "description": "2-3 sentences citing the PDB status numbers "
                               "and the workload's replica and rollout state."},
            "risk": {"type": "string",
                "description": "What the budget is protecting and why this "
                               "action keeps that protection intact."},
        },
        "required": ["diagnosis", "action", "evidence", "risk"],
    },
}

SYSTEM = (
    "You plan node drains for an SRE team. A PodDisruptionBudget is a promise "
    "the service owner made to their users; you may work around it, never "
    "through it. Prefer scale_up_then_drain when replicas are healthy and the "
    "budget simply equals the replica count. Prefer wait_for_rollout when "
    "currentHealthy is below desiredHealthy. Single-replica workloads with a "
    "PDB are always ask_owner. Anything annotated do-not-disrupt is skip_node. "
    "If the PDB itself looks wrong (SyncFailed, or maxUnavailable: 0 on a "
    "20-replica stateless service), answer propose_pdb_fix_pr and explain. "
    "Never suggest deleting pods, editing a PDB in place, or force flags."
)

The risk field carries the weight, as it does in every agent in this series. Forcing the model to say what the budget protects before it is allowed to route around it is what separates a maintenance plan from a rationalization. In practice the split is boring in a good way: most blocked drains are budget_equals_replicas on a healthy 3-replica deployment, and the answer is scale to 4, drain, scale back to 3.

Step 3: Executing With Hard Limits

Actions run through a small executor whose limits live in code, not in the prompt. The agent's ServiceAccount, built the way the least-privilege RBAC post lays out, has exactly these verbs:

rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "patch"]              # cordon and uncordon
  - apiGroups: [""]
    resources: ["pods", "pods/eviction"]
    verbs: ["get", "list", "create"]             # evict, never delete
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list"]                       # read, never patch
  - apiGroups: ["apps"]
    resources: ["deployments/scale", "statefulsets/scale"]
    verbs: ["get", "patch"]                      # scale up by one, at most

There is no delete on pods, so --force and --disable-eviction are impossible rather than forbidden. There is no patch on PDBs, so propose_pdb_fix_pr can only become a pull request in the manifests repo, opened the way the GitOps-for-agents pattern describes.

The executor adds four rules the RBAC cannot express:

• One node per zone at a time, and never a node where the preflight found a pod whose PDB has disruptionsAllowed: 1 shared with a pod on another node currently draining. Two agents racing the same budget is how you turn "safe" evictions into an outage.
• A burn-rate gate. Before each cordon it queries the fast-burn SLO alerts the error budget agent maintains. Any firing page-severity burn alert pauses the run. Maintenance during an incident is a second incident.
• A per-node clock. A drain gets 20 minutes. When it expires the node is uncordoned, scale-ups are reverted, and the node lands in the skip report with the reason. Cordoned-and-forgotten nodes are the other classic maintenance failure, and they quietly shrink capacity until something goes Pending.
• Scale-ups are always reverted, whether the drain succeeded or not, and the revert is verified against the Deployment's .spec.replicas in Git. If Git says 3 and the cluster says 4 after the run, the agent has left drift and the run is marked failed.

The scale_up_then_drain path in full:

def scale_up_then_drain(node, ns, kind, name, scale_to):
    scale = apps.read_namespaced_deployment_scale(name, ns)
    original = scale.spec.replicas
    assert scale_to == original + 1, "agent may only add one replica"
    apps.patch_namespaced_deployment_scale(
        name, ns, {"spec": {"replicas": scale_to}})
    try:
        wait_until(lambda: pdb_allows_disruption(ns, name), timeout=300)
        cordon(node)
        evict_all(node, timeout=1200)      # POST pods/eviction, honours 429 + Retry-After
    finally:
        apps.patch_namespaced_deployment_scale(
            name, ns, {"spec": {"replicas": original}})

The finally is not optional. The single most common bug in homegrown drain automation is an exception between the scale-up and the scale-down that leaves an extra replica running for a month.

A Real Run

A 14-node EKS worker pool ahead of a kubelet upgrade. The preflight cleared 11 nodes outright. Three came back blocked:

ip-10-0-41-7    payments/ledger      pdb=ledger   healthy=2 desired=2 expected=2  budget_equals_replicas
ip-10-0-41-7    search/indexer       pdb=indexer  healthy=1 desired=2 expected=2  unhealthy_replicas
ip-10-0-52-19   batch/report-gen     pdb=reports  healthy=1 desired=1 expected=1  single_replica_with_pdb
ip-10-0-63-3    ml/embedder-gpu      annotation karpenter.sh/do-not-disrupt      explicit_do_not_disrupt

The agent scaled ledger to 3, drained, scaled back, all inside four minutes. For indexer it chose wait_for_rollout with 15 minutes, because the second replica was in a CrashLoopBackOff from a bad config push, and drained the node once the rollback landed. report-gen went to its owner as a Slack message with the PDB status pasted in, and the owner replied with a restart window that night. The GPU node was skipped and reported. Nobody was paged, and the two hard decisions were made by the people who own the services, with the evidence already gathered.

The ask_owner and propose_pdb_fix_pr paths both stop at a human, which makes this a supervised loop rather than an autonomous one. Fully autonomous is the right setting only for the boring majority: healthy replicas, budget equals replica count, add one and go.

Honest Limits

The preflight reads the PDB status the controller computed, which lags reality by up to the controller's resync period. A pod that just became unhealthy can still show disruptionsAllowed: 1, and the drain will hit a 429 anyway. The executor tolerates this by honouring Retry-After for the full node clock, not by trusting the prediction.

unhealthyPodEvictionPolicy: AlwaysAllow, GA since Kubernetes 1.31, changes the unhealthy_replicas case: with it set, already-unhealthy pods can be evicted even when the budget is exhausted. The agent reads the field and adjusts the diagnosis, but you should decide as a team whether to set it. It makes drains faster and makes a degraded service slightly more degraded during maintenance.

Finally, this agent solves drains, not capacity. If evicted pods have nowhere to go, the drain "succeeds" and your users lose the replica anyway. Check headroom before the maintenance window, not during it, and on Karpenter clusters let its own disruption controller handle the routine churn as the spot interruption post describes. This agent earns its keep on the drains Karpenter refuses to do: the ones a PodDisruptionBudget is, correctly, standing in front of.

---

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.