DEV Community: bruce way

Fixing AKS NodePressure Memory Eviction with Azure Disk CSI: Root Cause & Recovery Guide

bruce way — Fri, 17 Jul 2026 02:37:58 +0000

The Incident (What Does the Error Mean?)

You will see this in kubectl describe pod <pod-name>:

Status:     Failed
Reason:     Evicted
Message:    The node was low on resource: memory.
            Threshold quantity: 100Mi, available: 42Mi.
            Container <name> was using 1.9Gi, request is 0.

And on the node:

kubectl describe node <node-name>

Conditions:
  Type                 Status
  MemoryPressure       True     # <-- kubelet has declared war
  DiskPressure         False
  PIDPressure          False
  Ready                False

Immediate consequence: Kubelet terminates pods in
priority order — BestEffort first (no requests/limits set), then
Burstable, then Guaranteed. If your pod had no resources.requests, it is BestEffort and dies first, every time. The Azure Disk CSI driver then fails to detach the PersistentVolume cleanly because the pod was force-killed, leaving a VolumeAttachment object in Terminating state that blocks the pod from starting on a new node.

The Attack Vector / Blast Radius

This is not a single-pod problem. The cascade goes:

One noisy-neighbor pod (no memory limit) balloons and consumes node memory.
Kubelet evicts BestEffort pods — likely your stateful workloads with CSI disks, since those are often misconfigured.
Azure Disk CSI VolumeAttachment gets stuck — the external-attacher sidecar in the CSI controller can't issue a ControllerUnpublishVolume to the Azure API fast enough before the node is marked NotReady.
Pod reschedule is blocked — Kubernetes scheduler sees the PV still attached to the dead node. The pod sits in ContainerCreating or Pending indefinitely with Multi-Attach error for volume.
If you have no PodDisruptionBudget, a rolling eviction can take down your entire Deployment simultaneously.
Node pool autoscaler may spin up a new node, but the stuck VolumeAttachment prevents the PV from binding to it — so you now have a new node and still a broken pod.

Blast radius: Full service outage for any stateful workload (databases, message queues, file processors) running on the affected node pool.

How to Fix It

Step 1: Unblock the Stuck VolumeAttachment (Immediate)

If your pod is stuck in ContainerCreating on a new node:

# Identify the stuck attachment
kubectl get volumeattachment

# Force-delete the finalizer — do this ONLY if the source node is confirmed dead/drained
kubectl patch volumeattachment <va-name> \
  -p '{"metadata":{"finalizers":null}}' \
  --type=merge

Warning: Only remove the finalizer if you have
confirmed the Azure Disk is not actually attached to any running VM.
Check in the Azure Portal under the disk's "Managed by" field.

Step 2: Fix the Root Cause — Resource Requests & Limits

The BestEffort QoS class is the direct cause of priority eviction. Every container needs explicit requests.

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: my-stateful-app
 spec:
   template:
     spec:
       containers:
       - name: app
         image: myrepo/app:1.4.2
-        # No resources defined — BestEffort QoS, evicted first
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"

Enterprise Best Practice: Use Guaranteed QoS by setting requests == limits. This makes kubelet treat your pod as the last resort for eviction.

         resources:
           requests:
-            memory: "512Mi"
-            cpu: "250m"
+            memory: "1Gi"
+            cpu: "500m"
           limits:
             memory: "1Gi"
-            cpu: "1000m"
+            cpu: "500m"
+        # requests == limits => Guaranteed QoS class

Step 3: Tune Kubelet Eviction Thresholds via AKS KubeletConfig

The default memory.available<100Mi hard eviction threshold is dangerously low for nodes running CSI
workloads. Raise the soft threshold so kubelet starts reclaiming memory
gracefully before hitting the hard wall.

 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: kubelet-config
 # Apply via AKS node pool KubeletConfig in the ARM template or Bicep:
---
 apiVersion: 2023-01-01
 type: Microsoft.ContainerService/managedClusters/agentPools
 properties:
   kubeletConfig:
-    # Default: no custom eviction thresholds
+    evictionSoft:
+      memory.available: "300Mi"
+      nodefs.available: "10%"
+    evictionSoftGracePeriod:
+      memory.available: "2m"
+      nodefs.available: "2m"
+    evictionHard:
+      memory.available: "150Mi"
+      nodefs.available: "5%"
+    evictionMaxPodGracePeriod: 90

Apply via Azure CLI:

az aks nodepool update \
  --resource-group <rg> \
  --cluster-name <cluster> \
  --name <nodepool> \
  --kubelet-config kubelet-config.json

Step 4: Add a PodDisruptionBudget

+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: my-stateful-app-pdb
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app: my-stateful-app

Prevention in CI/CD

OPA/Gatekeeper Policy — Block BestEffort QoS at Admission

Deploy this ConstraintTemplate to reject any pod without resource requests:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: requireresourcelimits
spec:
  crd:
    spec:
      names:
        kind: RequireResourceLimits
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |        package requireresourcelimits        violation[{"msg": msg}] {          container := input.review.object.spec.containers[_]          not container.resources.requests.memory          msg := sprintf("Container '%v' missing memory request. BestEffort QoS is prohibited.", [container.name])        }

Checkov — Scan IaC Before Apply

# In your CI pipeline (GitHub Actions, Azure DevOps)
checkov -d ./k8s-manifests \
  --check CKV_K8S_11 \ # CPU limits
  --check CKV_K8S_13 \ # Memory limits
  --check CKV_K8S_10 \ # CPU requests
  --check CKV_K8S_12    # Memory requests

Node Pool Sizing — Right-Size Before You Tune Thresholds

# Check actual memory consumption vs allocatable
kubectl describe node <node> | grep -A5 "Allocated resources"

# If requests > 80% of allocatable memory, scale the node pool
az aks nodepool scale \
  --resource-group <rg> \
  --cluster-name <cluster> \
  --name <nodepool> \
  --node-count <n+2>

Rule of thumb: Keep total pod memory requests below 70% of node allocatable memory to leave headroom for kubelet,
system daemons, and CSI driver sidecars (which themselves consume
50–150Mi per node).

Related Diagnostics

Fixing AKS ‘Failed to Get Node Info’ During NodePool Scale-Up: Root Cause & Resolution

How to Fix PostgreSQL 'cannot execute CREATE TABLE in a read-only transaction' Error

bruce way — Fri, 17 Jul 2026 02:28:41 +0000

The Incident

ERROR: cannot execute CREATE TABLE in a read-only transaction
CONTEXT: SQL function "migrate_schema" during inlining

Postgres enforces a hard wall: no DDL, no DML writes, no sequence increments are permitted inside a read-only transaction. The moment the executor hits CREATE TABLE, it throws and rolls back the entire transaction block. If this fires mid-migration (Flyway, Liquibase, Alembic, Django), your schema is left in a partial state — the migration tool may mark the version as failed, requiring manual repair of the migration history table before the next deploy can proceed.

The Attack Vector / Blast Radius

This is not a security exploit in the traditional sense — but the blast radius in a production pipeline is severe:

Replica misrouting (most common prod incident): A connection pooler (PgBouncer, RDS Proxy, HAProxy) routes the migration runner to a read replica instead of the primary. Every DDL fails. The migration tool may retry in a loop, hammering the replica and spiking its replication lag.

default_transaction_read_only = on at the role or database level: A DBA hardened a reporting role or an entire database cluster with ALTER ROLE reporter SET default_transaction_read_only = on. If your migration runner inherits this role or connects to that DB, every transaction opens read-only silently. No warning. Just the error at DDL time.
Explicit SET TRANSACTION READ ONLY in a wrapper: A shared DB utility function or ORM hook sets SET TRANSACTION READ ONLY for "safe" read operations and the DDL accidentally executes inside that context.
RDS/Aurora Multi-AZ failover window: During failover, the old primary becomes a replica. Connections that survive the failover are now on a read-only node. Migrations queued during this window fail with exactly this error.

Cascading risk: Liquibase/Flyway lock the DATABASECHANGELOGLOCK table at migration start. If the transaction fails mid-flight, the lock row may remain set to TRUE, blocking all future migration runs until manually cleared.
How to Fix It
Basic Fix — Verify and Override Transaction Mode

-- Implicit read-only context inherited from role or pooler
BEGIN;
CREATE TABLE orders (id SERIAL PRIMARY KEY, total NUMERIC);
COMMIT;
-- Explicitly assert READ WRITE before DDL
BEGIN;
SET LOCAL default_transaction_read_only = off; -- override session GUC for this txn
SET TRANSACTION READ WRITE; -- belt-and-suspenders
CREATE TABLE orders (id SERIAL PRIMARY KEY, total NUMERIC);
COMMIT;

Note: SET LOCAL only survives the current transaction block. This is intentional — do not use SET (session-level) in a shared connection pool.

Verify You Are on the Primary

-- Blindly running migration against pool VIP
psql -h db-pool.internal -U migrator -d appdb -f migrate.sql
-- Confirm primary before running DDL
psql -h db-pool.internal -U migrator -d appdb \
-c "SELECT CASE WHEN pg_is_in_recovery() THEN 'REPLICA - ABORT' ELSE 'PRIMARY - OK' END AS node_role;"
Only proceed if output is 'PRIMARY - OK'
psql -h db-pool.internal -U migrator -d appdb -f migrate.sql

Enterprise Best Practice — Role Hygiene + Connection Routing

-- Migration runner uses shared 'app_user' role with read-only default
ALTER ROLE app_user SET default_transaction_read_only = on;
-- Dedicated migration role, never read-only, explicitly scoped
CREATE ROLE migrator NOINHERIT LOGIN PASSWORD '...';
GRANT CREATE, CONNECT ON DATABASE appdb TO migrator;
-- DO NOT set default_transaction_read_only on this role
-- In PgBouncer: route migrator to primary_dsn ONLY
; pgbouncer.ini
[databases]
appdb_migrate = host=primary.db.internal port=5432 dbname=appdb auth_user=migrator
appdb = host=pool.db.internal port=5432 dbname=appdb

In RDS/Aurora, always resolve the cluster writer endpoint (cluster.cluster-xxxx.rds.amazonaws.com) — never the reader endpoint — for migration runners. Enforce this in your Terraform:

db_host = aws_rds_cluster.main.reader_endpoint
db_host = aws_rds_cluster.main.endpoint # writer/primary endpoint

Prevention in CI/CD

Pre-migration node role check (shell gate):

!/bin/bash

ROLE=$(psql "$DATABASE_URL" -tAc "SELECT pg_is_in_recovery();")
if [ "$ROLE" = "t" ]; then
echo "FATAL: Connected to a replica. Aborting migration."
exit 1
fi

Drop this as a pre-step in your GitHub Actions / GitLab CI job before Flyway or Alembic runs.
Become a Medium member

Checkov / kics policy for Terraform RDS:

Ensure your Terraform never wires a migration runner to a reader endpoint. Write a custom Checkov check or use OPA:

OPA: deny migration jobs targeting reader endpoints

deny[msg] {
input.resource.type == "kubernetes_deployment"
input.resource.spec.template.spec.containers[_].env[e]
e.name == "DATABASE_URL"
contains(e.value, "reader")
msg := "Migration workload must not target a reader/replica endpoint."
}

Liquibase / Flyway — fail fast on read-only:

Flyway 9+: set flyway.connectRetries=0 and flyway.validateOnMigrate=true. Do not let it retry silently into a replica loop.

Audit role defaults in your IaC:

Detect any role with read-only default in your DB

psql -c "SELECT rolname, rolconfig FROM pg_roles WHERE 'default_transaction_read_only=on' = ANY(rolconfig);"

Run this as a nightly CI job and alert if your migration role appears in the output.

How I built a pure client-side sanitizer to stop leaking Stripe tokens to ChatGPT.

bruce way — Sat, 16 May 2026 14:26:02 +0000

Like most developers, my go-to debugging strategy often involves dumping massive Nginx configs or React error traces straight into AI tools like Claude and ChatGPT. A few weeks ago, I made a classic mistake: I accidentally included a production DB URI and a Stripe token in my prompt. It made me realize how dangerously easy it is to leak credentials when you're moving fast and frustrated by a bug.

I searched around for a lightweight client-side scrubber to sanitize my text, but came up empty. The tools I found either choked on multi-line secrets (like RSA private keys) or completely failed on greedy regex traps (like the @ symbol in database credentials).

So I spent the last couple of weekends building GhostSanitizer — a pure TypeScript/Regex engine that runs entirely in the browser. It intercepts your text, tokenizes any high-entropy secrets (AWS keys, JWTs, URIs) into dummy tokens (e.g., STACK_SEC_1), and only sends the safe structure to the LLM. When the LLM replies with the refactored code, the browser locally maps the tokens back to your real secrets so you can 1-click copy it.

I open-sourced the core sanitizer logic here if anyone wants to use it for their own AI wrappers:

https://github.com/abests/ghost-sanitizer-js

The Live Sandbox:

To actually make it useful for myself, I wrapped it into a UI and used a Python script to map out and generate specific pre-loaded prompts for about 500 annoying WebDev/DevOps error scenarios (mostly React hydration errors, Nginx 502s/CORS issues). Yes, it's a generated matrix, but it forces the LLM to stay highly context-specific instead of giving generic advice.

You can test the live decryption theater (press F12 to watch it redact before the network request) here: stackengine.dev

(Link to a specific issue like PostgreSQL Deadlock ShareLock if you want to see the specific system prompts in action).

Full transparency on the model:

I threw in a tiny free tier (3 uses) routed through my own Claude 3.5 Haiku key just so people can test the decryption UI. After that, you have to drop in your own OpenAI/Anthropic key (BYOK). When you use your own key, the requests go straight from your browser to the LLM provider. I have zero backend database and literally store nothing.

Let me know if you manage to break the local regex mask. I'm actively trying to find edge cases to patch in the repo. Cheers!

IAM Access Analyzer nuked our prod hotfix because I fundamentally misunderstood how Zelkova evaluates wildcards

bruce way — Sat, 02 May 2026 13:30:49 +0000

TL;DR: Spent 6 hours debugging why our GitOps pipeline kept blocking a critical deployment. Turns out IAM Access Analyzer doesn't care about your Permission Boundaries when evaluating trust policies. Principal: "AWS: *" + a StringLike ARN condition is still globally exploitable. Fixed it with aws:PrincipalOrgID.

The Incident

Our zero-critical-finding security gate hard-blocked a hotfix for our order processing engine. The Terraform pipeline died during validation with:

[FATAL] IAM Access Analyzer finding [RESOURCE_PUBLICLY_ACCESSIBLE] 
detected on aws_iam_role.cross_account_event_bus. 
Trust policy allows Principal 'AWS:*'. Deployment halted.

The False Leads (aka me being an idiot)

First, I assumed IAM eventual consistency was screwing with us. Forced a state refresh, manually triggered aws accessanalyzer start-resource-scan, finding came right back.

Second hypothesis: I had a strict Permission Boundary on the role with aws:SourceVpc and aws:SourceIp conditions. I thought Zelkova (the automated reasoning engine behind Access Analyzer) would be smart enough to calculate the intersection of the trust policy + boundary and realize no external actor could satisfy the network requirements.

Wrong. Access Analyzer evaluates trust policies in complete isolation. It doesn't aggregate Permission Boundaries or SCPs when determining if something is publicly accessible.

The Root Cause

The role's trust policy had Principal: { "AWS": "*" } to support dynamic cross-account access for worker nodes across sub-accounts. To "secure" it, I added:

Condition: {
  StringLike: { "aws:PrincipalArn": "arn:aws:iam::*:role/worker-node-*" }
}

Looks safe, right? Nope. Zelkova uses formal logic. Since AWS account IDs are globally addressable, any attacker could create a role named worker-node-exploit in their own AWS account. That ARN would match the StringLike condition. Zelkova correctly flagged this as exploitable by the entire AWS universe.

The Fix

Replaced the non-deterministic ARN wildcard with aws:PrincipalOrgID:

condition {
  test     = "StringEquals"
  variable = "aws:PrincipalOrgID"
  values   = ["o-xyz123abc9"]
}

This mathematically proves to Zelkova that AWS: * is bounded to our AWS Organization. Finding cleared instantly.

Honestly, I got so annoyed debugging IAM policy logic at 2 AM that I refuse to paste our configs into ChatGPT or third-party linters because of compliance. So over the weekend, I just hacked together a pure client-side WASM utility that runs locally in the browser and redacts secrets before checking for this exact issue. Put it up here if anyone else wants to validate their trust policies without leaking data to the cloud:
IAM Access Analyzer Public Principal Auditor

DEV Community: bruce way

Fixing AKS NodePressure Memory Eviction with Azure Disk CSI: Root Cause & Recovery Guide

The Incident (What Does the Error Mean?)

The Attack Vector / Blast Radius

How to Fix It

Step 1: Unblock the Stuck VolumeAttachment (Immediate)

Step 2: Fix the Root Cause — Resource Requests & Limits

Step 3: Tune Kubelet Eviction Thresholds via AKS KubeletConfig

Step 4: Add a PodDisruptionBudget

Prevention in CI/CD

OPA/Gatekeeper Policy — Block BestEffort QoS at Admission

Checkov — Scan IaC Before Apply

Node Pool Sizing — Right-Size Before You Tune Thresholds

Related Diagnostics

How to Fix PostgreSQL 'cannot execute CREATE TABLE in a read-only transaction' Error

The Attack Vector / Blast Radius

Only proceed if output is 'PRIMARY - OK'

!/bin/bash

OPA: deny migration jobs targeting reader endpoints

Detect any role with read-only default in your DB

How I built a pure client-side sanitizer to stop leaking Stripe tokens to ChatGPT.

IAM Access Analyzer nuked our prod hotfix because I fundamentally misunderstood how Zelkova evaluates wildcards

The Incident

The False Leads (aka me being an idiot)

The Root Cause

The Fix