DEV Community: Usman Khan

Kubernetes Pod Churn & Node Thrashing: Ephemeral Storage Exhaustion and Eviction Cascades

Usman Khan — Thu, 15 Oct 2026 20:52:32 +0000

When background worker pods continuously write temporary scratch files, un-rotated log buffers, or dynamic assets directly to container filesystems without explicit ephemeral storage limits, worker nodes quietly enter local disk pressure states. The kubelet then triggers aggressive pod evictions, sparking cluster-wide pod churn, Karpenter/Cluster Autoscaler thrashing, and AWS bill inflation. Here is an architectural deep dive into diagnosing local disk exhaustion and configuring resilient kubelet eviction boundaries.

In high-throughput B2B SaaS platforms running containerized microservices, background processing workloads (e.g., generating PDF reports, processing video/image assets, compiling code, or handling CSV batch imports) frequently rely on local container file systems for temporary scratch space. Under heavy traffic, un-capped ephemeral storage usage creates an invisible cluster-level failure mode: Cascading Node Eviction and Auto-Scaler Thrashing.

1. The Mechanics of OverlayFS and Ephemeral Storage Eviction

By default, when a container writes files to its own filesystem (outside of explicitly mounted Persistent Volumes), those writes land on the node's local disk through the container runtime's writable layer, typically the OverlayFS storage driver. Disk-backed emptyDir volumes and container logs consume the same node disk. If a pod uses significant local disk space without ephemeral storage requests and limits in its manifest, it triggers a chain reaction across the host node.

The Anatomy of a Node Eviction Cascade

Unbounded Disk Growth: A burst of PDF generation worker tasks writes 15GB of temporary files to /tmp inside the container overlay filesystem.
Kubelet Disk Pressure Breach: The node's disk crosses a kubelet hard eviction threshold (by default, nodefs.available below 10% or imagefs.available below 15%). The kubelet marks the node DiskPressure = True, the node is tainted, and the scheduler stops placing new pods on it.
Aggressive Pod Eviction: The kubelet first garbage-collects dead containers and unused images, then starts evicting pods. It ranks pods by whether their disk usage exceeds their ephemeral storage request, then by pod priority, then by usage. With no requests set, every pod counts as over its request, so priority dominates, and lower-priority core services can be evicted before the high-priority worker that is actually filling the disk.
Auto-Scaler Thrashing (Karpenter / Cluster Autoscaler): Rescheduled evicted pods flood neighboring nodes, pushing those nodes into disk pressure as well. The autoscaler provisions new EC2 instances for pending pods while consolidation removes others, so nodes churn continuously and compute spend can multiply for the duration of the incident.

2. Diagnosing Ephemeral Storage Bloat in Production

Pod objects do not report disk usage, so kubectl get pods cannot show it. The kubelet's stats summary endpoint does. Run the following diagnostic commands to find which pods are consuming node disk and driving instability (requires jq):

# Top 20 pods by ephemeral storage usage (MiB) across all nodes
for node in $(kubectl get nodes -o name | cut -d/ -f2); do
  kubectl get --raw "/api/v1/nodes/$node/proxy/stats/summary" \
    | jq -r --arg node "$node" '.pods[]
        | [$node, .podRef.namespace, .podRef.name,
           ((.["ephemeral-storage"].usedBytes // 0) / 1048576 | floor)]
        | @tsv'
done | sort -t$'\t' -k4 -rn | head -20

# Inspect recent eviction events
kubectl get events --all-namespaces --field-selector reason=Evicted | grep -i "ephemeral"

# Detailed check of Node Disk Pressure conditions
kubectl describe node <node-name> | grep -A 8 "Conditions:"

A pod status displaying Evicted with the message The node was low on resource: ephemeral-storage confirms your cluster is actively suffering from un-capped container write exhaustion.

3. Production-Ready Remediation: Strict Resource Limits & In-Memory `emptyDir`

Step 1: Enforce Ephemeral Storage Requests and Limits

Every Kubernetes manifest deployed to production must explicitly define ephemeral-storage resource boundaries alongside CPU and memory parameters. This lets the scheduler place pods according to available disk, and when a container exceeds its storage limit, the kubelet evicts only that pod instead of letting the whole node reach disk pressure.

# PRODUCTION-GRADE DEPLOYMENT MANIFEST WITH EPHEMERAL LIMITS
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pdf-generator-worker
  namespace: production
spec:
  replicas: 8
  selector:
    matchLabels:
      app: pdf-generator-worker
  template:
    metadata:
      labels:
        app: pdf-generator-worker
    spec:
      containers:
      - name: worker
        image: registry.internal.net/pdf-worker:v2.1.0
        resources:
          requests:
            cpu: "500m"
            memory: "2Gi"
            ephemeral-storage: "1Gi" # Scheduler reserves this much node disk
          limits:
            cpu: "2000m"
            memory: "4Gi"            # Includes up to 2Gi of tmpfs scratch below
            ephemeral-storage: "4Gi" # Hard ceiling: kubelet evicts this pod if exceeded
        volumeMounts:
        - mountPath: /tmp/scratch
          name: scratch-volume
      volumes:
      - name: scratch-volume
        emptyDir:
          medium: Memory # Back temporary writes with RAM (tmpfs)
          sizeLimit: 2Gi # Cap scratch space so it cannot consume all memory

Step 2: Redirect High-Frequency Scratch Writes to In-Memory `tmpfs`

For workloads requiring ultra-fast, short-lived file creation (e.g., image resizing or PDF rendering), mount an emptyDir volume backed by RAM (medium: Memory). Writes to this volume bypass the node disk entirely, removing them as a source of disk pressure and speeding up file I/O. One catch: files in a memory-backed emptyDir count toward the container's memory limit, not its ephemeral storage limit. Size the memory limit to cover the application plus the scratch volume, as in the manifest above, or a full scratch directory will get the pod OOM-killed.

4. Tuning Kubelet Eviction Thresholds and Karpenter Node Consolidation

Reduce premature evictions by tuning kubelet eviction settings in your node configuration (for example, the kubelet config in your EKS node group launch template or Karpenter EC2NodeClass user data):

# Kubelet Eviction Configuration Parameters (kubelet-config.json)
{
  "evictionHard": {
    "memory.available": "5%",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%",
    "imagefs.available": "15%"
  },
  "evictionSoft": {
    "nodefs.available": "15%"
  },
  "evictionSoftGracePeriod": {
    "nodefs.available": "2m"
  },
  "evictionMaxPodGracePeriod": 60
}

Then slow down Karpenter's consolidation so short traffic spikes don't cause node churn (Karpenter v1 API):

# Karpenter v1 NodePool with churn guardrails
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h # Rotate nodes every 30 days for patching
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m # Wait before consolidating so spikes can settle
    budgets:
      - nodes: "10%"     # Never disrupt more than 10% of nodes at once

Soft Eviction Grace Periods: With a soft threshold, the kubelet only evicts if disk stays below 15% free for a full 2 minutes. Short spikes that clear on their own (a batch job finishing, logs rotating, image garbage collection) no longer trigger evictions. The hard threshold at 10% still protects the node.
Karpenter Consolidation Guardrails: consolidateAfter delays consolidation so nodes are not removed the moment utilization dips, and disruption budgets cap how many nodes Karpenter can replace at once. expireAfter handles routine node rotation; it is not a churn control on its own.

5. Executive Summary & Systems Rules

Always Specify ephemeral-storage Requests and Limits: Unbounded container writes will fill host node disks and trigger cascading evictions across unrelated pods.
Use RAM-Backed emptyDir for Transient Files: Mount temporary scratch directories with medium: Memory to keep high-frequency file I/O off node disks, and raise the memory limit to cover the scratch size.
Set Soft Kubelet Eviction Windows: Configure grace periods on nodes so brief disk spikes can clear before the kubelet evicts pods, and add consolidation delays and disruption budgets to your autoscaler.

Are pod evictions and node churn driving up your Kubernetes bill? I help SaaS teams right-size workloads, tune kubelet and Karpenter settings, and build clusters that stay stable under heavy batch traffic. Book a consultation →

Originally published on ctousman.com.

Distributed Idempotency Key Collisions: Clock Skew Mutex Leaks, Race Conditions, and Double-Execution Vulnerabilities

Usman Khan — Thu, 15 Oct 2026 20:41:01 +0000

Wrapping API handlers with naive Redis key locks (SET key val NX EX 30) to enforce idempotency creates an illusion of safety. Under concurrent traffic bursts, worker stalls, network partitions, and non-atomic status releases can cause duplicate executions: charging credit cards twice or creating duplicate database records. Here is an architectural deep dive into diagnosing distributed idempotency lock breaches and engineering atomic state transition engines.

In mission-critical B2B SaaS platforms, especially those handling payment processing, billing generation, or external state provisioning, enforcing strict exactly-once execution via idempotency keys is a core architectural requirement. However, naive Redis lock implementations often contain subtle, high-concurrency race conditions that silently break under network partitions, worker thread stalls, and distributed clock skew.

1. The Fallacy of Naive Redis Mutex Locks

A standard anti-pattern in Node.js and Go microservices is using a simple Redis string set operation to acquire an execution lock prior to invoking business logic:

// NAÏVE ANTI-PATTERN: Vulnerable Idempotency Lock
async function handlePaymentNaive(req: Request, res: Response) {
  const idempotencyKey = req.headers['idempotency-key'];
  const lockKey = `idempotency:${idempotencyKey}`;

  // Attempt to acquire execution lock for 10 seconds
  const acquired = await redis.set(lockKey, 'LOCKED', 'NX', 'EX', 10);

  if (!acquired) {
    return res.status(409).json({ error: 'Concurrent request in progress' });
  }

  // Execute heavy business logic (e.g., Stripe API call + DB mutation)
  const result = await processPayment(req.body);

  // Cache response for future retries
  await redis.set(`response:${idempotencyKey}`, JSON.stringify(result), 'EX', 86400);

  return res.json(result);
}

Three Silent Failure Vectors

The Worker Stall / GC Pause Breach: If a Garbage Collection (GC) pause, unhandled event loop block, or downstream Stripe API latency keeps the worker busy for longer than 10 seconds, the Redis TTL expires automatically. Redis deletes the lock while the first request is still executing. A concurrent retry attempt instantly acquires the lock and executes the payment a second time.
The Non-Atomic Release Bug: If a worker finishes execution and calls redis.del(lockKey) or overwrites the key without verifying ownership, it might delete or clobber a lock that was newly acquired by a second worker after the first worker's lock expired, unlocking the door for a third incoming request.
Clock Skew & Failover Split-Brain: With a single Redis primary, TTLs run on the Redis server's own clock, so application-server skew does not shorten them. Problems appear in distributed setups: multi-master schemes like Redlock depend on bounded clock drift across independent Redis nodes, and Redis replication is asynchronous, so a lock written to a primary can vanish during failover. In both cases, two workers can believe they hold the same lock simultaneously.

2. Architectural Pattern: Atomic Two-Phase State Engine

To close these race conditions, replace primitive boolean lock flags with a 3-state state engine backed by atomic Lua scripts inside Redis. Every state change, both acquiring the key and finalizing it, happens in a single atomic script that checks who owns the key.

State Machine States

PROCESSING: A request is actively executing. Contains a unique, randomly generated worker UUID (the owner) and an acquisition timestamp.
COMPLETED: The request finished successfully. Holds the serialized HTTP response status and body payload for replay.
RELEASED (key deleted): The operation failed with a retryable error. The owning worker deletes the key so the client can retry cleanly.

Phase 1: Atomic Acquisition via Redis Lua Scripting

-- ACQUIRE_LUA_SCRIPT: Atomic Idempotency Lock Acquisition
local lockKey = KEYS[1]
local workerUuid = ARGV[1]
local lockTtlSeconds = tonumber(ARGV[2])

local existingState = redis.call('GET', lockKey)

if not existingState then
    -- Key does not exist: create PROCESSING state with owner UUID
    local payload = cjson.encode({
        status = "PROCESSING",
        owner = workerUuid,
        created_at = ARGV[3]
    })
    redis.call('SET', lockKey, payload, 'EX', lockTtlSeconds)
    return { "ACQUIRED", payload }
else
    -- Key exists: return current state payload to application
    return { "LOCKED", existingState }
end

Phase 2: Ownership-Checked Finalization

Completing or releasing the key must be just as atomic. This script only lets the worker that still owns the PROCESSING lock write the final state. A stale worker whose lock already expired gets 0 back and leaves the key untouched.

-- FINALIZE_LUA_SCRIPT: Only the current owner may complete or release
local lockKey = KEYS[1]
local workerUuid = ARGV[1]
local action = ARGV[2] -- "COMPLETE" or "RELEASE"

local current = redis.call('GET', lockKey)
if not current then return 0 end

local state = cjson.decode(current)
if state.status ~= 'PROCESSING' or state.owner ~= workerUuid then
    return 0 -- Lock expired and was taken over: do not touch it
end

if action == 'COMPLETE' then
    redis.call('SET', lockKey, ARGV[3], 'EX', tonumber(ARGV[4]))
else
    redis.call('DEL', lockKey)
end
return 1

3. Production-Ready TypeScript Middleware Implementation

The application layer acquires the key before running the handler, replays cached responses for completed keys, and finalizes through the ownership-checked script, never with a blind SET or DEL.

// PRODUCTION-GRADE IDEMPOTENCY ENGINE (TypeScript)
import { Request, Response, NextFunction } from 'express';
import { v4 as uuidv4 } from 'uuid';
import { Redis } from 'ioredis';

// ACQUIRE_LUA_SCRIPT and FINALIZE_LUA_SCRIPT hold the two Lua scripts above

export function createIdempotencyMiddleware(redis: Redis) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const idempotencyKey = req.headers['idempotency-key'] as string;
    if (!idempotencyKey) return next();

    const tenantId = req.user?.tenantId || 'global';
    const lockKey = `idempotency:${tenantId}:${idempotencyKey}`;
    const workerUuid = uuidv4();
    const lockTtl = 30; // 30 second execution ceiling

    // 1. Execute Atomic Lua Acquisition Script
    const [action, rawData] = await redis.eval(
      ACQUIRE_LUA_SCRIPT,
      1,
      lockKey,
      workerUuid,
      lockTtl,
      Date.now().toString()
    ) as [string, string];

    if (action === 'LOCKED') {
      const state = JSON.parse(rawData);

      if (state.status === 'PROCESSING') {
        // Request is actively being processed by another worker
        return res.status(409).json({
          error: 'CONCURRENT_REQUEST',
          message: 'An identical request with this Idempotency-Key is currently executing.'
        });
      }

      if (state.status === 'COMPLETED') {
        // Replay identical response directly from cache
        res.setHeader('X-Cache-Lookup', 'HIT-IDEMPOTENT');
        return res.status(state.statusCode).json(state.body);
      }
    }

    // 2. Intercept the response and finalize ONLY if this worker still owns the lock
    const originalJson = res.json.bind(res);
    res.json = (body: any) => {
      const succeeded = res.statusCode >= 200 && res.statusCode < 300;
      const completedPayload = JSON.stringify({
        status: 'COMPLETED',
        statusCode: res.statusCode,
        body
      });

      redis
        .eval(
          FINALIZE_LUA_SCRIPT,
          1,
          lockKey,
          workerUuid,
          succeeded ? 'COMPLETE' : 'RELEASE',
          completedPayload,
          86400 // 24-hour replay retention
        )
        .then((owned) => {
          if (owned === 0) {
            // Our lock expired mid-request and another worker took over.
            // Redis state is protected; the DB constraint below must catch any duplicate side effect.
            console.warn('[IDEMPOTENCY] Lock lost before completion:', lockKey);
          }
        })
        .catch((err) => console.error('[IDEMPOTENCY] Finalize failed:', err));

      return originalJson(body);
    };

    next();
  };
}

This sketch hooks res.json. If your handlers also respond with res.send or res.end, wrap those the same way, or the key will sit in PROCESSING until its TTL expires.

4. Secondary Guardrail: Database-Level Unique Constraints

The ownership check stops a stale worker from corrupting Redis state, but it cannot undo work that worker already did. If a lock expires mid-request, two workers can still both reach the payment step. Redis should act as your fast, first-line filter. For financial mutations, never rely on Redis alone. Always back distributed application locks with strict relational database constraints at the storage engine level.

-- Database-Level Fail-Safe: Unique Idempotency Key Index
CREATE TABLE payment_transactions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL,
    amount_cents BIGINT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT CLOCK_TIMESTAMP(),

    -- Absolute DB Guardrail against double execution
    CONSTRAINT unique_tenant_idempotency UNIQUE (tenant_id, idempotency_key)
);

Insert this row before calling the payment provider, so the reservation happens first. If a Redis lock expires under an extreme 30-second stall, the database will reject the second worker's duplicate insert with a 23505 unique_violation error before any money moves. For the external call itself, pass the same idempotency key to the provider (Stripe, for example, accepts an Idempotency-Key header), so even a duplicate API request is charged only once.

5. Executive Summary & Systems Rules

Never Use Plain SET NX EX for Idempotency: Always store a structured state payload containing a unique worker UUID to verify lock ownership.
Never Release Un-Owned Locks: Use Lua scripts to ensure a worker completes or deletes a lock key ONLY if its worker UUID matches the key payload.
Enforce Storage-Layer Unique Constraints: Always back Redis idempotency checks with unique compound indexes (e.g., (tenant_id, idempotency_key)) inside PostgreSQL or MySQL, and forward the key to external payment providers.

Worried about double charges or duplicate records in your payment flows? I help SaaS teams design idempotent APIs, safe distributed locking, and database guardrails that hold up under retries and failures. Book a consultation →

Originally published on ctousman.com.

PostgreSQL JSONB Write Amplification: TOAST Chunk Fragmentation and Unseen I/O Thrashing

Usman Khan — Tue, 13 Oct 2026 16:03:10 +0000

Storing flexible schema payloads inside PostgreSQL JSONB columns feels like the ultimate developer productivity shortcut. However, mutating a single nested field inside a large JSONB document forces PostgreSQL to rewrite the entire compressed TOAST object on disk, triggering massive write amplification, Write-Ahead Log (WAL) flooding, and IOPS exhaustion. Here is an architectural deep dive into TOAST storage mechanics and how to eliminate JSONB write amplification.

In growth-stage B2B SaaS platforms, using PostgreSQL JSONB columns for multi-tenant configuration blocks, feature flags, or dynamic custom attributes is a common architectural pattern. While schema-less flexibility speeds up initial feature shipping, running high-frequency UPDATE statements against JSONB documents creates an invisible, compounding infrastructure trap: catastrophic Write Amplification driven by the PostgreSQL TOAST (The Oversized-Attribute Storage Technique) engine.

1. The Mechanics of TOAST Storage and Partial JSONB Mutations

PostgreSQL stores data in fixed-size pages of 8KB by default. When a row exceeds roughly 2KB, PostgreSQL transparently compresses its large values (such as long text blocks or serialized JSONB documents) and, if they are still too big, moves them out of line into an auxiliary storage area known as the TOAST table.

The Hidden Rewrite Penalty

A widespread architectural misconception is assuming that updating a single key inside a JSONB document using functions and operators like jsonb_set() or || performs a localized, in-place edit on disk. In reality, PostgreSQL has no partial-update path for TOASTed values: any change produces a brand-new copy of the whole value.

Whole-Document Serialization: When you update a 1-byte boolean inside a 60KB JSONB column, PostgreSQL reads the compressed TOAST chunks, decompresses the document, applies the mutation, re-compresses the entire 60KB payload, and writes completely new TOAST chunks to disk.
WAL Flooding: Because every new TOAST chunk is WAL-logged, changing a single character generates tens of kilobytes of WAL data. This floods replication streams, exhausts disk write IOPS, and drastically delays read-replica synchronization.
Double MVCC Overhead: The parent table record gets a new dead tuple (forcing a primary heap write), while the TOAST table acquires multiple new dead chunk tuples (forcing secondary heap and TOAST index writes), doubling autovacuum cleanup workloads.

2. Measuring the Amplification Factor: Code & Query Proof

To demonstrate the magnitude of JSONB write amplification, consider a simple settings table holding tenant feature toggles:

-- Production Target Schema with Large JSONB Document
CREATE TABLE tenant_settings (
    tenant_id UUID PRIMARY KEY,
    company_name VARCHAR(255) NOT NULL,
    config_payload JSONB NOT NULL -- Large nested user preferences & permissions
);

-- Seed a ~70KB document of 1,500 keys; random values defeat compression,
-- so the payload is forced into out-of-line TOAST storage
INSERT INTO tenant_settings (tenant_id, company_name, config_payload)
SELECT
    'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11',
    'Acme Corp',
    jsonb_build_object('feature_a', true)
      || jsonb_object_agg('pref_' || i, md5(random()::text))
FROM generate_series(1, 1500) AS i;

Now flip a single flag and measure exactly how much WAL that one statement generates (run in psql):

-- Capture the WAL position before the update
SELECT pg_current_wal_lsn() AS before_lsn \gset

-- Seemingly innocent partial JSONB update
UPDATE tenant_settings
SET config_payload = jsonb_set(config_payload, '{feature_a}', 'false'::jsonb)
WHERE tenant_id = 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11';

-- WAL bytes written by that single-flag change
SELECT pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), :'before_lsn')
) AS wal_generated;

The logical change is a single boolean, but PostgreSQL rewrites and WAL-logs the entire compressed document. In a test on PostgreSQL 16 with this exact setup, the single flag flip wrote about 58KB of WAL. The same change stored in a plain boolean column (Strategy A below) wrote under 200 bytes, roughly 300x less. Multiply that by every settings save across thousands of tenants and the I/O cost becomes the dominant load on your database.

3. Architectural Solutions: Eliminating JSONB Write Traps

Strategy A: Hybrid Column Decomposition (Extract Fast-Changing Attributes)

Identify attributes within the JSONB payload that experience high mutation frequency (such as last_active_at, login_counter, or active_status) and promote them to dedicated, typed relational columns on the main table. When an UPDATE leaves the JSONB column untouched, PostgreSQL reuses the existing TOAST pointer instead of rewriting the document.

-- Remediation: Decouple Hot Fields
-- Use non-volatile defaults (now(), not clock_timestamp()) so ADD COLUMN
-- stays a fast metadata-only change instead of a full table rewrite
ALTER TABLE tenant_settings
ADD COLUMN is_active BOOLEAN DEFAULT true,
ADD COLUMN updated_at TIMESTAMPTZ DEFAULT now();

-- Now updates hit inline page tuples WITHOUT touching the large TOAST object
UPDATE tenant_settings
SET is_active = false, updated_at = now()
WHERE tenant_id = 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11';

Strategy B: Key-Value EAV Partitioning for Unstructured Data

If you must store thousands of highly dynamic key-value pairs per tenant, decompose the monolithic JSONB document into an explicit narrow key-value table. This turns each update from a cost proportional to the whole document size into a small, constant-size single-row write.

-- Normalized EAV Storage for Highly Dynamic Attributes
CREATE TABLE tenant_attributes (
    tenant_id UUID NOT NULL REFERENCES tenant_settings(tenant_id),
    attribute_key VARCHAR(128) NOT NULL,
    attribute_value JSONB NOT NULL,
    PRIMARY KEY (tenant_id, attribute_key)
);

-- Mutating a single setting rewrites ONLY one small row, keeping I/O minimal
INSERT INTO tenant_attributes (tenant_id, attribute_key, attribute_value)
VALUES ('a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11', 'feature_a', 'false'::jsonb)
ON CONFLICT (tenant_id, attribute_key)
DO UPDATE SET attribute_value = EXCLUDED.attribute_value;

4. TOAST Storage Tuning for Read-Heavy JSONB Documents

For JSONB columns that are large but rarely updated, adjust the TOAST compression strategy to minimize CPU overhead during read operations. Both settings apply only to values written after the change; existing rows keep their current format until they are rewritten.

-- Switch TOAST compression from default pglz to LZ4 (PostgreSQL 14+)
ALTER TABLE tenant_settings ALTER COLUMN config_payload SET COMPRESSION lz4;

-- Or skip compression entirely: more disk space, less CPU on every read
ALTER TABLE tenant_settings ALTER COLUMN config_payload SET STORAGE EXTERNAL;

5. Executive Summary & Systems Rules

Never Mutate Large JSONB Documents Frequently: If a JSONB value is large enough to be TOASTed (roughly 2KB+) and undergoes frequent updates, it WILL trigger extreme TOAST write amplification and WAL bloat.
Promote Hot Keys to Relational Columns: Extract frequently updated keys out of JSONB blocks and store them as top-level typed columns.
Monitor WAL Generation Rates: Use pg_wal_lsn_diff() to track unexpected WAL volume spikes tied to JSONB update workloads.

Are JSONB updates quietly eating your database I/O budget? I help SaaS teams find hidden write amplification, redesign hot JSONB schemas, and tune Postgres storage for heavy write loads. Book a consultation →

Originally published on ctousman.com.

Server-Sent Events (SSE) Backpressure Breaks: Memory Leaks and Socket Starvation in High-Throughput LLM Pipelines

Usman Khan — Sat, 10 Oct 2026 14:27:10 +0000

Streaming LLM completions over Server-Sent Events (SSE) looks trivial in sandbox demos, but unhandled backpressure under real-world network conditions can destroy backend application servers. When slow clients fail to consume token chunks as fast as upstream model APIs generate them, unbounded Node.js internal stream buffers trigger memory leaks and OOM process crashes. Here is an architectural deep dive into enforcing stream backpressure, managing drain events, and protecting socket thread pools.

Integrating generative AI features into enterprise SaaS interfaces relies heavily on streaming response tokens via Server-Sent Events (SSE) to lower Time-To-First-Token (TTFT). However, naïve stream piping implementations assume the client's network connection can consume data at the same rate the upstream model provider (e.g., OpenAI or Anthropic) yields tokens. Under real-world network conditions, this assumption breaks down catastrophically.

1. The Mechanics of Unhandled Backpressure in Event Streams

In Node.js and similar asynchronous event-driven runtimes, HTTP responses are writable streams backed by internal memory buffers. When an upstream LLM API yields token deltas at high velocity (e.g., 80 to 120 tokens per second), the application server forwards these chunks directly into the client's HTTP response socket.

How Slow Clients Trigger Node.js Heap Exhaustion

Producer-Consumer Velocity Mismatch: An upstream LLM provider generates text rapidly, while a client on a high-latency mobile connection or throttled browser tab consumes data slowly.
Unbounded Internal Buffering: When res.write(chunk) is called faster than the TCP socket can flush frames over the wire, Node.js buffers the overflow chunks in process memory.
Cascading Out-Of-Memory (OOM) Crashes: Under high concurrency (e.g., 500 parallel streaming sessions), unbounded buffer growth consumes gigabytes of memory, ending in JavaScript heap out of memory process crashes across backend instances.

2. The Flaw in Naive SSE Controller Implementations

A standard anti-pattern in Node.js/Express controllers involves directly listening to LLM token events and pushing them into the response without inspecting return values or handling stream backpressure:

// NAÏVE ANTI-PATTERN: Zero Backpressure Control
export async function streamLLMResponseNaive(req: Request, res: Response) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  const upstreamStream = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: req.body.prompt }],
    stream: true,
  });

  // FAILURE VECTOR: If client socket is slow, res.write returns FALSE
  // Naïve code ignores this signal, buffering unbounded chunks in RAM!
  for await (const chunk of upstreamStream) {
    const token = chunk.choices[0]?.delta?.content || '';
    res.write(`data: ${JSON.stringify({ token })}\n\n`);
  }

  res.write('data: [DONE]\n\n');
  res.end();
}

When res.write() returns false, it explicitly signals that the stream's internal highWaterMark threshold has been breached. Continuing to push data ignores this backpressure boundary.

3. Production-Ready Backpressure-Aware SSE Pipeline

To safely stream LLM tokens at scale, backend services must inspect res.write() signals. If a write returns false, the worker must pause the upstream reader until the underlying TCP socket fires a 'drain' event. The drain wait must also be cancellable: if the client disconnects while the buffer is full, 'drain' never fires, and an un-cancellable wait becomes its own memory leak.

// PRODUCTION PATTERN: Explicit Backpressure & Abort Handling
import { Request, Response } from 'express';
import { once } from 'events';

export async function streamLLMResponseResilient(req: Request, res: Response) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('X-Accel-Buffering', 'no'); // Disable Nginx proxy buffering

  const abortController = new AbortController();

  // Clean up upstream request if client abruptly disconnects.
  // Listen on res, not req: req 'close' fires as soon as the request body is read.
  res.on('close', () => {
    if (!res.writableEnded) abortController.abort();
  });

  try {
    const upstreamStream = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: req.body.prompt }],
      stream: true,
    }, { signal: abortController.signal });

    for await (const chunk of upstreamStream) {
      if (abortController.signal.aborted) break;

      const token = chunk.choices[0]?.delta?.content || '';
      if (!token) continue;

      const payload = `data: ${JSON.stringify({ token })}\n\n`;

      // 1. Write chunk and inspect backpressure buffer capacity
      const canContinue = res.write(payload);

      // 2. If internal buffer is full, pause until the socket drains
      //    (or the client disconnects, which rejects with AbortError)
      if (!canContinue) {
        await once(res, 'drain', { signal: abortController.signal });
      }
    }

    if (!abortController.signal.aborted) {
      res.write('data: [DONE]\n\n');
      res.end();
    }
  } catch (error: any) {
    if (error.name === 'AbortError') {
      console.log('[SSE_STREAM] Client disconnected mid-stream. Resource freed.');
    } else {
      console.error('[SSE_STREAM_ERROR]', error);
      // Headers are already sent once streaming starts, so just close the stream
      if (!res.headersSent) res.status(500);
      res.end();
    }
  }
}

4. Reverse Proxy Tuning: Nginx and AWS ALB Buffering Traps

Backpressure handling in application code is rendered useless if reverse proxies in your network path silently buffer event streams in memory.

Nginx Proxy Buffering: By default, Nginx buffers upstream HTTP responses before forwarding them to clients. This breaks real-time token delivery and obscures backpressure signals. Always pass the X-Accel-Buffering: no header or set proxy_buffering off; in location blocks.
AWS ALB Idle Timeouts: Ensure your AWS Application Load Balancer idle timeout exceeds the maximum expected LLM completion duration (e.g., set to 120+ seconds) to prevent premature 504 Gateway Timeouts on long reasoning chains.

5. Executive Summary & Systems Rules

Respect res.write() Return Values: Never write to an HTTP response stream blindly. Pause upstream generation when res.write() returns false and await the 'drain' event, with an abort signal so the wait can't hang forever.
Attach Abort Controllers to Client Sockets: Always listen for res.on('close') to terminate upstream LLM API connections immediately when users navigate away or close tab sockets.
Disable Proxy Buffering: Configure edge proxies and gateways explicitly to allow immediate, unbuffered chunk passthrough for text/event-stream content types.

Are your LLM streaming endpoints leaking memory or dropping connections under load? I help SaaS teams build production-grade AI streaming pipelines with proper backpressure, abort handling, and proxy configuration. Book a consultation →

Originally published on ctousman.com.

PostgreSQL MVCC Autovacuum Starvation: HOT Update Breakdown & Table Bloat under Heavy Write Loads

Usman Khan — Fri, 09 Oct 2026 15:32:22 +0000

When PostgreSQL tables scale past hundreds of millions of rows under high-frequency updates, default autovacuum configurations quietly break. The failure of Heap Only Tuple (HOT) updates triggers exponential index bloat, disk I/O thrashing, and database starvation. Here is an architectural deep dive into diagnosing HOT update degradation, tuning fillfactors, and optimizing autovacuum for heavy SaaS write workloads.

In high-throughput B2B SaaS platforms, relational databases frequently handle thousands of state mutations per second: updating tenant session tokens, subscription statuses, or webhook dispatch counters. Under default PostgreSQL settings, Multi-Version Concurrency Control (MVCC) mechanics can silently trigger catastrophic table and index bloat, degrading query performance from single-digit milliseconds to multi-second timeouts.

1. The Mechanics of MVCC and the HOT Update Failure Vector

PostgreSQL handles concurrent operations via MVCC. When an application executes an UPDATE statement, Postgres does not overwrite the existing row on disk in place. Instead, it creates a new row version (a new tuple) and marks the old tuple as dead. Dead tuples remain on disk until cleaned up by the autovacuum daemon process.

How Heap Only Tuple (HOT) Updates Prevent Index Bloat

Normally, inserting a new tuple requires updating every index attached to the table. To avoid this massive I/O penalty, Postgres uses Heap Only Tuple (HOT) updates. A HOT update occurs when:

The UPDATE does not modify any column referenced by a table index.
The target page in heap memory has sufficient free space to store the new tuple version alongside the old one.

When HOT criteria are met, Postgres links the new tuple directly from the old tuple inside the same data page. B-Tree indexes continue pointing to the original root tuple, completely avoiding index write amplification.

The Silent Breakdown Under Production Load

When an application updates an indexed column (such as updating updated_at or last_seen_at where an index exists), or when heap pages fill up completely (100% page density), HOT updates fail completely. Every row update forces new index pointers into every B-tree index on the table, resulting in severe index bloat that autovacuum cannot reclaim efficiently.

2. Diagnosing HOT Update Failure and Autovacuum Starvation

To detect if your write-heavy tables are suffering from HOT update degradation, run the following diagnostic query on your production PostgreSQL instance:

-- Diagnostic Query: Evaluating HOT Update Success Ratios across Tables
SELECT
    schemaname,
    relname AS table_name,
    n_tup_upd AS total_updates,
    n_tup_hot_upd AS hot_updates,
    ROUND(
        (n_tup_hot_upd::float / NULLIF(n_tup_upd, 0)::float) * 100
    ) AS hot_update_ratio_pct,
    n_dead_tup AS dead_tuple_count,
    last_autovacuum
FROM pg_stat_user_tables
WHERE n_tup_upd > 10000
ORDER BY n_tup_upd DESC;

If your hot_update_ratio_pct falls below 80–90% on a table experiencing heavy updates, your database is actively incurring massive index write amplification and page fragmentation.

3. Architectural Remediation: Table Fillfactor & Autovacuum Tuning

Step 1: Setting Table-Level Fillfactors for HOT Space Allocation

By default, PostgreSQL fills heap pages to 100% capacity during initial INSERT operations. Consequently, subsequent UPDATE calls have zero free space on the page for a new HOT tuple. Reduce the table fillfactor to reserve space on heap pages explicitly for row updates.

-- Reduce page density to 80% to reserve 20% page space for HOT updates
ALTER TABLE tenant_session_states SET (fillfactor = 80);

-- Rewrite existing pages so the new fillfactor applies to old data too.
-- VACUUM FULL takes an exclusive lock; in production, use pg_repack instead.
VACUUM FULL tenant_session_states;

Step 2: Aggressive Per-Table Autovacuum Tuning

Global default autovacuum parameters are far too conservative for high-volume SaaS databases. Instead of waiting for 20% of a multi-million row table to turn into dead tuples before vacuuming, configure aggressive, scale-factor thresholds on write-heavy tables:

-- Configure Aggressive Autovacuum Parameters for High-Throughput Write Tables
ALTER TABLE tenant_session_states SET (
    autovacuum_vacuum_scale_factor = 0.02,     -- Trigger vacuum after 2% row changes (default is 20%)
    autovacuum_vacuum_threshold = 1000,        -- Minimum dead tuple threshold
    autovacuum_vacuum_cost_limit = 2000,       -- Increase I/O credit allocation for this table
    autovacuum_vacuum_cost_delay = 2           -- Reduce I/O delay between vacuum cycles (ms)
);

4. Removing Redundant & Unused Indexes

Every additional index on a table reduces the probability of a HOT update. Eliminate synthetic indexes on frequently updated columns (such as timestamps used strictly for auditing rather than filtering) and drop duplicate B-tree indexes.

-- Identifying Unused Indexes Causing HOT Update Blockers
SELECT
    schemaname,
    relname AS table_name,
    indexrelname AS index_name,
    idx_scan AS number_of_scans,
    pg_size_pretty(pg_relation_size(indexrelid)) AS index_size
FROM pg_stat_user_indexes
JOIN pg_index USING (indexrelid)
WHERE idx_scan = 0
  AND indisunique = false
ORDER BY pg_relation_size(indexrelid) DESC;

5. Executive Summary & Systems Rules

Isolate Frequently Mutated Attributes: Move fast-changing counter columns (e.g., last_active_at, login_count) into a dedicated secondary state table to protect the primary table from HOT update failures.
Never Leave High-Write Tables at Fillfactor 100: Set fillfactor = 70-80 on high-frequency update tables to give PostgreSQL the required heap page room for HOT tuples.
Tune Autovacuum per Table: Override global defaults on high-volume tables to vacuum dead tuples incrementally before index bloat degrades query execution plans.

Is table bloat or autovacuum lag slowing down your Postgres database? I help SaaS teams audit database performance, tune autovacuum and fillfactors, and design multi-tenant data architectures that hold up under heavy write loads. Book a consultation →

Originally published on ctousman.com.

Production AI & LLM Pipelines: Guardrails, Streaming Resilience, and Cost-Aware Fallbacks

Usman Khan — Thu, 08 Oct 2026 21:09:12 +0000

Moving generative AI features from prototype scripts to mission-critical SaaS production requires far more than wrapping an OpenAI or Anthropic API client. Upstream API timeouts, rate-limit spikes, context window overflows, and malformed JSON outputs cause cascading UI crashes if unhandled. Here is an architectural playbook for building streaming-resilient LLM pipelines with structural JSON guardrails, dynamic model fallbacks, and token budget safety nets.

Deploying Large Language Models into enterprise SaaS platforms introduces failure modes never encountered in traditional REST microservices. When upstream LLM providers suffer transient latency spikes, HTTP 429 rate limits, or produce hallucinated schema structures, your application backend must maintain strict availability SLAs.

1. The Vulnerability of Unhedged LLM Integration

Integrating generative AI features via synchronous inline API calls exposes SaaS platforms to three critical operational risks:

Non-Deterministic Latencies: Inference responses can span anywhere from 800ms to 30 seconds, causing web server socket timeouts and user experience degradation.
Upstream Outages and Rate Limiting: Single-vendor dependence means an upstream outage or sudden tenant spike instantly breaks all AI capabilities across your product.
Schema Drift & Unstructured Outputs: Prompt completion models occasionally produce invalid JSON formats, breaking downstream application parser logic and triggering unhandled runtime exceptions.

2. Architectural Blueprint: Resilience and Fallback Layer

A production-ready LLM orchestration layer wraps model provider SDKs in an abstraction client featuring automated circuit breaking, schema validation retries, and dynamic model tier fallback routing.

Dynamic Model Fallback Execution Pattern

If the primary model endpoint (e.g., Claude 3.5 Sonnet) exceeds a latency budget or throws a transient HTTP error, the pipeline immediately reroutes the execution to a secondary provider or smaller, faster model tier (e.g., GPT-4o-mini or Llama 3) without failing the client request.

// Production-Grade Resilient LLM Invocation Engine (TypeScript)
import { ZodSchema } from 'zod';

interface ModelRequestPayload<T> {
  prompt: string;
  systemPrompt: string;
  schema: ZodSchema<T>;
  timeoutMs?: number;
}

export class ResilientLLMPipeline {
  private primaryTimeout = 6000; // 6 second primary SLA limit

  async executeWithFallback<T>(payload: ModelRequestPayload<T>): Promise<T> {
    try {
      // 1. Attempt Primary High-Capability Model (e.g., Primary Provider)
      return await this.invokeModelWithTimeout(
        'primary-tier-1',
        payload,
        payload.timeoutMs || this.primaryTimeout
      );
    } catch (error) {
      console.warn('[LLM_FALLBACK] Primary model failed or timed out. Engaging secondary provider.', error);

      // 2. Fallback to Secondary Cost-Efficient Model (e.g., Secondary Provider)
      return await this.invokeModelWithTimeout(
        'secondary-tier-2',
        payload,
        10000 // Extended timeout for fallback
      );
    }
  }

  private async invokeModelWithTimeout<T>(
    modelId: string,
    payload: ModelRequestPayload<T>,
    timeout: number
  ): Promise<T> {
    const controller = new AbortController();
    const timeoutId = setTimeout(() => controller.abort(), timeout);

    try {
      const rawResult = await callLLMProvider(modelId, payload, controller.signal);

      // 3. Structural Guardrail Validation
      const validatedData = payload.schema.safeParse(rawResult);
      if (!validatedData.success) {
        throw new Error(`JSON_SCHEMA_VALIDATION_FAILED: ${validatedData.error.message}`);
      }

      return validatedData.data;
    } finally {
      clearTimeout(timeoutId);
    }
  }
}

3. Real-Time Streaming Resilience with Server-Sent Events (SSE)

For interactive user-facing AI applications (such as workspace copilots or conversational interfaces), holding an HTTP connection open until the complete completion is generated creates severe perceived latency. Streaming response tokens via Server-Sent Events (SSE) reduces Time-To-First-Token (TTFT) from seconds to milliseconds.

Decoupling Connection Management

Client Handshake: Establish an SSE connection (Content-Type: text/event-stream) from the client web application to the API gateway.
Token Chunking: Pipe upstream vendor streaming deltas directly to the response socket, minimizing internal buffer overhead.
Connection Termination Safeguards: If a client disconnects mid-stream, abort the upstream LLM API request immediately to prevent wasted token consumption on unread responses.

4. Enterprise Guardrails and Safety Controls

Strict Outbound Token Budget Caps: Enforce tenant-level token bucket limits in Redis to prevent rogue API scripts or prompt injection loops from generating multi-thousand dollar bill shocks.
Input Pre-Filtering & Sanitization: Validate and sanitize user inputs prior to prompt construction to mitigate prompt injection attacks and eliminate unnecessary whitespace bloat.
Asynchronous Telemetry Logging: Offload raw prompt payloads, completion metadata, latency metrics, and actual consumed token counts to cold storage (e.g., ClickHouse or S3) asynchronously via background queues.

5. Key Engineering Summary

Never Depend on a Single AI Provider: Abstract LLM integrations behind a unified client layer capable of seamless multi-vendor fallbacks.
Enforce Schema Validation at the Edge: Use typed structural validation libraries (e.g., Zod) to validate model completion outputs before returning data to application logic.
Optimize for TTFT via SSE: Use Server-Sent Events to stream tokens in real-time, drastically lowering perceived latency for end-users.

Shipping AI features that need to stay up when your LLM provider doesn't? I help SaaS teams design production LLM pipelines with model fallbacks, schema guardrails, and token budgets that keep costs and outages under control. Book a consultation →

Originally published on ctousman.com.

Advanced System Architecture: Designing Multi-Tenant Event-Driven Queues with Fair-Share Scheduling

Usman Khan — Wed, 07 Oct 2026 16:10:11 +0000

In multi-tenant SaaS systems, standard FIFO (First-In, First-Out) background job queues present a severe architectural risk: a single enterprise tenant queuing 100,000 asynchronous tasks can monopolize worker pools and starve interactive jobs for thousands of other tenants. Here is an engineering blueprint for implementing multi-tenant fair-share queue schedulers, dynamic concurrency isolation, and weighted round-robin dispatchers.

In multi-tenant SaaS platforms, background processing engines (handling webhooks, PDF generation, bulk data syncs, or video processing) must balance throughput against tenant isolation. A naive global First-In, First-Out (FIFO) queue allows a single high-volume customer to overwhelm worker pools, resulting in multi-hour processing delays for neighboring tenants.

1. The Noisy Neighbor Problem in Asynchronous Queues

Standard distributed queue setups, such as a single Redis list, SQS queue, or RabbitMQ exchange, treat all incoming payloads equally. Under light traffic, this abstraction works seamlessly. However, as tenant usage diverges between self-serve accounts and enterprise clients, systemic vulnerabilities emerge:

Worker Pool Starvation: Tenant A triggers a batch import containing 50,000 tasks. Tenant B triggers a single, time-sensitive transactional invoice email. Under FIFO, Tenant B waits behind 50,000 tasks.
Head-of-Line Blocking: Unhandled retries or slow downstream execution for one tenant's tasks stall worker threads, degrading processing SLAs for all tenants sharing the cluster.
Unpredictable Latency Variances: P99 queue wait times balloon, violating enterprise customer SLAs and triggering cascading API timeouts.

2. Architectural Pattern: Per-Tenant Virtual Queues with Fair-Share Dispatching

To guarantee tenant isolation without over-provisioning worker infrastructure, systems must decouple job ingestion from worker execution. Instead of pushing payloads to a single shared queue, jobs are routed into tenant-specific virtual queues managed by an intermediate Fair-Share Dispatcher.

The Deficit Round-Robin (DRR) Scheduler Engine

A proven strategy for fair-share queue orchestration is the Deficit Round-Robin (DRR) algorithm. The scheduler cycles through active tenant queues, dispensing a fixed processing credit (quantum) per turn. If a tenant's queue is empty or exhausts its quantum, the dispatcher advances to the next tenant.

// Production Fair-Share Tenant Queue Dispatcher (TypeScript / Redis Pattern)
import { Redis } from 'ioredis';

interface TenantJob {
  id: string;
  tenantId: string;
  payload: Record<string, any>;
  costWeight: number; // e.g., estimated execution time or complexity
}

export class FairShareQueueDispatcher {
  private redis: Redis;
  private quantumCost: number = 10; // Max execution credits per round

  constructor(redisClient: Redis) {
    this.redis = redisClient;
  }

  /**
   * Pushes a job to a tenant-isolated sub-queue
   */
  async enqueue(job: TenantJob): Promise<void> {
    const tenantQueueKey = `queue:tenant:${job.tenantId}`;
    const activeTenantsSet = 'queue:active_tenants';

    await this.redis.multi()
      .rpush(tenantQueueKey, JSON.stringify(job))
      .sadd(activeTenantsSet, job.tenantId)
      .exec();
  }

  /**
   * Dequeues the next fair-share job across active tenants
   */
  async fetchNextJob(): Promise<TenantJob | null> {
    const activeTenants = await this.redis.smembers('queue:active_tenants');
    if (activeTenants.length === 0) return null;

    for (const tenantId of activeTenants) {
      const tenantQueueKey = `queue:tenant:${tenantId}`;
      const rawJob = await this.redis.lpop(tenantQueueKey);

      if (!rawJob) {
        // Remove empty tenant queues from active rotation
        await this.redis.srem('queue:active_tenants', tenantId);
        continue;
      }

      const job: TenantJob = JSON.parse(rawJob);

      // Re-add tenant to active set if additional items remain
      const remainingCount = await this.redis.llen(tenantQueueKey);
      if (remainingCount > 0) {
        await this.redis.sadd('queue:active_tenants', tenantId);
      } else {
        await this.redis.srem('queue:active_tenants', tenantId);
      }

      return job; // Return job for worker execution frame
    }

    return null;
  }
}

3. Enforcing Tenant Concurrency Caps with Distributed Semaphores

Fair-share round-robin scheduling guarantees equal queue entry opportunities, but long-running tasks can still saturate worker threads. To prevent slow jobs from occupying all execution slots, enforce Dynamic Max Concurrency Limits per tenant tier using Redis semaphores.

-- Lua Script: Tenant Concurrency Semaphore Lock
local tenantLockKey = KEYS[1]
local maxAllowedConcurrency = tonumber(ARGV[1])
local lockTtlSeconds = tonumber(ARGV[2])

local currentConcurrency = redis.call('SCARD', tenantLockKey)

if currentConcurrency < maxAllowedConcurrency then
    local workerToken = redis.call('INCR', 'global:worker_nonce')
    redis.call('SADD', tenantLockKey, workerToken)
    redis.call('EXPIRE', tenantLockKey, lockTtlSeconds)
    return {1, tostring(workerToken)} -- Lock acquired
else
    return {0, "CONCURRENCY_LIMIT_EXCEEDED"} -- Max concurrent slots active
end

4. Tier-Based SLA Scheduling Strategies

While strict equality works well for homogeneous environments, enterprise SaaS business models often demand priority weighted scheduling for higher-tier accounts:

Weighted Deficit Allocations: Assign larger credit quanta to Enterprise tier tenants (e.g., Enterprise = 50 credits, Standard = 10 credits) so high-tier accounts drain faster while lower tiers remain protected from total starvation.
Dedicated High-Priority Worker Pools: Partition 20% of worker node infrastructure exclusively for low-latency interactive tasks (e.g., OAuth tokens, user-facing notifications) while routing background batch syncs to a flexible spot-instance cluster.
Adaptive Backpressure Notifications: Expose queue depth and estimated execution time metrics via tenant APIs so clients can adjust bulk upload speeds before hitting queue throttles.

5. Key Engineering Takeaways

Never Rely on a Single Global FIFO Queue: Partition asynchronous workloads into tenant-scoped virtual queues to eliminate noisy neighbor risks.
Decouple Ingestion from Dispatching: Use algorithms like Deficit Round-Robin to distribute worker processing capacity equitably across active tenants.
Combine Fair Scheduling with Concurrency Controls: Enforce strict per-tenant worker concurrency limits to prevent long-running tasks from occupying all compute resources.

Is one big customer slowing down background jobs for everyone else? I help SaaS teams design tenant-isolated queue architectures, fair-share schedulers, and concurrency limits that keep every customer fast. Book a consultation →

Originally published on ctousman.com.

Zero-Downtime Database Migrations: Shadow Tables, Dual-Writing, and Schema Expansion

Usman Khan — Tue, 06 Oct 2026 16:23:49 +0000

Taking a mission-critical B2B SaaS database offline for a multi-hour ALTER TABLE migration is no longer acceptable for high-growth engineering teams. Long-running DDL statements acquire exclusive table locks, holding incoming web transactions hostage and causing catastrophic application outages. Here is an actionable operational guide to executing zero-downtime schema migrations using shadow tables, dual-write abstraction layers, and zero-lock schema phase expansion.

In high-throughput relational database architectures, running a blocking ALTER TABLE ADD COLUMN or changing a column type on a table containing hundreds of millions of rows can trigger exclusive AccessExclusiveLocks. This blocks read and write queries, causes connection pool backpressure, and brings down web application layers. Scaling B2B SaaS applications requires decoupling schema evolution from application code deployments entirely.

1. The Anatomy of Database Locking Vulnerabilities

When an application attempts structural DDL modifications on relational databases like PostgreSQL or MySQL, the engine requests exclusive table locks. Under active write load, three major failure modes occur:

Lock Queue Contention: A heavy DDL statement waiting for existing read queries to finish will block all subsequent incoming SELECT and UPDATE queries behind it in the lock queue, exhausting connection pools in seconds.
Table Rewrite Overhead: Changing data types or setting default values without DEFAULT NULL on older engine versions forces full physical table rewrites, locking I/O channels for hours.
Application-Database Version Mismatch: Deploying code that expects a new column before the database finished updating, or dropping a column while old application nodes are still serving traffic, causes unhandled HTTP 500 runtime exceptions.

2. The Expand-Contract (Parallel Run) Migration Framework

To safely migrate production tables containing tens of millions of rows, engineering teams employ the Expand-Contract Pattern, breaking a single risky database alteration into discrete, backward-compatible phases.

Phase 1: Expand Schema (Non-Blocking DDL)

Add new columns, indices, or shadow tables without deleting or modifying existing columns. All new database structures must be optional or nullable to preserve backward compatibility with current application deployments.

-- PostgreSQL Safe Column Creation with Low Lock Timeout
SET lock_timeout = '2s';

-- Add column as NULLABLE first to avoid table rewrite
ALTER TABLE organization_subscriptions
ADD COLUMN billing_tier_v2 VARCHAR(64) DEFAULT NULL;

-- Create indices CONCURRENTLY to prevent blocking concurrent write transactions
CREATE INDEX CONCURRENTLY idx_org_sub_tier_v2
ON organization_subscriptions (billing_tier_v2)
WHERE billing_tier_v2 IS NOT NULL;

Phase 2: Enable Dual-Write Application Abstraction

Update backend application logic to write payload mutations to both the old schema location and the new schema location simultaneously. Primary reads continue pointing to the legacy column.

// TypeScript Dual-Write Abstraction Layer Example
async function updateSubscriptionTier(orgId: string, newTier: string): Promise<void> {
  await db.transaction(async (tx) => {
    // Write to Legacy Structure (Source of Truth for Reads)
    await tx.query(
      `UPDATE organization_subscriptions SET tier = $1 WHERE org_id = $2`,
      [newTier, orgId]
    );

    // Dual-Write to Expanded Structure (Shadow Destination)
    await tx.query(
      `UPDATE organization_subscriptions SET billing_tier_v2 = $1 WHERE org_id = $2`,
      [transformToNewFormat(newTier), orgId]
    );
  });
}

3. Backfilling Historical Data and Parity Auditing

With dual-writes active, newly modified records reflect in both schemas. However, historical records created prior to the expansion phase must be backfilled asynchronously without exceeding database CPU, memory, and IOPS budgets.

Batch Backfilling Pattern

Use chunked keyset pagination (never offset-based pagination) to iterate through historical records in background batch workers.

-- Batch Backfill Query using Keyset Pagination (Chunk Size: 1000)
-- $1 = last processed id from the previous batch
UPDATE organization_subscriptions
SET billing_tier_v2 = CASE
    WHEN tier = 'BASIC' THEN 'plan_tier_1'
    WHEN tier = 'ENTERPRISE' THEN 'plan_tier_3'
    ELSE 'plan_tier_2'
END
WHERE id IN (
    SELECT id FROM organization_subscriptions
    WHERE billing_tier_v2 IS NULL
    AND id > $1
    ORDER BY id ASC
    LIMIT 1000
);

Shadow Read Parity Auditing

Before switching read operations over to the new schema, run an asynchronous comparison worker that executes shadow reads, comparing legacy outputs against new schema outputs to verify 100% data parity.

4. Phase 3 & 4: Cutover Reads and Contract Legacy Schema

Once parity auditing reaches 100% data consistency and background backfills finish, execute the final cutover:

Cutover Reads: Deploy application code that reads primary queries from billing_tier_v2 while maintaining dual-writes to the old column for an observation period (e.g., 48 hours).
Stop Dual-Writes: Deploy application code that removes legacy column write references completely.
Contract Schema: Drop the old column asynchronously using ALTER TABLE ... DROP COLUMN ... with low lock timeouts.

5. Key Architectural Rules for Zero-Downtime Rollouts

Set Explicit Lock Timeouts: Never execute production DDL without setting SET lock_timeout = '2s';. It is significantly better for a migration script to fail and retry than to hold up the database lock queue.
Never Change Data Types In-Place: Altering a column type (e.g., INT to BIGINT or VARCHAR to JSONB) forces a full table rewrite. Always build a shadow column or shadow table instead.
Decouple Deployments: Every phase of an Expand-Contract migration must be a separate code deployment. Never attempt schema expansion, backfilling, and legacy column dropping in a single deployment pipeline.

Planning a risky schema change on a production database? I help SaaS teams design zero-downtime migration plans, from expand-contract rollouts to backfill pipelines that won't take your app offline. Book a consultation →

Originally published on ctousman.com.

Advanced System Architecture: Designing Multi-Region Read Replicas and Global Data Consistency

Usman Khan — Mon, 05 Oct 2026 09:38:07 +0000

TL;DR: As enterprise SaaS platforms expand internationally, serving all database queries from a single primary region causes severe latency degradation for global users. Distributing read replicas across multiple cloud regions solves latency, but introduces complex consistency guarantees. Here is an architectural deep dive into designing multi-region database topologies, handling read-after-write consistency, and mitigating replication lag.

Serving global enterprise clients from a single database cluster region introduces unavoidable physical network latency. A user in Singapore querying a database in US-East incurs a baseline 200ms round-trip delay before business logic even executes. Deploying multi-region read replicas minimizes read latencies to single-digit milliseconds, but requires precise handling of replication lag and global consistency guarantees.

1. The Multi-Region Topology Spectrum

When expanding database infrastructure across geographical regions, engineering leadership must choose a replication strategy based on write volume and consistency requirements:

Single-Leader Cross-Region Replication: A primary database cluster accepts all write traffic in one central region (e.g., us-east-1) and streams physical Write-Ahead Logs (WAL) or binary logs asynchronously to read-only replicas in target regions (e.g., eu-central-1, ap-southeast-1).
Multi-Leader (Active-Active) Replication: Primary databases in multiple regions accept local writes and synchronize changes asynchronously. Highly complex due to cross-region write conflict resolution policies (e.g., Last-Write-Wins vs. CRDTs).
Globally Distributed Consensus Engines: Engines like CockroachDB, Google Spanner, or YugabyteDB use Raft or Paxos consensus across regions to provide strict serializability at the cost of higher write latency.

For 95% of growth-stage B2B SaaS platforms, the Single-Leader Cross-Region Topology offers the optimal balance between operational simplicity and sub-50ms read latencies.

2. Solving the Read-After-Write Consistency Hazard

The primary architectural challenge with asynchronous cross-region replicas is replication lag. If a user updates their profile in Singapore, the write routes to us-east-1. If the user immediately refreshes their browser and the subsequent read hits the local Singapore replica before the WAL payload arrives, the user sees old data, causing support tickets and perceived system bugs.

Pattern: Causal Consistency via Write LSN Tracking

To prevent stale reads without routing all traffic back to the primary database, modern API gateways enforce Read-After-Write Consistency by tracking Log Sequence Numbers (LSN) in user session state.

// Middleware Handler for Monotonic Read Consistency (Node.js & PostgreSQL)
import { Request, Response, NextFunction } from 'express';
import { primaryDb, localReplicaDb } from './database-nodes';

export async function consistencyAwareReadHandler(req: Request, res: Response, next: NextFunction) {
  const lastWriteLsn = req.cookies['x-user-last-lsn'];

  // If user recently executed a write, verify local replica has caught up
  if (lastWriteLsn) {
    const isReplicaUpToDate = await checkReplicaLsn(localReplicaDb, lastWriteLsn);

    if (!isReplicaUpToDate) {
      // Fallback to Primary DB if replication lag has not caught up
      req.dbNode = primaryDb;
      res.setHeader('X-Route-Reason', 'REPLICA_LAG_FALLBACK');
      return next();
    }
  }

  // Default path: Route read to low-latency local region replica
  req.dbNode = localReplicaDb;
  res.setHeader('X-Route-Reason', 'LOCAL_REPLICA_HIT');
  next();
}

async function checkReplicaLsn(replicaClient: any, targetLsn: string): Promise<boolean> {
  const result = await replicaClient.query(
    `SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn AS is_caught_up;`,
    [targetLsn]
  );
  return result.rows[0]?.is_caught_up ?? false;
}

When a write transaction completes on the primary database, return the new commit LSN (SELECT pg_current_wal_lsn()) to the client via HTTP response headers or auth tokens, updating their local session state.

3. Connection Pooling and Intelligent Query Routing

Cross-region database architectures fail if application workers create direct un-pooled TCP connections to remote databases across WAN links. Network jitter and TLS handshake overhead will negate the benefits of regional proximity.

Architectural Rules for Global Routing

Deploy Regional Connection Proxies: Position PgBouncer or AWS RDS Proxy in every edge region alongside application compute nodes to maintain warm connection pools to local replicas.
Isolate Transactional Workloads: Explicitly split read-heavy background analytics, search filters, and API GET queries away from state-modifying POST/PUT/DELETE handlers at the application ORM level.
Circuit Breaking for Replica Failures: Implement automated health checks that automatically reroute regional reads back to the primary cluster if replication lag exceeds an operational SLA threshold (e.g., > 2.0 seconds).

4. Architectural Summary and Trade-Offs

Read Latency vs. Data Freshness: Asynchronous read replicas deliver ultra-low read latency globally, but trade off instant consistency. Use LSN session cookies to bridge the gap cleanly for active writers.
Cost vs. Availability: Running cross-region replicas increases database infrastructure bills and egress costs (inter-region WAL streaming). Ensure regional replica instance sizes match write traffic to prevent replication worker starvation.

Originally published on ctousman.com. I write about SaaS architecture, databases, and scaling infrastructure. Follow for more deep dives.

Microservices Event Pipelines: Outbox Pattern, CDC, and Idempotent Event Sourcing

Usman Khan — Sat, 03 Oct 2026 11:30:35 +0000

In distributed microservice architectures, synchronizing database updates with downstream event streams is a notorious source of silent data drift. Naive dual-writing within application code fails when database commits succeed but message broker network writes time out. Enforcing transactional consistency requires decoupled, log-based event publishing.

1. The Fallacy of Application-Level Dual Writing

Consider a standard e-commerce or SaaS subscription workflow: an application endpoint updates the user record in PostgreSQL and immediately dispatches a UserCreated event to Kafka or RabbitMQ.

Failure Scenario A: The database transaction commits successfully, but the message broker goes offline or drops the connection. The event is lost forever, resulting in state divergence downstream.
Failure Scenario B: The message broker accepts the event payload, but the local database transaction rolls back due to a constraint violation. Downstream workers process an event for entity state that does not exist in the primary datastore.

Distributed two-phase commits (2PC) solve this in theory, but introduce severe availability bottlenecks, network latency, and operational coupling that degrade microservice performance at scale.

2. Enforcing Atomicity: The Transactional Outbox Pattern

The Transactional Outbox Pattern eliminates dual-write risk by persisting domain entity mutations and event notifications within the same local ACID transaction. Instead of sending messages over the network directly, the application writes events to a dedicated outbox table.

Database Schema for Outbox Records

-- Outbox Table DDL in PostgreSQL
CREATE TABLE transactional_outbox (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT CLOCK_TIMESTAMP(),
    processed_at TIMESTAMPTZ NULL
);

-- Index for high-throughput polling or CDC sequence scan
CREATE INDEX idx_outbox_unprocessed
  ON transactional_outbox (created_at)
  WHERE processed_at IS NULL;

Atomic Transaction Execution

// Example Node.js/TypeScript Transaction Handler
async function createTenantAccount(tenantData: TenantPayload): Promise<Tenant> {
  return await db.transaction(async (tx) => {
    // 1. Insert primary entity record into operational database
    const tenant = await tx.tenants.create({ data: tenantData });

    // 2. Insert event payload into outbox table in the SAME transaction
    await tx.outbox.create({
      data: {
        aggregateType: 'TENANT',
        aggregateId: tenant.id,
        eventType: 'TENANT_PROVISIONED',
        payload: {
          tenantId: tenant.id,
          plan: tenant.plan,
          ownerEmail: tenant.email,
          createdAt: tenant.createdAt,
        },
      },
    });

    return tenant; // Guaranteed atomicity: both succeed or both roll back
  });
}

3. Change Data Capture (CDC) with Debezium and Kafka

Once events reside in the outbox table, they must be streamed out to external message brokers without polling overhead. Polling (e.g. SELECT * FROM outbox WHERE processed = false) causes lock contention and table bloat under high throughput.

Log-Based Change Data Capture

Using Debezium attached to the PostgreSQL Write-Ahead Log (WAL) or MySQL Binary Log (binlog), event records are read directly from the transaction log without issuing SQL queries against active tables.

No table locks: Debezium tails the transaction log asynchronously and takes no read locks on production application tables. The one thing to monitor is replication slot lag: if the connector stalls, PostgreSQL retains WAL on disk until it catches up.
At-least-once delivery: Every write to the outbox table produces a change event streamed into a dedicated Kafka topic.
Automated cleanup: A background job or partition-maintenance routine drops historical outbox records once Kafka has acknowledged ingestion.

4. Idempotent Consumer Design in Event Processing

Because log-based CDC and distributed brokers provide at-least-once delivery, downstream consumers must assume duplicate events will arrive. Handlers must guarantee strict idempotency using a lock or deduplication store.

// Event Consumer Handler with Redis Lock & Deduplication
async function handleTenantProvisionedEvent(event: KafkaEvent): Promise<void> {
  const { eventId, payload } = event;
  const dedupKey = `event:processed:${eventId}`;

  // 1. Claim the event with a short-lived lock (5 min),
  //    so a crashed worker can't block the event forever
  const claimed = await redis.set(dedupKey, 'PROCESSING', 'EX', 300, 'NX');

  if (!claimed) {
    const state = await redis.get(dedupKey);
    if (state === 'COMPLETED') {
      console.log(`[DEDUPLICATION] Event ${eventId} already processed. Skipping.`);
      return;
    }
    // Still in-flight on another worker: fail so the broker redelivers later
    throw new Error(`Event ${eventId} is in-flight on another worker`);
  }

  try {
    // 2. Execute idempotent downstream business logic
    await provisioningService.setupIsolatedResources(payload.tenantId);

    // 3. Mark completed and keep the dedup record for 7 days
    await redis.set(dedupKey, 'COMPLETED', 'EX', 604800);
  } catch (error) {
    // Release the lock so retries can execute
    await redis.del(dedupKey);
    throw error; // Trigger broker retry or Dead Letter Queue (DLQ) routing
  }
}

5. Key Architectural Takeaways

Never dual-write across network boundaries: Commit domain state changes and event records inside a single local database transaction.
Leverage database log tailing: Use log-based CDC (Debezium / Kafka Connect) instead of SQL table polling to stream events without locking overhead.
Enforce consumer-side idempotency: Design consumers with atomic deduplication locks that survive crashes, so unavoidable redeliveries are handled safely.

Originally published at ctousman.com.

About the Author:
I'm Usman Khan, Fractional CTO & Systems Architect. I advise high-growth SaaS and e-commerce platforms on event-driven architecture, distributed systems, and backend performance.

Need an architectural review of your event pipelines? Book a 30-min strategy call

Asynchronous ERP Sync Engine: High-Throughput Webhook Ingestion & Durable Queue Orchestration

Usman Khan — Sat, 03 Oct 2026 11:25:30 +0000

Processing inbound e-commerce order webhooks directly against downstream ERP systems (such as Odoo or SAP) exposes core applications to third-party API latency, rate limits, and transient network failures.

The async-erp-sync-engine microservice decouples webhook ingestion from execution, accepting payloads in under 10ms while ensuring durable background processing via BullMQ and Redis.

1. System Architecture & Queue Pipeline

The microservice separates the HTTP intake layer from worker processing loops so client webhooks are never blocked:

 ERP / Shopify             ┌────────────────────────────────────┐
 Webhook POST ────────────▶│        Express HTTP Server         │
 (order-created)           │  POST /api/v1/webhooks/order-created│
                           │                                    │
                           │  1. Zod payload validation         │
                           │  2. enqueueErpJob() → BullMQ       │
                           │  3. HTTP 202 Accepted returned     │
                           └─────────────────┬──────────────────┘
                                             │ enqueue
                                             ▼
                           ┌────────────────────────────────────┐
                           │      Redis 7 (BullMQ backend)      │
                           │                                    │
                           │  ┌──────────────────────────────┐  │
                           │  │  erp-sync-queue (active)     │  │
                           │  └──────────────┬───────────────┘  │
                           │                 │                  │
                           │  ┌──────────────▼───────────────┐  │
                           │  │  failed set (dead-letter)    │  │
                           │  └──────────────────────────────┘  │
                           └─────────────────┬──────────────────┘
                                             │ dequeue
                                             ▼
                           ┌────────────────────────────────────┐
                           │         ERP Worker (BullMQ)        │
                           │                                    │
                           │  • Concurrency: 5 (configurable)   │
                           │  • Rate limit: 20 jobs / second    │
                           │  • Retry: 5 attempts, exponential  │
                           │    backoff (1s → 2s → 4s → 8s)     │
                           │  • Permanent failures → failed     │
                           │    state (UnrecoverableError)      │
                           └─────────────────┬──────────────────┘
                                             │ sync
                                             ▼
                           ┌────────────────────────────────────┐
                           │     Odoo / SAP ERP API (external)  │
                           └────────────────────────────────────┘

2. Key Resiliency Patterns & Design Decisions

To guarantee message delivery and protect external ERP APIs from thundering herds, the microservice implements five architectural safeguards:

Immediate 202 Accepted responses: Inbound webhooks are validated with Zod runtime schemas and enqueued into BullMQ within milliseconds, immediately returning HTTP 202 Accepted.
Durable BullMQ state management: Redis 7 with Lua scripts guarantees atomic job state transitions, persistent queue storage across node restarts, and worker-level rate limiting (capped at 20 jobs/sec).
5-attempt exponential backoff: Jobs hitting transient errors retry with increasing delays (1s → 2s → 4s → 8s), backed by an inner retry loop for brief network blips.
Dead-letter queue (failed set): Unrecoverable errors, or jobs that exhaust all 5 attempts, move to a dedicated failed set for inspection and manual replay.
Graceful shutdown: The worker listens for SIGTERM/SIGINT so in-flight jobs finish before the process exits, enabling zero-downtime rolling container deployments.

3. Queue Implementation Snippet (TypeScript)

import { Queue } from 'bullmq';
import { z } from 'zod';
import { redisConnection } from '../config/redis';

export const OrderWebhookSchema = z.object({
  order_id: z.string().min(1),
  customer_ref: z.string().min(1),
  amount: z.number().positive(),
  currency: z.string().length(3),
});

export type OrderWebhookPayload = z.infer<typeof OrderWebhookSchema>;

export const erpQueue = new Queue('erp-sync-queue', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 1000, // retries after 1s -> 2s -> 4s -> 8s
    },
    removeOnComplete: true,
  },
});

export async function enqueueErpJob(payload: OrderWebhookPayload) {
  const jobId = `order-${payload.order_id}-${Date.now()}`;
  return await erpQueue.add('sync-order', payload, { jobId });
}

Originally published at ctousman.com.

About the Author:
I'm Usman Khan, Fractional CTO & Systems Architect. I advise high-growth SaaS and e-commerce platforms on high-concurrency backend architecture, distributed queues, and system performance.

Need an architectural audit of your backend queue pipelines? Book a 30-min strategy call

Cloud Infrastructure Cost Optimization: FinOps Engineering, Graviton Migrations, and Auto-Scaling

Usman Khan — Fri, 02 Oct 2026 05:29:16 +0000

Cloud cost runaway is the silent margin killer in scaling B2B SaaS platforms. Unoptimized auto-scaling groups, over-provisioned database instances, and idle compute nodes rapidly erode gross margins.

Here is an actionable FinOps engineering framework for migrating to ARM64, tuning Kubernetes pod bin-packing, and enforcing compute ceiling controls.

1. The Financial Engineering Framework: Unit Economics Over Flat Bills

Engineers often evaluate cloud costs by looking at total monthly spend. High-agency CTOs instead manage infrastructure spend through Cost Per Active Tenant (CPAT) or Cost Per API Transaction.

If infrastructure spend grows in line with revenue, gross margins stay healthy. If cloud costs outpace transaction growth, architectural inefficiency is taking hold.

Identify idle waste: EC2 instances or EKS nodes running under 15% average CPU/memory utilization.
Right-size compute: Shift workloads from memory-optimized (r6i) to compute or balanced types (c7g / m7g) based on profiling data.
Eliminate unused storage: Unattached EBS volumes, S3 buckets without lifecycle transition rules, and orphaned snapshot chains.

2. Architectural Leverage: x86 to Graviton (ARM64) Migration

Moving backend services from Intel/AMD x86 to AWS Graviton (ARM64) typically yields around a 20% cost reduction and up to 40% better price-performance. For containerized services (Node.js, Go, Python, Java), multi-architecture Docker builds make deployment straightforward.

Multi-Arch Dockerfile for Graviton

# Build stage runs natively on the CI machine's architecture
FROM --platform=$BUILDPLATFORM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci                      # dev dependencies needed for the build
COPY . .
RUN npm run build

# Runtime stage is built per target architecture (amd64 / arm64)
FROM node:20-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
COPY package*.json ./
RUN npm ci --omit=dev           # native modules compiled for the target arch
COPY --from=builder /app/dist ./dist

EXPOSE 3000
USER node
CMD ["node", "dist/main.js"]

Build and push multi-platform images directly from your CI/CD pipeline:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t registry.internal.net/saas-api:v2.4.0 \
  --push .

3. Kubernetes Pod Bin-Packing and Karpenter Auto-Scaling

The traditional Kubernetes Cluster Autoscaler (CAS) reacts slowly and often leaves nodes underutilized because it works with static node groups. Modern platforms use Karpenter for consolidated, right-sized node provisioning on demand.

Optimizing Pod Resource Requests

If a pod requests 2 vCPUs and 4 GB RAM but only uses 200m CPU at peak, the scheduler still reserves the full 2 vCPUs, which forces unnecessary node scale-outs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-pipeline-worker
  namespace: production
spec:
  replicas: 12
  selector:
    matchLabels:
      app: billing-pipeline-worker
  template:
    metadata:
      labels:
        app: billing-pipeline-worker
    spec:
      containers:
        - name: worker
          image: registry.internal.net/billing-worker:v1.8.2
          resources:
            requests:
              cpu: "250m"       # Minimal baseline for tight bin-packing
              memory: "512Mi"
            limits:
              cpu: "1000m"      # Burst capacity during high throughput
              memory: "1024Mi"

4. Spot Instance Orchestration for Fault-Tolerant Queue Workers

Stateless background workers, asynchronous LLM processors, and batch ETL jobs should run almost entirely on Spot Instances, which offer up to 90% off On-Demand pricing.

Handling Spot Interruptions Gracefully

Diversify Spot pools: Never restrict a Spot group to a single instance type. Configure Karpenter or your ASGs to draw from at least 4–8 instance types (e.g. c6g.large, c7g.large, m6g.large, m7g.large).
React to interruption signals: Handle both the EC2 Rebalance Recommendation (an early warning) and the 2-minute Spot Interruption Notice via EventBridge, so you can drain active tasks before the node is terminated.
Make tasks idempotent: Every job in BullMQ, SQS, or RabbitMQ should be safe to interrupt mid-execution and retry on another worker without duplicating business side effects.

Originally published at ctousman.com.

About the author: I'm Usman Khan, a Fractional CTO & Systems Architect. I advise high-growth SaaS platforms on backend performance, distributed systems, and cloud cost efficiency. Need an architectural audit of your infrastructure? Book a 30-min strategy call.

DEV Community: Usman Khan

Kubernetes Pod Churn & Node Thrashing: Ephemeral Storage Exhaustion and Eviction Cascades

1. The Mechanics of OverlayFS and Ephemeral Storage Eviction

The Anatomy of a Node Eviction Cascade

2. Diagnosing Ephemeral Storage Bloat in Production

3. Production-Ready Remediation: Strict Resource Limits & In-Memory emptyDir

Step 1: Enforce Ephemeral Storage Requests and Limits

Step 2: Redirect High-Frequency Scratch Writes to In-Memory tmpfs

4. Tuning Kubelet Eviction Thresholds and Karpenter Node Consolidation

5. Executive Summary & Systems Rules

Distributed Idempotency Key Collisions: Clock Skew Mutex Leaks, Race Conditions, and Double-Execution Vulnerabilities

1. The Fallacy of Naive Redis Mutex Locks

Three Silent Failure Vectors

2. Architectural Pattern: Atomic Two-Phase State Engine

State Machine States

Phase 1: Atomic Acquisition via Redis Lua Scripting

Phase 2: Ownership-Checked Finalization

3. Production-Ready TypeScript Middleware Implementation

4. Secondary Guardrail: Database-Level Unique Constraints

5. Executive Summary & Systems Rules

PostgreSQL JSONB Write Amplification: TOAST Chunk Fragmentation and Unseen I/O Thrashing

1. The Mechanics of TOAST Storage and Partial JSONB Mutations

The Hidden Rewrite Penalty

2. Measuring the Amplification Factor: Code & Query Proof

3. Architectural Solutions: Eliminating JSONB Write Traps

Strategy A: Hybrid Column Decomposition (Extract Fast-Changing Attributes)

Strategy B: Key-Value EAV Partitioning for Unstructured Data

4. TOAST Storage Tuning for Read-Heavy JSONB Documents

5. Executive Summary & Systems Rules

Server-Sent Events (SSE) Backpressure Breaks: Memory Leaks and Socket Starvation in High-Throughput LLM Pipelines

1. The Mechanics of Unhandled Backpressure in Event Streams

How Slow Clients Trigger Node.js Heap Exhaustion

2. The Flaw in Naive SSE Controller Implementations

3. Production-Ready Backpressure-Aware SSE Pipeline

4. Reverse Proxy Tuning: Nginx and AWS ALB Buffering Traps

5. Executive Summary & Systems Rules

PostgreSQL MVCC Autovacuum Starvation: HOT Update Breakdown & Table Bloat under Heavy Write Loads

1. The Mechanics of MVCC and the HOT Update Failure Vector

How Heap Only Tuple (HOT) Updates Prevent Index Bloat

The Silent Breakdown Under Production Load

2. Diagnosing HOT Update Failure and Autovacuum Starvation

3. Architectural Remediation: Table Fillfactor & Autovacuum Tuning

Step 1: Setting Table-Level Fillfactors for HOT Space Allocation

Step 2: Aggressive Per-Table Autovacuum Tuning

4. Removing Redundant & Unused Indexes

5. Executive Summary & Systems Rules

Production AI & LLM Pipelines: Guardrails, Streaming Resilience, and Cost-Aware Fallbacks

1. The Vulnerability of Unhedged LLM Integration

2. Architectural Blueprint: Resilience and Fallback Layer

Dynamic Model Fallback Execution Pattern

3. Real-Time Streaming Resilience with Server-Sent Events (SSE)

Decoupling Connection Management

4. Enterprise Guardrails and Safety Controls

5. Key Engineering Summary

Advanced System Architecture: Designing Multi-Tenant Event-Driven Queues with Fair-Share Scheduling

1. The Noisy Neighbor Problem in Asynchronous Queues

2. Architectural Pattern: Per-Tenant Virtual Queues with Fair-Share Dispatching

The Deficit Round-Robin (DRR) Scheduler Engine

3. Enforcing Tenant Concurrency Caps with Distributed Semaphores

4. Tier-Based SLA Scheduling Strategies

5. Key Engineering Takeaways

Zero-Downtime Database Migrations: Shadow Tables, Dual-Writing, and Schema Expansion

1. The Anatomy of Database Locking Vulnerabilities

2. The Expand-Contract (Parallel Run) Migration Framework

Phase 1: Expand Schema (Non-Blocking DDL)

Phase 2: Enable Dual-Write Application Abstraction

3. Backfilling Historical Data and Parity Auditing

Batch Backfilling Pattern

Shadow Read Parity Auditing

4. Phase 3 & 4: Cutover Reads and Contract Legacy Schema

5. Key Architectural Rules for Zero-Downtime Rollouts

Advanced System Architecture: Designing Multi-Region Read Replicas and Global Data Consistency

1. The Multi-Region Topology Spectrum

2. Solving the Read-After-Write Consistency Hazard

Pattern: Causal Consistency via Write LSN Tracking

3. Connection Pooling and Intelligent Query Routing

Architectural Rules for Global Routing

4. Architectural Summary and Trade-Offs

Microservices Event Pipelines: Outbox Pattern, CDC, and Idempotent Event Sourcing

1. The Fallacy of Application-Level Dual Writing

3. Production-Ready Remediation: Strict Resource Limits & In-Memory `emptyDir`

Step 2: Redirect High-Frequency Scratch Writes to In-Memory `tmpfs`