DEV Community: u11d

Stop One EKS Node Group From Blocking Scale-Down in Another

Daniel Kraszewski — Wed, 19 Aug 2026 10:00:00 +0000

If idle EKS nodes keep running while other node groups scale up around them, Cluster Autoscaler's global scale-down cooldown is the likely cause, and --scale-down-delay-type-local=true is the fix. The flag makes Cluster Autoscaler track post-scale-up cooldowns per node group instead of cluster-wide, so a GPU pool adding capacity no longer resets the clock on an idle node in an unrelated memory pool. The default is false, which means every cluster running multiple ASGs has this behavior until someone turns it off.

We hit this with Dagster. Jobs used separate pools for general work, large CPU and memory tasks, and GPU tasks, each with min_size = 0, so we expected an unused pool to disappear on its own schedule. It didn't. A job starting in one pool delayed scale-down in every other pool, and with jobs starting and ending throughout the day, nodes that were no longer needed stayed alive far longer than they should have.

Why idle nodes stay running

Take two node groups, one for memory-heavy batch jobs and one for GPU jobs, both with min_size = 0. At noon the last memory-heavy job completes, its node goes empty, and Cluster Autoscaler starts the normal waiting period before deletion. Ten minutes later that node has waited long enough - and at almost the same moment a GPU job appears, its pod can't schedule, and Cluster Autoscaler grows the GPU Auto Scaling Group. You'd expect the memory node to disappear now: it belongs to another group, has no useful work, and has already passed its own waiting period. It stays.

Time	Memory group	GPU group	Result
`12:00`	Last job ends	Idle	Memory node becomes unneeded
`12:10`	Node has waited 10 minutes	New job triggers scale-up	Global cooldown starts
`12:11`	Node is eligible for deletion	New node is starting	Memory node cannot scale down
`12:20`	Node has been idle 20 minutes	Cooldown expires	Deletion can proceed

One extra ten-minute wait doesn't look serious. Bursty workloads turn it into a loop: a CPU pool grows, then a GPU pool grows, then a general pool grows, and each event postpones unrelated scale-down work. The problem is easy to miss because every node group looks correctly configured - minimum size is zero, jobs finish, pods disappear, and the autoscaler still reports nodes as unneeded. Yet the EC2 instances stay active, and nothing in your Terraform or your Helm values looks wrong. We didn't identify the shared cooldown from configuration alone; it took finding upstream issue #4872, which describes the same cross-group delay.

What Cluster Autoscaler actually tracks

Cluster Autoscaler treats an AWS Auto Scaling Group as a Kubernetes node group and decides both when to grow that group and when nodes can be safely removed from it. Scale-down has two separate waiting mechanisms that are easy to confuse: one governs each node's eligibility, the other protects the cluster after recent scaling actions.

Setting	Default	Meaning
`--scale-down-unneeded-time`	`10m`	How long a node must remain unneeded
`--scale-down-delay-after-add`	`10m`	How long scale-down waits after scale-up
`--scale-down-delay-after-delete`	`0s`	How long scale-down waits after deletion
`--scale-down-delay-after-failure`	`3m`	How long scale-down waits after failed scale-down
`--scale-down-delay-type-local`	`false`	Whether post-action delays are per node group

scale-down-unneeded-time answers "has this node been unnecessary for long enough?" - Cluster Autoscaler checks resource requests, movable pods, scheduling rules, PodDisruptionBudgets, and other safety conditions to decide. The scale-down-delay-after-* settings answer a completely different question: "is scale-down allowed at all right now, given recent autoscaler activity?" A node can pass its unneeded timer while a cooldown still blocks deletion, and that's exactly what happens here. Another node group scaling up does not erase the idle node's unneeded history - it adds a separate gate on top of it. Before 2024 that gate was always global, and with the default false it still is. Support for per-group cooldowns arrived through kubernetes/autoscaler#5729, which lets one process remember scaling activity per node group instead of treating every action as cluster-wide.

Enable per-node-group cooldowns

Keep one Deployment managing all node groups and set the flag. If you install Cluster Autoscaler with Helm, the change belongs in your values file:

cloudProvider: aws

autoDiscovery:
  clusterName: production-eks

awsRegion: eu-west-1

extraArgs:
  scale-down-delay-type-local: "true"
  scale-down-delay-after-add: 10m
  scale-down-unneeded-time: 10m

A GPU group scale-up now starts a cooldown for the GPU group only, and an eligible node in the memory group can be removed while it runs. Keep the delay values at their defaults unless you have measurements that justify changing them - new nodes need time to register, start DaemonSets, advertise GPUs, and receive pending pods, and local scope fixes the isolation problem without removing that protection. Setting scale-down-delay-after-add: 0 is not an equivalent shortcut. It does avoid cross-group delay, but it does so by removing the cooldown everywhere, including from the group that just added nodes, which is the one group that actually needs it.

Check your binary supports the flag

The feature merged into the main branch in January 2024 and was backported to the 1.29 release branch that March through #6484. The original 1.29.0 release does not include it, so do not assume every binary labeled 1.29 supports the flag. Check the actual image and the rendered command:

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{range .spec.template.spec.containers[0].command[*]}{.}{"\n"}{end}'

If the logs contain unknown flag: --scale-down-delay-type-local, the image lacks the feature - upgrade the binary rather than quietly dropping the setting. That upgrade has a constraint of its own: Cluster Autoscaler simulates Kubernetes scheduling, so its minor version must match your cluster's Kubernetes minor version, and cross-version combinations are not supported. AWS repeats this in its EKS guidance. One more trap sits between you and the right binary: Helm chart version and Cluster Autoscaler application version are different numbers. Inspect the chart's appVersion or pin image.tag explicitly, so a chart that looks current doesn't deploy a binary that isn't.

Local cooldowns only isolate delays within a single Cluster Autoscaler process, which makes process ownership the next thing to confirm.

Make sure one autoscaler owns the ASGs

The guarantee is worthless if a second, unmanaged autoscaler also watches some of your ASGs, or if one process never discovers all of them. On AWS, auto-discovery selects ASGs by tags, and the standard Helm values look for two keys:

autoDiscovery:
  clusterName: production-eks
  tags:
    - k8s.io/cluster-autoscaler/enabled
    - k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}

Every ASG the autoscaler should own needs both keys attached before discovery picks it up. Tag values are ignored unless you include them in the discovery expression, but values like true and owned make ownership readable and give you something to write IAM conditions against.

Use separate node groups when capacity or scheduling actually differs, and give each one real labels, real taints, and scale limits that allow it to reach zero. With terraform-aws-modules/eks/aws (~> 21.0), the node group definition is the part that matters:

eks_managed_node_groups = {
  batch = {
    instance_types = ["m7i.2xlarge"]

    min_size     = 0   # allows the group to empty completely
    max_size     = 20
    desired_size = 0

    labels = {
      "node-role" = "batch"
    }

    taints = {
      dedicated = {
        key    = "dedicated"
        value  = "batch"
        effect = "NO_SCHEDULE"
      }
    }
  }
}

The ASG this creates is the resource your discovery tags must reach, and managed node groups generate ASG names you don't control. Module inputs and outputs also shift between versions, so verify the tags actually landed. Tags on launch templates, security groups, or EC2 instances do not count - auto-discovery reads ASG tags and nothing else:

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].{Name:AutoScalingGroupName,Tags:Tags[?starts_with(Key, `k8s.io/cluster-autoscaler/`)].{Key:Key,Value:Value}}' \
  --output table

If your module doesn't attach the discovery tags, read the generated ASG name and attach them yourself:

locals {
  batch_asg_name = module.eks.eks_managed_node_groups["batch"].node_group_resources[0].autoscaling_groups[0].name

  autoscaler_tags = {
    "k8s.io/cluster-autoscaler/enabled"        = "true"
    "k8s.io/cluster-autoscaler/production-eks" = "owned"
  }
}

resource "aws_autoscaling_group_tag" "autoscaler" {
  for_each = local.autoscaler_tags

  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = each.key
    value               = each.value
    propagate_at_launch = false  # this tag is for the ASG itself, not the instances it launches
  }
}

Do not combine static --nodes arguments with auto-discovery. Mixing the two makes ownership hard to reason about and will eventually have you debugging the state of a node group the process was never managing. The AWS provider documentation recommends auto-discovery and warns against the combination. Discovery controls what the process can see; IAM controls what it can change. Use IRSA or EKS Pod Identity and constrain SetDesiredCapacity and TerminateInstanceInAutoScalingGroup to ASGs carrying the cluster tags - AWS publishes a tag-conditioned IAM example.

Scale node groups back up from zero

Scale-to-zero introduces a different failure mode with the same symptom: a pool that sits at the wrong size and won't move. When a node group is empty, Kubernetes has no live Node object advertising that group's labels, taints, or custom resources, and Cluster Autoscaler still has to decide whether a pending pod would fit there. This bites hardest on pools selected by labels, protected by taints, or backed by GPUs.

Cluster Autoscaler infers CPU, memory, and GPU capacity from AWS configuration, and for EKS managed node groups it can also read labels and taints through eks:DescribeNodegroup, including at zero nodes. If that covers everything your pod's scheduling rules reference, you may not need explicit node-template tags at all. When it doesn't, put the metadata on the ASG, where it describes the simulated node used for scale-from-zero decisions:

resource "aws_autoscaling_group_tag" "batch_label" {
  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = "k8s.io/cluster-autoscaler/node-template/label/node-role"
    value               = "batch"
    propagate_at_launch = false
  }
}

resource "aws_autoscaling_group_tag" "batch_taint" {
  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = "k8s.io/cluster-autoscaler/node-template/taint/dedicated"
    value               = "batch:NoSchedule"  # value and effect packed into one string
    propagate_at_launch = false
  }
}

Node-template tags do not configure real nodes. They describe a hypothetical node for the simulator, so the same labels and taints must also exist on the managed node group itself - the labels and taints blocks shown earlier - and the pending pod must request that same contract:

spec:
  nodeSelector:
    node-role: batch
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: job
      image: example.com/data-job:2026-07-28
      resources:
        requests:
          cpu: "2"
          memory: 6Gi

Break that three-way match and you get one of two outcomes, both of which look like a stuck queue. Either the pool never scales up, because Cluster Autoscaler can't tell the pending pod would fit, or it scales up a node that still can't schedule the pod, because the real labels don't match what the pod asked for. Whichever source the metadata comes from, simulated and real must agree.

Diagnose and verify

Start by confirming the nodes are genuinely idle from Cluster Autoscaler's perspective. Low observed CPU is not enough - Cluster Autoscaler reasons about pod resource requests and whether pods can move elsewhere, not utilization. Run it at verbosity 4 while investigating and follow the Deployment logs:

kubectl -n kube-system logs deployment/cluster-autoscaler --follow \
  | grep -E 'unneeded|scale down|scaled up recently|removing'

You're looking for evidence that Cluster Autoscaler already considers the node removable, followed by a cooldown message instead of a deletion. Exact wording varies by version, but the pattern holds: one line marks a node unneeded or eligible, a later line skips scale-down because something scaled up recently. Compare those timestamps against ASG scaling activity in the other group - a scale-up in group B shortly before an eligible node in group A fails to disappear is the signature:

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name <group-b-asg-name> \
  --max-items 20

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Min:MinSize,Max:MaxSize}' \
  --output table

Rule out the ordinary blockers before blaming the cooldown. A node legitimately stays when it hosts local storage, a restrictive PodDisruptionBudget, an unreplicated pod, or safe-to-evict: "false". Scheduling constraints do the same: if the pods on that node can't fit anywhere else because of selectors, affinity, host ports, or resource requests, Cluster Autoscaler is keeping the node for good reason.

Once the flag is deployed, confirm that exactly one Deployment manages the intended ASGs and that the rendered command carries the setting:

kubectl -n kube-system get deployments | grep cluster-autoscaler

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{range .spec.template.spec.containers[0].command[*]}{.}{"\n"}{end}' \
  | grep scale-down-delay

Then run a controlled two-group test. Let a node in group A become unneeded, wait for the logs to show it as a candidate, and create a pod only group B can run - here a batch pool labeled node-role=batch and tainted dedicated=batch:NoSchedule:

apiVersion: v1
kind: Pod
metadata:
  name: force-batch-scale-up
spec:
  restartPolicy: Never
  nodeSelector:
    node-role: batch
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: pause
      image: public.ecr.aws/eks-distro/kubernetes/pause:3.10
      resources:
        requests:
          cpu: "2"
          memory: 6Gi

Group B's scale-up should no longer postpone the eligible node in group A. Deletion still depends on scan interval, drain time, AWS API latency, and the node staying unneeded throughout, so give it a couple of scan cycles before declaring the test failed.

Legacy: one autoscaler per ASG

Before per-node-group cooldowns existed, sharding was the practical workaround: run one Cluster Autoscaler per ASG, each process watching a disjoint discovery tag. Each shard then kept its own cooldown state, so a GPU scale-up couldn't delay the process responsible for memory nodes.

It's heavier than it looks. Every shard needs its own ASG ownership boundary, leader election, service account identity, and IAM scope, and the operational surface grows linearly with node group count. Worse, shards watch the same unschedulable pods without coordinating, so two autoscalers can each decide their ASG can help and add duplicate capacity for a single pending pod. AWS recommends sharding only as a last resort, and the only case that still qualifies is a cluster where the binary matching your Kubernetes minor version predates the feature and the upgrade is blocked.

What's next

Run one Cluster Autoscaler with --scale-down-delay-type-local enabled, keep the default safety delays, and let them apply to the group that earned them. Verify three things afterward: one process owns every ASG you care about, the deployed binary actually accepts the flag, and any group that reaches zero has matching metadata across ASG node-template tags, real node configuration, and pod scheduling rules.

Cluster Autoscaler's behavior shifts between releases, and flags have been added, renamed, and defaulted differently over time. Check the AWS provider documentation against the source tag matching your image before you ship, then test it with real workloads rather than a single pause pod.

Postgres LISTEN/NOTIFY in Production: Safe Reconnects & SSE

uninterrupted — Fri, 14 Aug 2026 07:24:09 +0000

If your app needs to push live updates to connected clients — a job progress bar, a balance that changes when a background worker finishes, a "this record was deleted elsewhere" toast — the reflex is to reach for Redis Pub/Sub or a message broker. But if you already run PostgreSQL, it ships with a pub/sub primitive built in: LISTEN / NOTIFY. No extra infrastructure, no extra failure domain.

What is Postgres LISTEN/NOTIFY? It's a built-in pub/sub mechanism: one session runs pg_notify(channel, payload) inside a transaction, and any other session that has issued LISTEN <channel> on a persistent connection receives the payload once that transaction commits. There's no persistence and no delivery guarantee — it's a live signal, not a queue — so it fits best as a "something changed, go re-check" nudge rather than a system of record.

This post walks through the implementation we actually run in production: a single shared LISTEN connection multiplexing many logical channels, wired into per-client Server-Sent Events (SSE) streams. Along the way: the reconnect logic and the race conditions that come with it.

How Postgres LISTEN/NOTIFY works

Postgres gives you two SQL-level building blocks:

SELECT pg_notify('job_status_123', '{"status":"done"}');

and, on a persistent connection:

LISTEN job_status_123;

Any session that has issued LISTEN on a channel receives the payload once the transaction that ran pg_notify commits, not the instant the call executes. Multiple identical notifications on the same channel/payload fired within one transaction get collapsed into one delivery. Several constraints shape everything that follows (see the Postgres NOTIFY reference for the authoritative spec):

Payload is capped at 8000 bytes. Fine for status pings, not for shipping large blobs.
Channel names are Postgres identifiers, truncated at 63 characters. Patterns like job_status_<uuid> need to fit under that limit or get silently truncated and collide.
No persistence, no delivery guarantee. If nobody is listening when pg_notify fires, the message is gone. This is a live signal, not a queue: any client relying on it needs a way to independently reconcile state, e.g. re-fetch from the DB.
The notification queue itself is bounded (8GB by default, per the Postgres async messaging docs). If a listening session sits idle inside an open transaction, it stops draining the queue for everyone, and pg_notify starts erroring once the queue fills. Keep LISTEN sessions out of long-running transactions.

Because LISTEN requires a session-scoped, long-lived connection, and most apps talk to Postgres through a connection pool, you can't just LISTEN from your normal query pool. Pooled connections get recycled and your subscription vanishes with them. You need one connection dedicated to listening, held open for the life of the process.

This also rules out transaction-mode poolers like pgbouncer or RDS Proxy for the LISTEN connection specifically. They can hand your session's underlying socket to a different client between transactions, which silently kills the subscription. The dedicated LISTEN connection needs to bypass the pooler and go straight to Postgres. If you run multiple app instances, each one holds its own LISTEN connection and each receives every NOTIFY independently, which is usually what you want: every instance can push updates to its own locally-connected clients.

One connection, many channels

Rather than open a Postgres connection per subscriber, a single physical LISTEN connection can multiplex an arbitrary number of logical channels. In-process, that's just a map from channel name to a set of callbacks:

class NotificationHub {
  private client: ReturnType<typeof postgres> | null = null;
  private readonly listeners = new Map<Channel, Set<MessageCallback>>();
  private readonly unlistenFns = new Map<Channel, () => Promise<void>>();
  ...

The connection itself is configured to never idle out and to identify itself for observability:

this.client = postgres({
  ...getAppConnectionConfig(),
  max: 1,
  idle_timeout: 0,
  connect_timeout: 10,
  connection: { application_name: "app-listen" },
  onclose: () => {
    /* ... */
  },
});

This pool is deliberately capped at one connection: it exists purely to host a dedicated LISTEN session, not to serve queries. Mixing it into a general-purpose pool would mean losing subscriptions every time the pool recycles a connection.

Handling reconnects: capped exponential backoff

A long-lived TCP connection to a database will eventually drop: network blips, database failover, load balancer idle timeouts. The interesting design question isn't "will it disconnect" but "what happens to every registered channel when it does."

onclose: () => {
  if (!this.closed) {
    dbLogger.warn("LISTEN connection closed unexpectedly");
    this.setState("disconnected");
    this.scheduleReconnect();
  }
},

Reconnect uses a capped exponential backoff so a flapping database doesn't get hammered with reconnect attempts:

const RECONNECT_DELAYS = [1_000, 2_000, 4_000, 8_000, 15_000] as const;

private scheduleReconnect(): void {
  if (this.closed || this.reconnectTimer) return;
  const delayIndex = Math.min(this.reconnectAttempt, RECONNECT_DELAYS.length - 1);
  const delay = RECONNECT_DELAYS[delayIndex];
  this.reconnectAttempt++;
  this.reconnectTimer = setTimeout(() => {
    this.reconnectTimer = null;
    void this.connect();
  }, delay);
}

reconnectAttempt resets to 0 on a successful connect(), so backoff only escalates across consecutive failures. A single flaky reconnect doesn't leave future reconnects artificially delayed.

On reconnect, every channel that had at least one active subscriber gets re-LISTENed from scratch:

const channels = Array.from(this.listeners.keys());
const failedChannels: Channel[] = [];
for (const channel of channels) {
  try {
    await this.listenToChannel(channel);
  } catch (err) {
    failedChannels.push(channel);
  }
}

if (failedChannels.length > 0) {
  this.setState("disconnected");
  this.scheduleReconnect();
  return;
}

Note the all-or-nothing treatment: if even one channel fails to re-LISTEN, the whole reconnect is considered failed and retried. A hub that's "half connected," silently missing notifications on some channels, is worse than one that's visibly disconnected and retrying. At least then the failure isn't invisible to callers.

The unsubscribe race, solved by ordering, not locking

Multiple parts of an app can subscribe to the same channel concurrently (two browser tabs watching the same job, for instance). The tricky moment is when the last subscriber for a channel unsubscribes at the same time a new subscriber for that same channel is arriving:

unsubscribe: async () => {
  for (const channel of channels) {
    const callbacks = this.listeners.get(channel);
    if (!callbacks) continue;
    callbacks.delete(onMessage);

    if (callbacks.size === 0) {
      this.listeners.delete(channel);
      const unlisten = this.unlistenFns.get(channel);
      // Delete before awaiting so a concurrent subscribe() for this
      // channel calls listenToChannel() again. PostgreSQL silently
      // ignores a duplicate LISTEN on the same connection, so the race
      // is harmless — the hub will have an active LISTEN either way.
      this.unlistenFns.delete(channel);
      if (unlisten) {
        try {
          await this.timeoutUnlisten(unlisten);
        } catch { /* timed out or connection gone — safe to ignore */ }
      }
    }
  }
},

The naive version would await UNLISTEN first, then clean up bookkeeping. But between "decide to unlisten" and "UNLISTEN finishes," a new subscriber could arrive, see the channel already has a bookkeeping entry, and skip re-LISTENing. That leaves the hub with no active LISTEN for a channel it thinks it's subscribed to.

By deleting the map entry before awaiting UNLISTEN, a concurrent subscribe() call always sees "no entry" and issues its own LISTEN unconditionally. Postgres treats a duplicate LISTEN on the same session as a no-op, so the worst case is two harmless LISTEN calls in flight instead of a silently dropped subscription. When you can't easily lock across an async boundary, look for an operation idempotent enough to make the race safe by construction.

Callers should never worry about connection state

The public subscribe() API registers callbacks unconditionally, regardless of whether the underlying Postgres connection is currently up. It only rejects if an opportunistic LISTEN call fails outright while attempting an already-connected session, and that case is recovered by the reconnect logic on its next attempt:

async subscribe(channels: Channel[], onMessage: MessageCallback): Promise<Subscription> {
  // Register callbacks regardless of connection state
  for (const channel of channels) {
    let callbacks = this.listeners.get(channel);
    if (!callbacks) {
      callbacks = new Set();
      this.listeners.set(channel, callbacks);
    }
    callbacks.add(onMessage);
  }

  if (this.state === "connected" && this.client) {
    for (const channel of channels) {
      await this.listenToChannel(channel); // no-op if already LISTENing
    }
  }

  if (this.state === "disconnected" && !this.connecting && !this.reconnectTimer) {
    void this.connect();
  }

  return { unsubscribe: async () => { /* ... */ } };
}

Callbacks are registered in memory first; the actual LISTEN happens opportunistically if connected, or gets picked up automatically on the next successful reconnect. Every consumer of the hub stays decoupled from the connection's health. They just subscribe, and messages show up when the pipe is open.

Wiring it into Server-Sent Events

The hub by itself is transport-agnostic; a thin SSE layer turns it into per-client streams with keep-alives and cleanup on disconnect:

const stream = new ReadableStream({
  start(controller) {
    let subscription: Subscription | null = null;

    const onAbort = () => {
      void cleanup();
    };
    options.request.signal.addEventListener("abort", onAbort);

    keepAliveInterval = setInterval(
      () => safeEnqueue(`: ping\n\n`),
      keepAliveMs,
    );

    const init = async () => {
      await options.onStart(safeEnqueue, () => {
        void cleanup();
      });
      if (!options.subscribe || isClosed) return;

      const { channels, onMessage } = options.subscribe;

      unsubscribeState = onHubStateChange((state) => {
        safeEnqueue(
          `data: ${JSON.stringify({ type: "info", liveUpdates: state === "connected" })}\n\n`,
        );
      });

      subscription = await subscribe(channels, (ch, message) => {
        if (!isClosed)
          onMessage(ch, message, safeEnqueue, () => {
            void cleanup();
          });
      });

      if (options.onAfterSubscribe) {
        await options.onAfterSubscribe(safeEnqueue, () => {
          void cleanup();
        });
      }
    };

    init().catch(() => {
      void cleanup();
    });
  },
  cancel() {
    return cleanupFn?.();
  },
});

Two details worth calling out.

The hub's connection state is surfaced to the client. When onHubStateChange reports "disconnected", an {"type":"info","liveUpdates":false} frame goes out over SSE so the UI can show "reconnecting..." instead of silently going stale. Since NOTIFY gives no delivery guarantee, telling the client when it might be missing updates is almost as important as delivering the updates themselves.

There's also a read-then-subscribe race window. onStart typically does an initial DB read to send current state, and only afterwards does the code call subscribe() to start receiving live updates. If a NOTIFY fires in the gap between that read and the LISTEN actually being established, it's lost for good: the classic "check-then-watch" race in any pub/sub system. The onAfterSubscribe hook closes that window. After the subscription is confirmed active, it re-checks state one more time, so any update that landed in the gap gets picked up on the second read instead of being silently missed. The client-side handler needs to tolerate receiving the same update twice (once from the catch-up read, once from a live NOTIFY that arrives moments later). Treat updates as idempotent, not as a strictly-once stream.

When this pattern fits

Postgres LISTEN/NOTIFY works well when:

Update volume is modest (dashboards, job status, per-user notifications), not a high-throughput event bus.
Clients treat NOTIFY as a nudge to re-sync, not as the sole source of truth. Always have a path to reconstruct state via a normal query if a notification is missed.
You want one fewer piece of infrastructure to run and monitor.

It's the wrong tool when you need guaranteed delivery, durable queues, replay, or very high fan-out. That's what a real message broker is for. But for "tell connected clients something changed," Postgres has had a message bus built in the whole time.

Keycloak at 100% CPU After Deploy: Root Cause and Fix

Bartek Gałęzowski — Fri, 14 Aug 2026 07:08:30 +0000

The Symptom: Recognizing This CPU Spike

You deploy a containerized Keycloak instance — even the official quay.io/keycloak/keycloak image, unmodified — to a managed container platform, and within seconds CPU utilization climbs to 100% and stays there — regardless of traffic. No requests are being served, no realm imports are running, and nothing in the application logs looks wrong. Health checks pass. HTTP endpoints respond normally.

This pattern shows up most often on:

DigitalOcean App Platform
Google Cloud Run
AWS ECS Fargate
Heroku
Self-managed or managed Kubernetes clusters without a multicast-aware CNI

Root Cause: Infinispan, JGroups, and the Multicast Problem

The Default: Distributed Caching via Infinispan

By default, Keycloak ships with a distributed cache mode built on Infinispan, an embedded data grid used to replicate session state, authentication sessions, and other transient data across cluster nodes. This is the ispn cache provider, and it's the default unless explicitly overridden — either at build time (kc.sh build --cache=<mode>) or at runtime (KC_CACHE).

Why JGroups Can't Discover Peers

Infinispan's clustering layer relies on JGroups, a toolkit for reliable group communication. JGroups needs a discovery protocol to find other cluster members, and out of the box, Keycloak's default JGroups stack uses UDP-based discovery (PING/MPING), which depends on IP multicast to broadcast "who else is out there?" messages across the network.

Managed container platforms don't expose multicast-capable networking. Their networking layers are software-defined, ephemeral, and built around dynamic scheduling of containers across shared infrastructure — a model that's fundamentally incompatible with IP multicast, which assumes a flat, broadcast-capable network segment. This holds almost universally: PaaS offerings, serverless container runtimes, and even many managed Kubernetes configurations without a multicast-aware CNI all impose the same restriction.

The Result: A Silent, Load-Independent CPU Burn

JGroups doesn't fail gracefully when discovery is unreachable. Instead, it enters an aggressive retry loop — broadcasting discovery packets, waiting for a timeout, retrying, re-forming view negotiations, and repeating indefinitely. Each cycle consumes CPU on packet construction, timeout scheduling, and thread contention, multiplied across the JGroups protocol stack (transport, discovery, failure detection, and flow control protocols all layered on top of one another).

None of this is visible from an application standpoint. A background thread pool is perpetually spinning, trying to solve a problem — finding cluster peers via multicast — that's structurally impossible in this environment. The result is sustained, load-independent CPU burn with no connection to request volume.

Setting KC_CACHE=local eliminates the problem — not by fixing discovery, but by disabling clustering altogether. In local mode, Keycloak uses in-memory, non-replicated caches with no Infinispan clustering, no JGroups stack, and therefore no discovery loop to fail.

Why It Happens After Redeploys, Not on First Boot

First Boot: A Clean, Uncontested Singleton

On the very first deploy, exactly one container exists. When JGroups' discovery protocol fires, it queries whatever peer list mechanism is configured — raw UDP multicast, or more commonly on managed platforms, a DNS-based or static-host discovery protocol that resolves the platform's internal service name to the currently running container IPs. Since this is the first and only container, that resolution returns exactly one address: itself. There's nothing to negotiate and no peer to merge with. JGroups times out once on the discovery request, finds no one else, and installs itself as a singleton cluster view — a cheap, one-shot operation that doesn't manifest as sustained CPU load.

Redeploy or Restart: A Two-Node Race That Never Resolves

A redeploy or restart on most managed platforms is a rolling operation, not an instant swap. The platform starts the new container before fully tearing down the old one, to avoid downtime. For a short window, both the outgoing and incoming containers are alive simultaneously, and depending on how peer discovery is configured, both may be resolvable under the same internal service identity.

This is where things diverge from the first-boot case: JGroups on the new node now believes there is another cluster member to reach. It attempts to establish a channel, negotiate a merged view, and start heartbeat-based failure detection with that peer — but the platform's networking still doesn't permit the actual cache-cluster transport traffic between containers (multicast is blocked, and often the cache port isn't even routed between sibling containers in the first place). The handshake can never complete: the peer is addressable enough to be discovered, but not reachable enough to actually communicate.

The result is a protocol deadlock rather than a clean failure. JGroups cycles through:

periodic heartbeat attempts to the discovered peer
failure-suspicion timers when heartbeats go unanswered
merge-coordinator election attempts to decide which node's view should win
state-transfer request/response handshakes that time out without completing

Each sub-protocol retries independently and indefinitely, because there's no terminal "peer confirmed dead, drop it forever" signal — the old container is still technically registered as existing from the discovery layer's point of view, even though it's unreachable at the transport layer. This composite retry storm is considerably more CPU-intensive than the one-shot singleton bootstrap on first boot, which is why the spike reliably appears after a redeploy or restart, but not on the initial one.

How to Fix It: Step-by-Step

Confirm you're hitting this issue. Check CPU metrics with no active traffic — a sustained 100% with zero requests is the signature. Confirm KC_CACHE is unset or set to ispn (the default).
Decide whether you actually need clustering. If you're running a single Keycloak instance, which covers most small and mid-sized deployments, you don't need Infinispan's distributed cache at all.
Single instance → set KC_CACHE=local. Add it as a runtime environment variable, or bake it in at build time with -cache=local. Redeploy and confirm CPU returns to baseline.
Multiple instances / HA → switch to JDBC_PING. Build with -cache-stack=jdbc-ping (or supply a custom cache-ispn.xml), pointing at your existing database. This keeps session replication working without depending on multicast.
Verify after a redeploy, not just first boot. Because the issue specifically reproduces on rolling restarts, don't declare victory after the first boot alone — trigger a redeploy and confirm CPU stays flat afterward too.

The Trade-Offs of KC_CACHE=local

KC_CACHE=local is a valid and common fix, but it comes with real trade-offs:

No safe horizontal scaling. Run more than one replica behind a load balancer, and users can be routed to a different instance mid-session — silently logged out, losing SSO state, or hitting inconsistent authorization decisions.
Restarts are lossy. A container restart — which managed platforms perform routinely during deploys, scaling events, or health-check failures — wipes all in-memory session state for that instance.

For a single-instance deployment, none of this is a compromise — there's no cluster to form in the first place, so disabling clustering is simply correct.

Need High Availability? Use JDBC_PING Instead

If horizontal scaling or high availability is a requirement, the architecturally correct fix isn't to disable clustering — it's to replace the discovery mechanism with one that doesn't depend on multicast.

Keycloak, via Infinispan/JGroups, supports a JDBC_PING discovery protocol that uses your existing relational database as a shared coordination point: each node writes its cluster address to a database table, and peers discover each other by querying that table instead of broadcasting UDP packets.

This is configured via a custom JGroups stack — --cache-stack=jdbc-ping at build time, or a custom cache-ispn.xml descriptor — and it works reliably on any managed platform, regardless of vendor, because it never depends on network-layer broadcast at all. The cost is a small, constant amount of additional load on your database (heartbeat writes/reads at a configurable interval) in exchange for genuine session replication across nodes.

KC_CACHE Settings: Quick Reference

Scenario	Recommended Setting	Why
Single Keycloak instance, no plans to scale horizontally	`KC_CACHE=local`	No cluster to form — disables Infinispan/JGroups entirely
Multiple instances, need session replication / HA	`KC_CACHE=ispn` with `--cache-stack=jdbc-ping`	Keeps clustering, swaps multicast discovery for database-backed discovery
Multiple instances, occasional failover session loss acceptable (rare)	`KC_CACHE=ispn` with default stack	Not recommended on managed platforms — reproduces the CPU issue

Faster Geospatial Animations with PMTiles: Multi-Layer MVT Time Series in OpenLayers

uninterrupted — Wed, 08 Jul 2026 06:00:00 +0000

Most people try to animate time series by stacking separate tile sources or flipping CSS styles mid-frame. That causes flicker and a surprising amount of wasted CPU/GPU work. The core takeaway is simple: bake your timesteps into a single vector-tile archive (PMTiles) with per-timestep named layers, load it once, and drive animation by re-styling already-decoded tiles.

I tested this with OpenLayers and noticed the difference right away: playback became smoother and far less CPU-bound once tiles were pre-baked and cached. The hard part is server-side — how you generate, package, and serve tiles matters more than which client library you pick.

When you animate a map by switching between separate tile sources, each frame causes one of two expensive operations: a network fetch of new tiles, or client-side re-decoding and re-styling. Baking timesteps into one archive avoids that by turning each frame into a cheap style re-evaluation on already-decoded features.

The architecture: single archive, dynamic layers

MVT (Mapbox Vector Tile) is a compact protobuf-based vector tile format that encodes geometry and attributes per z/x/y tile. PMTiles packages MVT tiles into a single file for efficient distribution and partial HTTP range retrieval.

The key workflow:

Server side — each timestep is a separate GeoJSON file. tippecanoe joins them into one PMTiles archive, giving each timestep its own named MVT layer (timestep_00, timestep_01, …).
Client side — one VectorTileSource loads the archive once. One VectorTileLayer uses a style function that reads a mutable currentLayerName variable. Advancing a frame means: update the variable, call layer.changed(). OpenLayers re-runs the style function on the already-decoded tile features and repaints — no network request, no re-decode.

Complete working example

!animation.gif

This section contains everything needed to reproduce the demo from scratch: 11 per-timestep GeoJSON files, a tile generation script, and the client HTML.

Step 1 — Per-timestep GeoJSON files

Create 11 files, one per timestep. Each contains a single large polygon with a value property. The temporal identity comes from the file name (and the MVT layer name tippecanoe will assign), not from a property on the feature.

for i in $(seq 0 10); do
  pad=$(printf "%02d" $i)
  value=$((i * 10))
  cat > "square_t${pad}.geojson" << EOF
{
  "type": "FeatureCollection",
  "features": [{
    "type": "Feature",
    "geometry": {
      "type": "Polygon",
      "coordinates": [[[-60,-50],[60,-50],[60,50],[-60,50],[-60,-50]]]
    },
    "properties": { "value": ${value} }
  }]
}
EOF
done

This produces square_t00.geojson (value=0) through square_t10.geojson (value=100).

Step 2 — Generate PMTiles with named layers (`generate-tiles.sh`)

#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

tippecanoe \
  --named-layer=timestep_00:"$SCRIPT_DIR/square_t00.geojson" \
  --named-layer=timestep_01:"$SCRIPT_DIR/square_t01.geojson" \
  --named-layer=timestep_02:"$SCRIPT_DIR/square_t02.geojson" \
  --named-layer=timestep_03:"$SCRIPT_DIR/square_t03.geojson" \
  --named-layer=timestep_04:"$SCRIPT_DIR/square_t04.geojson" \
  --named-layer=timestep_05:"$SCRIPT_DIR/square_t05.geojson" \
  --named-layer=timestep_06:"$SCRIPT_DIR/square_t06.geojson" \
  --named-layer=timestep_07:"$SCRIPT_DIR/square_t07.geojson" \
  --named-layer=timestep_08:"$SCRIPT_DIR/square_t08.geojson" \
  --named-layer=timestep_09:"$SCRIPT_DIR/square_t09.geojson" \
  --named-layer=timestep_10:"$SCRIPT_DIR/square_t10.geojson" \
  -o "$SCRIPT_DIR/square.pmtiles" \
  -z 8 -Z 0 \
  --no-simplification \
  --no-tile-size-limit \
  --no-feature-limit \
  -f

Run with: bash generate-tiles.sh

Step 3 — Start the server

npx serve

Step 4 — Client animation (`index.html`)

The full self-contained HTML file. No build step, no npm install — all dependencies come from esm.sh, which rewrites bare module specifiers (like pbf imported inside ol/format/MVT) to absolute CDN URLs that the browser can resolve.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>MVT Animation</title>
  <link rel="stylesheet" href="<https://cdn.jsdelivr.net/npm/ol@10.9.0/ol.css>" />
  <style>
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body { font-family: monospace; background: #111; color: #eee;
           display: flex; flex-direction: column; height: 100vh; }
    #map  { flex: 1; }
    #hud  { padding: 12px 20px; background: #1a1a1a; border-top: 1px solid #333;
            display: flex; align-items: center; gap: 24px; }
    #hud label { font-size: 13px; color: #aaa; }
    #hud span  { font-size: 16px; font-weight: bold; }
    #timestep-val { color: #7af; }
    #value-val    { color: #fa7; }
    #color-swatch { width: 28px; height: 28px; border-radius: 4px;
                    border: 1px solid #555; display: inline-block; }
    #progress-bar { flex: 1; height: 8px; background: #333; border-radius: 4px; overflow: hidden; }
    #progress-fill { height: 100%; background: linear-gradient(to right, #00f, #f00); border-radius: 4px; }
    #controls { display: flex; gap: 8px; }
    button { background: #333; color: #eee; border: 1px solid #555;
             padding: 5px 14px; border-radius: 4px; cursor: pointer; font-size: 13px; }
    button:hover { background: #444; }
    #loading { position: absolute; inset: 0; display: flex; align-items: center;
               justify-content: center; background: rgba(0,0,0,0.55); color: #fff;
               font-size: 15px; pointer-events: none; z-index: 999; }
    #loading.hidden { display: none; }
  </style>
</head>
<body>
  <div id="map"><div id="loading">Loading tiles…</div></div>
  <div id="hud">
    <div><label>Timestep </label><span id="timestep-val"></span></div>
    <div><label>Value </label><span id="value-val"></span></div>
    <div style="display:flex;align-items:center;gap:8px;">
      <label>Color </label><span id="color-swatch"></span>
    </div>
    <div id="progress-bar"><div id="progress-fill" style="width:0%"></div></div>
    <div id="controls">
      <button id="btn-playpause">Pause</button>
      <button id="btn-reset">Reset</button>
    </div>
  </div>

  <script type="module">
    import Map              from '<https://esm.sh/ol@10.9.0/Map>';
    import View             from '<https://esm.sh/ol@10.9.0/View>';
    import TileLayer        from '<https://esm.sh/ol@10.9.0/layer/Tile>';
    import OSM              from '<https://esm.sh/ol@10.9.0/source/OSM>';
    import VectorTileLayer  from '<https://esm.sh/ol@10.9.0/layer/VectorTile>';
    import VectorTileSource from '<https://esm.sh/ol@10.9.0/source/VectorTile>';
    import MVT              from '<https://esm.sh/ol@10.9.0/format/MVT>';
    import Style            from '<https://esm.sh/ol@10.9.0/style/Style>';
    import Fill             from '<https://esm.sh/ol@10.9.0/style/Fill>';
    import Stroke           from '<https://esm.sh/ol@10.9.0/style/Stroke>';
    import { fromLonLat }   from '<https://esm.sh/ol@10.9.0/proj>';
    import { PMTiles }      from '<https://esm.sh/pmtiles@4.4.1>';

    const LAYER_NAMES = [
      'timestep_00', 'timestep_01', 'timestep_02', 'timestep_03',
      'timestep_04', 'timestep_05', 'timestep_06', 'timestep_07',
      'timestep_08', 'timestep_09', 'timestep_10',
    ];
    const VALUES = LAYER_NAMES.map((_, i) => i * 10);

    const STYLES = LAYER_NAMES.map((_, i) => new Style({
      fill:   new Fill({ color: `rgb(${Math.round(255*i/10)},0,${Math.round(255*(1-i/10))})` }),
      stroke: new Stroke({ color: '#fff', width: 2 }),
    }));

    const archive = new PMTiles('square.pmtiles');

    const source = new VectorTileSource({
      format: new MVT(),
      url: 'square.pmtiles',
      maxZoom: 8,
    });

    let currentLayerIndex = 0;

    const squareLayer = new VectorTileLayer({
      source,
      style: (feature) => {
        return feature.get('layer') === LAYER_NAMES[currentLayerIndex]
          ? STYLES[currentLayerIndex] : null;
      },
    });

    const map = new Map({
      target: 'map',
      layers: [ new TileLayer({ source: new OSM() }), squareLayer ],
      view: new View({ center: fromLonLat([0, 0]), zoom: 2 }),
    });

    const FRAME_MS = 500;

    function animate(frames, onFrame = () => {}) {
      let i = 0;
      let intervalId = setInterval(() => {
        currentLayerIndex = (currentLayerIndex + 1) % frames.length;
        squareLayer.changed();
        onFrame(frames[i]);
        i = (i + 1) % frames.length;
      }, FRAME_MS);

      return {
        pause()  { clearInterval(intervalId); },
        resume() { intervalId = setInterval(() => {
          currentLayerIndex = (currentLayerIndex + 1) % frames.length;
          squareLayer.changed();
          onFrame(frames[i]);
          i = (i + 1) % frames.length;
        }, FRAME_MS); },
        reset()  { i = 0; },
      };
    }

    const timestepEl   = document.getElementById('timestep-val');
    const valueEl      = document.getElementById('value-val');
    const swatchEl     = document.getElementById('color-swatch');
    const progressFill = document.getElementById('progress-fill');

    function onFrame(layerName) {
      const i = LAYER_NAMES.indexOf(layerName);
      timestepEl.textContent    = layerName;
      valueEl.textContent       = VALUES[i];
      swatchEl.style.background = STYLES[i].getFill().getColor();
      progressFill.style.width  = `${(i / (LAYER_NAMES.length - 1)) * 100}%`;
    }

    onFrame(LAYER_NAMES[0]);

    const loadingEl = document.getElementById('loading');
    let player = null;
    let started = false;

    function startWhenReady() {
      if (started) return;
      started = true;
      loadingEl.classList.add('hidden');
      player = animate(LAYER_NAMES, onFrame);
    }

    source.once('tileloadend', startWhenReady);
    map.once('rendercomplete', startWhenReady);
    setTimeout(startWhenReady, 10000);

    let playing = true;
    document.getElementById('btn-playpause').addEventListener('click', (e) => {
      if (!player) return;
      playing = !playing;
      e.target.textContent = playing ? 'Pause' : 'Play';
      playing ? player.resume() : player.pause();
    });
    document.getElementById('btn-reset').addEventListener('click', () => {
      if (!player) return;
      player.reset();
      if (!playing) {
        playing = true;
        document.getElementById('btn-playpause').textContent = 'Pause';
        player.resume();
      }
    });
  </script>
</body>
</html>

Running the full setup

# 1. Install tippecanoe (macOS)
brew install tippecanoe

# 2. Create the 11 per-timestep GeoJSON files
for i in $(seq 0 10); do
  pad=$(printf "%02d" $i); value=$((i * 10))
  echo '{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-60,-50],[60,-50],[60,50],[-60,50],[-60,-50]]]},"properties":{"value":'$value'}}]}' \
    > square_t${pad}.geojson
done

# 3. Generate the PMTiles archive
bash generate-tiles.sh

# 4. Start the server
npx serve

# 5. Open the demo
open <http://localhost:3000/index.html>

Conclusion: paint time, not flip it

Don't stack separate tile sources for time-series animation. Bake timesteps into a single PMTiles archive as named MVT layers, load it once, and drive playback by updating a closure variable and calling layer.changed(). Serve the archive with npx serve. Start animation only after the first tile arrives.

The result: smooth playback from first load, no flicker on hard refresh, no blank map with cache disabled.

Argo CD GitLab Authentication: A Complete GitOps-Friendly Setup

Paweł Swiridow — Wed, 01 Jul 2026 10:00:00 +0000

Argo CD can use GitLab as an authentication provider, so users log in with their GitLab accounts instead of local Argo CD users. That gives you centralized identity, cleaner access control, and one less password database waiting to become a security incident with a logo.

The setup uses OIDC through Dex, stores the GitLab OAuth credentials as encrypted Kubernetes secrets with SOPS and age, and lets Argo CD decrypt them with KSOPS during reconciliation.

How authentication works

Argo CD does not talk to GitLab directly for user login. It uses Dex as an identity broker.

The flow looks like this:

User -> Argo CD -> Dex -> GitLab -> Dex -> Argo CD -> RBAC

The short version:

The user opens Argo CD.
Argo CD redirects the user to Dex.
Dex redirects the user to GitLab.
GitLab authenticates the user and returns OIDC claims.
Argo CD reads those claims and applies RBAC rules.

Authentication tells Argo CD who the user is. RBAC tells Argo CD what damage they are allowed to do. Ideally, not much.

Create a GitLab OAuth application

First, create an OAuth application in GitLab.

In GitLab, go to your user, group, or admin application settings and create a new application with these values:

Name: Argo CD
Redirect URI: https://argocd.example.com/api/dex/callback
Scopes: openid, read_user

GitLab gives you two important values:

Application ID
Secret

In the Argo CD Dex config, these become:

dex.gitlab.clientID: "<gitlab-application-id>"
dex.gitlab.clientSecret: "<gitlab-application-secret>"
dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Do not commit the secret in plain YAML. That is not GitOps. That is just leaving your keys under the doormat and calling it automation.

Encrypt the Argo CD secret with SOPS and age

Use SOPS with age to encrypt the Kubernetes secret before committing it.

Generate an age key if you do not have one yet:

age-keygen -o age.key

Store the public key in .sops.yaml:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1examplepublickeyreplacewithyourrealpublickey

Create argocd-secret.yml:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
type: Opaque
stringData:
  dex.gitlab.clientID: "<gitlab-application-id>"
  dex.gitlab.clientSecret: "<gitlab-application-secret>"
  dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Encrypt it:

sops --encrypt argocd-secret.yml > argocd-secret.sops.yml

Then remove the unencrypted file:

rm argocd-secret.yml

Yes, really remove it. Git has a long memory and an even longer blame history.

Add KSOPS to the Argo CD repo server

Argo CD does not decrypt SOPS files by default. To make this work with Kustomize, install the KSOPS plugin in the Argo CD repo server.

The plugin lives here:

https://github.com/viaduct-ai/kustomize-sops

If you deploy Argo CD with Helm, add this to your values:

repoServer:
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.5.1
      command:
        [
          "/usr/local/bin/ksops",
          "install",
          "--with-kustomize",
          "/custom-tools",
        ]
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

This installs ksops and a Kustomize binary with plugin support into the repo server container.

Provide the age private key to Argo CD

The repo server needs the age private key to decrypt the SOPS file.

Create a Kubernetes secret from your age key:

kubectl -n argocd create secret generic sops-age-key \
  --from-literal=key="$(cat age.key)"

Expose it to the repo server as SOPS_AGE_KEY:

repoServer:
  env:
    - name: SOPS_AGE_KEY
      valueFrom:
        secretKeyRef:
          name: sops-age-key
          key: key
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}

Protect this key properly. Anyone with the private key can decrypt the secrets. Kubernetes secrets are base64 encoded, not magically protected by tiny YAML elves.

Create the KSOPS generator

Create secret-generator.yml:

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: argocd-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - argocd-secret.sops.yml

This tells Kustomize to call ksops and decrypt argocd-secret.sops.yml during the build.

Reference the generator in Kustomize

In kustomization.yml, add the generator:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
  - secret-generator.yml

When Argo CD renders this application, Kustomize runs the KSOPS generator and produces the decrypted argocd-secret resource.

Configure Dex in Argo CD

Now configure Argo CD to use GitLab through Dex.

In Helm values, configure argocd-cm like this:

configs:
  cm:
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
    dex.config: |
      connectors:
        - type: gitlab
          id: gitlab
          name: GitLab
          config:
            baseURL: https://gitlab.example.com
            clientID: $dex.gitlab.clientID
            clientSecret: $dex.gitlab.clientSecret

The $dex.gitlab.* values are loaded from argocd-secret. That is why the secret names must match exactly.

The Kustomize build options matter:

kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

This allows Kustomize to execute the KSOPS plugin.

Do not enable exec plugins for repositories you do not trust. Otherwise, you have invented remote command execution with extra YAML. Security teams love that. Briefly.

Configure Argo CD RBAC

Authentication is not authorization. After GitLab login works, configure Argo CD RBAC.

For simple user mapping by email:

helm:
  valuesObject:
    configs:
      rbac:
        policy.csv: |
          g, alice@example.com, role:admin
          g, bob@example.com, role:readonly
        scopes: "[email,groups]"

This maps:

alice@example.com -> role:admin
bob@example.com -> role:readonly

For larger teams, prefer GitLab groups instead of individual users. User by user RBAC starts cute and ends as spreadsheet archaeology.

Apply the configuration

If Argo CD is managed by another Argo CD application, commit the files and let Argo CD sync them.

If you are applying locally for a test, run:

kustomize build --enable-alpha-plugins --enable-exec . | kubectl apply -f -

Then restart the Argo CD components if needed:

kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd rollout restart deployment argocd-dex-server
kubectl -n argocd rollout restart deployment argocd-repo-server

Verify the setup

Check that the secret exists:

kubectl -n argocd get secret argocd-secret

Check the Dex logs:

kubectl -n argocd logs deploy/argocd-dex-server

Check the Argo CD server logs:

kubectl -n argocd logs deploy/argocd-server

Check the repo server if decryption fails:

kubectl -n argocd logs deploy/argocd-repo-server

Open Argo CD in the browser:

https://argocd.example.com

You should see a GitLab login option. After login, the user should land in Argo CD with the expected role.

Troubleshoot common failures

If GitLab returns a redirect error, check the redirect URI.

It must match exactly:

https://argocd.example.com/api/dex/callback

If login works but RBAC does not, check which claim Argo CD receives. The email in policy.csv must match the user claim.

If the secret does not render, check the repo server pod:

kubectl -n argocd describe pod -l app.kubernetes.io/name=argocd-repo-server

Common causes:

KSOPS is not installed in the repo server.
-enable-alpha-plugins --enable-exec is missing.
The age private key is not available in the repo server environment.
SOPS_AGE_KEY is missing or points to the wrong Kubernetes secret key.
The encrypted file name does not match secret-generator.yml.
The secret keys do not match $dex.gitlab.clientID, $dex.gitlab.clientSecret, or $dex.gitlab.redirectURI.

You can test decryption locally if you have the age key:

SOPS_AGE_KEY="$(cat age.key)" sops --decrypt argocd-secret.sops.yml

Security notes

Keep the boring security parts boring. Boring is good. Boring means nobody is explaining an outage in a meeting with too many directors.

Encrypt the GitLab OAuth secret with SOPS.
Protect the age private key.
Limit who can change Argo CD applications that use Kustomize exec plugins.
Use GitLab groups for larger teams.
Rotate the GitLab OAuth secret periodically.
Audit role:admin assignments.
Avoid giving admin access to users because they asked nicely in Slack.

Final result

With this setup, GitLab becomes the login provider for Argo CD, secrets stay encrypted in Git, and Argo CD RBAC controls what authenticated users can do.

The moving parts are not complicated:

GitLab OAuth app
SOPS encrypted argocd-secret
KSOPS in repo server
Dex GitLab connector
Argo CD RBAC policy

That is a reasonable amount of YAML for centralized authentication. Not small, obviously. This is Kubernetes. We measure configuration in geological layers.

Reciprocal Rank Fusion on free Elasticsearch: licensing, workarounds, and the OpenSearch alternative

Maciej Łopalewski — Wed, 24 Jun 2026 10:59:52 +0000

Reciprocal Rank Fusion (RRF) on Elasticsearch is gated to the Enterprise tier. On the Basic/free tier, querying with the rrf retriever returns a 403:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "current license is non-compliant for [Reciprocal Rank Fusion (RRF)]",
        "license.expired.feature": "Reciprocal Rank Fusion (RRF)"
      }
    ],
    "type": "security_exception",
    "reason": "current license is non-compliant for [Reciprocal Rank Fusion (RRF)]",
    "license.expired.feature": "Reciprocal Rank Fusion (RRF)"
  },
  "status": 403
}

This has held since RRF first appeared as a technical preview in 8.8 (May 2023), through the 8.16 GA release (November 2024), and into the 9.x line. The linear retriever added in 8.18 / 9.0 is gated the same way, also at Enterprise. Three practical workarounds exist: implement RRF yourself in application code, combine BM25 and vector results with a normalized linear combination, or move to OpenSearch - which ships RRF, score normalization (including z-score), and late-interaction reranking by a field under Apache 2.0 license.

What RRF actually does

Reciprocal Rank Fusion combines multiple ranked result sets into a single ranking based on document positions rather than raw scores. The formula is:

score(d) = Σ 1 / (k + rank_i(d))

where rank_i(d) is the document's position in the i-th result list and k is a smoothing constant. The original paper from Cormack, Clarke, and Büttcher (SIGIR 2009) used k=60 and noted the choice was not critical. Their Table 1 shows MAP barely moves across k ∈ [10, 100] - only at the extremes (k=0 or k=500) does the score drop meaningfully. Elastic and OpenSearch both default the RRF rank constant to 60, following the original paper's convention.

The reason RRF matters for hybrid search: BM25 scores are unbounded and query/corpus-dependent, while vector similarity scores live on a different scale and distribution depending on the similarity function and embedding model. Naively summing them lets one method dominate purely from scale. RRF discards the scores entirely and works only with ranks, which sidesteps the normalization problem with no tuning required.

What Basic actually gets you, and what it doesn't

The Elastic subscriptions matrix splits hybrid-search-related features across three paid tiers. Knowing what's behind each wall is more useful than the headline "RRF is paid":

Feature	Basic / free	Platinum	Enterprise
Vector search (kNN)	✓	✓	✓
Standard, kNN, pinned, rescorer retrievers	✓	✓	✓
Similarity functions for vector fields	✓	✓	✓
Synonym management	✓	✓	✓
ELSER (learned sparse encoder)	—	✓	✓
Elastic Rerank	—	✓	✓
Inference API	—	✓	✓
RRF for hybrid search	—	—	✓
Linear, rule, text similarity re-ranker retrievers	—	—	✓
Rank Vectors (for MaxSim)	—	—	✓
DiskBBQ	—	—	✓
Indexing vectors with GPUs	—	—	✓
Query Rules	—	—	✓
Learning to Rank	—	—	✓

Source: https://www.elastic.co/subscriptions

The pattern is clear. Basic gives you the building blocks for client-side hybrid search - kNN runs, BM25 runs, both can be queried separately. Platinum unlocks Elastic-managed inference and ELSER. Enterprise is where the actual modern hybrid search features live: rank fusion, learned sparse rerankers, late interaction, GPU vector indexing, learning to rank.

If you're on Basic and you want hybrid search, you're either reimplementing pieces of Enterprise in application code, switching engines, or starting a trial.

The licensing timeline

The gating has been continuous and has expanded, not relaxed. The August 2024 license change that re-added AGPLv3 to Elasticsearch made the source code open source again but did not change which features the default ELv2 distribution gates behind paid tiers.

Version	Date	What changed
8.4	August 2022	First hybrid search support
8.8	May 2023	RRF added as technical preview, gated
8.14	June 2024	Retrievers framework introduced
Pre-8.16	August 2024	Elastic announced AGPLv3 as an additional source-code license option; feature-tier gating in the default distribution unchanged
8.16	November 2024	RRF and retrievers reach GA, gated to Enterprise
8.18 / 9.0	April 2025	Linear retriever added, also gated to Enterprise
9.0.1 Basic	June 2025	Linear retriever still throws license error in production
Late 2025	September 2025	Per-retriever weights added to RRF retriever (still Enterprise-gated)

Self-managed deployments can start a 30-day trial that gives access to all subscription features, including Enterprise-tier features, for evaluation. Elastic Cloud trials are 14 days. Elastic also publishes a trial extension form that grants one additional 30-day extension on request.

Workaround 1: Implement RRF in your application

Run BM25 and kNN as two separate queries against Elasticsearch Basic, then fuse the result lists in application code. The fusion logic is roughly seven lines of Python:

from collections import defaultdict

def rrf_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Combine multiple ranked lists of document IDs using Reciprocal Rank Fusion."""
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Calling it with a BM25 result list and a kNN result list returns the fused ranking:

bm25_ids = [hit["_id"] for hit in es.search(index=idx, query=match_query)["hits"]["hits"]]
knn_ids = [hit["_id"] for hit in es.search(index=idx, knn=knn_query)["hits"]["hits"]]

fused = rrf_fusion([bm25_ids, knn_ids])
top_10_ids = [doc_id for doc_id, _ in fused[:10]]

The trade-offs are clear. You pay two round trips instead of one, you lose retrievers-specific features like inner_hits and the unified pagination model, and you have to rehydrate the document _source after fusion - typically with an mget call against the top IDs. In return, you get RRF ranking on the Basic tier with no licensing exposure. In many RAG-style systems embedding generation and kNN latency dominate, but the actual breakdown depends on cache state, candidate window size, and deployment topology - measure for your workload before assuming the extra round trip is free.

Workaround 2: Linear combination with manual normalization

The Basic tier still allows two separate queries fused with a weighted sum, as long as the math happens in your application rather than through the gated linear retriever. Min-max normalize both score sets per query - using the min and max from each result list as the range - then combine with a weight α:

from collections import defaultdict

def minmax_normalize(score: float, lo: float, hi: float) -> float:
    rng = hi - lo
    return 1.0 if rng == 0 else (score - lo) / rng

def linear_fusion(bm25_hits: list[dict], knn_hits: list[dict], alpha: float = 0.5):
    scores: dict[str, float] = defaultdict(float)

    if bm25_hits:
        s = [h["_score"] for h in bm25_hits]
        lo, hi = min(s), max(s)
        for h in bm25_hits:
            scores[h["_id"]] += alpha * minmax_normalize(h["_score"], lo, hi)

    if knn_hits:
        s = [h["_score"] for h in knn_hits]
        lo, hi = min(s), max(s)
        for h in knn_hits:
            scores[h["_id"]] += (1 - alpha) * minmax_normalize(h["_score"], lo, hi)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

When you have labeled query data, calibrated linear can beat RRF. Elastic's research on ELSER + BM25 across BEIR reports that around 40 annotated queries are enough for linear combination to start outperforming RRF, and with 300 calibration queries the optimized linear combination achieved a 6% NDCG@10 improvement over ELSER alone - compared to RRF's 1.4% improvement over the same baseline. Without calibration data, RRF is the safer default - it requires no tuning and is far less sensitive to score distribution mismatches.

Workaround 3: Switch to OpenSearch

OpenSearch ships hybrid search and RRF under Apache 2.0 with no feature-tier licensing. The native hybrid query and normalization-processor (min_max, L2) landed in 2.10 (September 25, 2023). Native RRF landed in 2.19 (February 11, 2025). The 3.x line has continued to invest heavily in vector and hybrid search since then.

Version	Date	Hybrid search additions
2.10	September 25, 2023	First `hybrid` query, `normalization-processor` (min_max, L2)
2.19	February 11, 2025	Native RRF (`score-ranker-processor`), pagination support, `hybrid_score_explanation`
3.0	May 6, 2025	z-score normalization, lower bound for min-max, inner hits in hybrid, GPU acceleration for vector index builds (experimental)
3.1	June 24, 2025	GPU acceleration for vector index builds GA, hybrid query performance improvements (up to 65% latency reduction)
3.3	October 14, 2025	Up to 20% faster hybrid for lexical subqueries, `lateInteractionScore` for reranking by a field using externally hosted ColBERT/ColPali models

The trend continued after 3.3: OpenSearch 3.4, released in December 2025, added further vector-search investment such as k-NN memory-optimized search warmup, native FP16 vector scoring, and JDK 25 support. OpenSearch 3.5 followed in February 2026, and 3.6.0 - the project's first long-term support release - was published April 7, 2026. Refer to the OpenSearch version history and downloads page for the current state.

The score-ranker-processor documentation also shows that custom subquery weights are supported via the parameters.weights array on RRF - useful when you want to weight the BM25 and vector legs differently rather than treat them equally. Elasticsearch added per-retriever weights to its RRF retriever in late 2025 as well, but because the RRF retriever itself is Enterprise-gated, this doesn't change the Basic-tier workaround story.

The mapping to Elasticsearch's paid tiers is striking. Several broadly comparable capabilities that are Enterprise-only in Elasticsearch's built-in implementation are open-source in OpenSearch:

Capability	Elasticsearch tier	OpenSearch tier
RRF rank fusion	Enterprise	Apache 2.0 (since 2.19)
Built-in weighted score fusion	Enterprise (`linear` retriever)	Apache 2.0 score-based hybrid normalization/combination via search pipelines (since 2.10)
Multiple normalization methods (min-max, L2, z-score)	Enterprise	Apache 2.0 (z-score in 3.0)
Late-interaction reranking by a field (ColBERT/ColPali workflows)	Enterprise (Rank Vectors)	Apache 2.0 (`lateInteractionScore` in 3.3)
GPU acceleration for vector index builds	Enterprise	Apache 2.0 (3.0 experimental, 3.1 GA)
Learning to Rank	Enterprise	Apache 2.0 (LTR plugin)

If the project is greenfield and does not depend on Elastic-specific features such as ELSER, Elastic Rerank, ES|QL, or the managed Inference API integrations, OpenSearch is the cleanest path. The migration cost from an existing Elasticsearch deployment is non-trivial - index format compatibility, client library differences, and Kibana versus OpenSearch Dashboards differences all add up - but the functional gap between the free tiers has generally widened for hybrid and vector-search use cases.

When paying for Enterprise makes sense

The trial-then-pay path is reasonable when the team needs text_similarity_reranker for semantic reranking with hosted models, the calibrated linear retriever, MaxSim with rank vectors for ColBERT-style retrieval, GPU vector indexing for billion-scale corpora, learning-to-rank pipelines, or production support contracts. None of this comes for free, and reimplementing it is more expensive than the license for organizations with substantial search infrastructure. For teams whose only blocker is RRF specifically, the workarounds above usually win on cost and are straightforward to implement for simple two-leg hybrid retrieval.

What most teams actually need

Most teams that hit the RRF license error don't actually need RRF specifically - they need hybrid search to work. Manual linear combination with normalized scores is often sufficient for a first production hybrid-search implementation on any Elasticsearch version. The core RRF scoring loop is another twenty lines of code on top of that - production deployments will additionally want pagination, deduplication, an mget rehydration step, tie-breaking, and observability around the fused ranking. And for projects starting fresh in 2026, the open-source answer keeps getting better with every OpenSearch release.

Originally published at https://u11d.com on June 3, 2026.

How to Deploy Payload CMS on AWS Amplify with MongoDB Atlas for Free

Michał Miler — Wed, 17 Jun 2026 07:12:00 +0000

Organizations frequently over-invest in hosting infrastructure for low-traffic content sites. A landing page, blog, or small CMS rarely justifies the cost of a constantly-running server—whether VPS, platform-as-a-service, or managed container services. Serverless architectures solve this by shifting from per-minute billing to per-request pricing. When traffic is very low, your infrastructure costs may approach zero.

This guide demonstrates how to deploy Payload CMS with a Next.js frontend on AWS Amplify, AWS S3, and MongoDB Atlas in a way that can stay within free-tier thresholds for a small, low-traffic site. We'll walk through the stack rationale, step-by-step provisioning, caching strategy to minimize compute, and security trade-offs you inherit.

Architecture Overview

This setup combines AWS Amplify Hosting for serverless Next.js SSR, AWS S3 for media storage, MongoDB Atlas for the database, and Payload CMS for content operations. Amplify SSR runs on AWS Lambda, so compute is billed by request and duration rather than by always-on server time; Amplify pricing currently includes 1,000 build minutes, 500,000 SSR requests, 100 GB-hours of SSR duration, 15 GB data transfer out, and 5 GB stored on CDN per month at no cost. S3 keeps uploaded media outside of your app runtime with S3 pricing, while Atlas pricing offers a free M0 cluster with 512 MB storage for small content sites.

This model is especially useful when usage is low or uneven: marketing sites with occasional campaigns, content teams publishing periodically, early-stage products validating demand, and internal tools or software services that are active only for short windows, for example one to two hours per day. In these cases, paying for permanent infrastructure often brings little business value.

If you have sustained high traffic or volatile workloads, benchmark Amplify against Vercel and self-hosted options before committing. Vercel often handles cold starts and p95 latency more predictably for SSR-heavy apps.

Prerequisites

AWS account (sign up for 12 months of AWS Free Tier)
GitHub repository with Next.js and Payload CMS (we will use demo repo for this tutorial)
MongoDB Atlas free account
Node.js 20+ and npm installed locally

Step A: MongoDB Atlas Free Cluster

Create Cluster

Visit Atlas and log in.
Click Build a Cluster and select Free tier (512 MB storage, shared vCPU, no cost).
Choose a cloud provider and region matching your Amplify deployment (we'll use us-east-1 in AWS).
Unselect "Preload sample dataset."
Click Create Deployment. This takes 1–3 minutes.

Create Database User and Connection String

When prompted, create a database user. Use generated password or use your own password rules; copy it immediately—you'll need it to construct the connection string.
Click Choose a connection method and select Drivers.
Copy the connection string (example: mongodb+srv://username:password@cluster.mongodb.net/?appName=payload-cms). Replace <password> with your copied password. This string includes the username; the password is separate for security.
Store this securely—you'll add it to Amplify environment variables later.

Configure Network Access

By default, Atlas limits restrict external connections. For Amplify Lambda functions to reach the database:

Go to Security > Database & Network Access > IP Access List.
Click Add IP Address and enter 0.0.0.0/0 to allow all IPs.
Click Confirm.

Security trade-off: This opens the database to the internet. Authentication via username/password is your primary defense. For production with higher budget, use private network options like AWS PrivateLink or VPC peering.

Your MongoDB cluster is ready. Auto-pause can add a few seconds to the first request after inactivity. In practice, proper Next.js caching usually hides most of this for end users.

Step B: AWS S3 Bucket and IAM User

Payload CMS stores file uploads (images, documents) in backend storage. Lambda limits include payload and response constraints. By uploading directly to S3 and storing S3 URLs in the database, you avoid Lambda serialization bottlenecks. For image optimization, see image guide.

Create S3 Bucket

Log into AWS Console, select the same region where Amplify will be deployed (e.g., us-east-1) and navigate to S3.
Click Create bucket.
Enter a globally unique bucket name (e.g., blog-payloadcms-demo-12345).
Uncheck Block all public access so Amplify and browsers can retrieve media files. Public access applies only to objects with public ACLs or bucket policies; Payload uploads are private by default.
Accept remaining defaults and click Create bucket.

Create IAM User with S3 Permissions

Payload needs AWS credentials to upload and read media objects in your S3 bucket from the running app. Creating a dedicated IAM user gives the app programmatic access without reusing your root account and lets you limit permissions to only the S3 actions your project needs.

Navigate to Identity and Access Management (IAM) > IAM Users.
Click Create user and name it.
Uncheck Provide user access to the AWS Management Console (API access only).
Click Next.
Select Attach policies directly and search for AmazonS3FullAccess
Click Next and Create user.

For production, follow least privilege. Create a custom policy allowing only s3:PutObject, s3:GetObject, and s3:DeleteObject on your specific bucket.

Generate Access Keys

Select the new IAM user and go to Security credentials.
Click Create access key and choose Other.
Copy both the Access Key ID and Secret Access Key. You will not see the secret again. Store these securely (AWS Secrets Manager or a password manager). Never commit credentials to git.

You now have all configuration data needed for S3 integration: S3_BUCKET, S3_REGION, S3_ACCESS_KEY_ID, and S3_SECRET_ACCESS_KEY.

Payload S3 config

In our demonstration repository we used the following S3 plugin configuration in src/payload.config.ts. It utilizes environment variables mentioned above and uses S3 hosting for media collection.

s3Storage({
  collections: {
    media: true,
  },
  config: {
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID || '',
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || '',
    },
    region: process.env.S3_REGION || 'us-east-1',
    endpoint:
      process.env.S3_BUCKET && process.env.S3_REGION
        ? `https://s3.${process.env.S3_REGION}.amazonaws.com`
        : undefined,
    forcePathStyle: true,
  },
  bucket: process.env.S3_BUCKET || '',
  disableLocalStorage: true,
})

Why this setup works well on Amplify:

endpoint + forcePathStyle helps generate direct S3 URLs, which avoids routing large media responses through Lambda.
disableLocalStorage: true ensures uploaded media is persisted only in S3, which is important for stateless serverless runtimes.
Keeping credentials and bucket settings in environment variables makes the same code reusable across local, staging, and production deployments.

Your configuration may differ slightly, especially around endpoint strategy, custom domains, or local fallback behavior.

Step C: AWS Amplify Deployment

Amplify Hosting is a Git-based hosting service that watches your repository, runs your build in an isolated environment, deploys generated artifacts to the CDN and compute layer, and provisions HTTPS on an *.amplifyapp.com domain.

You can configure deployment fully in the Amplify UI, but keeping the build definition in amplify.yml is usually safer for teams: it is versioned, reviewable in pull requests, reproducible across environments, and avoids silent UI drift. This is a lightweight Infrastructure as Code (IaC) approach: the deployment behavior is declared in code, not only clicked in a console. For background, see IaC guide. Build configuration lives in the repository as amplify.yml. This ensures reproducible builds across your team.

Configure Build Settings

Before creating the Amplify app, make sure amplify.yml is committed to the repository root. The file below is our configuration. Yours may differ, especially in package manager commands (npm, pnpm, yarn) and which environment variables must be exported into .env.production.

Use this as a reference and adapt only what your app actually requires:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - echo "Installing dependencies..."
        - npm ci
    build:
      # <https://docs.aws.amazon.com/amplify/latest/userguide/ssr-environment-variables.html#:~:text=Amplify%20Hosting%20supports%20adding%20environment,in%20the%20build%20commands%20section>.
      commands:
        - echo "Building the application..."
        - rm -f .env.production
        - export NEXT_PUBLIC_SERVER_URL="${NEXT_PUBLIC_SERVER_URL:-<https://$>{AWS_BRANCH}.${AWS_APP_ID}.amplifyapp.com}"
        - echo "Using NEXT_PUBLIC_SERVER_URL=$NEXT_PUBLIC_SERVER_URL"
        - env | grep -e '^NEXT_PUBLIC_' >> .env.production || true
        - env | grep -e '^DATABASE_URL=' -e '^PAYLOAD_SECRET=' -e '^CRON_SECRET=' -e '^PREVIEW_SECRET=' -e '^S3_BUCKET=' -e '^S3_REGION=' -e '^S3_ACCESS_KEY_ID=' -e '^S3_SECRET_ACCESS_KEY=' >> .env.production || true
        - npm run build
    postBuild:
      commands:
        - echo "Post build complete"
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - .next/cache/**/*
      - node_modules/**/*

The key pattern is documented in env docs: expose only required variables during build, then write them to .env.production. Keep package manager commands and env exports aligned with your own repository.

Connect Repository to Amplify

Go to Amplify Console.
Make sure you are in the same region where S3 bucket and Mongo Cluster were created
Click Create new app.
Select your Git provider (GitHub in our case) and authorize Amplify to access your repository.
Choose your repository and the branch to deploy (typically main).
Click Next.

Add Environment Variables

During app creation, add environment variables in step 3: App settings -> Advanced settings.

Expand Advanced settings.
Add the same variables shown in the screenshot (retrieved from previous steps).
Continue deployment.

Deploy

Click Next and Save and deploy.
Amplify now builds and deploys your app. First builds take 3–10 minutes.
Once complete, you will see your application URL (e.g., https://myapp-12345.amplifyapp.com).

Verify Admin Access

Navigate to your admin panel (e.g. https://myapp-12345.amplifyapp.com/admin). You should see the Payload login page. Use user docs to create your first admin account.

Fix Media CORS (One-Time Setup)

If images or media files fail to load with CORS errors, your S3 bucket needs CORS configuration:

Go to S3 > Buckets and select your bucket.
Click Permissions and scroll to Cross-origin resource sharing (CORS).
Click Edit and paste:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "HEAD", "PUT"],
    "AllowedOrigins": ["<https://myapp-12345.amplifyapp.com>"],
    "ExposeHeaders": ["ETag", "x-amz-request-id"],
    "MaxAgeSeconds": 3000
  }
]

Replace myapp-12345.amplifyapp.com with your actual Amplify domain. Click Save. Media should now load without errors.

This CORS configuration allows your Amplify domain to read media from S3; it does not expose your bucket publicly.

Caching and Static Generation in Next.js

Next.js caching directly impacts Amplify compute costs. See caching docs.

Static Site Generation (SSG)

For content that changes infrequently (blog posts, landing pages), use Static Site Generation:

// app/posts/[slug]/page.tsx
export const revalidate = 86400 // Revalidate once per day (ISR)

export default async function PostPage({ params }: { params: { slug: string } }) {
  const post = await fetch(
    `${process.env.NEXT_PUBLIC_SERVER_URL}/api/posts?where[slug][equals]=${params.slug}`,
  )
  // ...
}

ISR docs explain how pages are pre-cached and revalidated in the background.

On-Demand Revalidation

When an editor publishes a new post, trigger revalidation immediately:

// `src/hooks/revalidateFrontend.ts` (Payload hook)
import { revalidateTag } from 'next/cache'

export default async function revalidateFrontend(doc) {
  if (!process.env.NEXT_PUBLIC_SERVER_URL) return

  // Tell Next.js to invalidate the blog archive page
  revalidateTag('posts-archive')

  return doc
}

revalidateTag docs cover tag invalidation across cached responses. This pattern helps keep content fresh without full rebuilds.

Cache Strategy Summary

For static pages such as landing pages and blog posts, SSG with a 24-hour ISR window minimizes compute and usually results in only periodic revalidation calls. Dynamic surfaces like search or personalized dashboards are still rendered per request, so they consume more SSR runtime. API routes can remain lightweight and use HTTP caching behavior where applicable. For low traffic, this mix typically stays inside Amplify free-tier limits.

Pricing Notes

AWS Amplify

Amplify pricing is usage-based:

Build minutes: 1,000/month included, then $0.01 per minute
SSR request count: 500,000/month included, then $0.30 per 1 million requests
SSR request duration: 100 GB-hours/month included, then $0.20 per GB-hour
Data transfer out: 15 GB/month included, then $0.15 per GB
CDN storage: 5 GB/month included, then $0.023 per GB-month

AWS S3

S3 pricing is also usage-based. For a small media library, storage and request costs are usually low, but the exact amount depends on object size, request volume, storage class, and data transfer.

MongoDB Atlas

Atlas pricing includes the free M0 cluster: 512 MB storage, shared RAM, shared vCPU, free forever. Atlas limits also note that auto-pause can slow the first request after inactivity.

When to Reconsider

High traffic or tail-latency sensitivity: benchmark Amplify against Vercel and self-hosted options.
Unpredictable spikes: verify how your cache hit rate and SSR load affect request count and duration.
Compliance or data residency: ensure your MongoDB and Amplify regions match your requirements. See AWS regions.

Security Trade-Offs

1. MongoDB IP Allowlist: Public Access

Configuring 0.0.0.0/0 means any IP can attempt to connect to your MongoDB cluster.

Use long random credentials, limit the Mongo user to the minimum database scope, and avoid admin privileges for application access. For stronger isolation in production, use network controls instead of public IP allowlists.

2. S3 Public Access: Bucket Policy and CORS

S3 blocks public access by default. Unchecking this allows you to create a public bucket policy.

Keep policies narrow: allow only public read (s3:GetObject) where needed and never expose write permissions publicly. If files are sensitive, prefer signed URLs with expiration instead of open object access; see storage docs. It also helps to separate public uploads from private objects by prefix and policy.

3. Secrets in Environment Variables

Amplify stores secrets in managed service configuration rather than in your repository. These values are encrypted at rest and should never be printed in build logs.

Keep IAM permissions minimal for deployment roles, avoid logging any secret values in build output, and consider Secrets Manager if you need managed rotation and stronger operational controls.

For a detailed security review, see security guide.

Bonus: Custom Domain Setup

Once your site is live on myapp-12345.amplifyapp.com, you can add a custom domain (e.g., blog.example.com):

In Amplify Console: Go to Domain management and click Add domain.
Point Route 53 DNS: If your domain is in Route 53 (AWS), Amplify auto-creates records. Otherwise, add your registrar's CNAME records.
SSL/TLS: Amplify generates a free certificate via AWS Certificate Manager. HTTPS is automatic.
Costs: Domain registration (~$10–15/year) + Route 53 hosting ($0.50/month) if applicable.

For detailed steps, see domain guide.

Troubleshooting

"502 Bad Gateway" or "413 Payload Too Large"

If media uploads fail, the issue is likely the 6 MB Lambda response limit. Ensure S3_ENDPOINT is configured in your Payload config to use direct S3 URLs, bypassing the Lambda gateway. See config file for the current setup.

"Database Connection Timeout"

Check that:

MongoDB Atlas IP allowlist includes 0.0.0.0/0 or your Amplify IP.
DATABASE_URL is correct and includes the password.
Database user exists and has correct credentials.

Media Not Loading (CORS Error)

Ensure your S3 bucket CORS policy includes your Amplify domain (see CORS step).

Logs and Debugging

Use logs docs to inspect runtime errors, environment variable issues, and database connectivity problems.

Summary and Conclusions

This architecture gives you a practical way to run a Payload CMS website with low baseline cost: Amplify for SSR compute, S3 for media, and MongoDB Atlas for managed data. For low-traffic landing pages and blogs, it can stay within free-tier thresholds when caching is configured correctly and media is served directly from S3.

The key trade-off is operational behavior under low traffic: you may see cold starts and first-request latency, so monitor logs, keep SSR scope minimal, and add warming only when needed. As traffic, compliance, or latency requirements increase, benchmark against Vercel and self-hosted options before scaling the same setup.

Next Steps

Now that your site is deployed, consider:

Optimize images: See image guide.
Reduce cold starts: If you notice slow first loads after inactivity, use warming guide. Vercel usually hides this problem better, especially for p95 latency, so benchmark both if first-response consistency matters.
Monitor costs: Set up AWS Budgets.
Scale content: As you grow, plan for MongoDB upgrade to a paid cluster if storage exceeds 512 MB.

References

Dynamic Email Domain Validation in Keycloak with a Custom Authenticator

Bartek Gałęzowski — Wed, 10 Jun 2026 06:00:00 +0000

Introduction

Keycloak ships with a built-in mechanism for restricting user registration by email domain — but it's static. Changing the allow-list means touching realm configuration and redeploying. For B2B SaaS products that onboard new tenants regularly, that's an operational bottleneck you don't want.

The right solution is to move domain policy out of Keycloak entirely and delegate it to a backend service that can be updated at runtime. This article walks through building a custom Keycloak Authenticator — called domain-email-validator — that does exactly that: at login time, it calls an external API to decide whether the user's email domain is permitted.

By the end, you'll understand the full architecture, the Java implementation, how to wire it into both browser and IDP flows, and the operational tradeoffs involved.

Why Static Domain Restrictions Fall Short in B2B Products

Keycloak's native domain restriction works well for single-tenant deployments with a fixed list of approved domains. But in multi-tenant environments, you typically need:

Per-tenant domain rules — Tenant A allows acme.com; Tenant B allows globex.com and initech.com.
Runtime updates — A new customer signs a contract and needs access today, not after the next deploy.
Consistency across login methods — The same rule should apply whether a user logs in via username/password or via Google SSO.

A custom Authenticator SPI solves all three. It's evaluated on every login attempt, it reads policy from an external source, and it plugs into both browser and post-broker flows with the same implementation.

Architecture Overview

The approach is a thin decision layer inside Keycloak. The plugin does not own domain policy — it only enforces it.

The contract is intentionally minimal:

Keycloak sends { domain, realmId } to a backend endpoint.
Backend returns 200 to allow or any non-200 to deny.
Keycloak continues or blocks the flow accordingly.

This keeps all business logic — tenant configuration, domain lists, auditing — in your backend, where it belongs.

Request Lifecycle

Keycloak executes earlier steps (credential check or broker handshake).
The domain validator step runs.
It reads the user's email from the active context.
It sends a small HTTP POST to the policy service.
The backend returns allow or deny.
Keycloak continues the flow or shows a denial message.

Because the decision is made per login attempt, domain policy changes take effect immediately — no cache warmup, no redeployment.

Implementation

The authenticator is split into two standard Keycloak SPI parts: a factory that registers the provider and exposes configuration fields, and an executor that runs the validation logic at login time.

Factory: Registering the Provider

DomainValidatorAuthenticatorFactory defines two configuration properties visible in the Keycloak Admin UI under the flow step's Config tab:

.property()
    .name(DomainValidatorAuthenticator.CONFIG_VALIDATION_URL)
    .label("Domain Validation URL")
    .type(ProviderConfigProperty.STRING_TYPE)
    .defaultValue("https://my-server/api/keycloak/domain-check")
    .add()
.property()
    .name(DomainValidatorAuthenticator.CONFIG_SHARED_SECRET)
    .label("Shared Secret")
    .type(ProviderConfigProperty.PASSWORD)
    .secret(true)
    .add()

There is no hidden YAML or server-side config file. Everything is per-flow-execution and editable through the UI, which makes it easy to configure different endpoints per realm.

Executor: The Core Validation Logic

The central method is authenticate(AuthenticationFlowContext context). It first resolves the email from context — either from a submitted form or from a brokered identity:

String email = resolveBrokeredEmail(context);
String flowType;

if (email != null) {
    // Post-broker flow: email came from the IDP identity token
    flowType = "idp";
} else {
    // Standard browser flow: read from the submitted form
    MultivaluedMap<String, String> formData =
            context.getHttpRequest().getDecodedFormParameters();
    email = formData.getFirst("username");
    flowType = "form";
}

// If no email or no @ sign, pass through — Keycloak handles invalid credentials
if (email == null || !email.contains("@")) {
    context.success();
    return;
}

Then it reads the flow config and calls the policy service:

Map<String, String> config = context.getAuthenticatorConfig().getConfig();
String validationUrl = config.get(CONFIG_VALIDATION_URL);
String sharedSecret = config.get(CONFIG_SHARED_SECRET);

HttpPost post = new HttpPost(validationUrl);
post.setEntity(new StringEntity(
    String.format("{\"domain\":\"%s\",\"realmId\":\"%s\"}",
        escapeJson(domain), escapeJson(realmId)),
    ContentType.APPLICATION_JSON
));

if (sharedSecret != null && !sharedSecret.isBlank()) {
    post.setHeader("Authorization", "Bearer " + sharedSecret);
}

Finally, it interprets the response:

int statusCode;
try (CloseableHttpResponse response = httpClient.execute(post)) {
    statusCode = response.getStatusLine().getStatusCode();
} catch (IOException e) {
    LOG.errorf(e, "DomainValidatorAuthenticator: HTTP call failed for domain '%s'" +
                    " in realm '%s'", domain, realmId);
    failWithError(context, "domainValidationUnavailable");
    return;
}

if (statusCode == 200) {
    context.success();
} else {
    LOG.infof("DomainValidatorAuthenticator: domain '%s' denied in realm '%s'" +
                " (HTTP %d)", domain, realmId, statusCode);
    failWithError(context, "domainNotAllowed");
}

Any IOException — network failure, timeout, connection refused — results in domainValidationUnavailable. Missing config results in domainValidatorMisconfigured. The validator is fail-closed: when in doubt, it denies.

Resolving the Brokered Email

For post-broker flows (e.g. after Google SSO), the email comes from the already-resolved user object, not the form:

private String resolveBrokeredEmail(AuthenticationFlowContext context) {
    UserModel user = context.getUser();
    if (user != null) {
        String userEmail = user.getEmail();
        if (userEmail != null && !userEmail.isBlank()) {
            return userEmail.trim();
        }
    }
    return null;
}

This means the same authenticator class handles both input sources. The only difference is where the email comes from — the rest of the flow is identical.

Docker Build

The Dockerfile wires the Maven build and the Keycloak image together cleanly:

FROM quay.io/keycloak/keycloak:26.6.0 AS base-keycloak

FROM maven:3.9-eclipse-temurin-21 AS maven-builder
WORKDIR /build
COPY keycloak-domain-validator/pom.xml ./pom.xml
RUN mvn dependency:resolve -q
COPY keycloak-domain-validator/src ./src
RUN mvn package -q -DskipTests

FROM base-keycloak AS keycloak-builder
COPY --from=maven-builder /build/target/domain-validator.jar /opt/keycloak/providers/
RUN /opt/keycloak/bin/kc.sh build

FROM base-keycloak
COPY --from=keycloak-builder /opt/keycloak/ /opt/keycloak/
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]

The multi-stage build keeps the final image lean: Maven is only present during the build stage, and the compiled JAR is dropped directly into Keycloak's providers directory.

Configuring Authentication Flows

1. Browser Login Flow (Username/Password)

Go to Authentication → Flows.

Duplicate the built-in browser flow

Inside the Forms sub-flow, add the Domain Email Validator step.

Place it directly after Username Password Form , and set its requirement to Required.

Click Config on the step and enter your validation URL and shared secret.

Bind the duplicated flow as the active browser flow under Bindings.

The validator reads the email from the submitted username field in the login form.

2. Google / IDP Login Flow (Post-Broker)

Create a new dedicated flow for post-broker login.
Add Domain Email Validator as a Required step.
Go to Identity Providers → Google → Post Login Flow and assign the new flow.

The validator reads the email from the resolved user identity returned by the IDP.

Best Practices

Keep the validation endpoint fast and internal. Since this check is synchronous and blocks login, endpoint latency directly affects user experience. Route it over internal networking and keep the response payload small.

Monitor the endpoint as authentication-critical infrastructure. Set up latency alerts and error rate tracking. A degraded policy service means users can't log in — treat it accordingly.

Rotate the shared secret regularly. The bearer token is your only authorization layer between Keycloak and the policy service. Automate rotation through your secrets management tooling.

Always set the step as Required. An Alternative or Disabled requirement silently bypasses the check, which defeats the purpose entirely.

Ensure Google OAuth includes the email scope. Without it, the IDP flow won't have an email to validate, and the validator will pass through silently.

Keep the API contract minimal and stable. The { domain, realmId } payload and the HTTP status response are a clean, versioned interface. Resist the temptation to add fields over time unless there's a clear reason.

Security Posture

The validator is intentionally conservative by design:

It derives the domain from the server-side authentication context, not from client-supplied input.
It supports a bearer secret for service-to-service authorization.
It denies access when validation cannot be completed for any reason — misconfiguration, network failure, or unexpected response codes all result in denial.

In other words: authentication proceeds only when policy can be positively verified.

Tradeoffs and Alternatives

Approach	Flexibility	Operational cost	Complexity
Keycloak built-in domain restriction	Low (static)	Low	Low
Custom Authenticator + external API	High (dynamic)	Medium	Medium
Keycloak scripting (deprecated)	Medium	Medium	Medium
Custom User Storage SPI	High	High	High

The custom Authenticator approach hits the right balance for most B2B products: it's dynamic, tenant-aware, independently deployable, and doesn't require deep Keycloak internals knowledge to maintain.

The main tradeoff is availability coupling — if your policy service goes down, so does login. Mitigate this with high-availability deployment, circuit breakers at the infrastructure layer, and solid monitoring.

FAQ

Q: What happens if the policy service is unreachable during a login attempt?

The authenticator catches the IOException and fails with domainValidationUnavailable. Login is denied. This is the intentional fail-closed behavior — an unreachable policy service is treated as a denial rather than a bypass.

Q: Can this authenticator be used across multiple realms with different endpoints?

Yes. The validation URL and shared secret are configured per flow execution, not globally. Each realm can have its own duplicated flow pointing to a different endpoint, or the same endpoint can handle per-realm routing using the realmId field in the request body.

Q: Does this affect performance at login time?

It adds one synchronous HTTP call per login attempt. In practice, if the policy service is on the same internal network and kept lightweight, the added latency is negligible — typically single-digit milliseconds. Treat the endpoint as latency-sensitive infrastructure.

Q: What error message does the user see when denied?

The authenticator calls failWithError(context, "domainNotAllowed"), which maps to a message key in Keycloak's theme messages file. You can customize the displayed text by overriding that key in your realm's login theme.

Conclusion

For multi-tenant B2B products, static email domain restrictions in Keycloak simply don't scale. A custom Authenticator SPI that delegates to an external policy service is a clean, maintainable pattern that keeps domain management out of Keycloak configuration and in the hands of your backend team.

Key takeaways:

The plugin is a thin decision layer — it enforces policy but doesn't own it.
The same authenticator class handles both browser and IDP (post-broker) login paths.
The API contract is minimal: { domain, realmId } in, HTTP status out.
The validator is fail-closed — network errors and misconfigurations result in denial, not bypass.
Configuration lives in the Keycloak Admin UI, scoped per flow execution, making it easy to manage per realm.
Treat the policy endpoint as authentication-critical infrastructure: monitor latency, alert on errors, and keep it highly available.

Next.js 16 Caching for E-Commerce: Cache Components, use cache, revalidateTag, and Fresh Product Data

uninterrupted — Tue, 02 Jun 2026 22:00:00 +0000

Caching in e-commerce is never just about speed. A fast storefront is useful only if it still shows the right price, the correct stock level, and the right experience for the current customer.

That is why caching in a Next.js storefront can be deceptively hard. Some data should be shared broadly and kept warm for SEO and performance. Some data should be refreshed often. Some should never be shared between users at all.

Next.js 16 gives teams a much clearer toolbox for solving this problem with Cache Components, use cache, tag-based invalidation, and explicit cache lifetime controls. Used properly, these features let you keep pages fast without drifting into stale commerce data.

In this guide, I will walk through a practical way to think about caching in a modern storefront and show how to combine use cache, cacheLife, and revalidateTag for real e-commerce use cases.

Why Caching Is Harder in E-Commerce Than in a Typical Content Site

On a standard marketing site, most content changes infrequently. If a page is cached for a few minutes or even a few hours, the business impact is usually negligible.

Commerce systems work differently.

The same product page may contain:

stable product descriptions and category copy
semi-dynamic data such as price, availability, shipping estimates, or promotion labels
private data such as cart state, recently viewed items, or customer-specific pricing

Treating all of that data the same way leads to one of two bad outcomes:

You cache too aggressively and serve stale prices or availability.
You disable caching everywhere and lose the performance benefits that help SEO and conversion.

The better approach is to split your data by volatility and audience.

The Three Cache Boundaries That Matter Most

For most commerce projects, the cleanest mental model is to divide data into three groups.

1. Stable catalog content

This is the part of the page that usually changes only when content editors or merchandisers update the catalog.

Examples:

product title and description
brand information
CMS blocks, FAQs, and long-form copy
category landing page copy
SEO metadata

This is the best candidate for shared caching because it improves performance for everyone and is usually safe to invalidate on demand when content changes.

2. Shared but fast-changing commerce data

This data is still shared between many users, but it changes more often and has stronger operational impact.

Examples:

current price for a product and region
inventory status
campaign badges
delivery promises
low-stock signals

This layer often benefits from a shorter cache lifetime and explicit invalidation.

3. User-specific or session-specific data

This is the data that should not be globally cached.

Examples:

cart contents
logged-in customer state
B2B contract pricing
personalized recommendations
account-specific discounts
checkout calculations tied to the active session

This is where teams most often make expensive mistakes. If the output depends on the current user, you need to keep that boundary private.

What Next.js 16 Changes in Practice

The major advantage of Next.js 16 is not just that it caches. It is that caching becomes more explicit.

Instead of treating an entire route as fully static or fully dynamic, you can choose more precise boundaries:

cache a component or function with use cache
assign cache tags to data and invalidate them later
control lifetime with cacheLife
keep per-user data private where needed
stream dynamic parts separately instead of forcing the entire page into request-time rendering

For e-commerce storefronts, that means you can serve a fast shell and still keep critical parts fresh.

A Practical Storefront Strategy

Let us look at a product page. In a typical headless stack, product content might come from Medusa or another commerce backend, while editorial content may come from a CMS.

The important point is not the backend itself. The important point is how you cache each layer.

Cache the stable product content

If product descriptions and media do not change every minute, cache them explicitly and tag them so they can be invalidated when merchandisers update the catalog.

import { cacheTag } from 'next/cache'

async function getProduct(slug: string) {
  'use cache'

  cacheTag(`product:${slug}`)

  const response = await fetch(
    `${process.env.COMMERCE_API_URL}/products/${slug}`
  )

  if (!response.ok) {
    throw new Error('Failed to load product')
  }

  return response.json()
}

This is a good fit for data that should be broadly reusable and revalidated only when a product actually changes.

Use a shorter cache boundary for price and stock

Price and stock need more care. In some stores, they can still be shared safely for a short period. In others, especially B2B or highly promotional environments, they may need user-level or request-level handling.

For shared price or inventory data, a short-lived cache can be a sensible compromise.

import { cacheLife, cacheTag } from 'next/cache'

async function getOffer(productId: string, regionId: string) {
  'use cache'

  cacheTag(`offer:${productId}:${regionId}`)
  cacheLife({ expire: 60 })

  const response = await fetch(
    `${process.env.COMMERCE_API_URL}/offers/${productId}?region=${regionId}`
  )

  if (!response.ok) {
    throw new Error('Failed to load offer')
  }

  return response.json()
}

Here the offer can be shared, but only briefly. That is often enough to reduce backend pressure while still keeping the storefront operationally safe.

Keep customer-specific data out of the shared cache

If pricing, recommendations, or entitlements depend on the active customer, do not treat them as generic shared data.

This applies especially to:

B2B price lists
customer group discounts
partner-only catalog visibility
personalized recommendation widgets

At that point, a private boundary or request-time fetching becomes more appropriate than a shared cache.

The principle is simple: if another customer must never see the same output, that data should not live in a shared cache.

If you do want to cache user-specific data briefly to avoid repeating the same work during a session, keep that cache private instead of shared.

import { cacheLife } from 'next/cache'

async function getRecommendations(productId: string, customerId: string) {
  'use cache: private'

  cacheLife({ expire: 60 })

  const response = await fetch(
    `${process.env.RECOMMENDATION_API_URL}/products/${productId}/recommendations?customer=${customerId}`
  )

  if (!response.ok) {
    throw new Error('Failed to load recommendations')
  }

  return response.json()
}

That keeps the result bounded to the current user context instead of letting it leak into global storefront output. It is also worth noting that 'use cache: private' does not store its cache entries on the server. The function still runs during server rendering, but the cached result is kept only in the browser’s memory and does not survive a full page reload.

Why This Matters for Medusa and Other Headless Commerce Backends

This pattern becomes especially valuable in headless commerce because the storefront is often responsible for combining multiple systems:

commerce backend
CMS
search service
recommendation engine
analytics and experiment layers

If all of those systems are fetched at request time for every page view, the frontend becomes slower and more fragile than it needs to be.

If all of them are aggressively cached together, the storefront becomes fast but operationally unsafe.

The right answer is to cache by business meaning, not by convenience.

For example:

CMS content can usually be cached until editors publish changes.
catalog copy can usually be cached and invalidated via tags.
regional shared pricing can often be cached briefly.
logged-in B2B pricing should stay private.
cart and checkout state should stay request-scoped.

That is the difference between a caching strategy that helps the business and one that only looks good in benchmarks.

Using `revalidateTag` to Keep Data Fresh

One of the most useful patterns in Next.js 16 is tag-based invalidation.

Instead of revalidating whole pages blindly, you can invalidate the exact data domain that changed. That is particularly useful when updates come from webhooks triggered by your CMS, PIM, or commerce backend.

For example, if a product is updated in Medusa or an editorial team changes CMS content linked to that product, you can invalidate the relevant tag immediately.

import { revalidateTag } from 'next/cache'

export async function POST(request: Request) {
  const payload = await request.json()

  revalidateTag(`product:${payload.slug}`, 'max')
  revalidateTag(`offer:${payload.id}:${payload.regionId}`, 'max')

  return Response.json({ ok: true })
}

In practice, this lets you keep product content cached most of the time while still reacting quickly to actual changes.

That is a much better fit for commerce than short global revalidation windows everywhere.

One operational note matters here: never expose a public revalidation endpoint without request validation. If invalidation is triggered by a CMS, Medusa, or ERP webhook, verify the source with a shared secret or signature check before calling revalidateTag.

Where Teams Usually Get It Wrong

There are a few caching mistakes that repeatedly appear in storefront projects.

Mistake 1: Caching personalized pricing globally

This is the most dangerous one. If the result depends on customer identity, customer group, contract terms, or session data, do not let it leak into a shared cache boundary.

Mistake 2: Making the whole page dynamic because one widget is dynamic

A product page does not need to become fully request-time just because the cart badge or recommendation box is personalized.

Keep the stable content cached and isolate the dynamic parts instead.

Mistake 3: Using `no-store` too broadly

no-store is sometimes necessary, but if it becomes the default reaction to uncertainty, you throw away much of the value of the App Router architecture.

Use it for truly request-bound data, not as a shortcut for unclear cache design.

Mistake 4: Forgetting business dimensions in cache keys

Prices are rarely just “product price.” They are often tied to region, currency, channel, campaign, or customer group.

If those dimensions are not reflected in the data boundary, the cache becomes unsafe.

Mistake 5: Mixing editorial freshness with operational freshness

A product description may be safe to cache until a content change occurs. Inventory and promotions may not be. These two layers should not be forced into the same lifetime.

A Good Default Model for Product Pages

If you need a practical starting point, this model works well for many storefronts:

Data type	Suggested approach
Product description, SEO copy, editorial blocks	`use cache` with tags and on-demand invalidation
Shared regional price and stock	short cache lifetime plus tags
Recommendations	private or isolated dynamic boundary
Cart state	request-time or session-specific logic
Checkout totals and payment readiness	request-time only
B2B contract pricing	private boundary, never generic shared output

This is not a framework rule. It is a business-safe baseline.

Why This Works Well With Streaming and PPR Thinking

If you have already explored Partial Prerendering in Next.js 16, this caching model should feel familiar.

The idea is the same:

keep the stable shell fast
isolate dynamic or high-volatility parts
avoid blocking the entire page for data that does not deserve that privilege

For commerce, that means a product page can still load quickly for users and search engines while the truly dynamic fragments remain fresh.

That is a much more scalable model than forcing every route into either “fully static” or “fully dynamic.”

A Note on Webhooks and Operational Ownership

Caching only works well if invalidation is treated as part of the architecture, not as an afterthought.

If your team updates product data in Medusa, content in a CMS, or prices in an ERP, your storefront should know how those changes trigger cache invalidation.

Typical patterns include:

Medusa webhooks invalidating product and offer tags
CMS publish events invalidating page or content tags
ERP sync jobs invalidating pricing-related tags
stock update events invalidating inventory-related tags only

This is where many storefronts move from “fast in development” to “reliable in production.”

Final Thoughts

The real value of caching in Next.js 16 is not that it makes everything static. It is that it helps you decide what should be shared, what should be refreshed, and what should remain private.

For e-commerce teams, that distinction matters more than raw speed numbers. The fastest storefront is not the one that caches the most. It is the one that caches the right things.

If you separate stable catalog content, shared operational data, and user-specific state into different cache boundaries, you can keep your storefront fast without risking stale or incorrect business data.

Kubernetes HPA Scale to Zero Without KEDA: Native Autoscaling for Idle Workloads

Daniel Kraszewski — Wed, 27 May 2026 10:00:00 +0000

If you run queue processors, batch workers, or event-driven workloads that sit idle for hours between bursts, you're paying for compute you don't need. Kubernetes HPA can scale these deployments to zero replicas — no KEDA, no Knative, no external controllers required. You need one feature gate, an external metrics source, and about twenty minutes of setup. When the queue is empty your pods disappear, and if you pair this with cluster autoscaler, the nodes disappear too. Real scale-to-zero, using nothing but native Kubernetes primitives.

Quick Reference

Requirement	Detail
Feature gate	`HPAScaleToZero=true` (available since Kubernetes 1.16)
Minimum replicas	`minReplicas: 0` in the HPA spec
Metrics source	Must use external or custom metrics (not CPU/memory)
Scale-to-zero trigger	Metric value drops to zero
Scale-from-zero trigger	Metric value rises above zero
Why not CPU/memory?	No pods means no resource metrics to observe — the HPA controller needs a signal that exists independently of the pods

Local Setup with kind

Since HPAScaleToZero requires an explicit feature gate, we need a cluster that has it enabled on both the API server and the controller manager. kind makes this straightforward — especially on Kubernetes 1.36, which at the time of writing is too new for most managed providers.

If you haven't already built the node image:

kind build node-image --type release v1.36.0

Create a cluster config that enables the feature gate:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  HPAScaleToZero: true
nodes:
  - role: control-plane
  - role: worker

kind create cluster --name hpa-scale-to-zero --config kind-config.yaml --image kindest/node:v1.36.0

The featureGates field at the cluster level propagates the flag to all relevant components (kube-apiserver, kube-controller-manager, kubelet). No need to manually patch component configs.

The Metrics Pipeline

The HPA controller can't scale from zero based on CPU or memory — there are no pods to measure. You need a metric that exists outside the workload itself. For queue-based workloads, the natural choice is the queue length: how many items are waiting to be processed.

We'll use Redis as the queue (specifically a Redis list), expose its length via a redis-exporter sidecar, scrape it with Prometheus, and surface it to the Kubernetes metrics API through prometheus-adapter. This is the same pipeline pattern you'd use in production — the only difference is we're running everything in a single kind cluster.

Deploy Redis with an Exporter

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install redis bitnami/redis \
  --set architecture=standalone \
  --set auth.enabled=false \
  --set metrics.enabled=true \
  --set metrics.extraArgs.check-keys=work-queue

The metrics.enabled=true deploys a redis-exporter sidecar alongside Redis. The --check-keys work-queue argument tells the exporter to emit redis_key_size{key="work-queue"} — the length of our Redis list. That's the metric we'll use to drive the HPA. The chart also sets up the correct Prometheus scrape annotations automatically.

Deploy Prometheus and the Adapter

The prometheus-community Helm charts get us running in two commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm install prometheus prometheus-community/prometheus \
  --set server.service.type=ClusterIP \
  --set alertmanager.enabled=false \
  --set kube-state-metrics.enabled=false \
  --set prometheus-node-exporter.enabled=false \
  --set prometheus-pushgateway.enabled=false

This gives us a minimal Prometheus that auto-discovers pods annotated with prometheus.io/scrape: "true" — which the Bitnami Redis chart already configures.

Now deploy the prometheus-adapter with a custom external metrics rule:

helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --set prometheus.url="http://prometheus-server" \
  --set prometheus.port=80 \
  --set-json 'rules.external=[{"seriesQuery":"{__name__=\"redis_key_size\",key=\"work-queue\"}","metricsQuery":"sum(<<.Series>>{<<.LabelMatchers>>})","name":{"as":"redis_queue_length"},"resources":{"namespaced":false}}]'

This rule tells the adapter: take the redis_key_size metric where key="work-queue", expose it as an external metric called redis_queue_length, and don't filter by namespace (since the queue is a cluster-wide resource — there's only one Redis).

After a minute or so, verify the external metric is available:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/redis_queue_length" | jq .

You should see a response with a value of 0 (since the queue is empty).

The Worker and HPA

The worker is deliberately simple — a shell script that pops items from the Redis list and "processes" them by sleeping for a second. In production this would be your actual consumer code.

Important: start the deployment with replicas: 1, not zero. The HPA controller uses a ScaledToZero condition internally — it only scales from zero if it was the one that previously scaled the workload to zero. A deployment that starts at zero replicas with a fresh HPA will never scale up. Let the HPA handle the initial scale-down to zero on its own.

# worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: queue-worker
spec:
  replicas: 1  # start at 1 — the HPA will scale to zero once metrics are available
  selector:
    matchLabels:
      app: queue-worker
  template:
    metadata:
      labels:
        app: queue-worker
    spec:
      containers:
        - name: worker
          image: redis:7
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                item=$(redis-cli -h redis-master BRPOP work-queue 30)
                if [ -n "$item" ]; then
                  echo "Processing: $item"
                  sleep 1  # simulate work
                fi
              done

Now the HPA that scales this deployment between 0 and 10 replicas based on the queue length:

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0  # this is what HPAScaleToZero enables
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: redis_queue_length
        target:
          type: Value
          value: "5"  # one pod per 5 queue items
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15

The target.value: "5" means the HPA will try to maintain a ratio of one pod per five pending items. When the queue has 20 items, you get 4 pods. When it drops to zero, you get zero pods. The behavior.scaleDown section shortens the stabilization window from the default 5 minutes to 60 seconds — useful for demos and workloads where you want faster scale-to-zero.

Seeing It in Action

Apply the worker and HPA:

kubectl apply -f worker.yaml -f hpa.yaml

Open a watch in one terminal:

kubectl get hpa queue-worker -w

Since the queue is empty and the deployment starts at 1 replica, the HPA will first scale it down to zero. This initial scale-down is critical — it sets the internal ScaledToZero condition that enables future scale-from-zero behavior. After about a minute (our shortened stabilization window), you'll see replicas drop to 0.

Now push 20 items into the queue:

kubectl exec redis-master-0 -c redis -- sh -c '
  for i in $(seq 1 20); do
    redis-cli LPUSH work-queue "job-$i"
  done
'

Within 30-60 seconds (Prometheus scrape interval + adapter refresh + HPA sync period), you'll see the HPA scale the deployment from 0 to 4 replicas. The workers will process the items — one per second per worker — and as the queue drains, the replica count drops. Once the queue is empty and the metric reads zero, the HPA scales the deployment back to zero replicas. Pods gone.

Here's what the kubectl get hpa queue-worker -w output looks like over the full cycle:

NAME           REFERENCE                 TARGETS   MINPODS   MAXPODS   REPLICAS
queue-worker   Deployment/queue-worker   0/5       0         10        1
queue-worker   Deployment/queue-worker   0/5       0         10        0
queue-worker   Deployment/queue-worker   20/5      0         10        0
queue-worker   Deployment/queue-worker   20/5      0         10        4
queue-worker   Deployment/queue-worker   8/5       0         10        4
queue-worker   Deployment/queue-worker   0/5       0         10        2
queue-worker   Deployment/queue-worker   0/5       0         10        0

Push items again — the HPA scales from zero immediately, no manual intervention needed. The cycle repeats indefinitely.

Cluster Autoscaler Integration

When a deployment scales to zero, its pods vanish. If those pods were the only workload on a node, the node becomes empty. Cluster autoscaler (or any node lifecycle manager) will notice the idle node and remove it after its configurable cool-down period — typically 10 minutes.

This is where the real cost savings live. You're not just saving pod-level resources; you're saving entire node-hours. For a workload that processes a queue for two hours a day and sits idle for twenty-two, you go from paying for 24 node-hours to paying for roughly 2.5 (accounting for scale-up overhead).

The cold-start chain matters. When an item hits the queue after a period of silence, here's what happens in sequence:

Prometheus scrapes the redis-exporter (scrape interval, typically 15s)
Prometheus-adapter picks up the new value (adapter refresh, typically 30s)
HPA sync loop fires (default 15s) and decides to scale up
Scheduler assigns the pod to a node — if no nodes are available, cluster autoscaler provisions one (60-120s for cloud providers)
Kubelet pulls the container image and starts the pod
Pod becomes Ready

In a warm cluster (nodes already present), steps 1-5 take about 30-60 seconds total. When a new node must be provisioned, add 1-2 minutes. For workloads where a minute or two of latency is unacceptable, consider these mitigations:

Smaller container images reduce pull time. Distroless or scratch-based images start faster.
Priority-based overprovisioning: deploy a low-priority "placeholder" pod that reserves a node. When real work arrives, the placeholder gets preempted and the worker starts immediately — no node provisioning needed.
Shorter scrape intervals on Prometheus reduce the detection lag.

For batch and queue workloads, the 30-60 second cold start is typically fine. You're processing a backlog, not serving interactive requests.

When to Use This vs. KEDA

KEDA is the most popular alternative for event-driven autoscaling in Kubernetes. It supports 60+ scalers (Redis, SQS, RabbitMQ, Kafka, Azure Queue, and many more), handles the metrics plumbing internally, and can scale to zero without any feature gate.

Use native HPA scale-to-zero when:

You already have Prometheus and a metrics adapter deployed
You want fewer moving parts and no additional CRDs
Your scaling logic is simple (one metric, linear scaling)
You prefer to stay within the boundaries of native Kubernetes APIs

Use KEDA when:

You don't have Prometheus or don't want to maintain the adapter
You need advanced scaling logic (multiple triggers, complex cooldowns, cron-based scaling)
You want out-of-the-box support for dozens of event sources without manual adapter configuration
You need ScaledJobs (scaling Kubernetes Jobs rather than Deployments)

There's no shame in either choice. KEDA is excellent software. The native approach is simpler when your infrastructure already includes Prometheus — you're just connecting pieces that already exist.

Gotchas

You cannot use CPU or memory metrics to scale from zero. This is the most common mistake. When replica count is zero, there are no pods, so there's no CPU or memory to measure. The HPA controller requires at least one non-resource metric (external or custom) to make scaling decisions at zero replicas. If you configure only resource metrics, the HPA will refuse to scale from zero.

The deployment must not start at zero replicas. This is the subtlest gotcha and I haven't seen it documented clearly anywhere. The HPA controller maintains a ScaledToZero condition internally. It only scales FROM zero if it was the one that previously scaled the workload TO zero. If you deploy with replicas: 0 and create a fresh HPA, the controller sees zero replicas but no ScaledToZero condition, and sets ScalingActive: False with reason ScalingDisabled. The fix is simple: start with replicas: 1 and let the HPA scale it down naturally. Once the HPA has performed that initial scale-to-zero, the ScaledToZero condition is set to True and all subsequent scale-from-zero operations work correctly.

The stabilization window still applies at zero. By default, the HPA waits 5 minutes of sustained low metrics before scaling down. This applies to the transition from 1 to 0 as well. If you want faster scale-to-zero, configure the behavior.scaleDown section with a shorter stabilizationWindowSeconds.

The feature gate is alpha — but don't let that scare you. HPAScaleToZero has been available since Kubernetes 1.16 without breaking changes. It hasn't graduated to beta primarily due to KEP process inertia, not because of instability. The implementation is a small conditional in the HPA controller that allows minReplicas: 0 when the feature gate is enabled. It's been running in production clusters for years.

PodDisruptionBudgets at zero replicas. A PDB with minAvailable: 1 on a deployment that's at zero replicas won't prevent the scale-to-zero — PDBs apply to voluntary eviction of running pods, and scaling down is a different path. However, consider what happens if you have a PDB and scale from zero to one: the single pod is protected by the PDB immediately upon becoming ready.

Wrapping Up

The combination of native HPA scale-to-zero, external metrics from Prometheus, and cluster autoscaler gives you genuine pay-for-what-you-use economics on idle workloads. No third-party controllers, no additional CRDs, no vendor lock-in. The feature gate has been stable for twenty releases. Enable it, point your HPA at a queue metric, set minReplicas: 0, and stop paying for compute that's doing nothing.

The KEP tracking this feature is KEP-2021: HPA Scale to Zero — if you want to see it graduate to beta and become the default, that's the issue to watch and engage with.

AWS CloudFront Cache Policies: Complete Guide

Maciej Łopalewski — Wed, 20 May 2026 11:30:24 +0000

A CloudFront cache policy controls two things: the cache key (which combination of URL, headers, cookies, and query strings makes a request unique) and the TTL (how long CloudFront keeps an object at the edge before re-checking the origin). Those two settings together determine your cache hit ratio.

The CloudFront console currently shows fifteen managed cache policies. Five are broadly useful on a typical self-managed distribution: CachingOptimized, CachingOptimizedForUncompressedObjects, CachingDisabled, UseOriginCacheControlHeaders, and UseOriginCacheControlHeaders-QueryStrings. One (Elemental-MediaPackage) is for AWS Elemental MediaPackage video origins. The remaining nine are Amplify-related — a standalone Amplify policy for Amplify origins plus eight Amplify-* policies that Amplify Hosting attaches to its own distributions automatically. You can also build a custom policy when none of the managed ones fit the shape of your traffic.

This is a follow-up to my earlier post on CloudFront's three policy types, going one level deeper on just cache policies.

What a cache policy controls

A cache policy has three categories of settings: policy information (name and description, just metadata), TTL settings (Minimum, Maximum, Default), and cache key settings (which headers, cookies, and query strings to include). Everything that affects caching behavior lives in the latter two.

The cache key is the unique identifier CloudFront uses to look up an object at an edge location. If two viewer requests produce the same cache key, the second one is a cache hit. If they differ — say, one request includes a ?ref=twitter query string and the other does not, and your policy includes query strings in the cache key — they get treated as separate objects, even when the response body is identical. Cache key shape is the single biggest lever for hit ratio.

The TTL settings work alongside Cache-Control and Expires headers from your origin to determine how long a cached object stays valid at the edge. They behave subtly differently from each other; more on that next.

How Minimum, Maximum, and Default TTL actually work

The three TTL settings are not redundant. Each one applies in a different scenario, and getting them confused is one of the more common ways to either over-cache stale content or hammer your origin unnecessarily.

Default TTL is used when your origin sends no Cache-Control or Expires header at all. CloudFront falls back to this value, subject to the Minimum TTL floor — if Minimum TTL is greater than Default TTL, CloudFront caches for at least the Minimum TTL. The default for the AWS managed CachingOptimized policy is 86,400 seconds (24 hours).

Maximum TTL caps the TTL when your origin does send Cache-Control or Expires. If your origin says Cache-Control: max-age=5184000 (60 days) and Maximum TTL is 31,536,000 (365 days), CloudFront honors the origin's 60 days. If your origin says max-age=63072000 (730 days), CloudFront caps it at 365 days. This setting only matters when you want to override an origin that's claiming overly aggressive cache durations.

One nuance worth knowing: if your origin sends both max-age and s-maxage, CloudFront uses s-maxage for its own caching decisions and lets browsers use max-age. This is how you get different cache durations at the edge versus the browser without writing two policies.

Minimum TTL is the floor. CloudFront keeps the object for at least this long, no matter what. This one comes with a sharp edge: if Minimum TTL is greater than 0, CloudFront ignores Cache-Control: no-cache, no-store, and private directives from your origin. The object gets cached anyway, for at least the Minimum TTL duration. Both CachingOptimized and CachingOptimizedForUncompressedObjects have Minimum TTL of 1 second, and the standalone Amplify policy has Minimum TTL of 2 seconds — meaning you cannot reliably stop caching with origin headers alone if you are using them.

If you set all three TTLs to 0, caching is effectively disabled — which is exactly what CachingDisabled does.

Cache key settings: headers, cookies, query strings, and compression

Each of the three viewer-data sources (headers, cookies, query strings) can be configured independently. Cookies and query strings have four modes: include none, include all, include specifically named ones, or include all-except-named-ones. Headers are the exception — they support only "none" or a specific list. There is no "all headers" option, because that would be unbounded and would fragment the cache catastrophically.

When you include a header, cookie, or query string in the cache key, CloudFront also automatically forwards it to the origin on cache misses. The cache key and the origin request are coupled by default; the only way to forward something to the origin without affecting the cache key is to add it to a separate origin request policy. This is the most common confusion with cache policies and the reason CloudFront split them apart in the first place.

A subtle but important detail: cache key matching uses header, cookie, and query string names, but the matching is on the full name+value. Specifying session_id in the cache key means every distinct value of session_id produces a different cache key — so if every visitor has a unique session ID, every request is a cache miss. This is why "include all cookies" rarely makes sense for general traffic.

The compression settings (EnableAcceptEncodingGzip, EnableAcceptEncodingBrotli) tell CloudFront to normalize the Accept-Encoding header before adding it to the cache key. With both enabled, the cache key sees one of br,gzip, gzip, or br (depending on what the viewer supports), or no Accept-Encoding at all when the viewer supports neither — in that last case CloudFront sends Accept-Encoding: identity to the origin instead. Without normalization, every browser variation of Accept-Encoding: gzip, deflate, br, zstd would produce a distinct cache key. Enable this when your origin returns compressed responses or when CloudFront edge compression is on. Leave it off otherwise.

One gotcha: if you enable Gzip or Brotli in the cache policy, do not also include Accept-Encoding in an origin request policy attached to the same behavior. CloudFront handles that header itself when compression is enabled, and adding it to the origin request policy has no effect.

The AWS managed cache policies

This is the comparison table for the policies you'll actually attach to a self-managed distribution by hand. The Amplify Hosting policies (Amplify-* and Amplify-*-V2) are intentionally excluded — see the Amplify section below for why.

Policy	TTL (min/default/max)	Cookies	Query strings	Headers in cache key	Compression
`CachingOptimized`	1s / 24h / 365d	None	None	None	Gzip + Brotli
`CachingOptimizedForUncompressedObjects`	1s / 24h / 365d	None	None	None	Off
`CachingDisabled`	0 / 0 / 0	None	None	None	Off
`UseOriginCacheControlHeaders`	0 / 0 / 365d	All	None	Host, Origin, method overrides	Gzip + Brotli
`UseOriginCacheControlHeaders-QueryStrings`	0 / 0 / 365d	All	All	Host, Origin, method overrides	Gzip + Brotli
`Elemental-MediaPackage`	0 / 24h / 365d	None	aws.manifestfilter, start, end, m	Origin	Gzip

CachingOptimized

ID: 658327ea-f89d-4fab-a63d-7e88639e58f6

The default choice for static content — S3 buckets, image assets, JS and CSS bundles, anything that does not change based on who is asking. The cache key is just the requested object plus the normalized Accept-Encoding. No headers, no cookies, no query strings. This produces the highest possible cache hit ratio for static content.

Watch the Minimum TTL of 1 second: even with Cache-Control: no-store on your origin, CloudFront holds the object for at least one second. That is usually fine, but if you are using this policy on something where origin-side cache busting needs to take effect immediately, swap to UseOriginCacheControlHeaders or CachingDisabled.

CachingOptimizedForUncompressedObjects

ID: b2884449-e4de-46a7-ac36-70bc7f1ddd6d

Identical to CachingOptimized except compression is off. Use this when your origin does not return Gzip or Brotli (raw binary files, video segments, pre-compressed media formats like MP4 or WebP) and you are not using CloudFront edge compression. With compression off, the Accept-Encoding header is excluded from the cache key entirely, which keeps things simple.

If you cannot decide between this and CachingOptimized, default to CachingOptimized. The compression setting only causes problems when your origin produces objects that do not benefit from compression — and even then, it is a minor inefficiency rather than a correctness bug.

CachingDisabled

ID: 4135ea2d-6df8-44a3-9df3-4b5a84be39ad

All TTLs at 0. Nothing in the cache key. Every request goes straight through to the origin. Use this for API behaviors, dynamic GET endpoints that should never be cached, real-time data, WebSockets, and anywhere caching would be incorrect rather than just suboptimal. (POST, PUT, and DELETE aren't cached by CloudFront in the first place — only GET, HEAD, and optionally OPTIONS are — so attaching CachingDisabled to those methods is more about being explicit than functionally necessary.)

UseOriginCacheControlHeaders, UseOriginCacheControlHeaders-QueryStrings, and Elemental-MediaPackage also have Minimum TTL of 0, so they too will respect Cache-Control: no-store from your origin. The difference is that CachingDisabled never caches anything regardless of origin headers, while the others cache when the origin tells them to.

UseOriginCacheControlHeaders

ID: 83da9c7e-98b4-4e11-a168-04f0df8e2c65

Defers caching duration to your origin's Cache-Control and Expires headers. If the origin says max-age=600, CloudFront caches for 10 minutes. If the origin says no-store, CloudFront does not cache at all (because Minimum TTL is 0).

This is the right choice for CMS-backed sites, mixed static-and-dynamic apps, and anywhere your application code already knows what should be cached and for how long. WordPress, Drupal, server-rendered Next.js, and most traditional web apps fit here.

The cache key includes all cookies plus Host, Origin, and three method-override headers (X-HTTP-Method-Override, X-HTTP-Method, X-Method-Override). The cookie inclusion is significant: if your application sets per-user session cookies on every response, you will fragment the cache per user. Either avoid setting cookies on cacheable responses, or stick with CachingOptimized for paths that should be shared across users.

UseOriginCacheControlHeaders-QueryStrings

ID: 4cc15a8a-d715-48a4-82b8-cc0b614638fe

Same as UseOriginCacheControlHeaders but with all query strings included in the cache key. Use this when your origin returns different responses based on query string values — search results, filtered listings, paginated content — and you want CloudFront to cache each variant separately.

The trade-off is cache fragmentation. URLs that include UTM parameters, click IDs, or other tracking junk get cached independently, which both lowers your hit ratio and bloats CloudFront's storage of your content. If you can strip those server-side or with a CloudFront Function before this policy applies, do.

Amplify (and the eight related Amplify policies)

ID: 2e54312d-136d-493c-8eb9-b001f22f67d2

Two distinct things share the Amplify name. The standalone Amplify policy is documented as a regular CloudFront managed policy designed for use with an Amplify web app origin — Min TTL 2 seconds, Max TTL 600 seconds (10 minutes), Default TTL 2 seconds, with Authorization, CloudFront-Viewer-Country, and Host in the cache key plus all cookies and all query strings. AWS doesn't warn against using it; it's just narrowly tuned for Amplify-shaped workloads. The 2-second Minimum TTL means even no-store responses get cached briefly, which is rarely what you want outside of that specific architecture.

The eight Amplify-* policies are something else entirely: Amplify-Default, Amplify-DefaultNoCookies, Amplify-ImageOptimization, Amplify-StaticContent, plus a -V2 variant of each. These are managed by Amplify Hosting itself — Amplify attaches them to the distributions it provisions and resets them on every deploy. AWS explicitly says "we don't recommend that you use these policies for your distributions." If you need similar cache key shapes for a non-Amplify app, copy the settings into a custom policy.

The V2 variants appear to be tied to the August 2024 Amplify Hosting caching overhaul, which raised default static asset cache duration from 2 seconds to 1 year and Maximum TTL from 10 minutes to 1 year. AWS hasn't documented the V2 policies in the public CloudFront developer guide as of this writing — they're visible in the console but the documentation only covers the four originals.

Elemental-MediaPackage

ID: 08627262-05a9-4f76-9ded-b50ca2e3a84f

For AWS Elemental MediaPackage origins specifically — HLS and DASH video streaming. The cache key includes the four query string parameters that MediaPackage actually uses for manifest filtering and time-shifted playback (aws.manifestfilter, start, end, m) plus the Origin header for CORS. Other query strings are excluded, which keeps the cache key tight even when player libraries append cache-busting noise.

If you are not using MediaPackage, do not use this policy. If you are, use it — it is tuned for the specific request shape MediaPackage produces.

How to choose a cache policy

Match your origin and content type to the policy that is already tuned for it.

Use case	Recommended policy
S3 static site, image bucket, asset CDN	`CachingOptimized`
Video files (MP4, WebP), pre-compressed binaries	`CachingOptimizedForUncompressedObjects`
API endpoints, dynamic GET responses, real-time data	`CachingDisabled`
WordPress, Drupal, server-rendered apps with origin Cache-Control	`UseOriginCacheControlHeaders`
Same as above, but query strings affect the response	`UseOriginCacheControlHeaders-QueryStrings`
AWS Elemental MediaPackage HLS/DASH origin	`Elemental-MediaPackage`
Custom requirements that do not match any of the above	Custom cache policy

For a single distribution serving multiple content types, you typically attach different policies to different cache behaviors. A common setup: CachingOptimized on the default behavior (static assets), CachingDisabled on /api/*, UseOriginCacheControlHeaders on /blog/* if blog pages set their own Cache-Control.

When to create a custom cache policy

Reach for a custom policy when you need to include something specific in the cache key that the managed policies do not cover. The most common reasons are device-type segmentation using CloudFront-Is-Mobile-Viewer (serving different markup to phones versus desktops), country-based variation using CloudFront-Viewer-Country (geo-targeted content where edge handling beats origin-side detection), language negotiation via Accept-Language, and tier-based content where a coarse-grained auth bucket determines which response to serve.

Here is a Terraform example for a custom policy that varies on viewer country and a coarse auth-tier cookie:

resource "aws_cloudfront_cache_policy" "country_and_tier" {
  name        = "country-and-tier"
  default_ttl = 3600       # 1 hour when origin sends no Cache-Control
  max_ttl     = 86400      # cap origin Cache-Control at 24 hours
  min_ttl     = 0          # respect origin no-store

  parameters_in_cache_key_and_forwarded_to_origin {
    # normalize Accept-Encoding so cache key collapses across browsers
    enable_accept_encoding_gzip   = true
    enable_accept_encoding_brotli = true

    headers_config {
      header_behavior = "whitelist"
      headers {
        items = ["CloudFront-Viewer-Country"]  # geo-target at the edge
      }
    }

    cookies_config {
      cookie_behavior = "whitelist"
      cookies {
        # coarse bucket like anonymous/free/premium — NOT a per-user session ID
        items = ["auth_tier"]
      }
    }

    query_strings_config {
      query_string_behavior = "none"
    }
  }
}

The cookie name matters here. auth_tier with values like anonymous, free, and premium produces three cache variants per country, which is reasonable. A per-user session token in the same slot would produce one cache variant per user, which is the cache-buster pattern the article warns against earlier.

A few things to know when going custom. CloudFront-generated headers like CloudFront-Viewer-Country are not sent to the origin by default — you need to either include them in the cache key (which automatically forwards them to the origin) or add them to an origin request policy if you want them at the origin without varying the cache. The cache compression toggle should match what your origin returns; turning it on when your origin does not compress can fragment the cache without benefit. And keep Minimum TTL at 0 unless you specifically want to override origin no-store directives, because the surprise-caching behavior burns a lot of debugging hours when you forget about it.

Common pitfalls

Cache hit ratio tanks after a policy change. Almost always means the cache key got too specific. Check whether the new policy includes cookies or query strings the previous one did not, and re-check whether your origin sets per-user cookies on responses that should be shared across users.

Origin keeps getting hit despite a long Default TTL. Default TTL only applies when the origin sends no Cache-Control or Expires header. If your origin sets Cache-Control: max-age=60, CloudFront uses 60 seconds, not the policy's Default TTL. Either change the origin or use Maximum TTL to cap origin-controlled durations.

no-cache from origin is being ignored. Several managed policies (CachingOptimized, CachingOptimizedForUncompressedObjects, and Amplify) have a Minimum TTL greater than 0, which overrides origin no-cache directives. If you need origin-side cache busting to take effect immediately, switch to CachingDisabled or build a custom policy with Minimum TTL = 0.

Forwarding CloudFront-Viewer-Country to origin does not work. For CloudFront-generated headers like CloudFront-Viewer-Country, make sure the header is explicitly enabled via a cache policy or an origin request policy — they aren't sent to your origin by default. If it's in the cache key, it's also forwarded to the origin automatically. If you only need it at the origin and don't want to vary the cache, put it in an origin request policy instead. (Note: a few CloudFront-* headers, including CloudFront-Viewer-Address, CloudFront-Viewer-ASN, and the TLS-related ones, can only be added via an origin request policy and not in a cache policy.)

Authorization header is not reaching the origin. CloudFront removes it by default. To forward it, either include Authorization in the cache key with a cache policy, or use an origin request policy that forwards all viewer headers (the managed Managed-AllViewer policy does this). You cannot forward only Authorization via an origin request policy. Be careful: putting Authorization in the cache key creates per-token cache variants and is usually inappropriate for broadly shared cacheable content.

What's next

If you came here looking for "which cache policy should I use," the short answer is CachingOptimized for static content, CachingDisabled for APIs, UseOriginCacheControlHeaders for content where your origin can express its own caching intent. Everything else is tuning at the margins.

The next post in this series goes deep on origin request policies — where the separation between cache key and origin forwarding really earns its keep — followed by response headers policies, which can replace a lot of security middleware with a single CloudFront config.

Secure Access to Private EKS Clusters Without Bastion Hosts Using SSM

Paweł Swiridow — Wed, 13 May 2026 13:00:00 +0000

Accessing Private EKS Clusters Without Losing Your Mind

Locking down your Kubernetes control plane is a basic requirement for any production environment. Exposing the EKS API server to the public internet is just asking for automated scanners to ruin your weekend. However, securing the endpoint creates an operational headache: how do you actually run kubectl when the API is sealed inside a private subnet?

The traditional answer was a bastion host. But managing SSH keys, rotating credentials, and maintaining yet another publicly exposed EC2 instance is tedious. We all know that a "temporary" bastion host spun up on a Friday afternoon will inevitably become a load-bearing production pillar by Monday.

Instead, we can use AWS Systems Manager (SSM) Session Manager. By leveraging the SSM agent already running on your EKS worker nodes, we can securely tunnel our local traffic directly to the private API endpoint without opening inbound ports or managing SSH keys.

The Mechanics of the SSM Tunnel

The flow is straightforward:

Your local machine initiates an SSM port forwarding session targeting a specific EKS worker node.
The SSM session is instructed to forward traffic to a remote host (the private EKS API endpoint URL) on port 443.
You update your kubeconfig to point to localhost on your chosen forwarded port.

Because the worker node is already in the VPC and authorized to talk to the EKS control plane, it acts as a highly secure, identity-aware proxy. Access is governed entirely by IAM, meaning you can audit every connection via CloudTrail.

Prerequisite: IAM Configuration

For this to work, your EKS worker nodes must have the SSM agent installed (the official EKS optimized AMIs have this by default) and the correct IAM permissions.

Here is a Terraform snippet demonstrating how to attach the necessary SSM policy to your existing EKS node IAM role.

# Assumes you already have an aws_iam_role defined for your worker nodes
# named 'eks_node_role'

resource "aws_iam_role_policy_attachment" "ssm_managed_instance_core" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  role       = aws_iam_role.eks_node_role.name
}

# Optional but recommended: Restrict who can start sessions in IAM
resource "aws_iam_policy" "ssm_user_access" {
  name        = "EKS-SSM-Tunnel-Access"
  description = "Allows users to port forward to EKS nodes"
  policy      = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "ssm:StartSession"
        Resource = [
          "arn:aws:ec2:*:*:instance/*",
          "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost"
        ]
        # In a real environment, restrict the instance resource via tags
        Condition = {
          StringEquals = {
            "ssm:resourceTag/eks:cluster-name" = "my-production-cluster"
          }
        }
      }
    ]
  })
}

This ensures your nodes can communicate with the SSM service and restricts which IAM users can actually initiate the tunnel.

Establishing the Tunnel

Once the nodes are registered in SSM, you need a script to extract a valid instance ID, locate the cluster API endpoint, and start the tunnel.

Here is a Bash script you can execute locally to handle the heavy lifting. It requires the AWS CLI and the Session Manager plugin to be installed on your workstation.

#!/bin/bash
set -euo pipefail

CLUSTER_NAME="my-production-cluster"
REGION="us-east-1"
LOCAL_PORT="8443"

# Fetch the private endpoint of the EKS cluster
echo "Fetching EKS endpoint for ${CLUSTER_NAME}..."
EKS_ENDPOINT=$(aws eks describe-cluster \
  --name "${CLUSTER_NAME}" \
  --region "${REGION}" \
  --query "cluster.endpoint" \
  --output text | sed 's/https:\/\///')

# Find an active worker node instance ID using tags
echo "Finding an active worker node..."
INSTANCE_ID=$(aws ec2 describe-instances \
  --region "${REGION}" \
  --filters "Name=tag:eks:cluster-name,Values=${CLUSTER_NAME}" "Name=instance-state-name,Values=running" \
  --query "Reservations[0].Instances[0].InstanceId" \
  --output text)

if [ "$INSTANCE_ID" == "None" ]; then
  echo "Error: No running worker nodes found."
  exit 1
fi

echo "Establishing SSM tunnel through ${INSTANCE_ID} to ${EKS_ENDPOINT}..."
echo "Leave this terminal open. Access EKS via https://localhost:${LOCAL_PORT}"

# Start the port forwarding session
aws ssm start-session \
  --region "${REGION}" \
  --target "${INSTANCE_ID}" \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters "{\"host\":[\"${EKS_ENDPOINT}\"],\"portNumber\":[\"443\"],\"localPortNumber\":[\"${LOCAL_PORT}\"]}"

Run this script, and it will bind localhost:8443 to the private API endpoint.

Updating Kubeconfig

The final step is modifying your local Kubernetes configuration. You cannot simply run aws eks update-kubeconfig and call it a day, because that will configure the private AWS endpoint, which your machine still cannot route to directly.

You need to manually alter the server field for your cluster to point to the local port.

When you port-forward the EKS API server to your local machine, connecting to https://localhost:8443 introduces a new problem. The API server presents a TLS certificate minted for its internal AWS endpoint (e.g., 1234567890ABCDEF.yl4.us-east-1.eks.amazonaws.com), not localhost.

The quick, dirty fix is to add insecure-skip-tls-verify: true to your kubeconfig. But nothing screams "I definitely passed my SOC2 audit" quite like explicitly disabling TLS validation in production. It is the infrastructure equivalent of putting black tape over a check engine light.

Instead of turning off validation, we can instruct kubectl to connect via our local port but validate the TLS certificate against the actual EKS endpoint hostname. We do this by utilizing the tls-server-name parameter.

apiVersion: v1
clusters:
- cluster:
    server: https://localhost:8443
    # Validate the certificate against the real AWS endpoint
    tls-server-name: 1234567890ABCDEF.yl4.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:123456789012:cluster/my-production-cluster
# ... contexts and users remain unchanged

Once saved, kubectl get pods will route securely through the SSM tunnel, across the worker node, and hit the control plane.

Wrap-Up

Relying on SSM port forwarding eliminates the need for VPNs, bastion hosts, and complex routing rules just to run operational commands against an isolated EKS cluster. By utilizing the existing IAM-integrated agent on your worker nodes, you shrink your external attack surface while maintaining strict audit trails for developer access.

DEV Community: u11d

Stop One EKS Node Group From Blocking Scale-Down in Another

Why idle nodes stay running

What Cluster Autoscaler actually tracks

Enable per-node-group cooldowns

Check your binary supports the flag

Make sure one autoscaler owns the ASGs

Scale node groups back up from zero

Diagnose and verify

Legacy: one autoscaler per ASG

What's next

Postgres LISTEN/NOTIFY in Production: Safe Reconnects & SSE

How Postgres LISTEN/NOTIFY works

One connection, many channels

Handling reconnects: capped exponential backoff

The unsubscribe race, solved by ordering, not locking

Callers should never worry about connection state

Wiring it into Server-Sent Events

When this pattern fits

Keycloak at 100% CPU After Deploy: Root Cause and Fix

The Symptom: Recognizing This CPU Spike

Root Cause: Infinispan, JGroups, and the Multicast Problem

The Default: Distributed Caching via Infinispan

Why JGroups Can't Discover Peers

The Result: A Silent, Load-Independent CPU Burn

Why It Happens After Redeploys, Not on First Boot

First Boot: A Clean, Uncontested Singleton

Redeploy or Restart: A Two-Node Race That Never Resolves

How to Fix It: Step-by-Step

The Trade-Offs of KC_CACHE=local

Need High Availability? Use JDBC_PING Instead

KC_CACHE Settings: Quick Reference

Faster Geospatial Animations with PMTiles: Multi-Layer MVT Time Series in OpenLayers

The architecture: single archive, dynamic layers

Complete working example

Step 1 — Per-timestep GeoJSON files

Step 2 — Generate PMTiles with named layers (generate-tiles.sh)

Step 3 — Start the server

Step 4 — Client animation (index.html)

Running the full setup

Conclusion: paint time, not flip it

Argo CD GitLab Authentication: A Complete GitOps-Friendly Setup

How authentication works

Create a GitLab OAuth application

Encrypt the Argo CD secret with SOPS and age

Add KSOPS to the Argo CD repo server

Provide the age private key to Argo CD

Create the KSOPS generator

Reference the generator in Kustomize

Configure Dex in Argo CD

Configure Argo CD RBAC

Apply the configuration

Verify the setup

Troubleshoot common failures

Security notes

Final result

Reciprocal Rank Fusion on free Elasticsearch: licensing, workarounds, and the OpenSearch alternative

What RRF actually does

What Basic actually gets you, and what it doesn't

The licensing timeline

Workaround 1: Implement RRF in your application

Workaround 2: Linear combination with manual normalization

Workaround 3: Switch to OpenSearch

When paying for Enterprise makes sense

What most teams actually need

How to Deploy Payload CMS on AWS Amplify with MongoDB Atlas for Free

Architecture Overview

Prerequisites

Step A: MongoDB Atlas Free Cluster

Create Cluster

Create Database User and Connection String

Configure Network Access

Step B: AWS S3 Bucket and IAM User

Create S3 Bucket

Create IAM User with S3 Permissions

Generate Access Keys

Payload S3 config

Step C: AWS Amplify Deployment

Configure Build Settings

Connect Repository to Amplify

Step 2 — Generate PMTiles with named layers (`generate-tiles.sh`)

Step 4 — Client animation (`index.html`)

Using `revalidateTag` to Keep Data Fresh

Mistake 3: Using `no-store` too broadly