DEV Community: u11d

Faster Geospatial Animations with PMTiles: Multi-Layer MVT Time Series in OpenLayers

uninterrupted — Wed, 08 Jul 2026 06:00:00 +0000

Most people try to animate time series by stacking separate tile sources or flipping CSS styles mid-frame. That causes flicker and a surprising amount of wasted CPU/GPU work. The core takeaway is simple: bake your timesteps into a single vector-tile archive (PMTiles) with per-timestep named layers, load it once, and drive animation by re-styling already-decoded tiles.

I tested this with OpenLayers and noticed the difference right away: playback became smoother and far less CPU-bound once tiles were pre-baked and cached. The hard part is server-side — how you generate, package, and serve tiles matters more than which client library you pick.

When you animate a map by switching between separate tile sources, each frame causes one of two expensive operations: a network fetch of new tiles, or client-side re-decoding and re-styling. Baking timesteps into one archive avoids that by turning each frame into a cheap style re-evaluation on already-decoded features.

The architecture: single archive, dynamic layers

MVT (Mapbox Vector Tile) is a compact protobuf-based vector tile format that encodes geometry and attributes per z/x/y tile. PMTiles packages MVT tiles into a single file for efficient distribution and partial HTTP range retrieval.

The key workflow:

Server side — each timestep is a separate GeoJSON file. tippecanoe joins them into one PMTiles archive, giving each timestep its own named MVT layer (timestep_00, timestep_01, …).
Client side — one VectorTileSource loads the archive once. One VectorTileLayer uses a style function that reads a mutable currentLayerName variable. Advancing a frame means: update the variable, call layer.changed(). OpenLayers re-runs the style function on the already-decoded tile features and repaints — no network request, no re-decode.

Complete working example

!animation.gif

This section contains everything needed to reproduce the demo from scratch: 11 per-timestep GeoJSON files, a tile generation script, and the client HTML.

Step 1 — Per-timestep GeoJSON files

Create 11 files, one per timestep. Each contains a single large polygon with a value property. The temporal identity comes from the file name (and the MVT layer name tippecanoe will assign), not from a property on the feature.

for i in $(seq 0 10); do
  pad=$(printf "%02d" $i)
  value=$((i * 10))
  cat > "square_t${pad}.geojson" << EOF
{
  "type": "FeatureCollection",
  "features": [{
    "type": "Feature",
    "geometry": {
      "type": "Polygon",
      "coordinates": [[[-60,-50],[60,-50],[60,50],[-60,50],[-60,-50]]]
    },
    "properties": { "value": ${value} }
  }]
}
EOF
done

This produces square_t00.geojson (value=0) through square_t10.geojson (value=100).

Step 2 — Generate PMTiles with named layers (`generate-tiles.sh`)

#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

tippecanoe \
  --named-layer=timestep_00:"$SCRIPT_DIR/square_t00.geojson" \
  --named-layer=timestep_01:"$SCRIPT_DIR/square_t01.geojson" \
  --named-layer=timestep_02:"$SCRIPT_DIR/square_t02.geojson" \
  --named-layer=timestep_03:"$SCRIPT_DIR/square_t03.geojson" \
  --named-layer=timestep_04:"$SCRIPT_DIR/square_t04.geojson" \
  --named-layer=timestep_05:"$SCRIPT_DIR/square_t05.geojson" \
  --named-layer=timestep_06:"$SCRIPT_DIR/square_t06.geojson" \
  --named-layer=timestep_07:"$SCRIPT_DIR/square_t07.geojson" \
  --named-layer=timestep_08:"$SCRIPT_DIR/square_t08.geojson" \
  --named-layer=timestep_09:"$SCRIPT_DIR/square_t09.geojson" \
  --named-layer=timestep_10:"$SCRIPT_DIR/square_t10.geojson" \
  -o "$SCRIPT_DIR/square.pmtiles" \
  -z 8 -Z 0 \
  --no-simplification \
  --no-tile-size-limit \
  --no-feature-limit \
  -f

Run with: bash generate-tiles.sh

Step 3 — Start the server

npx serve

Step 4 — Client animation (`index.html`)

The full self-contained HTML file. No build step, no npm install — all dependencies come from esm.sh, which rewrites bare module specifiers (like pbf imported inside ol/format/MVT) to absolute CDN URLs that the browser can resolve.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>MVT Animation</title>
  <link rel="stylesheet" href="<https://cdn.jsdelivr.net/npm/ol@10.9.0/ol.css>" />
  <style>
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body { font-family: monospace; background: #111; color: #eee;
           display: flex; flex-direction: column; height: 100vh; }
    #map  { flex: 1; }
    #hud  { padding: 12px 20px; background: #1a1a1a; border-top: 1px solid #333;
            display: flex; align-items: center; gap: 24px; }
    #hud label { font-size: 13px; color: #aaa; }
    #hud span  { font-size: 16px; font-weight: bold; }
    #timestep-val { color: #7af; }
    #value-val    { color: #fa7; }
    #color-swatch { width: 28px; height: 28px; border-radius: 4px;
                    border: 1px solid #555; display: inline-block; }
    #progress-bar { flex: 1; height: 8px; background: #333; border-radius: 4px; overflow: hidden; }
    #progress-fill { height: 100%; background: linear-gradient(to right, #00f, #f00); border-radius: 4px; }
    #controls { display: flex; gap: 8px; }
    button { background: #333; color: #eee; border: 1px solid #555;
             padding: 5px 14px; border-radius: 4px; cursor: pointer; font-size: 13px; }
    button:hover { background: #444; }
    #loading { position: absolute; inset: 0; display: flex; align-items: center;
               justify-content: center; background: rgba(0,0,0,0.55); color: #fff;
               font-size: 15px; pointer-events: none; z-index: 999; }
    #loading.hidden { display: none; }
  </style>
</head>
<body>
  <div id="map"><div id="loading">Loading tiles…</div></div>
  <div id="hud">
    <div><label>Timestep </label><span id="timestep-val"></span></div>
    <div><label>Value </label><span id="value-val"></span></div>
    <div style="display:flex;align-items:center;gap:8px;">
      <label>Color </label><span id="color-swatch"></span>
    </div>
    <div id="progress-bar"><div id="progress-fill" style="width:0%"></div></div>
    <div id="controls">
      <button id="btn-playpause">Pause</button>
      <button id="btn-reset">Reset</button>
    </div>
  </div>

  <script type="module">
    import Map              from '<https://esm.sh/ol@10.9.0/Map>';
    import View             from '<https://esm.sh/ol@10.9.0/View>';
    import TileLayer        from '<https://esm.sh/ol@10.9.0/layer/Tile>';
    import OSM              from '<https://esm.sh/ol@10.9.0/source/OSM>';
    import VectorTileLayer  from '<https://esm.sh/ol@10.9.0/layer/VectorTile>';
    import VectorTileSource from '<https://esm.sh/ol@10.9.0/source/VectorTile>';
    import MVT              from '<https://esm.sh/ol@10.9.0/format/MVT>';
    import Style            from '<https://esm.sh/ol@10.9.0/style/Style>';
    import Fill             from '<https://esm.sh/ol@10.9.0/style/Fill>';
    import Stroke           from '<https://esm.sh/ol@10.9.0/style/Stroke>';
    import { fromLonLat }   from '<https://esm.sh/ol@10.9.0/proj>';
    import { PMTiles }      from '<https://esm.sh/pmtiles@4.4.1>';

    const LAYER_NAMES = [
      'timestep_00', 'timestep_01', 'timestep_02', 'timestep_03',
      'timestep_04', 'timestep_05', 'timestep_06', 'timestep_07',
      'timestep_08', 'timestep_09', 'timestep_10',
    ];
    const VALUES = LAYER_NAMES.map((_, i) => i * 10);

    const STYLES = LAYER_NAMES.map((_, i) => new Style({
      fill:   new Fill({ color: `rgb(${Math.round(255*i/10)},0,${Math.round(255*(1-i/10))})` }),
      stroke: new Stroke({ color: '#fff', width: 2 }),
    }));

    const archive = new PMTiles('square.pmtiles');

    const source = new VectorTileSource({
      format: new MVT(),
      url: 'square.pmtiles',
      maxZoom: 8,
    });

    let currentLayerIndex = 0;

    const squareLayer = new VectorTileLayer({
      source,
      style: (feature) => {
        return feature.get('layer') === LAYER_NAMES[currentLayerIndex]
          ? STYLES[currentLayerIndex] : null;
      },
    });

    const map = new Map({
      target: 'map',
      layers: [ new TileLayer({ source: new OSM() }), squareLayer ],
      view: new View({ center: fromLonLat([0, 0]), zoom: 2 }),
    });

    const FRAME_MS = 500;

    function animate(frames, onFrame = () => {}) {
      let i = 0;
      let intervalId = setInterval(() => {
        currentLayerIndex = (currentLayerIndex + 1) % frames.length;
        squareLayer.changed();
        onFrame(frames[i]);
        i = (i + 1) % frames.length;
      }, FRAME_MS);

      return {
        pause()  { clearInterval(intervalId); },
        resume() { intervalId = setInterval(() => {
          currentLayerIndex = (currentLayerIndex + 1) % frames.length;
          squareLayer.changed();
          onFrame(frames[i]);
          i = (i + 1) % frames.length;
        }, FRAME_MS); },
        reset()  { i = 0; },
      };
    }

    const timestepEl   = document.getElementById('timestep-val');
    const valueEl      = document.getElementById('value-val');
    const swatchEl     = document.getElementById('color-swatch');
    const progressFill = document.getElementById('progress-fill');

    function onFrame(layerName) {
      const i = LAYER_NAMES.indexOf(layerName);
      timestepEl.textContent    = layerName;
      valueEl.textContent       = VALUES[i];
      swatchEl.style.background = STYLES[i].getFill().getColor();
      progressFill.style.width  = `${(i / (LAYER_NAMES.length - 1)) * 100}%`;
    }

    onFrame(LAYER_NAMES[0]);

    const loadingEl = document.getElementById('loading');
    let player = null;
    let started = false;

    function startWhenReady() {
      if (started) return;
      started = true;
      loadingEl.classList.add('hidden');
      player = animate(LAYER_NAMES, onFrame);
    }

    source.once('tileloadend', startWhenReady);
    map.once('rendercomplete', startWhenReady);
    setTimeout(startWhenReady, 10000);

    let playing = true;
    document.getElementById('btn-playpause').addEventListener('click', (e) => {
      if (!player) return;
      playing = !playing;
      e.target.textContent = playing ? 'Pause' : 'Play';
      playing ? player.resume() : player.pause();
    });
    document.getElementById('btn-reset').addEventListener('click', () => {
      if (!player) return;
      player.reset();
      if (!playing) {
        playing = true;
        document.getElementById('btn-playpause').textContent = 'Pause';
        player.resume();
      }
    });
  </script>
</body>
</html>

Running the full setup

# 1. Install tippecanoe (macOS)
brew install tippecanoe

# 2. Create the 11 per-timestep GeoJSON files
for i in $(seq 0 10); do
  pad=$(printf "%02d" $i); value=$((i * 10))
  echo '{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-60,-50],[60,-50],[60,50],[-60,50],[-60,-50]]]},"properties":{"value":'$value'}}]}' \
    > square_t${pad}.geojson
done

# 3. Generate the PMTiles archive
bash generate-tiles.sh

# 4. Start the server
npx serve

# 5. Open the demo
open <http://localhost:3000/index.html>

Conclusion: paint time, not flip it

Don't stack separate tile sources for time-series animation. Bake timesteps into a single PMTiles archive as named MVT layers, load it once, and drive playback by updating a closure variable and calling layer.changed(). Serve the archive with npx serve. Start animation only after the first tile arrives.

The result: smooth playback from first load, no flicker on hard refresh, no blank map with cache disabled.

Argo CD GitLab Authentication: A Complete GitOps-Friendly Setup

Paweł Swiridow — Wed, 01 Jul 2026 10:00:00 +0000

Argo CD can use GitLab as an authentication provider, so users log in with their GitLab accounts instead of local Argo CD users. That gives you centralized identity, cleaner access control, and one less password database waiting to become a security incident with a logo.

The setup uses OIDC through Dex, stores the GitLab OAuth credentials as encrypted Kubernetes secrets with SOPS and age, and lets Argo CD decrypt them with KSOPS during reconciliation.

How authentication works

Argo CD does not talk to GitLab directly for user login. It uses Dex as an identity broker.

The flow looks like this:

User -> Argo CD -> Dex -> GitLab -> Dex -> Argo CD -> RBAC

The short version:

The user opens Argo CD.
Argo CD redirects the user to Dex.
Dex redirects the user to GitLab.
GitLab authenticates the user and returns OIDC claims.
Argo CD reads those claims and applies RBAC rules.

Authentication tells Argo CD who the user is. RBAC tells Argo CD what damage they are allowed to do. Ideally, not much.

Create a GitLab OAuth application

First, create an OAuth application in GitLab.

In GitLab, go to your user, group, or admin application settings and create a new application with these values:

Name: Argo CD
Redirect URI: https://argocd.example.com/api/dex/callback
Scopes: openid, read_user

GitLab gives you two important values:

Application ID
Secret

In the Argo CD Dex config, these become:

dex.gitlab.clientID: "<gitlab-application-id>"
dex.gitlab.clientSecret: "<gitlab-application-secret>"
dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Do not commit the secret in plain YAML. That is not GitOps. That is just leaving your keys under the doormat and calling it automation.

Encrypt the Argo CD secret with SOPS and age

Use SOPS with age to encrypt the Kubernetes secret before committing it.

Generate an age key if you do not have one yet:

age-keygen -o age.key

Store the public key in .sops.yaml:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1examplepublickeyreplacewithyourrealpublickey

Create argocd-secret.yml:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
type: Opaque
stringData:
  dex.gitlab.clientID: "<gitlab-application-id>"
  dex.gitlab.clientSecret: "<gitlab-application-secret>"
  dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Encrypt it:

sops --encrypt argocd-secret.yml > argocd-secret.sops.yml

Then remove the unencrypted file:

rm argocd-secret.yml

Yes, really remove it. Git has a long memory and an even longer blame history.

Add KSOPS to the Argo CD repo server

Argo CD does not decrypt SOPS files by default. To make this work with Kustomize, install the KSOPS plugin in the Argo CD repo server.

The plugin lives here:

https://github.com/viaduct-ai/kustomize-sops

If you deploy Argo CD with Helm, add this to your values:

repoServer:
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.5.1
      command:
        [
          "/usr/local/bin/ksops",
          "install",
          "--with-kustomize",
          "/custom-tools",
        ]
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

This installs ksops and a Kustomize binary with plugin support into the repo server container.

Provide the age private key to Argo CD

The repo server needs the age private key to decrypt the SOPS file.

Create a Kubernetes secret from your age key:

kubectl -n argocd create secret generic sops-age-key \
  --from-literal=key="$(cat age.key)"

Expose it to the repo server as SOPS_AGE_KEY:

repoServer:
  env:
    - name: SOPS_AGE_KEY
      valueFrom:
        secretKeyRef:
          name: sops-age-key
          key: key
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}

Protect this key properly. Anyone with the private key can decrypt the secrets. Kubernetes secrets are base64 encoded, not magically protected by tiny YAML elves.

Create the KSOPS generator

Create secret-generator.yml:

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: argocd-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - argocd-secret.sops.yml

This tells Kustomize to call ksops and decrypt argocd-secret.sops.yml during the build.

Reference the generator in Kustomize

In kustomization.yml, add the generator:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
  - secret-generator.yml

When Argo CD renders this application, Kustomize runs the KSOPS generator and produces the decrypted argocd-secret resource.

Configure Dex in Argo CD

Now configure Argo CD to use GitLab through Dex.

In Helm values, configure argocd-cm like this:

configs:
  cm:
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
    dex.config: |
      connectors:
        - type: gitlab
          id: gitlab
          name: GitLab
          config:
            baseURL: https://gitlab.example.com
            clientID: $dex.gitlab.clientID
            clientSecret: $dex.gitlab.clientSecret

The $dex.gitlab.* values are loaded from argocd-secret. That is why the secret names must match exactly.

The Kustomize build options matter:

kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

This allows Kustomize to execute the KSOPS plugin.

Do not enable exec plugins for repositories you do not trust. Otherwise, you have invented remote command execution with extra YAML. Security teams love that. Briefly.

Configure Argo CD RBAC

Authentication is not authorization. After GitLab login works, configure Argo CD RBAC.

For simple user mapping by email:

helm:
  valuesObject:
    configs:
      rbac:
        policy.csv: |
          g, alice@example.com, role:admin
          g, bob@example.com, role:readonly
        scopes: "[email,groups]"

This maps:

alice@example.com -> role:admin
bob@example.com -> role:readonly

For larger teams, prefer GitLab groups instead of individual users. User by user RBAC starts cute and ends as spreadsheet archaeology.

Apply the configuration

If Argo CD is managed by another Argo CD application, commit the files and let Argo CD sync them.

If you are applying locally for a test, run:

kustomize build --enable-alpha-plugins --enable-exec . | kubectl apply -f -

Then restart the Argo CD components if needed:

kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd rollout restart deployment argocd-dex-server
kubectl -n argocd rollout restart deployment argocd-repo-server

Verify the setup

Check that the secret exists:

kubectl -n argocd get secret argocd-secret

Check the Dex logs:

kubectl -n argocd logs deploy/argocd-dex-server

Check the Argo CD server logs:

kubectl -n argocd logs deploy/argocd-server

Check the repo server if decryption fails:

kubectl -n argocd logs deploy/argocd-repo-server

Open Argo CD in the browser:

https://argocd.example.com

You should see a GitLab login option. After login, the user should land in Argo CD with the expected role.

Troubleshoot common failures

If GitLab returns a redirect error, check the redirect URI.

It must match exactly:

https://argocd.example.com/api/dex/callback

If login works but RBAC does not, check which claim Argo CD receives. The email in policy.csv must match the user claim.

If the secret does not render, check the repo server pod:

kubectl -n argocd describe pod -l app.kubernetes.io/name=argocd-repo-server

Common causes:

KSOPS is not installed in the repo server.
-enable-alpha-plugins --enable-exec is missing.
The age private key is not available in the repo server environment.
SOPS_AGE_KEY is missing or points to the wrong Kubernetes secret key.
The encrypted file name does not match secret-generator.yml.
The secret keys do not match $dex.gitlab.clientID, $dex.gitlab.clientSecret, or $dex.gitlab.redirectURI.

You can test decryption locally if you have the age key:

SOPS_AGE_KEY="$(cat age.key)" sops --decrypt argocd-secret.sops.yml

Security notes

Keep the boring security parts boring. Boring is good. Boring means nobody is explaining an outage in a meeting with too many directors.

Encrypt the GitLab OAuth secret with SOPS.
Protect the age private key.
Limit who can change Argo CD applications that use Kustomize exec plugins.
Use GitLab groups for larger teams.
Rotate the GitLab OAuth secret periodically.
Audit role:admin assignments.
Avoid giving admin access to users because they asked nicely in Slack.

Final result

With this setup, GitLab becomes the login provider for Argo CD, secrets stay encrypted in Git, and Argo CD RBAC controls what authenticated users can do.

The moving parts are not complicated:

GitLab OAuth app
SOPS encrypted argocd-secret
KSOPS in repo server
Dex GitLab connector
Argo CD RBAC policy

That is a reasonable amount of YAML for centralized authentication. Not small, obviously. This is Kubernetes. We measure configuration in geological layers.

Reciprocal Rank Fusion on free Elasticsearch: licensing, workarounds, and the OpenSearch alternative

Maciej Łopalewski — Wed, 24 Jun 2026 10:59:52 +0000

Reciprocal Rank Fusion (RRF) on Elasticsearch is gated to the Enterprise tier. On the Basic/free tier, querying with the rrf retriever returns a 403:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "current license is non-compliant for [Reciprocal Rank Fusion (RRF)]",
        "license.expired.feature": "Reciprocal Rank Fusion (RRF)"
      }
    ],
    "type": "security_exception",
    "reason": "current license is non-compliant for [Reciprocal Rank Fusion (RRF)]",
    "license.expired.feature": "Reciprocal Rank Fusion (RRF)"
  },
  "status": 403
}

This has held since RRF first appeared as a technical preview in 8.8 (May 2023), through the 8.16 GA release (November 2024), and into the 9.x line. The linear retriever added in 8.18 / 9.0 is gated the same way, also at Enterprise. Three practical workarounds exist: implement RRF yourself in application code, combine BM25 and vector results with a normalized linear combination, or move to OpenSearch - which ships RRF, score normalization (including z-score), and late-interaction reranking by a field under Apache 2.0 license.

What RRF actually does

Reciprocal Rank Fusion combines multiple ranked result sets into a single ranking based on document positions rather than raw scores. The formula is:

score(d) = Σ 1 / (k + rank_i(d))

where rank_i(d) is the document's position in the i-th result list and k is a smoothing constant. The original paper from Cormack, Clarke, and Büttcher (SIGIR 2009) used k=60 and noted the choice was not critical. Their Table 1 shows MAP barely moves across k ∈ [10, 100] - only at the extremes (k=0 or k=500) does the score drop meaningfully. Elastic and OpenSearch both default the RRF rank constant to 60, following the original paper's convention.

The reason RRF matters for hybrid search: BM25 scores are unbounded and query/corpus-dependent, while vector similarity scores live on a different scale and distribution depending on the similarity function and embedding model. Naively summing them lets one method dominate purely from scale. RRF discards the scores entirely and works only with ranks, which sidesteps the normalization problem with no tuning required.

What Basic actually gets you, and what it doesn't

The Elastic subscriptions matrix splits hybrid-search-related features across three paid tiers. Knowing what's behind each wall is more useful than the headline "RRF is paid":

Feature	Basic / free	Platinum	Enterprise
Vector search (kNN)	✓	✓	✓
Standard, kNN, pinned, rescorer retrievers	✓	✓	✓
Similarity functions for vector fields	✓	✓	✓
Synonym management	✓	✓	✓
ELSER (learned sparse encoder)	—	✓	✓
Elastic Rerank	—	✓	✓
Inference API	—	✓	✓
RRF for hybrid search	—	—	✓
Linear, rule, text similarity re-ranker retrievers	—	—	✓
Rank Vectors (for MaxSim)	—	—	✓
DiskBBQ	—	—	✓
Indexing vectors with GPUs	—	—	✓
Query Rules	—	—	✓
Learning to Rank	—	—	✓

Source: https://www.elastic.co/subscriptions

The pattern is clear. Basic gives you the building blocks for client-side hybrid search - kNN runs, BM25 runs, both can be queried separately. Platinum unlocks Elastic-managed inference and ELSER. Enterprise is where the actual modern hybrid search features live: rank fusion, learned sparse rerankers, late interaction, GPU vector indexing, learning to rank.

If you're on Basic and you want hybrid search, you're either reimplementing pieces of Enterprise in application code, switching engines, or starting a trial.

The licensing timeline

The gating has been continuous and has expanded, not relaxed. The August 2024 license change that re-added AGPLv3 to Elasticsearch made the source code open source again but did not change which features the default ELv2 distribution gates behind paid tiers.

Version	Date	What changed
8.4	August 2022	First hybrid search support
8.8	May 2023	RRF added as technical preview, gated
8.14	June 2024	Retrievers framework introduced
Pre-8.16	August 2024	Elastic announced AGPLv3 as an additional source-code license option; feature-tier gating in the default distribution unchanged
8.16	November 2024	RRF and retrievers reach GA, gated to Enterprise
8.18 / 9.0	April 2025	Linear retriever added, also gated to Enterprise
9.0.1 Basic	June 2025	Linear retriever still throws license error in production
Late 2025	September 2025	Per-retriever weights added to RRF retriever (still Enterprise-gated)

Self-managed deployments can start a 30-day trial that gives access to all subscription features, including Enterprise-tier features, for evaluation. Elastic Cloud trials are 14 days. Elastic also publishes a trial extension form that grants one additional 30-day extension on request.

Workaround 1: Implement RRF in your application

Run BM25 and kNN as two separate queries against Elasticsearch Basic, then fuse the result lists in application code. The fusion logic is roughly seven lines of Python:

from collections import defaultdict

def rrf_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Combine multiple ranked lists of document IDs using Reciprocal Rank Fusion."""
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Calling it with a BM25 result list and a kNN result list returns the fused ranking:

bm25_ids = [hit["_id"] for hit in es.search(index=idx, query=match_query)["hits"]["hits"]]
knn_ids = [hit["_id"] for hit in es.search(index=idx, knn=knn_query)["hits"]["hits"]]

fused = rrf_fusion([bm25_ids, knn_ids])
top_10_ids = [doc_id for doc_id, _ in fused[:10]]

The trade-offs are clear. You pay two round trips instead of one, you lose retrievers-specific features like inner_hits and the unified pagination model, and you have to rehydrate the document _source after fusion - typically with an mget call against the top IDs. In return, you get RRF ranking on the Basic tier with no licensing exposure. In many RAG-style systems embedding generation and kNN latency dominate, but the actual breakdown depends on cache state, candidate window size, and deployment topology - measure for your workload before assuming the extra round trip is free.

Workaround 2: Linear combination with manual normalization

The Basic tier still allows two separate queries fused with a weighted sum, as long as the math happens in your application rather than through the gated linear retriever. Min-max normalize both score sets per query - using the min and max from each result list as the range - then combine with a weight α:

from collections import defaultdict

def minmax_normalize(score: float, lo: float, hi: float) -> float:
    rng = hi - lo
    return 1.0 if rng == 0 else (score - lo) / rng

def linear_fusion(bm25_hits: list[dict], knn_hits: list[dict], alpha: float = 0.5):
    scores: dict[str, float] = defaultdict(float)

    if bm25_hits:
        s = [h["_score"] for h in bm25_hits]
        lo, hi = min(s), max(s)
        for h in bm25_hits:
            scores[h["_id"]] += alpha * minmax_normalize(h["_score"], lo, hi)

    if knn_hits:
        s = [h["_score"] for h in knn_hits]
        lo, hi = min(s), max(s)
        for h in knn_hits:
            scores[h["_id"]] += (1 - alpha) * minmax_normalize(h["_score"], lo, hi)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

When you have labeled query data, calibrated linear can beat RRF. Elastic's research on ELSER + BM25 across BEIR reports that around 40 annotated queries are enough for linear combination to start outperforming RRF, and with 300 calibration queries the optimized linear combination achieved a 6% NDCG@10 improvement over ELSER alone - compared to RRF's 1.4% improvement over the same baseline. Without calibration data, RRF is the safer default - it requires no tuning and is far less sensitive to score distribution mismatches.

Workaround 3: Switch to OpenSearch

OpenSearch ships hybrid search and RRF under Apache 2.0 with no feature-tier licensing. The native hybrid query and normalization-processor (min_max, L2) landed in 2.10 (September 25, 2023). Native RRF landed in 2.19 (February 11, 2025). The 3.x line has continued to invest heavily in vector and hybrid search since then.

Version	Date	Hybrid search additions
2.10	September 25, 2023	First `hybrid` query, `normalization-processor` (min_max, L2)
2.19	February 11, 2025	Native RRF (`score-ranker-processor`), pagination support, `hybrid_score_explanation`
3.0	May 6, 2025	z-score normalization, lower bound for min-max, inner hits in hybrid, GPU acceleration for vector index builds (experimental)
3.1	June 24, 2025	GPU acceleration for vector index builds GA, hybrid query performance improvements (up to 65% latency reduction)
3.3	October 14, 2025	Up to 20% faster hybrid for lexical subqueries, `lateInteractionScore` for reranking by a field using externally hosted ColBERT/ColPali models

The trend continued after 3.3: OpenSearch 3.4, released in December 2025, added further vector-search investment such as k-NN memory-optimized search warmup, native FP16 vector scoring, and JDK 25 support. OpenSearch 3.5 followed in February 2026, and 3.6.0 - the project's first long-term support release - was published April 7, 2026. Refer to the OpenSearch version history and downloads page for the current state.

The score-ranker-processor documentation also shows that custom subquery weights are supported via the parameters.weights array on RRF - useful when you want to weight the BM25 and vector legs differently rather than treat them equally. Elasticsearch added per-retriever weights to its RRF retriever in late 2025 as well, but because the RRF retriever itself is Enterprise-gated, this doesn't change the Basic-tier workaround story.

The mapping to Elasticsearch's paid tiers is striking. Several broadly comparable capabilities that are Enterprise-only in Elasticsearch's built-in implementation are open-source in OpenSearch:

Capability	Elasticsearch tier	OpenSearch tier
RRF rank fusion	Enterprise	Apache 2.0 (since 2.19)
Built-in weighted score fusion	Enterprise (`linear` retriever)	Apache 2.0 score-based hybrid normalization/combination via search pipelines (since 2.10)
Multiple normalization methods (min-max, L2, z-score)	Enterprise	Apache 2.0 (z-score in 3.0)
Late-interaction reranking by a field (ColBERT/ColPali workflows)	Enterprise (Rank Vectors)	Apache 2.0 (`lateInteractionScore` in 3.3)
GPU acceleration for vector index builds	Enterprise	Apache 2.0 (3.0 experimental, 3.1 GA)
Learning to Rank	Enterprise	Apache 2.0 (LTR plugin)

If the project is greenfield and does not depend on Elastic-specific features such as ELSER, Elastic Rerank, ES|QL, or the managed Inference API integrations, OpenSearch is the cleanest path. The migration cost from an existing Elasticsearch deployment is non-trivial - index format compatibility, client library differences, and Kibana versus OpenSearch Dashboards differences all add up - but the functional gap between the free tiers has generally widened for hybrid and vector-search use cases.

When paying for Enterprise makes sense

The trial-then-pay path is reasonable when the team needs text_similarity_reranker for semantic reranking with hosted models, the calibrated linear retriever, MaxSim with rank vectors for ColBERT-style retrieval, GPU vector indexing for billion-scale corpora, learning-to-rank pipelines, or production support contracts. None of this comes for free, and reimplementing it is more expensive than the license for organizations with substantial search infrastructure. For teams whose only blocker is RRF specifically, the workarounds above usually win on cost and are straightforward to implement for simple two-leg hybrid retrieval.

What most teams actually need

Most teams that hit the RRF license error don't actually need RRF specifically - they need hybrid search to work. Manual linear combination with normalized scores is often sufficient for a first production hybrid-search implementation on any Elasticsearch version. The core RRF scoring loop is another twenty lines of code on top of that - production deployments will additionally want pagination, deduplication, an mget rehydration step, tie-breaking, and observability around the fused ranking. And for projects starting fresh in 2026, the open-source answer keeps getting better with every OpenSearch release.

Originally published at https://u11d.com on June 3, 2026.

How to Deploy Payload CMS on AWS Amplify with MongoDB Atlas for Free

Michał Miler — Wed, 17 Jun 2026 07:12:00 +0000

Organizations frequently over-invest in hosting infrastructure for low-traffic content sites. A landing page, blog, or small CMS rarely justifies the cost of a constantly-running server—whether VPS, platform-as-a-service, or managed container services. Serverless architectures solve this by shifting from per-minute billing to per-request pricing. When traffic is very low, your infrastructure costs may approach zero.

This guide demonstrates how to deploy Payload CMS with a Next.js frontend on AWS Amplify, AWS S3, and MongoDB Atlas in a way that can stay within free-tier thresholds for a small, low-traffic site. We'll walk through the stack rationale, step-by-step provisioning, caching strategy to minimize compute, and security trade-offs you inherit.

Architecture Overview

This setup combines AWS Amplify Hosting for serverless Next.js SSR, AWS S3 for media storage, MongoDB Atlas for the database, and Payload CMS for content operations. Amplify SSR runs on AWS Lambda, so compute is billed by request and duration rather than by always-on server time; Amplify pricing currently includes 1,000 build minutes, 500,000 SSR requests, 100 GB-hours of SSR duration, 15 GB data transfer out, and 5 GB stored on CDN per month at no cost. S3 keeps uploaded media outside of your app runtime with S3 pricing, while Atlas pricing offers a free M0 cluster with 512 MB storage for small content sites.

This model is especially useful when usage is low or uneven: marketing sites with occasional campaigns, content teams publishing periodically, early-stage products validating demand, and internal tools or software services that are active only for short windows, for example one to two hours per day. In these cases, paying for permanent infrastructure often brings little business value.

If you have sustained high traffic or volatile workloads, benchmark Amplify against Vercel and self-hosted options before committing. Vercel often handles cold starts and p95 latency more predictably for SSR-heavy apps.

Prerequisites

AWS account (sign up for 12 months of AWS Free Tier)
GitHub repository with Next.js and Payload CMS (we will use demo repo for this tutorial)
MongoDB Atlas free account
Node.js 20+ and npm installed locally

Step A: MongoDB Atlas Free Cluster

Create Cluster

Visit Atlas and log in.
Click Build a Cluster and select Free tier (512 MB storage, shared vCPU, no cost).
Choose a cloud provider and region matching your Amplify deployment (we'll use us-east-1 in AWS).
Unselect "Preload sample dataset."
Click Create Deployment. This takes 1–3 minutes.

Create Database User and Connection String

When prompted, create a database user. Use generated password or use your own password rules; copy it immediately—you'll need it to construct the connection string.
Click Choose a connection method and select Drivers.
Copy the connection string (example: mongodb+srv://username:password@cluster.mongodb.net/?appName=payload-cms). Replace <password> with your copied password. This string includes the username; the password is separate for security.
Store this securely—you'll add it to Amplify environment variables later.

Configure Network Access

By default, Atlas limits restrict external connections. For Amplify Lambda functions to reach the database:

Go to Security > Database & Network Access > IP Access List.
Click Add IP Address and enter 0.0.0.0/0 to allow all IPs.
Click Confirm.

Security trade-off: This opens the database to the internet. Authentication via username/password is your primary defense. For production with higher budget, use private network options like AWS PrivateLink or VPC peering.

Your MongoDB cluster is ready. Auto-pause can add a few seconds to the first request after inactivity. In practice, proper Next.js caching usually hides most of this for end users.

Step B: AWS S3 Bucket and IAM User

Payload CMS stores file uploads (images, documents) in backend storage. Lambda limits include payload and response constraints. By uploading directly to S3 and storing S3 URLs in the database, you avoid Lambda serialization bottlenecks. For image optimization, see image guide.

Create S3 Bucket

Log into AWS Console, select the same region where Amplify will be deployed (e.g., us-east-1) and navigate to S3.
Click Create bucket.
Enter a globally unique bucket name (e.g., blog-payloadcms-demo-12345).
Uncheck Block all public access so Amplify and browsers can retrieve media files. Public access applies only to objects with public ACLs or bucket policies; Payload uploads are private by default.
Accept remaining defaults and click Create bucket.

Create IAM User with S3 Permissions

Payload needs AWS credentials to upload and read media objects in your S3 bucket from the running app. Creating a dedicated IAM user gives the app programmatic access without reusing your root account and lets you limit permissions to only the S3 actions your project needs.

Navigate to Identity and Access Management (IAM) > IAM Users.
Click Create user and name it.
Uncheck Provide user access to the AWS Management Console (API access only).
Click Next.
Select Attach policies directly and search for AmazonS3FullAccess
Click Next and Create user.

For production, follow least privilege. Create a custom policy allowing only s3:PutObject, s3:GetObject, and s3:DeleteObject on your specific bucket.

Generate Access Keys

Select the new IAM user and go to Security credentials.
Click Create access key and choose Other.
Copy both the Access Key ID and Secret Access Key. You will not see the secret again. Store these securely (AWS Secrets Manager or a password manager). Never commit credentials to git.

You now have all configuration data needed for S3 integration: S3_BUCKET, S3_REGION, S3_ACCESS_KEY_ID, and S3_SECRET_ACCESS_KEY.

Payload S3 config

In our demonstration repository we used the following S3 plugin configuration in src/payload.config.ts. It utilizes environment variables mentioned above and uses S3 hosting for media collection.

s3Storage({
  collections: {
    media: true,
  },
  config: {
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID || '',
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || '',
    },
    region: process.env.S3_REGION || 'us-east-1',
    endpoint:
      process.env.S3_BUCKET && process.env.S3_REGION
        ? `https://s3.${process.env.S3_REGION}.amazonaws.com`
        : undefined,
    forcePathStyle: true,
  },
  bucket: process.env.S3_BUCKET || '',
  disableLocalStorage: true,
})

Why this setup works well on Amplify:

endpoint + forcePathStyle helps generate direct S3 URLs, which avoids routing large media responses through Lambda.
disableLocalStorage: true ensures uploaded media is persisted only in S3, which is important for stateless serverless runtimes.
Keeping credentials and bucket settings in environment variables makes the same code reusable across local, staging, and production deployments.

Your configuration may differ slightly, especially around endpoint strategy, custom domains, or local fallback behavior.

Step C: AWS Amplify Deployment

Amplify Hosting is a Git-based hosting service that watches your repository, runs your build in an isolated environment, deploys generated artifacts to the CDN and compute layer, and provisions HTTPS on an *.amplifyapp.com domain.

You can configure deployment fully in the Amplify UI, but keeping the build definition in amplify.yml is usually safer for teams: it is versioned, reviewable in pull requests, reproducible across environments, and avoids silent UI drift. This is a lightweight Infrastructure as Code (IaC) approach: the deployment behavior is declared in code, not only clicked in a console. For background, see IaC guide. Build configuration lives in the repository as amplify.yml. This ensures reproducible builds across your team.

Configure Build Settings

Before creating the Amplify app, make sure amplify.yml is committed to the repository root. The file below is our configuration. Yours may differ, especially in package manager commands (npm, pnpm, yarn) and which environment variables must be exported into .env.production.

Use this as a reference and adapt only what your app actually requires:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - echo "Installing dependencies..."
        - npm ci
    build:
      # <https://docs.aws.amazon.com/amplify/latest/userguide/ssr-environment-variables.html#:~:text=Amplify%20Hosting%20supports%20adding%20environment,in%20the%20build%20commands%20section>.
      commands:
        - echo "Building the application..."
        - rm -f .env.production
        - export NEXT_PUBLIC_SERVER_URL="${NEXT_PUBLIC_SERVER_URL:-<https://$>{AWS_BRANCH}.${AWS_APP_ID}.amplifyapp.com}"
        - echo "Using NEXT_PUBLIC_SERVER_URL=$NEXT_PUBLIC_SERVER_URL"
        - env | grep -e '^NEXT_PUBLIC_' >> .env.production || true
        - env | grep -e '^DATABASE_URL=' -e '^PAYLOAD_SECRET=' -e '^CRON_SECRET=' -e '^PREVIEW_SECRET=' -e '^S3_BUCKET=' -e '^S3_REGION=' -e '^S3_ACCESS_KEY_ID=' -e '^S3_SECRET_ACCESS_KEY=' >> .env.production || true
        - npm run build
    postBuild:
      commands:
        - echo "Post build complete"
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - .next/cache/**/*
      - node_modules/**/*

The key pattern is documented in env docs: expose only required variables during build, then write them to .env.production. Keep package manager commands and env exports aligned with your own repository.

Connect Repository to Amplify

Go to Amplify Console.
Make sure you are in the same region where S3 bucket and Mongo Cluster were created
Click Create new app.
Select your Git provider (GitHub in our case) and authorize Amplify to access your repository.
Choose your repository and the branch to deploy (typically main).
Click Next.

Add Environment Variables

During app creation, add environment variables in step 3: App settings -> Advanced settings.

Expand Advanced settings.
Add the same variables shown in the screenshot (retrieved from previous steps).
Continue deployment.

Deploy

Click Next and Save and deploy.
Amplify now builds and deploys your app. First builds take 3–10 minutes.
Once complete, you will see your application URL (e.g., https://myapp-12345.amplifyapp.com).

Verify Admin Access

Navigate to your admin panel (e.g. https://myapp-12345.amplifyapp.com/admin). You should see the Payload login page. Use user docs to create your first admin account.

Fix Media CORS (One-Time Setup)

If images or media files fail to load with CORS errors, your S3 bucket needs CORS configuration:

Go to S3 > Buckets and select your bucket.
Click Permissions and scroll to Cross-origin resource sharing (CORS).
Click Edit and paste:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "HEAD", "PUT"],
    "AllowedOrigins": ["<https://myapp-12345.amplifyapp.com>"],
    "ExposeHeaders": ["ETag", "x-amz-request-id"],
    "MaxAgeSeconds": 3000
  }
]

Replace myapp-12345.amplifyapp.com with your actual Amplify domain. Click Save. Media should now load without errors.

This CORS configuration allows your Amplify domain to read media from S3; it does not expose your bucket publicly.

Caching and Static Generation in Next.js

Next.js caching directly impacts Amplify compute costs. See caching docs.

Static Site Generation (SSG)

For content that changes infrequently (blog posts, landing pages), use Static Site Generation:

// app/posts/[slug]/page.tsx
export const revalidate = 86400 // Revalidate once per day (ISR)

export default async function PostPage({ params }: { params: { slug: string } }) {
  const post = await fetch(
    `${process.env.NEXT_PUBLIC_SERVER_URL}/api/posts?where[slug][equals]=${params.slug}`,
  )
  // ...
}

ISR docs explain how pages are pre-cached and revalidated in the background.

On-Demand Revalidation

When an editor publishes a new post, trigger revalidation immediately:

// `src/hooks/revalidateFrontend.ts` (Payload hook)
import { revalidateTag } from 'next/cache'

export default async function revalidateFrontend(doc) {
  if (!process.env.NEXT_PUBLIC_SERVER_URL) return

  // Tell Next.js to invalidate the blog archive page
  revalidateTag('posts-archive')

  return doc
}

revalidateTag docs cover tag invalidation across cached responses. This pattern helps keep content fresh without full rebuilds.

Cache Strategy Summary

For static pages such as landing pages and blog posts, SSG with a 24-hour ISR window minimizes compute and usually results in only periodic revalidation calls. Dynamic surfaces like search or personalized dashboards are still rendered per request, so they consume more SSR runtime. API routes can remain lightweight and use HTTP caching behavior where applicable. For low traffic, this mix typically stays inside Amplify free-tier limits.

Pricing Notes

AWS Amplify

Amplify pricing is usage-based:

Build minutes: 1,000/month included, then $0.01 per minute
SSR request count: 500,000/month included, then $0.30 per 1 million requests
SSR request duration: 100 GB-hours/month included, then $0.20 per GB-hour
Data transfer out: 15 GB/month included, then $0.15 per GB
CDN storage: 5 GB/month included, then $0.023 per GB-month

AWS S3

S3 pricing is also usage-based. For a small media library, storage and request costs are usually low, but the exact amount depends on object size, request volume, storage class, and data transfer.

MongoDB Atlas

Atlas pricing includes the free M0 cluster: 512 MB storage, shared RAM, shared vCPU, free forever. Atlas limits also note that auto-pause can slow the first request after inactivity.

When to Reconsider

High traffic or tail-latency sensitivity: benchmark Amplify against Vercel and self-hosted options.
Unpredictable spikes: verify how your cache hit rate and SSR load affect request count and duration.
Compliance or data residency: ensure your MongoDB and Amplify regions match your requirements. See AWS regions.

Security Trade-Offs

1. MongoDB IP Allowlist: Public Access

Configuring 0.0.0.0/0 means any IP can attempt to connect to your MongoDB cluster.

Use long random credentials, limit the Mongo user to the minimum database scope, and avoid admin privileges for application access. For stronger isolation in production, use network controls instead of public IP allowlists.

2. S3 Public Access: Bucket Policy and CORS

S3 blocks public access by default. Unchecking this allows you to create a public bucket policy.

Keep policies narrow: allow only public read (s3:GetObject) where needed and never expose write permissions publicly. If files are sensitive, prefer signed URLs with expiration instead of open object access; see storage docs. It also helps to separate public uploads from private objects by prefix and policy.

3. Secrets in Environment Variables

Amplify stores secrets in managed service configuration rather than in your repository. These values are encrypted at rest and should never be printed in build logs.

Keep IAM permissions minimal for deployment roles, avoid logging any secret values in build output, and consider Secrets Manager if you need managed rotation and stronger operational controls.

For a detailed security review, see security guide.

Bonus: Custom Domain Setup

Once your site is live on myapp-12345.amplifyapp.com, you can add a custom domain (e.g., blog.example.com):

In Amplify Console: Go to Domain management and click Add domain.
Point Route 53 DNS: If your domain is in Route 53 (AWS), Amplify auto-creates records. Otherwise, add your registrar's CNAME records.
SSL/TLS: Amplify generates a free certificate via AWS Certificate Manager. HTTPS is automatic.
Costs: Domain registration (~$10–15/year) + Route 53 hosting ($0.50/month) if applicable.

For detailed steps, see domain guide.

Troubleshooting

"502 Bad Gateway" or "413 Payload Too Large"

If media uploads fail, the issue is likely the 6 MB Lambda response limit. Ensure S3_ENDPOINT is configured in your Payload config to use direct S3 URLs, bypassing the Lambda gateway. See config file for the current setup.

"Database Connection Timeout"

Check that:

MongoDB Atlas IP allowlist includes 0.0.0.0/0 or your Amplify IP.
DATABASE_URL is correct and includes the password.
Database user exists and has correct credentials.

Media Not Loading (CORS Error)

Ensure your S3 bucket CORS policy includes your Amplify domain (see CORS step).

Logs and Debugging

Use logs docs to inspect runtime errors, environment variable issues, and database connectivity problems.

Summary and Conclusions

This architecture gives you a practical way to run a Payload CMS website with low baseline cost: Amplify for SSR compute, S3 for media, and MongoDB Atlas for managed data. For low-traffic landing pages and blogs, it can stay within free-tier thresholds when caching is configured correctly and media is served directly from S3.

The key trade-off is operational behavior under low traffic: you may see cold starts and first-request latency, so monitor logs, keep SSR scope minimal, and add warming only when needed. As traffic, compliance, or latency requirements increase, benchmark against Vercel and self-hosted options before scaling the same setup.

Next Steps

Now that your site is deployed, consider:

Optimize images: See image guide.
Reduce cold starts: If you notice slow first loads after inactivity, use warming guide. Vercel usually hides this problem better, especially for p95 latency, so benchmark both if first-response consistency matters.
Monitor costs: Set up AWS Budgets.
Scale content: As you grow, plan for MongoDB upgrade to a paid cluster if storage exceeds 512 MB.

References

Dynamic Email Domain Validation in Keycloak with a Custom Authenticator

Bartek Gałęzowski — Wed, 10 Jun 2026 06:00:00 +0000

Introduction

Keycloak ships with a built-in mechanism for restricting user registration by email domain — but it's static. Changing the allow-list means touching realm configuration and redeploying. For B2B SaaS products that onboard new tenants regularly, that's an operational bottleneck you don't want.

The right solution is to move domain policy out of Keycloak entirely and delegate it to a backend service that can be updated at runtime. This article walks through building a custom Keycloak Authenticator — called domain-email-validator — that does exactly that: at login time, it calls an external API to decide whether the user's email domain is permitted.

By the end, you'll understand the full architecture, the Java implementation, how to wire it into both browser and IDP flows, and the operational tradeoffs involved.

Why Static Domain Restrictions Fall Short in B2B Products

Keycloak's native domain restriction works well for single-tenant deployments with a fixed list of approved domains. But in multi-tenant environments, you typically need:

Per-tenant domain rules — Tenant A allows acme.com; Tenant B allows globex.com and initech.com.
Runtime updates — A new customer signs a contract and needs access today, not after the next deploy.
Consistency across login methods — The same rule should apply whether a user logs in via username/password or via Google SSO.

A custom Authenticator SPI solves all three. It's evaluated on every login attempt, it reads policy from an external source, and it plugs into both browser and post-broker flows with the same implementation.

Architecture Overview

The approach is a thin decision layer inside Keycloak. The plugin does not own domain policy — it only enforces it.

The contract is intentionally minimal:

Keycloak sends { domain, realmId } to a backend endpoint.
Backend returns 200 to allow or any non-200 to deny.
Keycloak continues or blocks the flow accordingly.

This keeps all business logic — tenant configuration, domain lists, auditing — in your backend, where it belongs.

Request Lifecycle

Keycloak executes earlier steps (credential check or broker handshake).
The domain validator step runs.
It reads the user's email from the active context.
It sends a small HTTP POST to the policy service.
The backend returns allow or deny.
Keycloak continues the flow or shows a denial message.

Because the decision is made per login attempt, domain policy changes take effect immediately — no cache warmup, no redeployment.

Implementation

The authenticator is split into two standard Keycloak SPI parts: a factory that registers the provider and exposes configuration fields, and an executor that runs the validation logic at login time.

Factory: Registering the Provider

DomainValidatorAuthenticatorFactory defines two configuration properties visible in the Keycloak Admin UI under the flow step's Config tab:

.property()
    .name(DomainValidatorAuthenticator.CONFIG_VALIDATION_URL)
    .label("Domain Validation URL")
    .type(ProviderConfigProperty.STRING_TYPE)
    .defaultValue("https://my-server/api/keycloak/domain-check")
    .add()
.property()
    .name(DomainValidatorAuthenticator.CONFIG_SHARED_SECRET)
    .label("Shared Secret")
    .type(ProviderConfigProperty.PASSWORD)
    .secret(true)
    .add()

There is no hidden YAML or server-side config file. Everything is per-flow-execution and editable through the UI, which makes it easy to configure different endpoints per realm.

Executor: The Core Validation Logic

The central method is authenticate(AuthenticationFlowContext context). It first resolves the email from context — either from a submitted form or from a brokered identity:

String email = resolveBrokeredEmail(context);
String flowType;

if (email != null) {
    // Post-broker flow: email came from the IDP identity token
    flowType = "idp";
} else {
    // Standard browser flow: read from the submitted form
    MultivaluedMap<String, String> formData =
            context.getHttpRequest().getDecodedFormParameters();
    email = formData.getFirst("username");
    flowType = "form";
}

// If no email or no @ sign, pass through — Keycloak handles invalid credentials
if (email == null || !email.contains("@")) {
    context.success();
    return;
}

Then it reads the flow config and calls the policy service:

Map<String, String> config = context.getAuthenticatorConfig().getConfig();
String validationUrl = config.get(CONFIG_VALIDATION_URL);
String sharedSecret = config.get(CONFIG_SHARED_SECRET);

HttpPost post = new HttpPost(validationUrl);
post.setEntity(new StringEntity(
    String.format("{\"domain\":\"%s\",\"realmId\":\"%s\"}",
        escapeJson(domain), escapeJson(realmId)),
    ContentType.APPLICATION_JSON
));

if (sharedSecret != null && !sharedSecret.isBlank()) {
    post.setHeader("Authorization", "Bearer " + sharedSecret);
}

Finally, it interprets the response:

int statusCode;
try (CloseableHttpResponse response = httpClient.execute(post)) {
    statusCode = response.getStatusLine().getStatusCode();
} catch (IOException e) {
    LOG.errorf(e, "DomainValidatorAuthenticator: HTTP call failed for domain '%s'" +
                    " in realm '%s'", domain, realmId);
    failWithError(context, "domainValidationUnavailable");
    return;
}

if (statusCode == 200) {
    context.success();
} else {
    LOG.infof("DomainValidatorAuthenticator: domain '%s' denied in realm '%s'" +
                " (HTTP %d)", domain, realmId, statusCode);
    failWithError(context, "domainNotAllowed");
}

Any IOException — network failure, timeout, connection refused — results in domainValidationUnavailable. Missing config results in domainValidatorMisconfigured. The validator is fail-closed: when in doubt, it denies.

Resolving the Brokered Email

For post-broker flows (e.g. after Google SSO), the email comes from the already-resolved user object, not the form:

private String resolveBrokeredEmail(AuthenticationFlowContext context) {
    UserModel user = context.getUser();
    if (user != null) {
        String userEmail = user.getEmail();
        if (userEmail != null && !userEmail.isBlank()) {
            return userEmail.trim();
        }
    }
    return null;
}

This means the same authenticator class handles both input sources. The only difference is where the email comes from — the rest of the flow is identical.

Docker Build

The Dockerfile wires the Maven build and the Keycloak image together cleanly:

FROM quay.io/keycloak/keycloak:26.6.0 AS base-keycloak

FROM maven:3.9-eclipse-temurin-21 AS maven-builder
WORKDIR /build
COPY keycloak-domain-validator/pom.xml ./pom.xml
RUN mvn dependency:resolve -q
COPY keycloak-domain-validator/src ./src
RUN mvn package -q -DskipTests

FROM base-keycloak AS keycloak-builder
COPY --from=maven-builder /build/target/domain-validator.jar /opt/keycloak/providers/
RUN /opt/keycloak/bin/kc.sh build

FROM base-keycloak
COPY --from=keycloak-builder /opt/keycloak/ /opt/keycloak/
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]

The multi-stage build keeps the final image lean: Maven is only present during the build stage, and the compiled JAR is dropped directly into Keycloak's providers directory.

Configuring Authentication Flows

1. Browser Login Flow (Username/Password)

Go to Authentication → Flows.

Duplicate the built-in browser flow

Inside the Forms sub-flow, add the Domain Email Validator step.

Place it directly after Username Password Form , and set its requirement to Required.

Click Config on the step and enter your validation URL and shared secret.

Bind the duplicated flow as the active browser flow under Bindings.

The validator reads the email from the submitted username field in the login form.

2. Google / IDP Login Flow (Post-Broker)

Create a new dedicated flow for post-broker login.
Add Domain Email Validator as a Required step.
Go to Identity Providers → Google → Post Login Flow and assign the new flow.

The validator reads the email from the resolved user identity returned by the IDP.

Best Practices

Keep the validation endpoint fast and internal. Since this check is synchronous and blocks login, endpoint latency directly affects user experience. Route it over internal networking and keep the response payload small.

Monitor the endpoint as authentication-critical infrastructure. Set up latency alerts and error rate tracking. A degraded policy service means users can't log in — treat it accordingly.

Rotate the shared secret regularly. The bearer token is your only authorization layer between Keycloak and the policy service. Automate rotation through your secrets management tooling.

Always set the step as Required. An Alternative or Disabled requirement silently bypasses the check, which defeats the purpose entirely.

Ensure Google OAuth includes the email scope. Without it, the IDP flow won't have an email to validate, and the validator will pass through silently.

Keep the API contract minimal and stable. The { domain, realmId } payload and the HTTP status response are a clean, versioned interface. Resist the temptation to add fields over time unless there's a clear reason.

Security Posture

The validator is intentionally conservative by design:

It derives the domain from the server-side authentication context, not from client-supplied input.
It supports a bearer secret for service-to-service authorization.
It denies access when validation cannot be completed for any reason — misconfiguration, network failure, or unexpected response codes all result in denial.

In other words: authentication proceeds only when policy can be positively verified.

Tradeoffs and Alternatives

Approach	Flexibility	Operational cost	Complexity
Keycloak built-in domain restriction	Low (static)	Low	Low
Custom Authenticator + external API	High (dynamic)	Medium	Medium
Keycloak scripting (deprecated)	Medium	Medium	Medium
Custom User Storage SPI	High	High	High

The custom Authenticator approach hits the right balance for most B2B products: it's dynamic, tenant-aware, independently deployable, and doesn't require deep Keycloak internals knowledge to maintain.

The main tradeoff is availability coupling — if your policy service goes down, so does login. Mitigate this with high-availability deployment, circuit breakers at the infrastructure layer, and solid monitoring.

FAQ

Q: What happens if the policy service is unreachable during a login attempt?

The authenticator catches the IOException and fails with domainValidationUnavailable. Login is denied. This is the intentional fail-closed behavior — an unreachable policy service is treated as a denial rather than a bypass.

Q: Can this authenticator be used across multiple realms with different endpoints?

Yes. The validation URL and shared secret are configured per flow execution, not globally. Each realm can have its own duplicated flow pointing to a different endpoint, or the same endpoint can handle per-realm routing using the realmId field in the request body.

Q: Does this affect performance at login time?

It adds one synchronous HTTP call per login attempt. In practice, if the policy service is on the same internal network and kept lightweight, the added latency is negligible — typically single-digit milliseconds. Treat the endpoint as latency-sensitive infrastructure.

Q: What error message does the user see when denied?

The authenticator calls failWithError(context, "domainNotAllowed"), which maps to a message key in Keycloak's theme messages file. You can customize the displayed text by overriding that key in your realm's login theme.

Conclusion

For multi-tenant B2B products, static email domain restrictions in Keycloak simply don't scale. A custom Authenticator SPI that delegates to an external policy service is a clean, maintainable pattern that keeps domain management out of Keycloak configuration and in the hands of your backend team.

Key takeaways:

The plugin is a thin decision layer — it enforces policy but doesn't own it.
The same authenticator class handles both browser and IDP (post-broker) login paths.
The API contract is minimal: { domain, realmId } in, HTTP status out.
The validator is fail-closed — network errors and misconfigurations result in denial, not bypass.
Configuration lives in the Keycloak Admin UI, scoped per flow execution, making it easy to manage per realm.
Treat the policy endpoint as authentication-critical infrastructure: monitor latency, alert on errors, and keep it highly available.

Next.js 16 Caching for E-Commerce: Cache Components, use cache, revalidateTag, and Fresh Product Data

uninterrupted — Tue, 02 Jun 2026 22:00:00 +0000

Caching in e-commerce is never just about speed. A fast storefront is useful only if it still shows the right price, the correct stock level, and the right experience for the current customer.

That is why caching in a Next.js storefront can be deceptively hard. Some data should be shared broadly and kept warm for SEO and performance. Some data should be refreshed often. Some should never be shared between users at all.

Next.js 16 gives teams a much clearer toolbox for solving this problem with Cache Components, use cache, tag-based invalidation, and explicit cache lifetime controls. Used properly, these features let you keep pages fast without drifting into stale commerce data.

In this guide, I will walk through a practical way to think about caching in a modern storefront and show how to combine use cache, cacheLife, and revalidateTag for real e-commerce use cases.

Why Caching Is Harder in E-Commerce Than in a Typical Content Site

On a standard marketing site, most content changes infrequently. If a page is cached for a few minutes or even a few hours, the business impact is usually negligible.

Commerce systems work differently.

The same product page may contain:

stable product descriptions and category copy
semi-dynamic data such as price, availability, shipping estimates, or promotion labels
private data such as cart state, recently viewed items, or customer-specific pricing

Treating all of that data the same way leads to one of two bad outcomes:

You cache too aggressively and serve stale prices or availability.
You disable caching everywhere and lose the performance benefits that help SEO and conversion.

The better approach is to split your data by volatility and audience.

The Three Cache Boundaries That Matter Most

For most commerce projects, the cleanest mental model is to divide data into three groups.

1. Stable catalog content

This is the part of the page that usually changes only when content editors or merchandisers update the catalog.

Examples:

product title and description
brand information
CMS blocks, FAQs, and long-form copy
category landing page copy
SEO metadata

This is the best candidate for shared caching because it improves performance for everyone and is usually safe to invalidate on demand when content changes.

2. Shared but fast-changing commerce data

This data is still shared between many users, but it changes more often and has stronger operational impact.

Examples:

current price for a product and region
inventory status
campaign badges
delivery promises
low-stock signals

This layer often benefits from a shorter cache lifetime and explicit invalidation.

3. User-specific or session-specific data

This is the data that should not be globally cached.

Examples:

cart contents
logged-in customer state
B2B contract pricing
personalized recommendations
account-specific discounts
checkout calculations tied to the active session

This is where teams most often make expensive mistakes. If the output depends on the current user, you need to keep that boundary private.

What Next.js 16 Changes in Practice

The major advantage of Next.js 16 is not just that it caches. It is that caching becomes more explicit.

Instead of treating an entire route as fully static or fully dynamic, you can choose more precise boundaries:

cache a component or function with use cache
assign cache tags to data and invalidate them later
control lifetime with cacheLife
keep per-user data private where needed
stream dynamic parts separately instead of forcing the entire page into request-time rendering

For e-commerce storefronts, that means you can serve a fast shell and still keep critical parts fresh.

A Practical Storefront Strategy

Let us look at a product page. In a typical headless stack, product content might come from Medusa or another commerce backend, while editorial content may come from a CMS.

The important point is not the backend itself. The important point is how you cache each layer.

Cache the stable product content

If product descriptions and media do not change every minute, cache them explicitly and tag them so they can be invalidated when merchandisers update the catalog.

import { cacheTag } from 'next/cache'

async function getProduct(slug: string) {
  'use cache'

  cacheTag(`product:${slug}`)

  const response = await fetch(
    `${process.env.COMMERCE_API_URL}/products/${slug}`
  )

  if (!response.ok) {
    throw new Error('Failed to load product')
  }

  return response.json()
}

This is a good fit for data that should be broadly reusable and revalidated only when a product actually changes.

Use a shorter cache boundary for price and stock

Price and stock need more care. In some stores, they can still be shared safely for a short period. In others, especially B2B or highly promotional environments, they may need user-level or request-level handling.

For shared price or inventory data, a short-lived cache can be a sensible compromise.

import { cacheLife, cacheTag } from 'next/cache'

async function getOffer(productId: string, regionId: string) {
  'use cache'

  cacheTag(`offer:${productId}:${regionId}`)
  cacheLife({ expire: 60 })

  const response = await fetch(
    `${process.env.COMMERCE_API_URL}/offers/${productId}?region=${regionId}`
  )

  if (!response.ok) {
    throw new Error('Failed to load offer')
  }

  return response.json()
}

Here the offer can be shared, but only briefly. That is often enough to reduce backend pressure while still keeping the storefront operationally safe.

Keep customer-specific data out of the shared cache

If pricing, recommendations, or entitlements depend on the active customer, do not treat them as generic shared data.

This applies especially to:

B2B price lists
customer group discounts
partner-only catalog visibility
personalized recommendation widgets

At that point, a private boundary or request-time fetching becomes more appropriate than a shared cache.

The principle is simple: if another customer must never see the same output, that data should not live in a shared cache.

If you do want to cache user-specific data briefly to avoid repeating the same work during a session, keep that cache private instead of shared.

import { cacheLife } from 'next/cache'

async function getRecommendations(productId: string, customerId: string) {
  'use cache: private'

  cacheLife({ expire: 60 })

  const response = await fetch(
    `${process.env.RECOMMENDATION_API_URL}/products/${productId}/recommendations?customer=${customerId}`
  )

  if (!response.ok) {
    throw new Error('Failed to load recommendations')
  }

  return response.json()
}

That keeps the result bounded to the current user context instead of letting it leak into global storefront output. It is also worth noting that 'use cache: private' does not store its cache entries on the server. The function still runs during server rendering, but the cached result is kept only in the browser’s memory and does not survive a full page reload.

Why This Matters for Medusa and Other Headless Commerce Backends

This pattern becomes especially valuable in headless commerce because the storefront is often responsible for combining multiple systems:

commerce backend
CMS
search service
recommendation engine
analytics and experiment layers

If all of those systems are fetched at request time for every page view, the frontend becomes slower and more fragile than it needs to be.

If all of them are aggressively cached together, the storefront becomes fast but operationally unsafe.

The right answer is to cache by business meaning, not by convenience.

For example:

CMS content can usually be cached until editors publish changes.
catalog copy can usually be cached and invalidated via tags.
regional shared pricing can often be cached briefly.
logged-in B2B pricing should stay private.
cart and checkout state should stay request-scoped.

That is the difference between a caching strategy that helps the business and one that only looks good in benchmarks.

Using `revalidateTag` to Keep Data Fresh

One of the most useful patterns in Next.js 16 is tag-based invalidation.

Instead of revalidating whole pages blindly, you can invalidate the exact data domain that changed. That is particularly useful when updates come from webhooks triggered by your CMS, PIM, or commerce backend.

For example, if a product is updated in Medusa or an editorial team changes CMS content linked to that product, you can invalidate the relevant tag immediately.

import { revalidateTag } from 'next/cache'

export async function POST(request: Request) {
  const payload = await request.json()

  revalidateTag(`product:${payload.slug}`, 'max')
  revalidateTag(`offer:${payload.id}:${payload.regionId}`, 'max')

  return Response.json({ ok: true })
}

In practice, this lets you keep product content cached most of the time while still reacting quickly to actual changes.

That is a much better fit for commerce than short global revalidation windows everywhere.

One operational note matters here: never expose a public revalidation endpoint without request validation. If invalidation is triggered by a CMS, Medusa, or ERP webhook, verify the source with a shared secret or signature check before calling revalidateTag.

Where Teams Usually Get It Wrong

There are a few caching mistakes that repeatedly appear in storefront projects.

Mistake 1: Caching personalized pricing globally

This is the most dangerous one. If the result depends on customer identity, customer group, contract terms, or session data, do not let it leak into a shared cache boundary.

Mistake 2: Making the whole page dynamic because one widget is dynamic

A product page does not need to become fully request-time just because the cart badge or recommendation box is personalized.

Keep the stable content cached and isolate the dynamic parts instead.

Mistake 3: Using `no-store` too broadly

no-store is sometimes necessary, but if it becomes the default reaction to uncertainty, you throw away much of the value of the App Router architecture.

Use it for truly request-bound data, not as a shortcut for unclear cache design.

Mistake 4: Forgetting business dimensions in cache keys

Prices are rarely just “product price.” They are often tied to region, currency, channel, campaign, or customer group.

If those dimensions are not reflected in the data boundary, the cache becomes unsafe.

Mistake 5: Mixing editorial freshness with operational freshness

A product description may be safe to cache until a content change occurs. Inventory and promotions may not be. These two layers should not be forced into the same lifetime.

A Good Default Model for Product Pages

If you need a practical starting point, this model works well for many storefronts:

Data type	Suggested approach
Product description, SEO copy, editorial blocks	`use cache` with tags and on-demand invalidation
Shared regional price and stock	short cache lifetime plus tags
Recommendations	private or isolated dynamic boundary
Cart state	request-time or session-specific logic
Checkout totals and payment readiness	request-time only
B2B contract pricing	private boundary, never generic shared output

This is not a framework rule. It is a business-safe baseline.

Why This Works Well With Streaming and PPR Thinking

If you have already explored Partial Prerendering in Next.js 16, this caching model should feel familiar.

The idea is the same:

keep the stable shell fast
isolate dynamic or high-volatility parts
avoid blocking the entire page for data that does not deserve that privilege

For commerce, that means a product page can still load quickly for users and search engines while the truly dynamic fragments remain fresh.

That is a much more scalable model than forcing every route into either “fully static” or “fully dynamic.”

A Note on Webhooks and Operational Ownership

Caching only works well if invalidation is treated as part of the architecture, not as an afterthought.

If your team updates product data in Medusa, content in a CMS, or prices in an ERP, your storefront should know how those changes trigger cache invalidation.

Typical patterns include:

Medusa webhooks invalidating product and offer tags
CMS publish events invalidating page or content tags
ERP sync jobs invalidating pricing-related tags
stock update events invalidating inventory-related tags only

This is where many storefronts move from “fast in development” to “reliable in production.”

Final Thoughts

The real value of caching in Next.js 16 is not that it makes everything static. It is that it helps you decide what should be shared, what should be refreshed, and what should remain private.

For e-commerce teams, that distinction matters more than raw speed numbers. The fastest storefront is not the one that caches the most. It is the one that caches the right things.

If you separate stable catalog content, shared operational data, and user-specific state into different cache boundaries, you can keep your storefront fast without risking stale or incorrect business data.

Kubernetes HPA Scale to Zero Without KEDA: Native Autoscaling for Idle Workloads

Daniel Kraszewski — Wed, 27 May 2026 10:00:00 +0000

If you run queue processors, batch workers, or event-driven workloads that sit idle for hours between bursts, you're paying for compute you don't need. Kubernetes HPA can scale these deployments to zero replicas — no KEDA, no Knative, no external controllers required. You need one feature gate, an external metrics source, and about twenty minutes of setup. When the queue is empty your pods disappear, and if you pair this with cluster autoscaler, the nodes disappear too. Real scale-to-zero, using nothing but native Kubernetes primitives.

Quick Reference

Requirement	Detail
Feature gate	`HPAScaleToZero=true` (available since Kubernetes 1.16)
Minimum replicas	`minReplicas: 0` in the HPA spec
Metrics source	Must use external or custom metrics (not CPU/memory)
Scale-to-zero trigger	Metric value drops to zero
Scale-from-zero trigger	Metric value rises above zero
Why not CPU/memory?	No pods means no resource metrics to observe — the HPA controller needs a signal that exists independently of the pods

Local Setup with kind

Since HPAScaleToZero requires an explicit feature gate, we need a cluster that has it enabled on both the API server and the controller manager. kind makes this straightforward — especially on Kubernetes 1.36, which at the time of writing is too new for most managed providers.

If you haven't already built the node image:

kind build node-image --type release v1.36.0

Create a cluster config that enables the feature gate:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  HPAScaleToZero: true
nodes:
  - role: control-plane
  - role: worker

kind create cluster --name hpa-scale-to-zero --config kind-config.yaml --image kindest/node:v1.36.0

The featureGates field at the cluster level propagates the flag to all relevant components (kube-apiserver, kube-controller-manager, kubelet). No need to manually patch component configs.

The Metrics Pipeline

The HPA controller can't scale from zero based on CPU or memory — there are no pods to measure. You need a metric that exists outside the workload itself. For queue-based workloads, the natural choice is the queue length: how many items are waiting to be processed.

We'll use Redis as the queue (specifically a Redis list), expose its length via a redis-exporter sidecar, scrape it with Prometheus, and surface it to the Kubernetes metrics API through prometheus-adapter. This is the same pipeline pattern you'd use in production — the only difference is we're running everything in a single kind cluster.

Deploy Redis with an Exporter

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install redis bitnami/redis \
  --set architecture=standalone \
  --set auth.enabled=false \
  --set metrics.enabled=true \
  --set metrics.extraArgs.check-keys=work-queue

The metrics.enabled=true deploys a redis-exporter sidecar alongside Redis. The --check-keys work-queue argument tells the exporter to emit redis_key_size{key="work-queue"} — the length of our Redis list. That's the metric we'll use to drive the HPA. The chart also sets up the correct Prometheus scrape annotations automatically.

Deploy Prometheus and the Adapter

The prometheus-community Helm charts get us running in two commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm install prometheus prometheus-community/prometheus \
  --set server.service.type=ClusterIP \
  --set alertmanager.enabled=false \
  --set kube-state-metrics.enabled=false \
  --set prometheus-node-exporter.enabled=false \
  --set prometheus-pushgateway.enabled=false

This gives us a minimal Prometheus that auto-discovers pods annotated with prometheus.io/scrape: "true" — which the Bitnami Redis chart already configures.

Now deploy the prometheus-adapter with a custom external metrics rule:

helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --set prometheus.url="http://prometheus-server" \
  --set prometheus.port=80 \
  --set-json 'rules.external=[{"seriesQuery":"{__name__=\"redis_key_size\",key=\"work-queue\"}","metricsQuery":"sum(<<.Series>>{<<.LabelMatchers>>})","name":{"as":"redis_queue_length"},"resources":{"namespaced":false}}]'

This rule tells the adapter: take the redis_key_size metric where key="work-queue", expose it as an external metric called redis_queue_length, and don't filter by namespace (since the queue is a cluster-wide resource — there's only one Redis).

After a minute or so, verify the external metric is available:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/redis_queue_length" | jq .

You should see a response with a value of 0 (since the queue is empty).

The Worker and HPA

The worker is deliberately simple — a shell script that pops items from the Redis list and "processes" them by sleeping for a second. In production this would be your actual consumer code.

Important: start the deployment with replicas: 1, not zero. The HPA controller uses a ScaledToZero condition internally — it only scales from zero if it was the one that previously scaled the workload to zero. A deployment that starts at zero replicas with a fresh HPA will never scale up. Let the HPA handle the initial scale-down to zero on its own.

# worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: queue-worker
spec:
  replicas: 1  # start at 1 — the HPA will scale to zero once metrics are available
  selector:
    matchLabels:
      app: queue-worker
  template:
    metadata:
      labels:
        app: queue-worker
    spec:
      containers:
        - name: worker
          image: redis:7
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                item=$(redis-cli -h redis-master BRPOP work-queue 30)
                if [ -n "$item" ]; then
                  echo "Processing: $item"
                  sleep 1  # simulate work
                fi
              done

Now the HPA that scales this deployment between 0 and 10 replicas based on the queue length:

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0  # this is what HPAScaleToZero enables
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: redis_queue_length
        target:
          type: Value
          value: "5"  # one pod per 5 queue items
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15

The target.value: "5" means the HPA will try to maintain a ratio of one pod per five pending items. When the queue has 20 items, you get 4 pods. When it drops to zero, you get zero pods. The behavior.scaleDown section shortens the stabilization window from the default 5 minutes to 60 seconds — useful for demos and workloads where you want faster scale-to-zero.

Seeing It in Action

Apply the worker and HPA:

kubectl apply -f worker.yaml -f hpa.yaml

Open a watch in one terminal:

kubectl get hpa queue-worker -w

Since the queue is empty and the deployment starts at 1 replica, the HPA will first scale it down to zero. This initial scale-down is critical — it sets the internal ScaledToZero condition that enables future scale-from-zero behavior. After about a minute (our shortened stabilization window), you'll see replicas drop to 0.

Now push 20 items into the queue:

kubectl exec redis-master-0 -c redis -- sh -c '
  for i in $(seq 1 20); do
    redis-cli LPUSH work-queue "job-$i"
  done
'

Within 30-60 seconds (Prometheus scrape interval + adapter refresh + HPA sync period), you'll see the HPA scale the deployment from 0 to 4 replicas. The workers will process the items — one per second per worker — and as the queue drains, the replica count drops. Once the queue is empty and the metric reads zero, the HPA scales the deployment back to zero replicas. Pods gone.

Here's what the kubectl get hpa queue-worker -w output looks like over the full cycle:

NAME           REFERENCE                 TARGETS   MINPODS   MAXPODS   REPLICAS
queue-worker   Deployment/queue-worker   0/5       0         10        1
queue-worker   Deployment/queue-worker   0/5       0         10        0
queue-worker   Deployment/queue-worker   20/5      0         10        0
queue-worker   Deployment/queue-worker   20/5      0         10        4
queue-worker   Deployment/queue-worker   8/5       0         10        4
queue-worker   Deployment/queue-worker   0/5       0         10        2
queue-worker   Deployment/queue-worker   0/5       0         10        0

Push items again — the HPA scales from zero immediately, no manual intervention needed. The cycle repeats indefinitely.

Cluster Autoscaler Integration

When a deployment scales to zero, its pods vanish. If those pods were the only workload on a node, the node becomes empty. Cluster autoscaler (or any node lifecycle manager) will notice the idle node and remove it after its configurable cool-down period — typically 10 minutes.

This is where the real cost savings live. You're not just saving pod-level resources; you're saving entire node-hours. For a workload that processes a queue for two hours a day and sits idle for twenty-two, you go from paying for 24 node-hours to paying for roughly 2.5 (accounting for scale-up overhead).

The cold-start chain matters. When an item hits the queue after a period of silence, here's what happens in sequence:

Prometheus scrapes the redis-exporter (scrape interval, typically 15s)
Prometheus-adapter picks up the new value (adapter refresh, typically 30s)
HPA sync loop fires (default 15s) and decides to scale up
Scheduler assigns the pod to a node — if no nodes are available, cluster autoscaler provisions one (60-120s for cloud providers)
Kubelet pulls the container image and starts the pod
Pod becomes Ready

In a warm cluster (nodes already present), steps 1-5 take about 30-60 seconds total. When a new node must be provisioned, add 1-2 minutes. For workloads where a minute or two of latency is unacceptable, consider these mitigations:

Smaller container images reduce pull time. Distroless or scratch-based images start faster.
Priority-based overprovisioning: deploy a low-priority "placeholder" pod that reserves a node. When real work arrives, the placeholder gets preempted and the worker starts immediately — no node provisioning needed.
Shorter scrape intervals on Prometheus reduce the detection lag.

For batch and queue workloads, the 30-60 second cold start is typically fine. You're processing a backlog, not serving interactive requests.

When to Use This vs. KEDA

KEDA is the most popular alternative for event-driven autoscaling in Kubernetes. It supports 60+ scalers (Redis, SQS, RabbitMQ, Kafka, Azure Queue, and many more), handles the metrics plumbing internally, and can scale to zero without any feature gate.

Use native HPA scale-to-zero when:

You already have Prometheus and a metrics adapter deployed
You want fewer moving parts and no additional CRDs
Your scaling logic is simple (one metric, linear scaling)
You prefer to stay within the boundaries of native Kubernetes APIs

Use KEDA when:

You don't have Prometheus or don't want to maintain the adapter
You need advanced scaling logic (multiple triggers, complex cooldowns, cron-based scaling)
You want out-of-the-box support for dozens of event sources without manual adapter configuration
You need ScaledJobs (scaling Kubernetes Jobs rather than Deployments)

There's no shame in either choice. KEDA is excellent software. The native approach is simpler when your infrastructure already includes Prometheus — you're just connecting pieces that already exist.

Gotchas

You cannot use CPU or memory metrics to scale from zero. This is the most common mistake. When replica count is zero, there are no pods, so there's no CPU or memory to measure. The HPA controller requires at least one non-resource metric (external or custom) to make scaling decisions at zero replicas. If you configure only resource metrics, the HPA will refuse to scale from zero.

The deployment must not start at zero replicas. This is the subtlest gotcha and I haven't seen it documented clearly anywhere. The HPA controller maintains a ScaledToZero condition internally. It only scales FROM zero if it was the one that previously scaled the workload TO zero. If you deploy with replicas: 0 and create a fresh HPA, the controller sees zero replicas but no ScaledToZero condition, and sets ScalingActive: False with reason ScalingDisabled. The fix is simple: start with replicas: 1 and let the HPA scale it down naturally. Once the HPA has performed that initial scale-to-zero, the ScaledToZero condition is set to True and all subsequent scale-from-zero operations work correctly.

The stabilization window still applies at zero. By default, the HPA waits 5 minutes of sustained low metrics before scaling down. This applies to the transition from 1 to 0 as well. If you want faster scale-to-zero, configure the behavior.scaleDown section with a shorter stabilizationWindowSeconds.

The feature gate is alpha — but don't let that scare you. HPAScaleToZero has been available since Kubernetes 1.16 without breaking changes. It hasn't graduated to beta primarily due to KEP process inertia, not because of instability. The implementation is a small conditional in the HPA controller that allows minReplicas: 0 when the feature gate is enabled. It's been running in production clusters for years.

PodDisruptionBudgets at zero replicas. A PDB with minAvailable: 1 on a deployment that's at zero replicas won't prevent the scale-to-zero — PDBs apply to voluntary eviction of running pods, and scaling down is a different path. However, consider what happens if you have a PDB and scale from zero to one: the single pod is protected by the PDB immediately upon becoming ready.

Wrapping Up

The combination of native HPA scale-to-zero, external metrics from Prometheus, and cluster autoscaler gives you genuine pay-for-what-you-use economics on idle workloads. No third-party controllers, no additional CRDs, no vendor lock-in. The feature gate has been stable for twenty releases. Enable it, point your HPA at a queue metric, set minReplicas: 0, and stop paying for compute that's doing nothing.

The KEP tracking this feature is KEP-2021: HPA Scale to Zero — if you want to see it graduate to beta and become the default, that's the issue to watch and engage with.

AWS CloudFront Cache Policies: Complete Guide

Maciej Łopalewski — Wed, 20 May 2026 11:30:24 +0000

A CloudFront cache policy controls two things: the cache key (which combination of URL, headers, cookies, and query strings makes a request unique) and the TTL (how long CloudFront keeps an object at the edge before re-checking the origin). Those two settings together determine your cache hit ratio.

The CloudFront console currently shows fifteen managed cache policies. Five are broadly useful on a typical self-managed distribution: CachingOptimized, CachingOptimizedForUncompressedObjects, CachingDisabled, UseOriginCacheControlHeaders, and UseOriginCacheControlHeaders-QueryStrings. One (Elemental-MediaPackage) is for AWS Elemental MediaPackage video origins. The remaining nine are Amplify-related — a standalone Amplify policy for Amplify origins plus eight Amplify-* policies that Amplify Hosting attaches to its own distributions automatically. You can also build a custom policy when none of the managed ones fit the shape of your traffic.

This is a follow-up to my earlier post on CloudFront's three policy types, going one level deeper on just cache policies.

What a cache policy controls

A cache policy has three categories of settings: policy information (name and description, just metadata), TTL settings (Minimum, Maximum, Default), and cache key settings (which headers, cookies, and query strings to include). Everything that affects caching behavior lives in the latter two.

The cache key is the unique identifier CloudFront uses to look up an object at an edge location. If two viewer requests produce the same cache key, the second one is a cache hit. If they differ — say, one request includes a ?ref=twitter query string and the other does not, and your policy includes query strings in the cache key — they get treated as separate objects, even when the response body is identical. Cache key shape is the single biggest lever for hit ratio.

The TTL settings work alongside Cache-Control and Expires headers from your origin to determine how long a cached object stays valid at the edge. They behave subtly differently from each other; more on that next.

How Minimum, Maximum, and Default TTL actually work

The three TTL settings are not redundant. Each one applies in a different scenario, and getting them confused is one of the more common ways to either over-cache stale content or hammer your origin unnecessarily.

Default TTL is used when your origin sends no Cache-Control or Expires header at all. CloudFront falls back to this value, subject to the Minimum TTL floor — if Minimum TTL is greater than Default TTL, CloudFront caches for at least the Minimum TTL. The default for the AWS managed CachingOptimized policy is 86,400 seconds (24 hours).

Maximum TTL caps the TTL when your origin does send Cache-Control or Expires. If your origin says Cache-Control: max-age=5184000 (60 days) and Maximum TTL is 31,536,000 (365 days), CloudFront honors the origin's 60 days. If your origin says max-age=63072000 (730 days), CloudFront caps it at 365 days. This setting only matters when you want to override an origin that's claiming overly aggressive cache durations.

One nuance worth knowing: if your origin sends both max-age and s-maxage, CloudFront uses s-maxage for its own caching decisions and lets browsers use max-age. This is how you get different cache durations at the edge versus the browser without writing two policies.

Minimum TTL is the floor. CloudFront keeps the object for at least this long, no matter what. This one comes with a sharp edge: if Minimum TTL is greater than 0, CloudFront ignores Cache-Control: no-cache, no-store, and private directives from your origin. The object gets cached anyway, for at least the Minimum TTL duration. Both CachingOptimized and CachingOptimizedForUncompressedObjects have Minimum TTL of 1 second, and the standalone Amplify policy has Minimum TTL of 2 seconds — meaning you cannot reliably stop caching with origin headers alone if you are using them.

If you set all three TTLs to 0, caching is effectively disabled — which is exactly what CachingDisabled does.

Cache key settings: headers, cookies, query strings, and compression

Each of the three viewer-data sources (headers, cookies, query strings) can be configured independently. Cookies and query strings have four modes: include none, include all, include specifically named ones, or include all-except-named-ones. Headers are the exception — they support only "none" or a specific list. There is no "all headers" option, because that would be unbounded and would fragment the cache catastrophically.

When you include a header, cookie, or query string in the cache key, CloudFront also automatically forwards it to the origin on cache misses. The cache key and the origin request are coupled by default; the only way to forward something to the origin without affecting the cache key is to add it to a separate origin request policy. This is the most common confusion with cache policies and the reason CloudFront split them apart in the first place.

A subtle but important detail: cache key matching uses header, cookie, and query string names, but the matching is on the full name+value. Specifying session_id in the cache key means every distinct value of session_id produces a different cache key — so if every visitor has a unique session ID, every request is a cache miss. This is why "include all cookies" rarely makes sense for general traffic.

The compression settings (EnableAcceptEncodingGzip, EnableAcceptEncodingBrotli) tell CloudFront to normalize the Accept-Encoding header before adding it to the cache key. With both enabled, the cache key sees one of br,gzip, gzip, or br (depending on what the viewer supports), or no Accept-Encoding at all when the viewer supports neither — in that last case CloudFront sends Accept-Encoding: identity to the origin instead. Without normalization, every browser variation of Accept-Encoding: gzip, deflate, br, zstd would produce a distinct cache key. Enable this when your origin returns compressed responses or when CloudFront edge compression is on. Leave it off otherwise.

One gotcha: if you enable Gzip or Brotli in the cache policy, do not also include Accept-Encoding in an origin request policy attached to the same behavior. CloudFront handles that header itself when compression is enabled, and adding it to the origin request policy has no effect.

The AWS managed cache policies

This is the comparison table for the policies you'll actually attach to a self-managed distribution by hand. The Amplify Hosting policies (Amplify-* and Amplify-*-V2) are intentionally excluded — see the Amplify section below for why.

Policy	TTL (min/default/max)	Cookies	Query strings	Headers in cache key	Compression
`CachingOptimized`	1s / 24h / 365d	None	None	None	Gzip + Brotli
`CachingOptimizedForUncompressedObjects`	1s / 24h / 365d	None	None	None	Off
`CachingDisabled`	0 / 0 / 0	None	None	None	Off
`UseOriginCacheControlHeaders`	0 / 0 / 365d	All	None	Host, Origin, method overrides	Gzip + Brotli
`UseOriginCacheControlHeaders-QueryStrings`	0 / 0 / 365d	All	All	Host, Origin, method overrides	Gzip + Brotli
`Elemental-MediaPackage`	0 / 24h / 365d	None	aws.manifestfilter, start, end, m	Origin	Gzip

CachingOptimized

ID: 658327ea-f89d-4fab-a63d-7e88639e58f6

The default choice for static content — S3 buckets, image assets, JS and CSS bundles, anything that does not change based on who is asking. The cache key is just the requested object plus the normalized Accept-Encoding. No headers, no cookies, no query strings. This produces the highest possible cache hit ratio for static content.

Watch the Minimum TTL of 1 second: even with Cache-Control: no-store on your origin, CloudFront holds the object for at least one second. That is usually fine, but if you are using this policy on something where origin-side cache busting needs to take effect immediately, swap to UseOriginCacheControlHeaders or CachingDisabled.

CachingOptimizedForUncompressedObjects

ID: b2884449-e4de-46a7-ac36-70bc7f1ddd6d

Identical to CachingOptimized except compression is off. Use this when your origin does not return Gzip or Brotli (raw binary files, video segments, pre-compressed media formats like MP4 or WebP) and you are not using CloudFront edge compression. With compression off, the Accept-Encoding header is excluded from the cache key entirely, which keeps things simple.

If you cannot decide between this and CachingOptimized, default to CachingOptimized. The compression setting only causes problems when your origin produces objects that do not benefit from compression — and even then, it is a minor inefficiency rather than a correctness bug.

CachingDisabled

ID: 4135ea2d-6df8-44a3-9df3-4b5a84be39ad

All TTLs at 0. Nothing in the cache key. Every request goes straight through to the origin. Use this for API behaviors, dynamic GET endpoints that should never be cached, real-time data, WebSockets, and anywhere caching would be incorrect rather than just suboptimal. (POST, PUT, and DELETE aren't cached by CloudFront in the first place — only GET, HEAD, and optionally OPTIONS are — so attaching CachingDisabled to those methods is more about being explicit than functionally necessary.)

UseOriginCacheControlHeaders, UseOriginCacheControlHeaders-QueryStrings, and Elemental-MediaPackage also have Minimum TTL of 0, so they too will respect Cache-Control: no-store from your origin. The difference is that CachingDisabled never caches anything regardless of origin headers, while the others cache when the origin tells them to.

UseOriginCacheControlHeaders

ID: 83da9c7e-98b4-4e11-a168-04f0df8e2c65

Defers caching duration to your origin's Cache-Control and Expires headers. If the origin says max-age=600, CloudFront caches for 10 minutes. If the origin says no-store, CloudFront does not cache at all (because Minimum TTL is 0).

This is the right choice for CMS-backed sites, mixed static-and-dynamic apps, and anywhere your application code already knows what should be cached and for how long. WordPress, Drupal, server-rendered Next.js, and most traditional web apps fit here.

The cache key includes all cookies plus Host, Origin, and three method-override headers (X-HTTP-Method-Override, X-HTTP-Method, X-Method-Override). The cookie inclusion is significant: if your application sets per-user session cookies on every response, you will fragment the cache per user. Either avoid setting cookies on cacheable responses, or stick with CachingOptimized for paths that should be shared across users.

UseOriginCacheControlHeaders-QueryStrings

ID: 4cc15a8a-d715-48a4-82b8-cc0b614638fe

Same as UseOriginCacheControlHeaders but with all query strings included in the cache key. Use this when your origin returns different responses based on query string values — search results, filtered listings, paginated content — and you want CloudFront to cache each variant separately.

The trade-off is cache fragmentation. URLs that include UTM parameters, click IDs, or other tracking junk get cached independently, which both lowers your hit ratio and bloats CloudFront's storage of your content. If you can strip those server-side or with a CloudFront Function before this policy applies, do.

Amplify (and the eight related Amplify policies)

ID: 2e54312d-136d-493c-8eb9-b001f22f67d2

Two distinct things share the Amplify name. The standalone Amplify policy is documented as a regular CloudFront managed policy designed for use with an Amplify web app origin — Min TTL 2 seconds, Max TTL 600 seconds (10 minutes), Default TTL 2 seconds, with Authorization, CloudFront-Viewer-Country, and Host in the cache key plus all cookies and all query strings. AWS doesn't warn against using it; it's just narrowly tuned for Amplify-shaped workloads. The 2-second Minimum TTL means even no-store responses get cached briefly, which is rarely what you want outside of that specific architecture.

The eight Amplify-* policies are something else entirely: Amplify-Default, Amplify-DefaultNoCookies, Amplify-ImageOptimization, Amplify-StaticContent, plus a -V2 variant of each. These are managed by Amplify Hosting itself — Amplify attaches them to the distributions it provisions and resets them on every deploy. AWS explicitly says "we don't recommend that you use these policies for your distributions." If you need similar cache key shapes for a non-Amplify app, copy the settings into a custom policy.

The V2 variants appear to be tied to the August 2024 Amplify Hosting caching overhaul, which raised default static asset cache duration from 2 seconds to 1 year and Maximum TTL from 10 minutes to 1 year. AWS hasn't documented the V2 policies in the public CloudFront developer guide as of this writing — they're visible in the console but the documentation only covers the four originals.

Elemental-MediaPackage

ID: 08627262-05a9-4f76-9ded-b50ca2e3a84f

For AWS Elemental MediaPackage origins specifically — HLS and DASH video streaming. The cache key includes the four query string parameters that MediaPackage actually uses for manifest filtering and time-shifted playback (aws.manifestfilter, start, end, m) plus the Origin header for CORS. Other query strings are excluded, which keeps the cache key tight even when player libraries append cache-busting noise.

If you are not using MediaPackage, do not use this policy. If you are, use it — it is tuned for the specific request shape MediaPackage produces.

How to choose a cache policy

Match your origin and content type to the policy that is already tuned for it.

Use case	Recommended policy
S3 static site, image bucket, asset CDN	`CachingOptimized`
Video files (MP4, WebP), pre-compressed binaries	`CachingOptimizedForUncompressedObjects`
API endpoints, dynamic GET responses, real-time data	`CachingDisabled`
WordPress, Drupal, server-rendered apps with origin Cache-Control	`UseOriginCacheControlHeaders`
Same as above, but query strings affect the response	`UseOriginCacheControlHeaders-QueryStrings`
AWS Elemental MediaPackage HLS/DASH origin	`Elemental-MediaPackage`
Custom requirements that do not match any of the above	Custom cache policy

For a single distribution serving multiple content types, you typically attach different policies to different cache behaviors. A common setup: CachingOptimized on the default behavior (static assets), CachingDisabled on /api/*, UseOriginCacheControlHeaders on /blog/* if blog pages set their own Cache-Control.

When to create a custom cache policy

Reach for a custom policy when you need to include something specific in the cache key that the managed policies do not cover. The most common reasons are device-type segmentation using CloudFront-Is-Mobile-Viewer (serving different markup to phones versus desktops), country-based variation using CloudFront-Viewer-Country (geo-targeted content where edge handling beats origin-side detection), language negotiation via Accept-Language, and tier-based content where a coarse-grained auth bucket determines which response to serve.

Here is a Terraform example for a custom policy that varies on viewer country and a coarse auth-tier cookie:

resource "aws_cloudfront_cache_policy" "country_and_tier" {
  name        = "country-and-tier"
  default_ttl = 3600       # 1 hour when origin sends no Cache-Control
  max_ttl     = 86400      # cap origin Cache-Control at 24 hours
  min_ttl     = 0          # respect origin no-store

  parameters_in_cache_key_and_forwarded_to_origin {
    # normalize Accept-Encoding so cache key collapses across browsers
    enable_accept_encoding_gzip   = true
    enable_accept_encoding_brotli = true

    headers_config {
      header_behavior = "whitelist"
      headers {
        items = ["CloudFront-Viewer-Country"]  # geo-target at the edge
      }
    }

    cookies_config {
      cookie_behavior = "whitelist"
      cookies {
        # coarse bucket like anonymous/free/premium — NOT a per-user session ID
        items = ["auth_tier"]
      }
    }

    query_strings_config {
      query_string_behavior = "none"
    }
  }
}

The cookie name matters here. auth_tier with values like anonymous, free, and premium produces three cache variants per country, which is reasonable. A per-user session token in the same slot would produce one cache variant per user, which is the cache-buster pattern the article warns against earlier.

A few things to know when going custom. CloudFront-generated headers like CloudFront-Viewer-Country are not sent to the origin by default — you need to either include them in the cache key (which automatically forwards them to the origin) or add them to an origin request policy if you want them at the origin without varying the cache. The cache compression toggle should match what your origin returns; turning it on when your origin does not compress can fragment the cache without benefit. And keep Minimum TTL at 0 unless you specifically want to override origin no-store directives, because the surprise-caching behavior burns a lot of debugging hours when you forget about it.

Common pitfalls

Cache hit ratio tanks after a policy change. Almost always means the cache key got too specific. Check whether the new policy includes cookies or query strings the previous one did not, and re-check whether your origin sets per-user cookies on responses that should be shared across users.

Origin keeps getting hit despite a long Default TTL. Default TTL only applies when the origin sends no Cache-Control or Expires header. If your origin sets Cache-Control: max-age=60, CloudFront uses 60 seconds, not the policy's Default TTL. Either change the origin or use Maximum TTL to cap origin-controlled durations.

no-cache from origin is being ignored. Several managed policies (CachingOptimized, CachingOptimizedForUncompressedObjects, and Amplify) have a Minimum TTL greater than 0, which overrides origin no-cache directives. If you need origin-side cache busting to take effect immediately, switch to CachingDisabled or build a custom policy with Minimum TTL = 0.

Forwarding CloudFront-Viewer-Country to origin does not work. For CloudFront-generated headers like CloudFront-Viewer-Country, make sure the header is explicitly enabled via a cache policy or an origin request policy — they aren't sent to your origin by default. If it's in the cache key, it's also forwarded to the origin automatically. If you only need it at the origin and don't want to vary the cache, put it in an origin request policy instead. (Note: a few CloudFront-* headers, including CloudFront-Viewer-Address, CloudFront-Viewer-ASN, and the TLS-related ones, can only be added via an origin request policy and not in a cache policy.)

Authorization header is not reaching the origin. CloudFront removes it by default. To forward it, either include Authorization in the cache key with a cache policy, or use an origin request policy that forwards all viewer headers (the managed Managed-AllViewer policy does this). You cannot forward only Authorization via an origin request policy. Be careful: putting Authorization in the cache key creates per-token cache variants and is usually inappropriate for broadly shared cacheable content.

What's next

If you came here looking for "which cache policy should I use," the short answer is CachingOptimized for static content, CachingDisabled for APIs, UseOriginCacheControlHeaders for content where your origin can express its own caching intent. Everything else is tuning at the margins.

The next post in this series goes deep on origin request policies — where the separation between cache key and origin forwarding really earns its keep — followed by response headers policies, which can replace a lot of security middleware with a single CloudFront config.

Secure Access to Private EKS Clusters Without Bastion Hosts Using SSM

Paweł Swiridow — Wed, 13 May 2026 13:00:00 +0000

Accessing Private EKS Clusters Without Losing Your Mind

Locking down your Kubernetes control plane is a basic requirement for any production environment. Exposing the EKS API server to the public internet is just asking for automated scanners to ruin your weekend. However, securing the endpoint creates an operational headache: how do you actually run kubectl when the API is sealed inside a private subnet?

The traditional answer was a bastion host. But managing SSH keys, rotating credentials, and maintaining yet another publicly exposed EC2 instance is tedious. We all know that a "temporary" bastion host spun up on a Friday afternoon will inevitably become a load-bearing production pillar by Monday.

Instead, we can use AWS Systems Manager (SSM) Session Manager. By leveraging the SSM agent already running on your EKS worker nodes, we can securely tunnel our local traffic directly to the private API endpoint without opening inbound ports or managing SSH keys.

The Mechanics of the SSM Tunnel

The flow is straightforward:

Your local machine initiates an SSM port forwarding session targeting a specific EKS worker node.
The SSM session is instructed to forward traffic to a remote host (the private EKS API endpoint URL) on port 443.
You update your kubeconfig to point to localhost on your chosen forwarded port.

Because the worker node is already in the VPC and authorized to talk to the EKS control plane, it acts as a highly secure, identity-aware proxy. Access is governed entirely by IAM, meaning you can audit every connection via CloudTrail.

Prerequisite: IAM Configuration

For this to work, your EKS worker nodes must have the SSM agent installed (the official EKS optimized AMIs have this by default) and the correct IAM permissions.

Here is a Terraform snippet demonstrating how to attach the necessary SSM policy to your existing EKS node IAM role.

# Assumes you already have an aws_iam_role defined for your worker nodes
# named 'eks_node_role'

resource "aws_iam_role_policy_attachment" "ssm_managed_instance_core" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  role       = aws_iam_role.eks_node_role.name
}

# Optional but recommended: Restrict who can start sessions in IAM
resource "aws_iam_policy" "ssm_user_access" {
  name        = "EKS-SSM-Tunnel-Access"
  description = "Allows users to port forward to EKS nodes"
  policy      = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "ssm:StartSession"
        Resource = [
          "arn:aws:ec2:*:*:instance/*",
          "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost"
        ]
        # In a real environment, restrict the instance resource via tags
        Condition = {
          StringEquals = {
            "ssm:resourceTag/eks:cluster-name" = "my-production-cluster"
          }
        }
      }
    ]
  })
}

This ensures your nodes can communicate with the SSM service and restricts which IAM users can actually initiate the tunnel.

Establishing the Tunnel

Once the nodes are registered in SSM, you need a script to extract a valid instance ID, locate the cluster API endpoint, and start the tunnel.

Here is a Bash script you can execute locally to handle the heavy lifting. It requires the AWS CLI and the Session Manager plugin to be installed on your workstation.

#!/bin/bash
set -euo pipefail

CLUSTER_NAME="my-production-cluster"
REGION="us-east-1"
LOCAL_PORT="8443"

# Fetch the private endpoint of the EKS cluster
echo "Fetching EKS endpoint for ${CLUSTER_NAME}..."
EKS_ENDPOINT=$(aws eks describe-cluster \
  --name "${CLUSTER_NAME}" \
  --region "${REGION}" \
  --query "cluster.endpoint" \
  --output text | sed 's/https:\/\///')

# Find an active worker node instance ID using tags
echo "Finding an active worker node..."
INSTANCE_ID=$(aws ec2 describe-instances \
  --region "${REGION}" \
  --filters "Name=tag:eks:cluster-name,Values=${CLUSTER_NAME}" "Name=instance-state-name,Values=running" \
  --query "Reservations[0].Instances[0].InstanceId" \
  --output text)

if [ "$INSTANCE_ID" == "None" ]; then
  echo "Error: No running worker nodes found."
  exit 1
fi

echo "Establishing SSM tunnel through ${INSTANCE_ID} to ${EKS_ENDPOINT}..."
echo "Leave this terminal open. Access EKS via https://localhost:${LOCAL_PORT}"

# Start the port forwarding session
aws ssm start-session \
  --region "${REGION}" \
  --target "${INSTANCE_ID}" \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters "{\"host\":[\"${EKS_ENDPOINT}\"],\"portNumber\":[\"443\"],\"localPortNumber\":[\"${LOCAL_PORT}\"]}"

Run this script, and it will bind localhost:8443 to the private API endpoint.

Updating Kubeconfig

The final step is modifying your local Kubernetes configuration. You cannot simply run aws eks update-kubeconfig and call it a day, because that will configure the private AWS endpoint, which your machine still cannot route to directly.

You need to manually alter the server field for your cluster to point to the local port.

When you port-forward the EKS API server to your local machine, connecting to https://localhost:8443 introduces a new problem. The API server presents a TLS certificate minted for its internal AWS endpoint (e.g., 1234567890ABCDEF.yl4.us-east-1.eks.amazonaws.com), not localhost.

The quick, dirty fix is to add insecure-skip-tls-verify: true to your kubeconfig. But nothing screams "I definitely passed my SOC2 audit" quite like explicitly disabling TLS validation in production. It is the infrastructure equivalent of putting black tape over a check engine light.

Instead of turning off validation, we can instruct kubectl to connect via our local port but validate the TLS certificate against the actual EKS endpoint hostname. We do this by utilizing the tls-server-name parameter.

apiVersion: v1
clusters:
- cluster:
    server: https://localhost:8443
    # Validate the certificate against the real AWS endpoint
    tls-server-name: 1234567890ABCDEF.yl4.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:123456789012:cluster/my-production-cluster
# ... contexts and users remain unchanged

Once saved, kubectl get pods will route securely through the SSM tunnel, across the worker node, and hit the control plane.

Wrap-Up

Relying on SSM port forwarding eliminates the need for VPNs, bastion hosts, and complex routing rules just to run operational commands against an isolated EKS cluster. By utilizing the existing IAM-integrated agent on your worker nodes, you shrink your external attack surface while maintaining strict audit trails for developer access.

Next.js Server Actions vs API Routes: Architecture, Performance, and Use Cases

Paweł Sobolewski — Tue, 05 May 2026 22:00:00 +0000

With the introduction of the App Router, Next.js added two powerful server-side primitives:

At first glance, they might seem interchangeable—both run on the server. But they're designed for different responsibilities, follow different execution models, and come with different constraints.

Understanding these differences is essential to avoid architectural pitfalls and performance issues.

What Are Server Actions?

According to the official documentation, Server Actions are a specific type of React Server Function used for mutations. They are invoked from the client by sending a POST request to the current page’s route (the same URL context in which the action was triggered), where Next.js internally routes the request to the corresponding server-side function based on unique server action identifier.

Here's an important clarification:

We only call them Server Actions when they're invoked from the client.

From the server's perspective, they're just regular asynchronous functions.

For example:

"use server"

export async function updateUser() {
    // mutation logic
}

Called from a Client Component → acts as a Server Action
Called from another server-side function (like server component) → just a normal function call

This distinction is crucial:

The network boundary only exists when invoked from the client,
The serialization and execution constraints only apply in that context.

Key characteristics

Must be asynchronous
Marked with the "use server" directive
Invoked via POST requests under the hood

Mutations + Rendering = Sequential Constraints

The official documentation states that React Server Functions are designed for server-side mutations. This design choice is directly tied to one of their most important limitations.

When a Server Action is invoked:

It mutates server-side state
It triggers a React Server Component (RSC) re-render
It produces a new UI snapshot
The client merges the updated result

This tight coupling between mutation and rendering means React must ensure each update applies to a consistent version of the UI tree.

This creates a critical constraint described in the docs:

The client currently dispatches and awaits them one at a time

What this means in practice

Even if multiple Server Actions are triggered “at once”:

useEffect(() => {
    actionA();
    actionB();
}, [])

They will execute sequentially, not in parallel:

actionA → re-render → actionB → re-render

This is not an implementation detail—it is a consequence of the architecture.

Running them in parallel would risk:

race conditions between mutations
inconsistent UI snapshots
stale data being rendered

So React prioritizes consistency over concurrency.

Future note

The documentation explicitly indicates that this behavior is current, which means it may change in the future. However, today:

You should design your system assuming Server Actions are sequential

What Are Route Handlers?

Route Handlers address a key limitation of Server Actions: the lack of true concurrent, independent calls.

Because Server Actions are tightly coupled with React's rendering lifecycle, they aren't suited for scenarios where multiple operations must run in parallel without UI-driven synchronization.

Route Handlers provide a decoupled HTTP-based execution model that restores full concurrency and independence between requests.

Definition

Route Handlers are standard HTTP endpoints defined in:

app/api/.../route.ts

Example:

import { NextResponse }from"next/server"

export async function POST(request: Request) {
    const body= await request.json();

    await db.cart.add(body.productId);

    return NextResponse.json({ success:true });
}

Key characteristics

Full HTTP interface
Support for all HTTP methods (GET, POST, etc.)
Publicly accessible endpoints
Independent of React rendering lifecycle
Fully parallelizable and concurrent by default

Good to know

In older versions of Next.js there were only Route Handlers, as Server Action concept came with React Server Components in Next.js 13. So they are improving specific use cases, not replacing them. You can still write the whole application using only Route Handlers and not writing a single Server Action at all.

Why Use Server Actions?

At this point, you might ask:

If Route Handlers provide full concurrency, HTTP control, and fewer constraints—why use Server Actions at all?

Server Actions aren't meant to replace Route Handlers. They solve a different class of problems: UI-driven mutations tightly integrated with React's rendering model, not transport-layer API design.

1. They eliminate the client–server API layer

With Route Handlers, every mutation typically requires:

defining an API endpoint
handling request/response manually
writing client-side fetch calls
managing loading and error states explicitly

Server Actions remove this entire layer:

<form action={createPost}>
    <input name="title"/>
    <button type="submit">Create</button>
</form>

No more:

manual fetch calls
endpoint definitions
request parsing boilerplate

The mutation becomes a direct extension of the UI.

2. They're first-class citizens of React Server Components

Server Actions are deeply integrated with the RSC model:

They trigger server component re-renders
They return updated UI state without manual client synchronization
They work seamlessly with Next.js caching and revalidation primitives

This creates a unique property:

A mutation can immediately reflect in the UI—without explicitly managing state on the client.

Route Handlers can't do this. They only return data.

3. They reduce client-side complexity

With Route Handlers, a typical mutation flow looks like:

await fetch("/api/posts", { method:"POST", body: ... });
setState(...);
handleLoading(...);
handleError(...);

With Server Actions:

no manual request lifecycle
no fetch boilerplate
no explicit client-state synchronization layer

This shifts complexity from the client to the framework boundary.

Conclusion

Server Actions are powerful but intentionally constrained.

They are:

React-integrated mutation primitives designed for UI-driven updates—not general-purpose concurrency.

Choosing between Server Actions and Route Handlers isn't about preference—it's about understanding whether your problem belongs to the UI layer or the transport/API layer.

Payload CMS Security Best Practices: Top 10 Threats & Mitigation Strategies in 2026

Michał Miler — Tue, 21 Apr 2026 08:20:00 +0000

Payload CMS is a powerful, developer-first headless CMS built on Node.js and TypeScript. It gives you complete control over authentication, access control, and API behavior - but with that flexibility comes responsibility for implementing robust security measures and following OWASP security best practices.

Security misconfigurations remain one of the leading causes of data breaches in modern web applications. According to IBM's Cost of a Data Breach Report, thousands of CMS-powered websites and APIs are compromised every year due to preventable issues like weak authentication, improper access control, and exposed admin panels.

From our experience working on production SaaS applications, eCommerce platforms, and multi-tenant systems at u11d, over 80% of Payload CMS projects lack proper implementation of critical security controls aligned with OWASP Top 10 risks - especially around authentication, authorization, API exposure, and infrastructure hardening.

In this comprehensive guide, we'll cover the most common Payload CMS security threats and practical, production-tested mitigation strategies you should implement to avoid costly vulnerabilities, data leaks, and security incidents.

Who This Guide is For:

Payload CMS developers building production applications and APIs
DevOps engineers securing Payload deployments on AWS, DigitalOcean, Vercel
Project managers and product owners overseeing headless CMS implementations
Security auditors reviewing Payload CMS implementations for compliance
Technical leads architecting secure headless CMS solutions with Next.js

What You'll Learn:

Critical security threats specific to Payload CMS (with OWASP mapping)
OWASP Top 10 aligned mitigation strategies for headless CMS
Production-ready implementation examples with TypeScript
Complete security checklist for production deployment
Infrastructure hardening techniques for Node.js applications
Real-world security incidents and lessons learned

I. Admin Account Compromise (Critical Priority)

The Security Risk

Admin accounts are the highest-value target in any CMS. In Payload CMS, administrators typically have unrestricted access to:

All content and collections
User management and permissions
System configuration
API access controls
Database operations

Attack Impact

If compromised, attackers can:

Modify or deface content
Inject malicious scripts (XSS)
Manipulate pricing or product data
Access sensitive user information
Delete or corrupt critical data

In real-world incidents, compromised admin access often leads to full platform takeover within minutes - especially in systems without audit logging or alerts.

The Solution: Multi-Layered Admin Protection

1. Enforce Modern Password Policies (NIST-Compliant)

Modern password policies prioritize length and uniqueness over complexity rules (NIST SP 800-63B). Best practices include:

Minimum 15+ characters (passphrases preferred over complexity)
Prevent password reuse (store hash history)
Block common and breached passwords (Have I Been Pwned API)
Encourage password managers
Avoid forced periodic password expiration (outdated practice)

Why This Matters:

Short, complex passwords (e.g., P@ssw0rd123) are far weaker than long passphrases (e.g., correct-horse-battery-staple-2025).

2. Enable Multi-Factor Authentication (MFA/2FA)

Critical: Payload CMS does not enforce 2FA by default for admin users. You must explicitly add this protection layer.

Recommended Solutions for Payload CMS:

Option A: TOTP-Based

payloadcms-tfa - Community plugin for Time-based OTP
payload-totp - Alternative TOTP implementation
Supports authenticator apps (Google Authenticator, Authy, 1Password)

Option B: Custom OTP Implementation

Email-based one-time codes
SMS-based codes (requires Twilio/similar)
Hardware tokens (YubiKey, FIDO2)

Option C: External Auth Providers

Auth.js (NextAuth) with 2FA providers
Keycloak with MFA policies
Zitadel with passkey support

Production Requirement

For production and SaaS systems, MFA for all admin users should be mandatory, not optional.

3. Enforce HTTPS Everywhere (TLS/SSL)

Never expose Payload admin panels over HTTP. This is a critical vulnerability that exposes:

Admin credentials during login
Session cookies
API tokens
All transmitted data

Recommended TLS Configuration:

TLS 1.3 preferred (TLS 1.2 minimum)
Strong cipher suites only
HSTS header with preload
Redirect all HTTP → HTTPS
Secure cookie flags (secure, httpOnly, sameSite)

Summary: Admin security is your first line of defense - weak authentication here leads to total system compromise.

II. Weak Authentication Strategy

The Risk

Payload provides flexible authentication, but that flexibility often leads to insecure defaults in real projects.

Common issues include:

Long-lived JWT tokens
Tokens stored in localStorage
No refresh token rotation
Mixing admin and public authentication flows

These mistakes significantly increase the risk of session hijacking and token theft.

The Solution

Secure Token Handling

Use short-lived access tokens
Implement refresh token rotation
Store tokens in HTTP-only cookies
Avoid localStorage for sensitive tokens

Consider External Identity Providers

For more advanced or scalable setups, integrate external auth systems:

Auth.js (NextAuth)
Better-Auth
Keycloak
Zitadel

These solutions provide:

OAuth & social login
Enterprise SSO
Centralized identity management
Advanced session control

Summary: A well-designed authentication layer reduces your attack surface and improves scalability.

III. Missing Access Control Rules

The Risk

Payload’s access control system is powerful - but optional. Many teams either skip it or implement overly permissive rules.

This can lead to:

Unauthorized data access
Privilege escalation
Exposure of sensitive fields via API

In many breaches, improper authorization - not authentication - is the root cause.

The Solution

Define Explicit Access Rules

Always define:

read
create
update
delete

For every collection.

Best practices:

Public content → read-only for anonymous users
Admin content → role-based restrictions
User data → owner-only access

Never rely on frontend restrictions — enforce everything server-side.

Summary: Authorization must be explicit and restrictive by default.

IV. Public API Exposure

The Risk

Payload automatically exposes REST and optionally GraphQL APIs, which can unintentionally leak data if not configured correctly.

Common risks:

Public access to internal collections
Exposure of sensitive fields
Endpoint enumeration and brute-force attacks

Attackers often scan APIs first - not your frontend.

The Solution

Limit API Surface

Disable GraphQL if unused
Restrict public endpoints
Use API gateways or reverse proxies

Protect Sensitive Fields

hidden: true
access: { read: () => false }

Add Rate Limiting

Implement at infrastructure level:

Cloudflare
AWS API Gateway
Reverse proxy throttling

Payload does not provide built-in rate limiting.

Summary: Reduce what is exposed - every public endpoint is a potential attack vector.

V. No Audit Logging

The Risk

Without audit logs, security incidents become invisible.

You won’t know:

Who changed what
When it happened
Whether malicious activity occurred

This makes incident response and compliance extremely difficult.

The Solution

Enable Versioning

Use Payload’s versioning for:

Pages
Products
Critical content

Centralize Logging

Track:

Login attempts
Failed logins
Content changes
Permission updates

Send logs to:

CloudWatch
Datadog
ELK stack

Summary: If you can’t see it, you can’t secure it.

VI. Database Security Misconfiguration

The Risk

Payload typically uses MongoDB or PostgreSQL. Misconfigured databases are a frequent source of major data breaches.

Risks include:

Public database exposure
Weak credentials
Lack of encryption
Lateral movement within infrastructure

The Solution

Never expose databases publicly
Use private VPC networking
Rotate credentials regularly
Use IAM-based authentication where possible
Encrypt data at rest and in transit

Summary: Infrastructure security is just as important as application security.

VII. Missing Content Validation (XSS Risk)

The Risk

Allowing rich text or HTML input without sanitization opens the door to stored XSS attacks.

Attackers can inject scripts that execute in:

Admin panel
Frontend applications
Other users’ browsers

The Solution

Sanitize HTML inputs
Use strict schema validation
Limit custom HTML fields
Escape output in frontend

Never trust user-generated content - even from “trusted” users.

Summary: Input validation is essential to prevent client-side attacks.

Final Thoughts: Security is a Feature, Not an Afterthought in Payload CMS

Payload CMS gives developers exceptional flexibility and control over authentication, authorization, and data access - but security must be explicitly designed and implemented from day one, not bolted on later.

Unlike managed SaaS CMS platforms (Contentful, Sanity, Hygraph), Payload assumes you understand authentication mechanisms, authorization patterns, and infrastructure security. That's powerful and flexible - but also a common source of critical vulnerabilities in production deployments.

Key Takeaways:

Payload CMS requires explicit security configuration - No secure-by-default settings
80% of projects have preventable security gaps - Based on real-world security audits
OWASP Top 10 alignment is critical - Authentication, access control, API security
Infrastructure security matters as much as application security - Database, network, TLS configuration
Security is continuous, not one-time - Regular audits, dependency updates, monitoring
Security impacts performance and UX - See our localization guide for secure field components
Secure scaling is possible - Our Connect211 case study shows 50+ domains secured

If you're running Payload CMS in production — especially for:

eCommerce platforms with payment processing
SaaS applications with sensitive user data
Fintech solutions requiring PCI-DSS compliance
Healthcare systems needing HIPAA compliance
Mobile app backends with millions of users
Multi-tenant platforms isolating customer data

Treat security as a first-class feature from the start, not a checkbox before launch.

Additional Resources

OWASP Top 10 - Web application security risks (updated 2021)
Payload CMS Authentication Documentation - Official authentication guide
NIST Password Guidelines - Modern password policy standards (SP 800-63B)
CIS Benchmarks - Infrastructure hardening guides for Linux, Docker, databases
Have I Been Pwned API - Password breach detection service
Payload Discord Community - Security discussions with Payload experts
Snyk Vulnerability Database - Node.js package vulnerabilities

Need Payload CMS Experts?

u11d specializes in Payload CMS development, migration, and deployment. We help you build secure, scalable Payload projects, migrate from legacy CMS platforms, and optimize your admin, API, and infrastructure for production. Get expert support for custom features, localization, and high-performance deployments.

Talk to Payload Experts

Auto-Translation in Payload CMS with Azure AI Translator: Complete 2026 Implementation Guide

Michał Miler — Tue, 14 Apr 2026 09:52:00 +0000

Manual translation in Payload CMS doesn't scale for production applications: it's slow, expensive, and error‑prone once you support multiple documents and locales. This comprehensive guide shows a production-tested automated approach using Azure AI Translator, Payload background jobs, and LRU caching to batch translations, preserve manual edits, and minimize API costs.

At u11d, we've implemented this solution across multiple high-traffic Payload CMS applications, processing millions of translations monthly. This guide covers everything from Azure setup to production deployment with real-world performance metrics.

Key features of this production-ready solution:

One-click batch translation for any Payload collection (100+ languages supported)
Async background jobs - No timeouts, scalable processing with Payload Jobs API
Intelligent caching - LRU cache to avoid duplicate API calls and reduce costs by 70%+
Preserves manual edits - Only empty fields are translated by default
Force mode - Re-translate all fields when needed (content updates, quality improvements)
Audit trail - Tracks translation metadata for compliance and debugging
Type-safe - Full TypeScript support with Payload's generated types
Production-tested - Handles millions of translations in real-world applications

Solution Overview

Architecture:

Translation Service: Azure + Cache
Payload Background Job: async, scalable
API Route Handler: secure job submission
Admin UI: button + modal for user control
Translation Metadata: audit what/when/force

This tutorial focuses on a single collection for clarity. For a full, type-safe, multi-collection solution, see the complete implementation on GitHub.

Prerequisites

Before implementing auto-translation, ensure you have:

Azure Account - Create free account ($200 credit included)
Azure Translator Resource - Setup guide
Payload Localization Enabled - See our localization best practices guide
Node.js 18+ - Required for Payload 3.0+
TypeScript 5+ - For type-safe implementation

Database Note: This tutorial uses SQLite for local development only. For production, configure DATABASE_URL to connect to PostgreSQL, MongoDB, or MySQL for better performance and reliability.

Step-by-Step Implementation

Step 1: Install Dependencies

First, install the Azure AI Translator SDK and the additional project dependencies:

npm install @azure-rest/ai-translation-text lru-cache radash

Step 2: Configure Environment Variables

Create or update your .env file:

# Azure Translator Configuration
AZURE_TRANSLATOR_KEY=your_azure_translator_key
AZURE_TRANSLATOR_REGION=your_azure_translator_region

Check .env.example to see what else can be configured.

Getting Azure Credentials:

Go to Azure Portal
Navigate to your Translator resource
Create Translator resource if it does not exist yet
Click "Keys and Endpoint" in the left menu
Copy KEY 1 and set AZURE_TRANSLATOR_KEY in .env
Copy Location/Region and set AZURE_TRANSLATOR_REGION in .env

Never commit API keys. Add .env.local to .gitignore, use secret managers or CI/CD secret stores, and rotate keys regularly.

Step 3: Create Translation Service

This service handles Azure API calls, batching, and caching:

// src/services/translationService.ts
import crypto from "crypto";
import TextTranslationClient, {
  isUnexpected,
} from "@azure-rest/ai-translation-text";
import { LRUCache } from "lru-cache";

export type TranslationEngine = "azure";

export interface TranslationResult {
  text: string;
  fromCache: boolean;
}

export interface BatchTranslationInput {
  id?: string;
  text: string;
  targetLocale: string;
}

export interface BatchTranslationResult {
  id?: string;
  text: string;
  translatedText: string;
  targetLocale: string;
  fromCache: boolean;
}

const CACHE_TTL_MS = 7 * 24 * 60 * 60 * 1000; // 7 days

const translationCache = new LRUCache<string, string>({
  max: 10000,
  ttl: CACHE_TTL_MS,
  ttlAutopurge: true,
});

function createTextHash(text: string): string {
  return crypto.createHash("sha256").update(text, "utf8").digest("hex");
}

function createCacheKey(locale: string, englishText: string): string {
  const hash = createTextHash(englishText);
  return `${locale}:${hash}`;
}

async function translateWithAzure(
  texts: string[],
  targetLocale: string,
): Promise<string[]> {
  const apiKey = process.env.AZURE_TRANSLATOR_KEY;
  const endpoint =
    process.env.AZURE_TRANSLATOR_ENDPOINT ||
    "<https://api.cognitive.microsofttranslator.com>";
  const region = process.env.AZURE_TRANSLATOR_REGION || "global";

  if (!apiKey) {
    throw new Error(
      "AZURE_TRANSLATOR_KEY is not configured in environment variables",
    );
  }

  const client = TextTranslationClient(endpoint, {
    key: apiKey,
    region: region,
  });

  const inputText = texts.map((text) => ({ text }));
  const translateResponse = await client.path("/translate").post({
    body: inputText,
    queryParameters: {
      to: targetLocale,
      from: "en",
    },
  });

  if (isUnexpected(translateResponse)) {
    throw new Error(
      `Azure translation failed: ${translateResponse.body.error?.message || "Unknown error"}`,
    );
  }

  return translateResponse.body.map((item) => item.translations[0].text);
}

export async function batchTranslate(
  inputs: BatchTranslationInput[],
): Promise<BatchTranslationResult[]> {
  const results: BatchTranslationResult[] = new Array(inputs.length);

  const inputsWithIndex = inputs.map((input, index) => ({ input, index }));

  const byLocale = inputsWithIndex.reduce<
    Record<string, typeof inputsWithIndex>
  >((acc, item) => {
    if (!acc[item.input.targetLocale]) {
      acc[item.input.targetLocale] = [];
    }
    acc[item.input.targetLocale].push(item);
    return acc;
  }, {});

  for (const [locale, localeInputs] of Object.entries(byLocale)) {
    const textsToTranslate: string[] = [];
    const itemsToTranslate: typeof inputsWithIndex = [];

    for (const item of localeInputs) {
      const cacheKey = createCacheKey(locale, item.input.text);
      const cached = translationCache.get(cacheKey);

      if (cached) {
        results[item.index] = {
          id: item.input.id,
          text: item.input.text,
          translatedText: cached,
          targetLocale: locale,
          fromCache: true,
        };
      } else {
        textsToTranslate.push(item.input.text);
        itemsToTranslate.push(item);
      }
    }

    if (textsToTranslate.length > 0) {
      const BATCH_SIZE_AZURE = 100;

      for (let i = 0; i < textsToTranslate.length; i += BATCH_SIZE_AZURE) {
        const batch = textsToTranslate.slice(i, i + BATCH_SIZE_AZURE);
        const batchItems = itemsToTranslate.slice(i, i + BATCH_SIZE_AZURE);

        const translations = await translateWithAzure(batch, locale);

        for (let j = 0; j < translations.length; j++) {
          const item = batchItems[j];
          const translatedText = translations[j];
          const cacheKey = createCacheKey(locale, item.input.text);

          translationCache.set(cacheKey, translatedText);

          results[item.index] = {
            id: item.input.id,
            text: item.input.text,
            translatedText,
            targetLocale: locale,
            fromCache: false,
          };
        }
      }
    }
  }

  return results;
}

export async function translate(
  text: string,
  targetLocale: string,
): Promise<TranslationResult> {
  const results = await batchTranslate([{ text, targetLocale }]);
  const result = results[0];

  return {
    text: result.translatedText,
    fromCache: result.fromCache,
  };
}

What This Does:

LRU Cache - Stores up to 10,000 translations in memory
Cache Key - SHA256 hash of source text + locale
7-Day TTL - Reasonable cache duration for production content
Batch Processing - Groups translations by locale for efficiency
Azure Integration - Sends up to 100 texts per request
Cache Hits - Returns cached translations instantly (< 1ms)
Cache Misses - Fetches from Azure and caches result
Type Safety - Full TypeScript interfaces for reliability

Caching reduces API calls, latency, and cost by reusing recent translations. Use the in-memory LRU cache for local development, but deploy a shared cache (for example, Redis) in production to persist entries across restarts, share results between instances, and scale capacity.

Step 4: Create Payload Background Job

Background jobs prevent timeout issues and allow progress tracking.

Payload's Jobs API runs translation work outside the request lifecycle so long-running batches won't block the admin UI or hit HTTP timeouts. Queued jobs can report progress, be retried on failure, and scale across worker processes, which makes them a reliable choice for large or repeated translation tasks. In this step we'll implement a job that fetches the English document, extracts fields to translate, batches calls to the translation service, and updates target locales in a safe, idempotent way.

// src/jobs/translateResource.ts
import { TaskConfig, TypedLocale } from "payload";
import { batchTranslate } from "../services/translationService";
import { retry } from "radash";
import { isValidLocale } from "@/locales";
import { Resource } from "@/payload-types";

interface FieldToTranslate {
  path: string;
  value: string;
  locale: string;
  id?: string;
}

function isEmpty(value: unknown): boolean {
  if (value === null || value === undefined) return true;
  if (typeof value === "string") return value.trim() === "";
  return false;
}

function shouldTranslate(
  force: boolean,
  sourceValue: string | null | undefined,
  targetValue: string | null | undefined,
): boolean {
  if (isEmpty(sourceValue)) return false;
  if (force) return true;
  return isEmpty(targetValue);
}

function extractFieldsToTranslate(
  englishDoc: Resource,
  targetDoc: Resource,
  force: boolean,
  targetLocale: TypedLocale,
): FieldToTranslate[] {
  const fieldsToTranslate: FieldToTranslate[] = [];

  // Extract title field
  if (shouldTranslate(force, englishDoc.title, targetDoc.title)) {
    fieldsToTranslate.push({
      path: "title",
      value: englishDoc.title,
      locale: targetLocale,
    });
  }

  // Extract list array fields
  if (Array.isArray(englishDoc.list)) {
    englishDoc.list.forEach((englishItem, index) => {
      if (!englishItem.id) return;

      const targetItem = Array.isArray(targetDoc.list)
        ? targetDoc.list.find((t) => t.id === englishItem.id)
        : null;

      if (shouldTranslate(force, englishItem.name, targetItem?.name)) {
        fieldsToTranslate.push({
          path: `list.${index}.name`,
          value: englishItem.name!,
          locale: targetLocale,
          id: englishItem.id,
        });
      }
    });
  }

  return fieldsToTranslate;
}

function buildUpdateData(
  englishDoc: Resource,
  targetDoc: Resource,
  translationsByPath: Record<string, string>,
): Partial<Resource> {
  const updateData: Partial<Resource> = {
    ...targetDoc,
  };

  // Update title
  if (translationsByPath["title"]) {
    updateData.title = translationsByPath["title"];
  } else if (isEmpty(targetDoc.title) && englishDoc.title) {
    updateData.title = englishDoc.title;
  }

  // Update list array
  if (Array.isArray(englishDoc.list)) {
    const targetItemsById = new Map(
      Array.isArray(targetDoc.list)
        ? targetDoc.list.map((item) => [item.id, item])
        : [],
    );

    updateData.list = englishDoc.list.map((englishItem, index) => {
      const targetItem = targetItemsById.get(englishItem.id);
      const mergedItem = {
        ...englishItem,
        ...(targetItem || {}),
        id: englishItem.id,
      };

      const namePath = `list.${index}.name`;
      if (translationsByPath[namePath]) {
        mergedItem.name = translationsByPath[namePath];
      } else if (isEmpty(targetItem?.name) && englishItem.name) {
        mergedItem.name = englishItem.name;
      }

      return mergedItem;
    });
  }

  return updateData;
}

async function executeBatchTranslation(
  fields: FieldToTranslate[],
): Promise<Record<string, string>> {
  const BATCH_SIZE_AZURE = 100;
  const translationsByPath: Record<string, string> = {};

  for (let i = 0; i < fields.length; i += BATCH_SIZE_AZURE) {
    const batch = fields.slice(i, i + BATCH_SIZE_AZURE);
    const translations = await retry({ times: 3, delay: 1000 }, () =>
      batchTranslate(
        batch.map((f) => ({
          id: f.path,
          text: f.value,
          targetLocale: f.locale,
        })),
      ),
    );
    translations.forEach((t) => {
      if (t?.id) translationsByPath[t.id] = t.translatedText;
    });
  }

  return translationsByPath;
}

export const translateResourceJob: TaskConfig<"translateResource"> = {
  slug: "translateResource",
  inputSchema: [
    {
      name: "documentId",
      type: "text",
      required: true,
    },
    {
      name: "locales",
      type: "array",
      required: true,
      fields: [
        {
          name: "locale",
          type: "text",
        },
      ],
    },
    {
      name: "force",
      type: "checkbox",
      defaultValue: false,
    },
  ],
  outputSchema: [
    {
      name: "success",
      type: "checkbox",
      required: true,
    },
    {
      name: "translated",
      type: "number",
      required: true,
    },
    {
      name: "details",
      type: "json",
    },
  ],
  handler: async ({ input, job, req }) => {
    const { payload } = req;
    const { documentId, force } = input;
    const locales = input.locales.map(({ locale }) => locale);

    console.log(
      `[Job ${job.id}] Translation started for resources:${documentId}`,
      { locales, force },
    );

    try {
      const englishDoc = await payload.findByID({
        collection: "resources",
        id: documentId,
        locale: "en",
        depth: 0,
      });

      if (!englishDoc) {
        throw new Error(`Resource not found: ${documentId}`);
      }

      const totalTranslatedCounts: Record<string, number> = {};

      for (const targetLocale of locales) {
        if (targetLocale === "en") continue;

        console.log(`[Job ${job.id}] Processing locale: ${targetLocale}`);

        if (!isValidLocale(targetLocale)) {
          console.warn(`[Job ${job.id}] Invalid locale: ${targetLocale}`);
          continue;
        }

        const targetDoc = await payload.findByID({
          collection: "resources",
          id: documentId,
          locale: targetLocale,
        });

        const fieldsToTranslate = extractFieldsToTranslate(
          englishDoc,
          targetDoc,
          force || false,
          targetLocale,
        );

        if (fieldsToTranslate.length === 0) {
          console.log(
            `[Job ${job.id}] No fields to translate for locale ${targetLocale}`,
          );
          continue;
        }

        console.log(
          `[Job ${job.id}] Translating ${fieldsToTranslate.length} fields for ${targetLocale}`,
        );

        const translationsByPath =
          await executeBatchTranslation(fieldsToTranslate);

        const updateData = buildUpdateData(
          englishDoc,
          targetDoc,
          translationsByPath,
        );

        updateData._translationMeta = {
          lastTranslatedAt: new Date().toISOString(),
          translatedBy: "auto",
        };

        await payload.update({
          collection: "resources",
          id: documentId,
          locale: targetLocale,
          data: updateData,
        });

        totalTranslatedCounts[targetLocale] = fieldsToTranslate.length;
      }

      const totalTranslated = Object.values(totalTranslatedCounts).reduce(
        (a, b) => a + b,
        0,
      );

      console.log(
        `[Job ${job.id}] Translation completed. Total fields translated: ${totalTranslated}`,
        totalTranslatedCounts,
      );

      return {
        output: {
          success: true,
          translated: totalTranslated,
          details: totalTranslatedCounts,
        },
      };
    } catch (error) {
      console.error(`[Job ${job.id}] Translation failed:`, error);
      throw error;
    }
  },
};

What This Does:

Simple & Focused - Handles only Resources collection for tutorial clarity
Selective Translation - Only translates empty fields by default
Force Mode - Option to re-translate all fields
Retry Logic - 3 attempts with 1-second delays
Structure Preservation - Maintains array order and IDs
Fallback - Uses English value if translation fails
Metadata Tracking - Records when and how translation occurred

Want to translate multiple collections? This tutorial simplifies the code for learning purposes. For a production-ready implementation with type-safe mappers, field extractors, and support for multiple collections, check out the complete codebase on GitHub. Don't forget to give it a star!

Step 5: Create API Route for Job Queueing

This endpoint triggers translation jobs. It validates the requesting admin session, sanitizes and checks the request body (documentId, locales, force), and queues the translateResource job with the provided input. The implementation below uses cookie-based Payload auth and returns the queued job's ID on success.

// src/app/api/translate/route.ts
import { NextRequest, NextResponse } from "next/server";
import { getPayload } from "payload";
import config from "@/payload.config";

export async function POST(request: NextRequest) {
  try {
    const payload = await getPayload({ config });

    const cookies = request.cookies;
    const payloadToken = cookies.get("payload-token");

    if (!payloadToken) {
      return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
    }

    try {
      const { user } = await payload.auth({ headers: request.headers });
      if (!user) {
        return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
      }
    } catch (authError) {
      return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
    }

    const body = await request.json();
    const { documentId, locales, force } = body;

    if (!documentId || !locales) {
      return NextResponse.json(
        {
          error: "Missing required fields: documentId, locales",
        },
        { status: 400 },
      );
    }

    if (!Array.isArray(locales) || locales.length === 0) {
      return NextResponse.json(
        { error: "locales must be a non-empty array" },
        { status: 400 },
      );
    }

    const job = await payload.jobs.queue({
      task: "translateResource",
      input: {
        documentId,
        locales: locales.map((locale: string) => ({ locale })),
        force: force || false,
      },
      queue: "translation",
    });

    console.log(
      `Translation job queued: ${job.id} for resources:${documentId}`,
      { locales },
    );

    return NextResponse.json({
      success: true,
      jobId: job.id,
      message: "Translation job queued successfully",
    });
  } catch (error) {
    console.error("Failed to queue translation job:", error);
    return NextResponse.json(
      {
        error: "Internal server error",
        message: error instanceof Error ? error.message : "Unknown error",
      },
      { status: 500 },
    );
  }
}

Step 6: Create Admin UI Components

This step adds an admin-side UI — a translate button and modal — that lets editors pick target locales, toggle force mode, and queue translation jobs without leaving the editor.

Translate Button:

// src/components/TranslateButton.tsx
"use client";

import React, { useState } from "react";
import { toast, Button, useModal, useDocumentInfo } from "@payloadcms/ui";
import { LOCALES_WITHOUT_EN, TranslateModal } from "./TranslateModal";

export const TranslateButton: React.FC = () => {
  const { id } = useDocumentInfo();
  const [isLoading, setIsLoading] = useState(false);
  const { openModal } = useModal();

  const handleTranslateClick = async () => {
    if (!id) {
      toast.error("Document ID is required");
      return;
    }

    setIsLoading(true);

    try {
      if (LOCALES_WITHOUT_EN.length === 0) {
        toast.error("No target locales available for translation");
        return;
      }

      openModal("translate-modal");
    } catch (error) {
      console.error("Error opening translate modal:", error);
      toast.error("Failed to open translation modal");
    } finally {
      setIsLoading(false);
    }
  };

  if (!id) {
    return null;
  }

  return (
    <>
      <Button
        buttonStyle="primary"
        onClick={handleTranslateClick}
        disabled={isLoading}
      >
        {isLoading ? "Loading..." : "Translate (BETA)"}
      </Button>
      <TranslateModal />
    </>
  );
};

Translate Modal:

// src/components/TranslateModal.tsx
"use client";

import React, { useState } from "react";
import {
  Button,
  toast,
  Modal,
  useModal,
  useDocumentInfo,
} from "@payloadcms/ui";
import { LOCALES_WITHOUT_EN } from "@/locales";

export const TranslateModal: React.FC = () => {
  const { closeModal } = useModal();
  const { id } = useDocumentInfo();
  const [selectedLocales, setSelectedLocales] = useState<Set<string>>(
    new Set(),
  );
  const [force, setForce] = useState(false);
  const [isSubmitting, setIsSubmitting] = useState(false);

  const handleLocaleToggle = (locale: string) => {
    const newSelected = new Set(selectedLocales);
    if (newSelected.has(locale)) {
      newSelected.delete(locale);
    } else {
      newSelected.add(locale);
    }
    setSelectedLocales(newSelected);
  };

  const handleSelectAll = () => {
    if (selectedLocales.size === LOCALES_WITHOUT_EN.length) {
      setSelectedLocales(new Set());
    } else {
      setSelectedLocales(new Set(LOCALES_WITHOUT_EN));
    }
  };

  const handleSubmit = async () => {
    if (selectedLocales.size === 0) {
      toast.error("Please select at least one locale");
      return;
    }

    if (!id) {
      toast.error("Document ID is required");
      return;
    }

    setIsSubmitting(true);

    const body = {
      documentId: id.toString(),
      locales: Array.from(selectedLocales),
      force,
    };

    try {
      const response = await fetch("/api/translate", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        credentials: "include",
        body: JSON.stringify(body),
      });

      if (!response.ok) {
        const error = await response.json();
        throw new Error(error.message || "Failed to start translation job");
      }

      const result = await response.json();

      toast.success(
        `Translation job #${result.jobId} queued! Check the Jobs section for progress.`,
      );

      closeModal("translate-modal");

      // Reset form
      setSelectedLocales(new Set());
      setForce(false);
    } catch (error) {
      console.error("Translation error:", error);
      toast.error(
        error instanceof Error
          ? error.message
          : "Failed to start translation job",
      );
    } finally {
      setIsSubmitting(false);
    }
  };

  return (
    <Modal slug="translate-modal">
      <div
        style={{
          display: "flex",
          alignItems: "center",
          justifyContent: "center",
          minHeight: "100vh",
          padding: "20px",
        }}
      >
        <div
          style={{
            background: "var(--theme-elevation-0)",
            borderRadius: "8px",
            padding: "32px",
            maxWidth: "600px",
            width: "100%",
            boxShadow: "0 0 16px rgba(0, 0, 0, 0.25)",
          }}
        >
          <h2 style={{ marginTop: "0", marginBottom: "24px" }}>
            Auto-Translate
          </h2>

          <div>
            <div
              style={{
                display: "flex",
                justifyContent: "space-between",
                alignItems: "center",
              }}
            >
              <label style={{ fontWeight: "bold" }}>Locales</label>
              <Button
                buttonStyle="secondary"
                size="small"
                onClick={handleSelectAll}
              >
                {selectedLocales.size === LOCALES_WITHOUT_EN.length
                  ? "Deselect All"
                  : "Select All"}
              </Button>
            </div>
            <div
              style={{
                border: "1px solid #ccc",
                borderRadius: "4px",
                padding: "12px",
                maxHeight: "200px",
                overflowY: "auto",
                gap: "8px",
                marginTop: "8px",
              }}
            >
              {LOCALES_WITHOUT_EN.map((locale) => (
                <div
                  key={locale}
                  style={{
                    display: "flex",
                    alignItems: "center",
                    marginBottom: "8px",
                  }}
                >
                  <input
                    type="checkbox"
                    id={`locale-${locale}`}
                    checked={selectedLocales.has(locale)}
                    onChange={() => handleLocaleToggle(locale)}
                    style={{ marginRight: "8px" }}
                  />
                  <label htmlFor={`locale-${locale}`}>{locale}</label>
                </div>
              ))}
            </div>
          </div>

          <div
            style={{ display: "flex", alignItems: "center", marginTop: "20px" }}
          >
            <input
              type="checkbox"
              id="force-translate"
              checked={force}
              onChange={(e) => setForce(e.target.checked)}
              style={{ marginRight: "16px" }}
            />
            <label htmlFor="force-translate">
              <strong>Force re-translate all fields</strong>
              <br />
              <span style={{ fontSize: "0.75em", color: "#666" }}>
                If unchecked, only empty fields will be translated
              </span>
            </label>
          </div>

          <div
            style={{
              display: "flex",
              justifyContent: "flex-end",
              gap: "10px",
              marginTop: "24px",
            }}
          >
            <Button
              buttonStyle="secondary"
              onClick={() => closeModal("translate-modal")}
              disabled={isSubmitting}
            >
              Cancel
            </Button>
            <Button onClick={handleSubmit} disabled={isSubmitting}>
              {isSubmitting ? "Starting..." : "Start Translation"}
            </Button>
          </div>
        </div>
      </div>
    </Modal>
  );
};

Step 7: Update Collection Configuration

Wire the translate button into the collection UI and add a _translationMeta group so automated translations and audit data are stored alongside your content.

// src/collections/Resources.ts
import type { CollectionConfig } from "payload";

export const Resources: CollectionConfig = {
  slug: "resources",
  admin: {
    useAsTitle: "title",
    components: {
      edit: {
        SaveButton: "@/components/TranslateButton",
      },
    },
  },
  fields: [
    {
      name: "title",
      type: "text",
      localized: true,
      required: true,
      admin: {
        components: {
          Field: "@/components/LocalizedTextField",
        },
      },
    },
    {
      name: "list",
      type: "array",
      fields: [
        {
          name: "name",
          type: "text",
          localized: true,
          admin: {
            components: {
              Field: "@/components/LocalizedTextField",
            },
          },
        },
      ],
    },
    {
      name: "_translationMeta",
      type: "group",
      admin: {
        description: "Metadata about automatic translations",
        condition: (_, siblingData) => !!siblingData._translationMeta,
      },
      fields: [
        {
          name: "lastTranslatedAt",
          type: "date",
          admin: {
            readOnly: true,
          },
        },
        {
          name: "translatedBy",
          type: "text",
          admin: {
            readOnly: true,
          },
        },
      ],
    },
  ],
};

Step 8: Update Payload Config

Register the translation job, enable localization settings, and ensure the job is included in Payload's jobs configuration so queued translations run reliably.

// src/payload.config.ts
import path from "path";
import { buildConfig } from "payload";
import { fileURLToPath } from "url";
import { sqliteAdapter } from "@payloadcms/db-sqlite";
import { lexicalEditor } from "@payloadcms/richtext-lexical";
import { Resources } from "./collections/Resources";
import { Users } from "./collections/Users";
import { translateResourceJob } from "./jobs/translateResource";
import { AVAILABLE_LOCALES } from "./locales";

const filename = fileURLToPath(import.meta.url);
const dirname = path.dirname(filename);

export default buildConfig({
  admin: {
    user: "users",
    importMap: {
      baseDir: path.resolve(dirname),
    },
  },

  // Localization config
  localization: {
    locales: AVAILABLE_LOCALES,
    defaultLocale: "en",
    fallback: true,
  },

  // Jobs config for auto-translation
  jobs: {
    tasks: [translateResourceJob],
  },

  collections: [Users, Resources],
  editor: lexicalEditor(),
  secret: process.env.PAYLOAD_SECRET || "dev-secret-change-me",
  typescript: {
    outputFile: path.resolve(dirname, "payload-types.ts"),
  },
  db: sqliteAdapter({
    client: {
      url: process.env.DATABASE_URL || "file:./payload.db",
    },
  }),
});

Step 9: Generate Import Map and Types

npm run generate:importmap
npm run generate:types

This registers your custom components and generates TypeScript types.

How It Works

Complete Translation Flow

User clicks "Translate" button in admin UI
Modal opens with locale selection and force mode option
User submits translation request
API route verifies authentication and validates parameters
Job is queued in Payload's background job system
Job handler fetches English document from database
For each target locale:
- Fetch existing locale document
- Identify fields needing translation
- Group by locale for batch processing
Translation service:
- Checks cache for each text
- Batches uncached texts (up to 100 per request)
- Calls Azure Translator API
- Stores results in cache
Job handler updates documents with translations
Metadata recorded (timestamp, auto-translated flag)
User sees success message with job ID

Monitoring and Debugging

View Job Status

Navigate to Admin → Jobs in Payload
Find your translation job by ID
View status: pending, processing, completed, or failed
Check output for translation counts

Console Logs

The job handler logs progress:

[Job abc123] Translation started for resources:65f7a...
[Job abc123] Processing locale: es
[Job abc123] Translating 15 fields for es
[Job abc123] Translation completed. Total: 15

Error Handling

Common errors and solutions:

Error	Cause	Solution
`AZURE_TRANSLATOR_KEY is not configured`	Missing env var	Add to `.env.local`
`Unauthorized`	Not logged into admin	Log in to Payload admin
`Document not found`	Invalid document ID	Verify document exists
`Translation failed: Rate limit`	Too many requests	Wait and retry

Advanced Customization

Adding More Locales

Update two places:

Locales file:

// src/locales.ts
export const AVAILABLE_LOCALES: string[] = ["en", "es", "fr", "de"];
export const LOCALES_WITHOUT_EN = AVAILABLE_LOCALES.filter(
  (value) => value !== "en",
);

Regenerate types:

npm run generate:types

The locales are automatically used by:

Payload's localization config (in payload.config.ts)
TranslateModal component (imports LOCALES_WITHOUT_EN)
Translation validation (in locales.ts)

Supporting More Field Types

Extend the job handler to support other localized fields:

// Add textarea support
if (englishDoc.description) {
  if (shouldTranslate(englishDoc.description, targetDoc.description)) {
    fieldsToTranslate.push({
      path: "description",
      value: englishDoc.description,
      locale: targetLocale,
    });
  }
}

// Add nested fields
if (englishDoc.metadata?.title) {
  if (shouldTranslate(englishDoc.metadata.title, targetDoc.metadata?.title)) {
    fieldsToTranslate.push({
      path: "metadata.title",
      value: englishDoc.metadata.title,
      locale: targetLocale,
    });
  }
}

Using Redis for Distributed Caching

For multi-server deployments, replace LRU cache with Redis:

import { createClient } from "redis";

const redis = createClient({
  url: process.env.REDIS_URL,
});

await redis.connect();

// Get from cache
const cached = await redis.get(cacheKey);

// Set in cache
await redis.set(cacheKey, translatedText, {
  EX: CACHE_TTL, // Expiration in seconds
});

Benefits:

Shared cache across server instances
Persistent cache across restarts
Higher capacity (GBs vs MBs)

Auto-Translate on Save (Delta Translation)

Use Payload hooks (for example beforeChange/afterChange) to detect which English fields were modified and queue translation jobs for only those deltas. This keeps translations in sync, reduces API usage, and avoids translating unchanged content. Consider UI indicators for in-progress translations, field-level locking, and debouncing to prevent rapid repeated jobs.

Multi-Provider Translation Support

Azure is a solid default, but it isn't the only option — some languages or content types can produce better results with DeepL, Google Cloud, OpenAI, or other services. Add a provider abstraction and selection rules so you can route specific locales or content categories to the best provider, implement fallbacks, and monitor costs/quotas to balance quality against price.

Production Checklist

Before deploying to production:

[ ] Environment variables configured in production
[ ] Azure Translator key has sufficient quota
[ ] Job queue working (test with small document)
[ ] Backup database before first bulk translation
[ ] Cache size appropriate for your data volume
[ ] Error monitoring set up (Sentry, etc.)
[ ] Rate limiting configured if needed
[ ] Cost alerts set in Azure portal
[ ] User permissions reviewed (who can translate?)

Troubleshooting Common Issues

Translations Not Appearing

Check job status: Admin → Jobs → Find your job ID
Review job logs: Look for errors in console output
Verify authentication: Ensure you're logged into Payload admin
Check document: Switch to target locale in admin UI
Validate locales: Ensure target locale exists in config

Cache Not Working

Verify LRU cache: Check import and initialization in service
Test cache hit: Translate same content twice, check logs for "fromCache: true"
Cache size limit: Increase max if needed (default: 10,000 entries)
Memory issues: Monitor Node.js heap usage with -max-old-space-size
Production caching: Consider Redis for multi-instance deployments

Azure API Errors

"Invalid API key": Double-check AZURE_TRANSLATOR_KEY in environment variables
"Rate limit exceeded": Wait 60 seconds or upgrade Azure plan (F0 → S1)
"Invalid language code": Verify locale codes match Azure supported languages
"Text too long": Azure limit is 50,000 characters per text - batch large content
"Authentication failed": Check AZURE_TRANSLATOR_REGION matches resource location

Performance Issues

Slow translations: Increase batch size or upgrade Azure tier
Job timeouts: Split large jobs into smaller batches
Memory leaks: Clear cache periodically or use Redis
Database locks: Add retry logic with exponential backoff
Network errors: Implement connection pooling and retry strategies

Conclusion: Production-Ready Azure AI Translation for Payload CMS 2026

You've successfully implemented an enterprise-grade, scalable auto-translation system for Payload CMS with:

Azure AI Translator - Enterprise-quality translation for 100+ languages with 99.9% uptime
LRU Cache (7-day TTL) - Minimize API costs with intelligent caching (70%+ cost reduction)
Payload Background Jobs - Reliable async processing without HTTP timeouts
Batch Translation (100 texts/request) - Optimal Azure API usage and performance
Selective Mode - Preserve manual edits, translate only empty fields by default
Force Mode - Re-translate all fields when needed for content updates
Metadata Tracking - Complete audit trail for compliance and debugging
Type-Safe Implementation - Full TypeScript support with zero runtime errors

Technical Stack Summary

Payload CMS: 3.79.0
Next.js: 15.4.11 (App Router + API Routes)
Azure AI Translator: @azure-rest/ai-translation-text v1.0.1
Caching: lru-cache v11.2.7 (7-day TTL)
Utilities: radash v12.1.1 (retry logic)
TypeScript: 5.7.2
React: 19.0.0

Real-World Performance Metrics

Based on our production implementations at u11d:

Translation Speed: 100 fields in ~3 seconds (with cache: <100ms)
API Cost: $10 per million characters (Azure S1 tier)
Cache Hit Rate: 70-85% for typical content updates
Uptime: 99.9% (Azure SLA)
Concurrent Jobs: 10+ simultaneous translations without issues
Database Impact: Minimal (batched updates, optimized queries)

Next Steps & Advanced Features

Consider these production enhancements:

Automatic translation triggers - Translate on document publish via hooks
Translation quality scoring - Flag low-confidence translations for review
Custom glossaries - Brand-specific term preservation (Azure feature)
Translation memory - Learn from manual corrections
Multi-provider support - Fallback to Google/DeepL/OpenAI
Batch document translation - Translate multiple documents in one job
Translation diff viewer - Compare original vs. translated side-by-side
Cost monitoring - Track Azure API usage and set budget alerts

Additional Resources for Payload CMS Translation

Azure AI Translator Documentation - Official Azure docs
Payload Jobs API Documentation - Background job implementation
Azure Supported Languages - 100+ language codes
Azure Custom Translator - Brand terminology glossaries
Azure Pricing Calculator - Estimate translation costs
LRU Cache Documentation - Caching implementation details
Payload Discord Community - Get help from Payload experts
Complete Code on GitHub - Full implementation

Need Payload CMS Experts?

Talk to Payload Experts

DEV Community: u11d

Faster Geospatial Animations with PMTiles: Multi-Layer MVT Time Series in OpenLayers

The architecture: single archive, dynamic layers

Complete working example

Step 1 — Per-timestep GeoJSON files

Step 2 — Generate PMTiles with named layers (generate-tiles.sh)

Step 3 — Start the server

Step 4 — Client animation (index.html)

Running the full setup

Conclusion: paint time, not flip it

Argo CD GitLab Authentication: A Complete GitOps-Friendly Setup

How authentication works

Create a GitLab OAuth application

Encrypt the Argo CD secret with SOPS and age

Add KSOPS to the Argo CD repo server

Provide the age private key to Argo CD

Create the KSOPS generator

Reference the generator in Kustomize

Configure Dex in Argo CD

Configure Argo CD RBAC

Apply the configuration

Verify the setup

Troubleshoot common failures

Security notes

Final result

Reciprocal Rank Fusion on free Elasticsearch: licensing, workarounds, and the OpenSearch alternative

What RRF actually does

What Basic actually gets you, and what it doesn't

The licensing timeline

Workaround 1: Implement RRF in your application

Workaround 2: Linear combination with manual normalization

Workaround 3: Switch to OpenSearch

When paying for Enterprise makes sense

What most teams actually need

How to Deploy Payload CMS on AWS Amplify with MongoDB Atlas for Free

Architecture Overview

Prerequisites

Step A: MongoDB Atlas Free Cluster

Create Cluster

Create Database User and Connection String

Configure Network Access

Step B: AWS S3 Bucket and IAM User

Create S3 Bucket

Create IAM User with S3 Permissions

Generate Access Keys

Payload S3 config

Step C: AWS Amplify Deployment

Configure Build Settings

Connect Repository to Amplify

Add Environment Variables

Deploy

Verify Admin Access

Fix Media CORS (One-Time Setup)

Caching and Static Generation in Next.js

Static Site Generation (SSG)

On-Demand Revalidation

Cache Strategy Summary

Pricing Notes

AWS Amplify

AWS S3

MongoDB Atlas

When to Reconsider

Security Trade-Offs

1. MongoDB IP Allowlist: Public Access

2. S3 Public Access: Bucket Policy and CORS

3. Secrets in Environment Variables

Bonus: Custom Domain Setup

Troubleshooting

"502 Bad Gateway" or "413 Payload Too Large"

"Database Connection Timeout"

Media Not Loading (CORS Error)

Logs and Debugging

Summary and Conclusions

Next Steps

References

See Also

Dynamic Email Domain Validation in Keycloak with a Custom Authenticator

Introduction

Why Static Domain Restrictions Fall Short in B2B Products

Architecture Overview

Step 2 — Generate PMTiles with named layers (`generate-tiles.sh`)

Step 4 — Client animation (`index.html`)

Using `revalidateTag` to Keep Data Fresh

Mistake 3: Using `no-store` too broadly