When One Keycloak Node Is a Single Point of Failure: A Keycloak Cluster Nginx Threat Model for Zero-Downtime Upgrades

The exposure

We ran Keycloak the way most teams start: one node, one VM, "we'll cluster it later." Later arrived as a 40-minute outage. A routine security patch required a restart, the node came back with a config typo, and for the duration nobody in the company could authenticate to anything — not the admin tools, not the customer portal, not the internal apps that all trusted it.

It is worth being precise about what class of exposure that is, because "make it not crash" is the wrong framing. Availability is a security property — it sits alongside confidentiality and integrity in the CIA triad — and a single identity provider is a single point of failure whose blast radius is every system that delegates authentication to it. When that one node is down, the failure is not "one service is slow"; it is that the entire organization loses the ability to prove who anyone is. A keycloak cluster nginx design is the control that bounds that blast radius: the postmortem action item was blunt — Keycloak must survive a node loss and support rolling upgrades so that a routine patch is never again an org-wide authentication outage. I did not want to invent an HA design from first principles, so I worked from a thorough third-party guide to running Keycloak behind Nginx as a real cluster → alongside the official docs.

Threat model

Framing the outage as a security problem dictated what we actually had to build.

• Total authentication loss on node failure. With one node, any crash, bad config, or patch takes down authentication for every dependent system simultaneously. The control objective is that a single node loss is survivable and invisible to users.
• Patch and upgrade as forced outage. If the only way to apply a security patch is to restart the sole node, then staying patched and staying available are in direct conflict — which pressures teams to defer security updates. Rolling upgrades remove that conflict, so the cluster is also a patch-hygiene control.
• Session loss as a re-authentication storm. If sessions live only on the failed node, a failover forces every active user to log in again at once. Beyond the user impact, a synchronized re-auth burst is itself a load event against the surviving node. Session state must survive a node loss.
• Proxy misconfiguration as an integrity risk. Putting a reverse proxy in front means Keycloak now trusts forwarded headers to build redirect URLs and evaluate the request origin. Mishandled X-Forwarded-* headers produce wrong issuer/redirect behavior — a correctness-and-trust problem, not just a cosmetic one.

Controls we added

Control 1 — two nodes with replicated session state

Clustering adds redundancy, and Infinispan provides the distributed caches that replicate sessions and tokens across nodes, so a user logged in on node 1 stays logged in if node 1 disappears (Keycloak's distributed-cache guide documents the session, token, and authentication-session caches). Discovery is the first real decision: on VMs, JGroups with JDBC_PING (nodes find each other through a shared database table — now Keycloak's default discovery mechanism); on Kubernetes, KUBE_PING via the API with a StatefulSet and headless Service. We were on VMs with a shared Postgres, so we started on JGroups/JDBC_PING and planned a later migration to KUBE_PING.

Control 2 — Nginx as the boundary, configured deliberately

Nginx terminates TLS, hides the internal cluster topology, and load-balances across the nodes — the reverse-proxy pattern documented for HTTP load balancing. The Keycloak side uses the modern Quarkus proxy flags, which are the crux of running behind a proxy safely:

command: >
  start --http-enabled=true
        --hostname-strict=false
        --proxy-headers=xforwarded
        --metrics-enabled=true

--proxy-headers=xforwarded is the line people miss; the legacy PROXY_ADDRESS_FORWARDING env var is gone. Keycloak's reverse-proxy guide is explicit that xforwarded enables parsing of the X-Forwarded-* headers and that you relax --hostname-strict (or pin --hostname) so redirect URLs are built correctly. Getting this wrong produces "invalid issuer" and redirect errors — and, more subtly, means Keycloak may be trusting client-supplied headers it should not, which is why the proxy must be the only thing setting them.

The Nginx config matches those flags — TLS termination, forwarded headers, passive health checks, and sticky sessions:

upstream keycloak_http {
  ip_hash;                      # simple session affinity
  server keycloak-1:8080 max_fails=3 fail_timeout=10s;
  server keycloak-2:8080 max_fails=3 fail_timeout=10s;
}

server {
  listen 443 ssl http2;
  ssl_protocols TLSv1.2 TLSv1.3;
  location / {
    proxy_pass http://keycloak_http;
    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Port  $server_port;
  }
}

Passive health checks (max_fails=3 fail_timeout=10s) evict a bad node after a few failures. On affinity: ip_hash hashes on the first three octets of the client's IPv4 address (nginx ngx_http_upstream_module), which is coarse — everyone behind one corporate NAT lands on the same node. Cookie-based stickiness on the AUTH_SESSION_ID cookie pins each browser session to the node that owns it; we started with ip_hash for simplicity and moved to AUTH_SESSION_ID affinity as traffic grew.

Control 3 — hardening the now-critical identity tier

Concentrating all authentication behind a cluster makes that cluster the highest-value asset on the network, so it earns defense in depth beyond mere redundancy:

• Pin --hostname so external URLs are consistent across nodes and cannot be influenced by a spoofed Host header.
• Enable --metrics-enabled=true, scrape with Prometheus, and alert on 5xx, login errors, and session spikes — a login-error spike is a security signal (credential stuffing, brute force), not just an ops one, so lost telemetry is lost detection.
• Run managed HA Postgres with DB TLS and tuned pools.
• Lock the admin console behind IP/VPN, rotate admin credentials, and require 2FA for admins — the admin console is a full realm-compromise surface and does not belong on the open internet.
• On Kubernetes, use a StatefulSet, a headless Service, and NGINX Ingress with correct timeout/header annotations.

Verifying the control, not just shipping it

An HA design is a hypothesis until failure is demonstrated on demand — assuming HA works because the diagram says so is how you end up with a more expensive single point of failure. Before pointing production DNS at the cluster we ran deliberate failure drills: with a colleague mid-session in the admin console, I killed keycloak-1 outright; Nginx's passive health check marked it down within fail_timeout, traffic shifted to keycloak-2, and because Infinispan had replicated the session, the colleague never saw a login prompt. We repeated it in reverse, then killed a node during an active token refresh. Only after all three drills passed did we cut over. Two lessons from that exercise are worth keeping: session affinity and replication are different controls (without stickiness, users bounced between nodes mid-login and hit failures in the brief window before a session propagated; replication is the safety net if a node actually dies — you want both), and JGroups is fussy about networking (our first attempt only half-formed a cluster because a firewall rule silently dropped discovery traffic, so confirm the cluster actually forms rather than assuming it did).

Residual risk / what we're still watching

Clustering bounded the blast radius of a node loss, but it introduced new failure modes and a more valuable target, and naming them is the point.

• The cluster is now the highest-value asset. Redundancy protects availability, but it does not protect against a compromise of the identity tier itself — a stolen admin credential or a Keycloak CVE is now a whole-org event. That is why the admin console is VPN-gated with mandatory 2FA, and why we track Keycloak advisories directly rather than waiting for a bump.
• Cluster-formation and discovery are silent failure modes. JGroups needs clean ports and coherent discovery, and inter-node latency degrades replication. A cluster that silently fails to form looks healthy until a failover drops sessions. We monitor cluster membership as a first-class signal.
• Forwarded-header trust boundary. Because Keycloak now trusts X-Forwarded-* from the proxy, the proxy must be the only source of those headers; a path that lets a client inject them would let it influence issuer/redirect behavior. We treat the proxy configuration as security-sensitive and reviewed.
• Session-affinity coarseness under NAT. ip_hash concentrates large NATed populations onto one node — an availability and load hot-spot; AUTH_SESSION_ID affinity mitigates it, and we watch per-node session distribution.
• Multi-AZ is still open. Two nodes in one availability zone still share an AZ-failure fate. We are finishing the migration to KUBE_PING on Kubernetes and spreading nodes multi-AZ so a zone loss is as boring as a node restart.

The morning after go-live we patched a node in the middle of the workday and nobody noticed — which was the entire point. The residual work is to keep proving that property holds as the topology changes, and to keep treating the identity tier as the critical asset a whole organization now depends on.

Sources & further reading

• Keycloak — Configuring a reverse proxy — the proxy-headers values and hostname flags, straight from the source.
• Keycloak — Configuring distributed caches — how Infinispan replicates sessions and how nodes discover each other.
• NGINX — HTTP Load Balancing — upstreams, passive health checks, and session persistence.
• nginx ngx_http_upstream_module — the ip_hash directive and its IPv4 hashing behavior.
• A Keycloak + Nginx clustering walkthrough — a useful third-party account of the Docker lab, production nginx.conf, and hardening checklist this analysis draws on.

Comments

[Alex Shev]

I like seeing Keycloak treated as infrastructure that needs a threat model, not just an auth box you install and forget. The zero-downtime part is especially important because auth incidents rarely fail gracefully. A small routing or session-state mistake can turn into every app looking broken at once.