The exposure
The riskiest asset in our environment was never a customer-facing app. It was the component every one of those apps delegated trust to. Our Keycloak deployment issued the OIDC tokens and enforced the login flow for every internal service, and it ran on a single, hand-configured EC2 instance, auth-prod-1, with PostgreSQL installed on the same host. No infrastructure-as-code, no reviewed change history, no standby. Roughly two years of realm, client, user, and role-mapping data lived on one EBS volume for which nobody had a tested restore path.
Framed in blast-radius terms, that box was the highest-value single point of failure we owned. Its loss did not degrade one service; it revoked the ability to authenticate to all of them. And because the identity store was co-located with the application, a host compromise was not merely a downtime event, it was a full credential-store disclosure: password hashes, client secrets, and active session state, all reachable from the same shell. The state had been built by clicking and SSH-ing over two years, which meant the exposure was also un-auditable: we could not enumerate, with confidence, what was actually reachable or how it was configured. That uncertainty is itself a security finding.
The goal of the rebuild was not "move to containers." It was to shrink the blast radius, make the configuration auditable, and put defensible boundaries between the parts of the system that currently fail together.
Threat model
Before touching Terraform I mapped out what we were actually defending, because "make it highly available" is not a threat model.
Assets, in priority order. The integrity of the token-issuance path (an attacker who can mint or alter tokens owns every downstream service); the confidentiality of the realm database (password hashes, client secrets, session data); and the availability of the identity provider itself.
Adversaries and failure modes we scoped in, roughly ordered by the residual risk they carried on the old design:
- Host compromise escalating to credential-store access. Application and database shared a trust boundary, so any RCE in Keycloak, any dependency CVE, any stray SSH key was one hop from the full user store. The blast radius of an app compromise equaled the value of the data on the same box.
- Loss-of-availability events. A single instance, a single AZ, a single disk: availability of the entire authentication plane depended on the least reliable component in the stack.
- Un-auditable configuration drift. Hand-applied changes with no review gate mean no way to reason about the current attack surface, or to tell an unauthorized change from an authorized one.
- Network-adjacent lateral movement. A flat network path to the database, secured by host-level assumptions rather than an explicit deny-by-default boundary.
- Application-layer identity flaws in Keycloak itself. Open-redirect and redirect-URI validation classes (for example the path-normalization bypass tracked as CVE-2024-1132, and the SSRF-via-request_uri class of CVE-2020-10770) are exploitable regardless of where Keycloak runs, and they leak or misdirect tokens. Hosting choices do not fix these; a patch cadence and strict hostname/redirect config do.
Explicitly out of scope for this pass: realm-level policy hardening (MFA enforcement, password policy, token lifetimes) and a formal disaster-recovery game day. Naming them as out of scope is deliberate. They are known residual risk, tracked below, not oversights.
Controls we added
The design applies defense in depth: no single control is load-bearing, and each one narrows the blast radius of the layer above it. We rebuilt the plane as Keycloak on ECS Fargate, Aurora PostgreSQL in private subnets, an Application Load Balancer, and Route 53, defined entirely as reviewed Terraform. A third-party writeup on the full Terraform setup covers the complete VPC/ALB/Route 53 wiring; below I focus on the controls and why each one earns its place in the threat model.
Control 1 — Remove the host to shrink the compromise surface
Moving Keycloak onto ECS Fargate removes the long-lived, hand-patched host from the picture entirely. There is no SSH surface, no persistent OS to accrue drift, and the container is replaced rather than patched in place. The task definition runs the official image in production mode against the managed database, following Keycloak's own production configuration guidance:
resource "aws_ecs_task_definition" "keycloak" {
family = "keycloak"
network_mode = "awsvpc" # mandatory on Fargate; gives each task its own ENI
requires_compatibilities = ["FARGATE"]
cpu = "1024"
memory = "2048"
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn # least-privilege pull + secrets read
task_role_arn = aws_iam_role.keycloak_task_role.arn
container_definitions = jsonencode([
{
name = "keycloak"
image = "quay.io/keycloak/keycloak:24.0.5"
essential = true
portMappings = [{ containerPort = 8080, protocol = "tcp" }]
environment = [
{ name = "KC_DB", value = "postgres" },
{ name = "KC_HOSTNAME", value = "auth.example.com" },
{ name = "KC_PROXY_HEADERS", value = "xforwarded" }
]
secrets = [
{ name = "KC_DB_URL", valueFrom = aws_secretsmanager_secret.kc_db_url.arn },
{ name = "KC_DB_USERNAME", valueFrom = aws_secretsmanager_secret.kc_db_user.arn },
{ name = "KC_DB_PASSWORD", valueFrom = aws_secretsmanager_secret.kc_db_pass.arn },
{ name = "KEYCLOAK_ADMIN", valueFrom = aws_secretsmanager_secret.kc_admin_user.arn },
{ name = "KEYCLOAK_ADMIN_PASSWORD", valueFrom = aws_secretsmanager_secret.kc_admin_pass.arn }
]
command = ["start", "--optimized"]
}
])
}
Two decisions are security-relevant. command = ["start", "--optimized"] runs Keycloak in production mode against a pre-built configuration rather than start-dev, disabling the dev-mode conveniences that widen the attack surface. And every credential moves out of environment into secrets, sourced from Secrets Manager at runtime, so plaintext passwords never sit in the task definition, in Terraform state as a resolved value, or in the ECS console.
Control 2 — Isolate the credential store behind a deny-by-default boundary
The single most important boundary in the old design that did not exist: the database is no longer on the application host. It moves to managed Aurora PostgreSQL in private subnets, and its ingress is scoped to the Keycloak service's security group rather than a CIDR range. That is least privilege expressed at the network layer: the database accepts 5432 from the identity of the calling service, and nothing else.
resource "aws_security_group" "database" {
name = "keycloak-database-sg"
vpc_id = module.vpc.vpc_id
ingress {
description = "PostgreSQL from Keycloak tasks only"
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [aws_security_group.keycloak.id] # SG-to-SG, not a CIDR
}
}
resource "aws_rds_cluster" "keycloak" {
cluster_identifier = "keycloak-cluster"
engine = "aurora-postgresql"
engine_version = "15.4"
database_name = "keycloak"
master_username = "keycloak"
master_password = var.database_password # bootstrap only; rotated into Secrets Manager
backup_retention_period = 14
storage_encrypted = true # encryption at rest for the user store
db_subnet_group_name = aws_db_subnet_group.keycloak.name
vpc_security_group_ids = [aws_security_group.database.id]
skip_final_snapshot = false
final_snapshot_identifier = "keycloak-final"
}
storage_encrypted = true protects the user store at rest, the SG-to-SG rule collapses the reachable network path to exactly one source, and 14-day retention with a real final snapshot gives us a recovery point the old ad-hoc setup never had. The host-compromise-to-credential-store path from the threat model is now a two-boundary problem instead of a one-shell problem.
Control 3 — Terminate TLS at the ALB and trust the proxy correctly
The ALB terminates TLS and is the only component with a public interface; the tasks sit in private subnets, unreachable except through it. This matters for correctness as well as exposure: behind a terminating proxy, Keycloak must be told to trust forwarded headers or it builds redirect and issuer URLs from the wrong host, which is both a broken login flow and, misconfigured, an open-redirect risk. Per the reverse-proxy guide we set KC_PROXY_HEADERS=xforwarded (the older KC_PROXY flag was deprecated in Keycloak 24), and per the hostname guide we pin KC_HOSTNAME to the exact Route 53 domain so issuer and admin URLs cannot be coerced to an attacker-supplied host. Locking the hostname is the direct mitigation for the redirect-validation CVE class from the threat model.
Verification
Controls that are not verified are assumptions. Before cutting over we exercised each one against the threat model:
- Network boundary. From a host outside the Keycloak security group, a connection to the Aurora endpoint on 5432 must time out; from a Keycloak task it must succeed. We confirmed both, so the SG-to-SG rule is enforcing, not merely declared.
- Secret hygiene. We grepped the rendered task definition and the ECS console for plaintext credentials and confirmed only Secrets Manager ARNs appear, and that resolved secret values are absent from Terraform state.
-
Proxy and hostname correctness. We validated that the OIDC discovery document, issuer, and every client redirect URI resolve to the canonical Route 53 hostname over TLS, and that requests spoofing a
HostorX-Forwarded-Hostheader do not change the issued URLs. -
Migration integrity. We exported the realms with
kc.sh export, imported into a fresh Aurora database, and checked every client and redirect URI against a checklist. The new stack ran in parallel on a temporary hostname and served real logins before we moved DNS. Because the old host stayed untouched, rollback was a one-line DNS revert, keeping the cutover's own blast radius small.
Residual risk / what we're still watching
Rebuilding the plane reduced the exposure; it did not eliminate risk, and it would be dishonest to present it that way. What we are still watching:
- Patch cadence is now the primary risk owner for application-layer flaws. Fargate solved host patching but not Keycloak's own CVE stream. Redirect-URI and SSRF classes like CVE-2024-1132 and CVE-2020-10770 are exploitable regardless of hosting, so we are wiring image-tag currency and CVE alerting into the pipeline and treating a stale image tag as a finding.
- Bootstrap credentials and rotation. The Aurora master password still enters the world as a Terraform variable before it is rotated into Secrets Manager, and rotation is not yet automated. That bootstrap window is real residual risk; automatic rotation is the next control.
- Distributed session state across tasks. Multiple Keycloak tasks behind the ALB without a shared cache can drop sessions as requests bounce between instances. Until the Infinispan/JDBC-Ping cache is configured we lean on sticky sessions, which is a workaround, not a design.
- Realm-level policy and a DR game day remain out of scope. MFA enforcement, token lifetimes, and a tested full-restore of the encrypted Aurora snapshot are named, tracked work. A backup you have never restored is a hypothesis.
- Admin console exposure. The admin endpoints currently share the ALB with the login flow. Splitting them onto a restricted path or internal listener is queued to further shrink the surface an external attacker can reach.
The core exposure, an entire company's authentication resting on one un-auditable pet server, is gone. What remains is a set of named, bounded risks with owners, which is a defensible position to operate from rather than a fragile one to hope over.
Sources
- Configuring Keycloak for production — production-mode baseline and hardening checklist.
-
Configuring a reverse proxy (Keycloak) — proxy header modes and why
KC_PROXY_HEADERSreplacedKC_PROXY. -
Configuring the hostname (Keycloak) — pinning
KC_HOSTNAMEand issuer URLs behind a load balancer. -
Amazon ECS task definition parameters for Fargate — the
awsvpcrequirement and per-task ENI behavior. - What is Amazon Aurora? (AWS docs) — the managed, encrypted HA database backing the user store.
- A thorough third-party writeup on the full Terraform setup — the complete RDS/ALB/VPC/Route 53 Terraform for this architecture.
Top comments (0)