DEV Community: Gaberial Sofie

One Realm for Every App and Tenant Was a Time Bomb: A Keycloak Realms Clients Roles Threat Model

Gaberial Sofie — Wed, 12 Aug 2026 12:32:06 +0000

The exposure

The first Keycloak rollout worked on the first try, which is the dangerous part. Under the hood it was one realm for every app and every tenant, wildcard redirect URIs, a client secret embedded in a single-page app, and roles assigned directly to individual users. None of it threw an error, and that is precisely why it festered for six months until we had three apps, two customer tenants, and a growing suspicion the whole thing was held together with tape.

Understanding keycloak realms clients roles correctly is a security exercise, not a modeling preference, because each of those four sins maps to a concrete class of vulnerability. A working login flow and a safe one look identical from the outside, and the console will happily let you build the former long before you have the latter. Before changing anything it was worth cataloguing exactly what each shortcut exposed us to. For the concept model and the remediation I worked from a thorough third-party guide to Keycloak's core concepts → alongside the official administration guide.

Threat model

No tenant isolation — shared blast radius. One realm carrying all apps and both tenants means a misconfiguration, a leaked admin credential, or a bad role change in one tenant's context has blast radius across every tenant. There is no boundary to contain a mistake, which is the opposite of what a realm is for.
Wildcard redirect URIs — open redirect and token exfiltration. http://localhost:3000/* was convenient in dev and quietly rode into prod. A wildcard redirect lets an attacker steer the authorization-code callback to a URL they control, turning a valid login into a token-leak primitive.
Client secret in a SPA — a secret that is not secret. The single-page app was registered as if it could hold a secret; the secret shipped in the browser bundle, readable by anyone. A confidential client whose secret is public is a confidential client in name only, and it undermines the entire client-authentication assumption.
Per-user role assignment — unauditable and unscalable. Assigning roles to individuals by hand means "who can do X?" is answerable only by enumerating every user, and every onboarding is a manual chance to over-grant. Least privilege is not enforceable if entitlements are scattered.

The organizing principle for the fix is the three-word model: realms isolate identities, clients integrate apps, roles authorize actions. Every defect above was a violation of one of those three boundaries, so the remediation is to restore each boundary deliberately.

Controls we added

Control 1 — realms as isolation boundaries

A realm is a self-contained boundary with its own users, groups, clients, and roles — Keycloak's definition is that "a realm manages a set of users, credentials, roles, and groups" (Server Administration Guide). We split by tenant, and at minimum by environment (dev/staging/prod), so a misconfiguration in one is contained to one. Users no longer span realms automatically, and that is the point: if identity genuinely needs to be shared, identity brokering or federation is the explicit, reviewable tool for it, rather than an implicit shared bucket.

There is a ceiling to this control worth stating honestly, because over-applying it creates its own risk. Realm-per-tenant is right for a handful of tenants but stops scaling around a couple dozen — past that you are operating an IAM platform instead of shipping a product, and a sprawl of realms becomes its own misconfiguration surface. Keycloak's Organizations feature (preview in 25, GA in 26 — announcement) provides first-class multi-tenancy inside a single realm for that case. The rule: separate realms to isolate genuinely different security domains (different admins, or dev/staging/prod); Organizations to partition many customers who share one app's trust boundary.

Control 2 — clients with an honest access type and PKCE

A client's access type is a security control, not a convenience setting:

public — SPAs and native apps; no secret, because anything shipped to a browser or device is readable.
confidential — server-side apps that can actually keep a secret.
bearer-only — APIs that verify tokens and never start a login flow.

The SPA became a public client, and the leaked secret was retired rather than rotated-in-place, because the fix is to stop pretending a browser bundle is private, not to ship a fresh secret into the same exposure. What replaces the secret is PKCE (S256): designed specifically for OAuth public clients that cannot keep a secret (RFC 7636), it stops an intercepted authorization code from being redeemed by an attacker (Auth0's PKCE explainer). Turning it on is non-negotiable for browser and native clients. Alongside it, the wildcards died: valid redirect URIs and web origins are now exact entries, no *, to close the open-redirect vector.

Control 3 — roles and groups for least privilege by construction

Roles model permissions, and we now use two flavors deliberately: realm roles for org-wide levels (admin, editor, viewer) and client roles for app-specific permissions (billing:read). Composite roles let one Admin assignment imply a bundle — though a composite is also where over-grant hides, so each composite is reviewed for exactly what it expands to. The biggest change was mapping roles to groups, not individuals: assigning to users does not scale and is not auditable; group-based mapping is both. A new hire joins a group and inherits precisely the intended roles, which makes least privilege the default path rather than a discipline someone has to remember.

Control 4 — authorize from the token, not from identity

The refactor only pays off if the application uses the structure. The early mistake was scattering permission checks keyed on usernames — a hard-coded allow-list is unauditable and drifts silently. The clean version reads roles straight from the token: in OIDC, Keycloak places realm roles under realm_access.roles and client roles under resource_access[client].roles. An authorization check becomes "does this token carry the editor role?" rather than "is this user in my admin list?" Because Admin is modeled as a composite, one role in the token expands to everything it should, consistently. Permissions become a configuration concern in Keycloak, not a deployment concern in the codebase — a new hire is added to a group, the roles land in their token, and the app authorizes them with no code change or redeploy.

Verifying the controls, not just shipping them

The rebuild started from a minimal, reviewable baseline rather than clicking around the console:

docker run -p 8080:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:25.0.2 start-dev

Then a five-step skeleton: a dedicated realm per security domain, each app as its own client with exact redirect URIs and the correct access type, realm roles like Admin and User, and assignment via groups. The verification that mattered most was that the controls are now demonstrable: "who can write billing?" used to mean grepping code and cross-referencing a user table; now it is "which groups carry a role that composes billing:write?", answerable in the admin console in seconds. An entitlement you can query is an entitlement you can audit, and negative-testing a redirect URI with a deliberately invalid value confirms the allow-list actually rejects what it should.

Residual risk / what we're still watching

Restoring the three boundaries removed the acute exposures, but each control carries its own residual risk.

Realm sprawl vs. Organizations. Separate realms isolate cleanly at small scale but become an operational and misconfiguration burden past a couple dozen tenants. We are watching the tenant count against the point where Organizations-in-one-realm is the safer trade, since the wrong choice in either direction reintroduces risk — sprawl on one side, a weaker isolation boundary on the other.
PKCE and redirect-URI drift. The public-client posture depends on PKCE staying enforced and redirect URIs staying exact. A future convenience edit that re-adds a wildcard, or a client that quietly disables PKCE, silently reopens the token-leak vector. Both live in reviewed configuration.
Composite-role over-grant. Composites are ergonomic but concentrate authority; a role added to a widely-assigned composite grants it to everyone downstream at once. We review composite membership as an entitlement change, not a convenience.
Group hygiene. Group-based assignment scales, but it moves the risk to group membership — a stale or over-broad group is now the thing to audit. We periodically reconcile group membership against intended access.
The leaked secret is retired but assume-compromised. Because the old SPA secret was public for months, we treat it as burned permanently and monitor for any client still configured to accept it.

The lesson that stuck is that the console lets you build something that works long before it is something that is safe, and the three-word model — isolate, integrate, authorize — is both the setup mantra and the operating model that keeps the system explainable, and auditable, a year later.

Sources & further reading

Keycloak — Server Administration Guide — realms, clients, roles, groups, and composite roles.
Keycloak — Organizations announcement — multi-tenancy inside a single realm (preview in 25, GA in 26).
RFC 7636 — Proof Key for Code Exchange — why public clients use PKCE instead of a secret.
Auth0 — Authorization Code Flow with PKCE — a clear walkthrough of the SPA/native login flow.
A Keycloak realms/clients/roles walkthrough — a useful third-party account of the pitfalls, the cross-realm FAQ, and a runnable five-step example.

A Typosquatted Gem Almost Shipped, Caught by Luck: A Ruby Supply Chain Security Playbook

Gaberial Sofie — Tue, 11 Aug 2026 19:26:01 +0000

The exposure

A teammate opened a PR adding a small utility gem. The name looked right at a glance — one transposed character from a popular package. It passed our tests. On a normal Friday it would have merged. Someone caught the typo in review by luck, not process, and the gem turned out to be a typosquat whose extconf.rb ran code at install time that had no business running.

"By luck, not process" is the phrase that reframes this from an anecdote into a security finding. Ruby supply chain security is the discipline of not depending on luck — or on someone else's controls — for the integrity of code you ship. We were trusting gem install blindly and leaning entirely on RubyGems.org to catch everything upstream. It usually does, but "usually" is a probability, not a guarantee, and a plan that works right up until it doesn't is not a control. RubyGems.org's own account of how it protects the ecosystem puts a number on it: its automated tooling catches roughly 70-80% of malicious packages before anyone reports them. That upstream layer is real and valuable, but the residual 20-30% is precisely the population your build has to assume it will meet. For the fuller hardening notes I worked from a thorough third-party writeup of a Ruby dependency-security playbook → alongside the registry's post.

Threat model

Typosquatting and look-alike names. A one-character-off gem name is a social-engineering primitive aimed at review and autocomplete. It is caught, if at all, by attention — an unreliable control at 2pm on a Friday.
Arbitrary code execution at install time. extconf.rb, native extension builds, and install hooks run on developer laptops and CI runners with whatever privileges those hosts hold. A malicious gem does not need to reach production to do damage; installing it is the exploit.
Lockfile injection and source substitution. A version number can be re-pointed at different bytes upstream, and a machine-generated Gemfile.lock that nobody reads during review can quietly point a dependency at a different set of bytes than the manifest implies. Mixed gem sources open a substitution path where a gem is pulled from an unexpected origin.
The post-disclosure window. A gem that is clean today gets a CVE tomorrow while sitting in your lockfile. Detection that only runs when code changes never revisits already-shipped dependencies.
Over-reliance on upstream controls. Delegating your production security entirely to the registry's 70-80% means the ecosystem's residual miss rate is your residual risk, unmediated.

The organizing principle is defense in depth: the registry is the first line, and the build must be a deliberate second line behind it, because malicious packages are not rare events — they are background noise of the modern software supply chain.

Controls we added

Control 1 — shrink the attack surface

Pin versions and platforms in Gemfile.lock and commit it. No floating ranges into production.
No git dependencies without a commit SHA. A branch can be rewritten; a SHA cannot.
Narrow ranges (~>) for critical gems, and prune ruthlessly — every unused and transitive gem is surface area, reviewed on a quarterly dependency-hygiene day.

Control 2 — checksum verification, the highest-leverage single addition

Pinning a version number is necessary but not sufficient, because the number can be re-pointed at different bytes. Bundler 2.6 makes checksum verification first-class (--add-checksums is a documented bundle lock flag):

bundle lock --add-checksums

This writes a CHECKSUMS section into Gemfile.lock, and from then on Bundler refuses to install a gem whose contents do not match the exact bytes you locked. It closes the nastiest gap in a pin-the-version-only strategy: a checksum cannot be quietly swapped the way a version can be re-pointed. If you adopt one new control this quarter, make it this one. Alongside it, a few Bundler settings harden source handling — disable_multisource true (the underrated one, blocking the source-substitution class), cache_all true, and clean 'true'.

Control 3 — make the human review a process, not a hero act

The near-miss was caught by a person, so we turned the person into a checklist. Adding or bumping a gem now requires a short "why this gem?" note and a CHANGELOG link in the PR, documented in the PR template where the work happens rather than a wiki nobody reads. Dependabot and GitHub alerts are enabled so advisories are not manually watched. The cultural shift — that a dependency is code we are adopting, not a freebie we are grabbing — mattered more than any single tool.

Control 4 — gate it in CI

A checklist people can forget is a suggestion; a CI gate is a rule. Two independent scanners catch different things, and layering them is cheap: bundler-audit (checking Gemfile.lock against the ruby-advisory-db) and Google's OSV-Scanner (fed by osv.dev). It runs on every dependency-touching PR and weekly on a schedule:

on:
  pull_request:
    paths: ['Gemfile', 'Gemfile.lock']
  schedule:
    - cron: '0 6 * * 1'   # weekly on Mondays

jobs:
  bundler-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with: { ruby-version: '3.3', bundler-cache: true }
      - run: gem install bundler-audit
      - run: bundle audit check --update
  osv-scanner:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google/osv-scanner-action@v1
        with: { scan-args: '-L -r .' }

The weekly cron addresses the post-disclosure window directly: a vulnerability disclosed on Wednesday fails the build even if nobody touches the code. For teams that want typosquat detection and SBOM generation on top, a Ruby-specific scanner like GemGuard — pulling from OSV.dev and the ruby-advisory-db — slots in beside bundler-audit rather than replacing it.

Verifying the control, not just shipping it

The fastest way to get a security process ignored is to drop a wall of red CI checks on a team mid-sprint, so we staged it and verified adoption as deliberately as coverage. Week one, the scanners ran in warn-only mode — reporting findings without failing the build. That surfaced the existing backlog of advisories, gave us an honest baseline instead of a nasty surprise, and let us triage and either fix or explicitly accept each item. Only then did we flip the gate to blocking, by which point there was nothing left for it to fail on. The second thing that kept adoption alive was making the checks fast and legible: bundler-audit and OSV both run in well under a minute with a warm cache, and their output points straight at the offending gem and advisory, so a failure has an obvious fix and nobody routes around it. A gate that takes ten minutes or emits inscrutable output gets bypassed, which is how a control becomes theater.

Residual risk / what we're still watching

The playbook meaningfully raised the cost of a supply-chain compromise, but it does not reduce the residual to zero, and the edges are where the honesty has to be.

Scanners detect known vulnerabilities only. A brand-new typosquat or an undisclosed backdoor has no advisory yet and passes clean. That is exactly why the human "why this gem?" review still carries load the automation cannot — the checklist is a control against the unknown-unknown, not redundant with the scanners.
Signatures are not a force field. Gem signing is one control among pinning, checksums, review, and CI gating — treating it as a silver bullet would be a false-confidence failure. We weight it accordingly.
The lockfile diff still needs human eyes. Checksum verification defuses most lockfile-injection risk, but a reviewer who reads only the Gemfile and skims the machine-generated Gemfile.lock can still miss a dependency quietly re-pointed. We review the lockfile diff on any dependency PR.
Patch-bump noise vs. coverage. A blanket manual-review policy drowns the team; letting CI and scans gate patch bumps while humans review major/minor keeps signal high, but it does accept that a malicious patch release is caught by scanners rather than eyes. We watch that trade.
We still depend on the registry and the advisory databases. Our second line assumes RubyGems.org's first line and assumes OSV.dev and the ruby-advisory-db are current; a gap or delay in any of them is a gap in us. Two independent scanners mitigate the advisory dependency, and — because that shared line of defense runs partly on a small group's effort — we treat a recurring sponsorship of the ecosystem as risk management, not charity: the CI gate protects our repo, sponsorship protects the line in front of it.

If a team ships Ruby, the near-miss described here is a matter of when, not if. The residual we keep watching is the undisclosed package and the lockfile change nobody reads closely — the two places where luck, not process, is still doing quiet work.

Sources & further reading

RubyGems.org — How RubyGems.org Protects Our Community's Critical OSS Infrastructure (Aug 25, 2025)
rubysec — bundler-audit and the ruby-advisory-db it checks against
Google — OSV-Scanner, backed by osv.dev
Bundler docs — bundle lock --add-checksums (Bundler 2.6 checksum verification)
wilburhimself — GemGuard: typosquat detection + SBOM for Ruby
A full Ruby supply-chain playbook — a useful third-party account of the Bundler hardening settings and legacy-app rollout FAQ.

Governance Risk Is Supply-Chain Risk: Auditing Every Dependency After the RubyGems Hostile Takeover

Gaberial Sofie — Mon, 10 Aug 2026 08:12:28 +0000

The exposure

In September 2025 the people who built and ran RubyGems and Bundler lost administrative access to the infrastructure they maintained, and several publicly quit. One side called it a hostile takeover; the other called it responsible stewardship. The verdict is not the point. The exposure is: infrastructure serving billions of downloads a month — the infrastructure every bundle install, every deploy, every CI run reaches into — demonstrated that control over the packages your builds depend on can change hands overnight.

Framed as a security problem, the rubygems hostile takeover is a governance-risk event, and governance risk is supply-chain risk that no vulnerability scanner will ever flag. A CVE scanner tells you a dependency has a known flaw; it says nothing about whether that dependency's maintainers just walked out or its publishing rights just moved to someone you did not vet. My company does not contribute to RubyGems — we only consume it, which means we had silently accepted a dependency on both the packages and the people, and the crisis made that dependency visible. The Register covered the control fight as it erupted, Ruby Central later published its own RubyGems Fracture Incident Report owning the execution mistakes, and I worked through a thorough third-party first-person timeline of what one engineer changed in response → while treating our own exposure as a fire drill rather than a hot take.

Threat model

The timeline matters only insofar as it names the threats. Between September 9 and 19, the GitHub org was renamed, maintainers were removed, administrative access to the RubyGems, Bundler, and RubyGems.org teams was revoked — including control of the bundler and rubygems-update gems themselves — and the creator of Bundler announced the team was effectively finished. From that, the concrete threats:

Publishing rights changing hands overnight. Control over the exact packages your builds pull can move to a different party without your consent or knowledge. If your security model assumes the right people hold the keys, it is not a model — it is a hope. Ruby Central's own incident report concedes the foundation "lacked the structural ability to make this change directly," i.e. an informal power structure that a single actor could reconfigure.
Maintainer-exodus as a failure mode. The people who understand a critical dependency's internals can all leave at once. A healthy maintainer team is itself a dependency, and its collapse is a real failure mode — a bus-factor risk at ecosystem scale.
Registry availability during the incident. If RubyGems.org had gone dark or a gem had been yanked mid-fracture, builds that fetch from upstream at deploy time would have failed. Reachability of a third party you do not control sits on your critical path.
Funding-pressure-driven consolidation. Reporting since made clear how much funding pressure sat underneath the fight — The Register relayed developer Joel Drapper's allegation that Shopify pressed for consolidated control of the bundler and rubygems-update gems under threat of pulled funding. Commercial leverage over volunteer infrastructure is a governance attack surface in its own right.
Ecosystem split. The ousted maintainers did not disappear; Martin Emde announced The Gem Cooperative (gem.coop), a community-run mirror, which The Register covered as the maintainers' new home. "Which registry does our bundle install actually hit" went from a question I would have shrugged at to a line item.

The organizing principle: reproducibility is resilience. If a build needs the internet — and a particular governance arrangement — to be calm in order to succeed, it will fail when they are not.

Controls we added

I gave the drill one week and one goal: make sure a governance blowup or a bad actor upstream could neither stop us shipping nor poison our builds.

Control 1 — freeze the ground we stand on

Committed and audited Gemfile.lock so every version and platform is pinned, not floating into production.
Vendored the critical path. bundle cache copies every .gem into vendor/cache and, per the docs, makes future installs "use the gems in the cache in preference to the ones on rubygems.org"; bundle config set cache_all true keeps git and path gems there too. If RubyGems.org went dark or a gem was yanked mid-incident, deploys still work from the cache. This directly retires the registry-availability threat.
Pinned git dependencies to commit SHAs, never bare branch names. A branch can be force-pushed under new ownership; a SHA cannot be quietly swapped — which addresses the "control changed hands" threat at the integrity level.

Control 2 — inventory governance risk, not just CVE risk

Because scanners like bundler-audit (which checks Gemfile.lock against the ruby-advisory-db) are silent on governance, I built a checklist and ran our top dependencies through it:

Who actually controls publishing rights to this package today?
How many maintainers — is it a bus-factor of one?
Has ownership or org name changed recently?
Is there a funded, formal body behind it, or informal goodwill?
If it vanished tomorrow, what is our replacement path?

That last question turns a vague anxiety into a ranked list of things to fix, which is the difference between worrying about governance risk and actually managing it.

Control 3 — reduce the surface

Fewer upstreams means fewer governance fires can burn you. I pruned unused and transitive bloat and added a "why this dependency, and who is behind it?" note to code review for anything new — the same hygiene the ecosystem needs at the macro level, applied to our own repo. Least privilege for dependencies: adopt only what you can justify and account for.

Verifying the controls, not just shipping them

A control the team does not believe in gets removed the first time it is inconvenient, so the verification here was as much organizational as technical. I made the risk concrete with three questions borrowed from the crisis: if the maintainers of our single most critical dependency all quit tomorrow, what breaks and how fast; who can publish a new version of that package right now, and do we trust the process that grants them that power; and if that package got hijacked and shipped a malicious release, would our pipeline catch it before production. For most dependencies the honest first answer was "we don't know," and "we don't know" is the actual risk — so the checklist and the vendoring exist to convert those unknowns into documented, tested answers. The vendoring in particular is verifiable in the most direct way: a build run with upstream network access blocked either succeeds from vendor/cache or it does not, and ours does. Unexpectedly, "we are hardening against a real, recent, widely-reported failure in our supply chain" turned out to be the easiest resilience work I have ever had to justify to leadership.

Residual risk / what we're still watching

The drill meaningfully reduced our exposure to an upstream governance shock, but it did not make us independent of the ecosystem, and the edges deserve honesty.

Vendored gems drift and stale. vendor/cache protects availability, but a frozen cache also stops receiving security updates; a vulnerability disclosed in a vendored gem is now on us to notice and refresh. Vendoring trades an availability risk for a patch-currency obligation, so we pair it with scheduled re-audits rather than treating the cache as done.
Scanners still cannot see governance. The checklist is a manual control against a threat no tool flags, which means it decays unless it is re-run. We re-inventory the top dependencies periodically, because ownership and maintainer health change quietly between reviews.
The ecosystem split is unresolved. With gem.coop now standing alongside RubyGems.org, "which registry are we actually trusting, and what is its provenance" is an ongoing decision, not a settled one. We are watching how it resolves without letting our builds depend on it resolving well.
Transitive dependencies we do not control. We can inventory our direct dependencies' governance, but the long tail of transitive gems carries the same risk with far less visibility. That surface is the hardest residual to close, and it is where we assume the next surprise lives.
Our second line still assumes a first line. Vendoring and pinning protect us from upstream disappearing, not from a malicious release we vendor before anyone notices it is malicious — that gap is covered by scanning and review, which are themselves only as current as their data.

The durable lesson is that you depend on people, not just packages, and that a small group quietly holds up infrastructure the whole ecosystem leans on. When that group fractures, everyone downstream discovers exactly how much they were depending on trust they never examined. What we keep watching is that trust — deliberately, on calm afternoons, rather than during the next incident.

Sources & further reading

The Register — Open source to closed doors: RubyGems control fight erupts (Sep 25, 2025)
Ruby Central — RubyGems Fracture Incident Report
Martin Emde — Announcing The Gem Cooperative (gem.coop) (Oct 5, 2025)
The Register — Kicked from RubyGems, maintainers forge new home at Gem Cooperative (Oct 6, 2025)
Bundler docs — bundle cache: vendoring gems into vendor/cache
rubysec — bundler-audit: scan Gemfile.lock against the ruby-advisory-db
A field-notes dependency-audit checklist from the week of the crisis — a useful third-party account of the same governance-risk drill applied to a real repo.

When One Keycloak Node Is a Single Point of Failure: A Keycloak Cluster Nginx Threat Model for Zero-Downtime Upgrades

Gaberial Sofie — Sun, 09 Aug 2026 08:50:41 +0000

The exposure

We ran Keycloak the way most teams start: one node, one VM, "we'll cluster it later." Later arrived as a 40-minute outage. A routine security patch required a restart, the node came back with a config typo, and for the duration nobody in the company could authenticate to anything — not the admin tools, not the customer portal, not the internal apps that all trusted it.

It is worth being precise about what class of exposure that is, because "make it not crash" is the wrong framing. Availability is a security property — it sits alongside confidentiality and integrity in the CIA triad — and a single identity provider is a single point of failure whose blast radius is every system that delegates authentication to it. When that one node is down, the failure is not "one service is slow"; it is that the entire organization loses the ability to prove who anyone is. A keycloak cluster nginx design is the control that bounds that blast radius: the postmortem action item was blunt — Keycloak must survive a node loss and support rolling upgrades so that a routine patch is never again an org-wide authentication outage. I did not want to invent an HA design from first principles, so I worked from a thorough third-party guide to running Keycloak behind Nginx as a real cluster → alongside the official docs.

Threat model

Framing the outage as a security problem dictated what we actually had to build.

Total authentication loss on node failure. With one node, any crash, bad config, or patch takes down authentication for every dependent system simultaneously. The control objective is that a single node loss is survivable and invisible to users.
Patch and upgrade as forced outage. If the only way to apply a security patch is to restart the sole node, then staying patched and staying available are in direct conflict — which pressures teams to defer security updates. Rolling upgrades remove that conflict, so the cluster is also a patch-hygiene control.
Session loss as a re-authentication storm. If sessions live only on the failed node, a failover forces every active user to log in again at once. Beyond the user impact, a synchronized re-auth burst is itself a load event against the surviving node. Session state must survive a node loss.
Proxy misconfiguration as an integrity risk. Putting a reverse proxy in front means Keycloak now trusts forwarded headers to build redirect URLs and evaluate the request origin. Mishandled X-Forwarded-* headers produce wrong issuer/redirect behavior — a correctness-and-trust problem, not just a cosmetic one.

Controls we added

Control 1 — two nodes with replicated session state

Clustering adds redundancy, and Infinispan provides the distributed caches that replicate sessions and tokens across nodes, so a user logged in on node 1 stays logged in if node 1 disappears (Keycloak's distributed-cache guide documents the session, token, and authentication-session caches). Discovery is the first real decision: on VMs, JGroups with JDBC_PING (nodes find each other through a shared database table — now Keycloak's default discovery mechanism); on Kubernetes, KUBE_PING via the API with a StatefulSet and headless Service. We were on VMs with a shared Postgres, so we started on JGroups/JDBC_PING and planned a later migration to KUBE_PING.

Control 2 — Nginx as the boundary, configured deliberately

Nginx terminates TLS, hides the internal cluster topology, and load-balances across the nodes — the reverse-proxy pattern documented for HTTP load balancing. The Keycloak side uses the modern Quarkus proxy flags, which are the crux of running behind a proxy safely:

command: >
  start --http-enabled=true
        --hostname-strict=false
        --proxy-headers=xforwarded
        --metrics-enabled=true

--proxy-headers=xforwarded is the line people miss; the legacy PROXY_ADDRESS_FORWARDING env var is gone. Keycloak's reverse-proxy guide is explicit that xforwarded enables parsing of the X-Forwarded-* headers and that you relax --hostname-strict (or pin --hostname) so redirect URLs are built correctly. Getting this wrong produces "invalid issuer" and redirect errors — and, more subtly, means Keycloak may be trusting client-supplied headers it should not, which is why the proxy must be the only thing setting them.

The Nginx config matches those flags — TLS termination, forwarded headers, passive health checks, and sticky sessions:

upstream keycloak_http {
  ip_hash;                      # simple session affinity
  server keycloak-1:8080 max_fails=3 fail_timeout=10s;
  server keycloak-2:8080 max_fails=3 fail_timeout=10s;
}

server {
  listen 443 ssl http2;
  ssl_protocols TLSv1.2 TLSv1.3;
  location / {
    proxy_pass http://keycloak_http;
    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Port  $server_port;
  }
}

Passive health checks (max_fails=3 fail_timeout=10s) evict a bad node after a few failures. On affinity: ip_hash hashes on the first three octets of the client's IPv4 address (nginx ngx_http_upstream_module), which is coarse — everyone behind one corporate NAT lands on the same node. Cookie-based stickiness on the AUTH_SESSION_ID cookie pins each browser session to the node that owns it; we started with ip_hash for simplicity and moved to AUTH_SESSION_ID affinity as traffic grew.

Control 3 — hardening the now-critical identity tier

Concentrating all authentication behind a cluster makes that cluster the highest-value asset on the network, so it earns defense in depth beyond mere redundancy:

Pin --hostname so external URLs are consistent across nodes and cannot be influenced by a spoofed Host header.
Enable --metrics-enabled=true, scrape with Prometheus, and alert on 5xx, login errors, and session spikes — a login-error spike is a security signal (credential stuffing, brute force), not just an ops one, so lost telemetry is lost detection.
Run managed HA Postgres with DB TLS and tuned pools.
Lock the admin console behind IP/VPN, rotate admin credentials, and require 2FA for admins — the admin console is a full realm-compromise surface and does not belong on the open internet.
On Kubernetes, use a StatefulSet, a headless Service, and NGINX Ingress with correct timeout/header annotations.

Verifying the control, not just shipping it

An HA design is a hypothesis until failure is demonstrated on demand — assuming HA works because the diagram says so is how you end up with a more expensive single point of failure. Before pointing production DNS at the cluster we ran deliberate failure drills: with a colleague mid-session in the admin console, I killed keycloak-1 outright; Nginx's passive health check marked it down within fail_timeout, traffic shifted to keycloak-2, and because Infinispan had replicated the session, the colleague never saw a login prompt. We repeated it in reverse, then killed a node during an active token refresh. Only after all three drills passed did we cut over. Two lessons from that exercise are worth keeping: session affinity and replication are different controls (without stickiness, users bounced between nodes mid-login and hit failures in the brief window before a session propagated; replication is the safety net if a node actually dies — you want both), and JGroups is fussy about networking (our first attempt only half-formed a cluster because a firewall rule silently dropped discovery traffic, so confirm the cluster actually forms rather than assuming it did).

Residual risk / what we're still watching

Clustering bounded the blast radius of a node loss, but it introduced new failure modes and a more valuable target, and naming them is the point.

The cluster is now the highest-value asset. Redundancy protects availability, but it does not protect against a compromise of the identity tier itself — a stolen admin credential or a Keycloak CVE is now a whole-org event. That is why the admin console is VPN-gated with mandatory 2FA, and why we track Keycloak advisories directly rather than waiting for a bump.
Cluster-formation and discovery are silent failure modes. JGroups needs clean ports and coherent discovery, and inter-node latency degrades replication. A cluster that silently fails to form looks healthy until a failover drops sessions. We monitor cluster membership as a first-class signal.
Forwarded-header trust boundary. Because Keycloak now trusts X-Forwarded-* from the proxy, the proxy must be the only source of those headers; a path that lets a client inject them would let it influence issuer/redirect behavior. We treat the proxy configuration as security-sensitive and reviewed.
Session-affinity coarseness under NAT. ip_hash concentrates large NATed populations onto one node — an availability and load hot-spot; AUTH_SESSION_ID affinity mitigates it, and we watch per-node session distribution.
Multi-AZ is still open. Two nodes in one availability zone still share an AZ-failure fate. We are finishing the migration to KUBE_PING on Kubernetes and spreading nodes multi-AZ so a zone loss is as boring as a node restart.

The morning after go-live we patched a node in the middle of the workday and nobody noticed — which was the entire point. The residual work is to keep proving that property holds as the topology changes, and to keep treating the identity tier as the critical asset a whole organization now depends on.

Sources & further reading

Keycloak — Configuring a reverse proxy — the proxy-headers values and hostname flags, straight from the source.
Keycloak — Configuring distributed caches — how Infinispan replicates sessions and how nodes discover each other.
NGINX — HTTP Load Balancing — upstreams, passive health checks, and session persistence.
nginx ngx_http_upstream_module — the ip_hash directive and its IPv4 hashing behavior.
A Keycloak + Nginx clustering walkthrough — a useful third-party account of the Docker lab, production nginx.conf, and hardening checklist this analysis draws on.

Deleting Hand-Rolled Auth From a Next.js App: A Keycloak Nextjs Threat Model and NextAuth Cutover

Gaberial Sofie — Sat, 08 Aug 2026 17:16:55 +0000

The exposure

Every codebase has one module nobody wants their name on. Ours was lib/auth.ts — a homegrown JWT system in our Next.js app that had been "good enough for the MVP" three years earlier. It signed its own tokens, hand-rolled refresh-token rotation, and stored sessions in a way two engineers actively disagreed about. Exactly one person understood the rotation logic, and when he took vacation we froze all auth-adjacent work out of fear.

That is a security exposure before it is an engineering one. Hand-rolled authentication concentrates several of the highest-consequence failure modes in software into code that is rarely reviewed and understood by one person. A keycloak nextjs integration was attractive not because Keycloak is fashionable but because delegating authentication moves password storage, MFA, token signing, and refresh — the parts where a subtle bug is a breach, not a bug — out of our codebase entirely. The trigger was concrete: a security review flagged four separate issues in that one file. Rather than patch a system we did not trust, we ripped it out and stood on an identity provider we already ran, leaning on the official Auth.js Keycloak provider and Next.js App Router route handlers. The result was about forty lines.

Threat model

Being specific about what a hand-rolled auth module gets you exposed to is what justified deleting it rather than patching it.

Self-implemented token cryptography. A module that signs and verifies its own JWTs is one weak-algorithm choice or one missing signature check away from token forgery. This is the class of defect where "it works" and "it is secure" look identical from the outside.
Refresh-token rotation errors. Refresh logic understood by a single engineer is fragile in the worst place: a bug can either lock users out (availability) or fail to invalidate a stolen refresh token (a persistent-access foothold). The four review findings clustered here.
Ambiguous session handling. Two engineers disagreeing on how sessions are stored is a session-fixation and session-invalidation risk waiting to surface — you cannot invalidate a session cleanly if you cannot agree on where it lives.
Knowledge concentration as a security risk. Auth that only one person understands cannot be safely reviewed, patched, or incident-handled. Bus factor one on the credential path is itself a finding.

The decision: reduce our attack surface by owning as little authentication code as possible, and let Keycloak — whose entire job is to get password storage, MFA, and token signing right — own the security-critical parts.

Controls we added

Control 1 — delegate authentication, own almost none of it

NextAuth.js (branded Auth.js in v5; the next-auth package and Keycloak provider are the same) ships a first-party Keycloak provider. You declare a provider, point it at your realm's issuer URL, and it handles the Authorization Code flow, callbacks, and session cookies. The security-relevant property is subtraction: the token exchange, password handling, and MFA all move behind Keycloak, so the surface where our own code can be wrong shrinks to configuration. I cross-referenced the official provider docs against a thorough third-party walkthrough of the NextAuth-Keycloak wiring → while doing this.

One trap that had burned us: do not mix Pages Router examples into an App Router app. Most stale tutorials use NextApiRequest and pages/api/auth; in App Router you export GET/POST from a route.ts, per the Next.js file convention. Getting that straight up front is a correctness control — a subtly wrong handler on the auth path is not a place to be copy-pasting.

The whole integration lives at app/api/auth/[...nextauth]/route.ts:

import NextAuth from 'next-auth'
import KeycloakProvider from 'next-auth/providers/keycloak'

const handler = NextAuth({
  providers: [
    KeycloakProvider({
      clientId: process.env.KEYCLOAK_CLIENT_ID!,
      clientSecret: process.env.KEYCLOAK_CLIENT_SECRET || '',
      issuer: `${process.env.KEYCLOAK_URL}/realms/${process.env.KEYCLOAK_REALM}`,
    }),
  ],
  secret: process.env.NEXTAUTH_SECRET,
  session: { strategy: 'jwt' },
})

export { handler as GET, handler as POST }

The issuer must include the realm (/realms/<name>) — the provider docs are blunt about it, and a wrong issuer path is a common and security-relevant misconfiguration because it changes which authority you are actually trusting.

Control 2 — client configuration as an enforced allow-list

In Keycloak we created a nextjs-client (confidential, with a client secret) and treated two fields as controls, not preferences:

Valid Redirect URIs: http://localhost:3000/api/auth/callback/*, matched character for character including protocol and trailing slash. A loose or wildcard redirect is an open-redirect and token-exfiltration vector, so it is scoped to exactly where NextAuth listens.
Access type: confidential for the server-side surface, holding a secret; public clients get PKCE instead. Getting this classification wrong is a genuine weakness, not a style choice.

Control 3 — hardening we adopted deliberately

PKCE for public clients, always — RFC 7636 exists precisely to stop authorization-code interception on clients that cannot hold a secret; modern Auth.js enables it for OIDC providers by default.
KEYCLOAK_CLIENT_SECRET and NEXTAUTH_SECRET generated per environment (openssl rand -base64 32), pushed through the secrets pipeline, never committed, and rotated on a cadence.
HTTPS end-to-end with secure cookies in production.
Explicit session strategy (jwt, chosen on purpose) rather than silently mixing jwt and database sessions, which is a classic footgun with real invalidation consequences.

Control 4 — a cutover that never locked anyone out

Swapping authentication on a live product is a high-risk change, so we staged it to stay defensible throughout. We shipped the Keycloak route behind a feature flag and ran it in parallel with the old system for a week, dogfooding internally. We migrated user identities into Keycloak ahead of time and mapped them by email so nobody had to re-register — a re-registration flow is itself a phishing pretext we did not want to create. We kept the old lib/auth.ts in the tree but unreferenced for one release as a rollback escape hatch, then deleted it once the dashboards stayed green — a dormant second auth system is standing attack surface, so it did not linger. The cutover was a single flag toggle at low traffic, and the rollback plan was toggling it back.

Verifying the control, not just shipping it

The failure that validated the approach was a redirect loop that appeared the instant we deployed to staging: no useful error. The cause was a mismatched redirect URI down to a trailing slash the Keycloak Valid Redirect URI did not have. That is worth internalizing as a security habit, not just a debugging tip — redirect-URI matching is the control that keeps authorization codes from going to the wrong place, so when it is strict enough to break on a trailing slash it is also strict enough to reject an attacker's callback. We aligned it and added the check to our deploy runbook. Because authentication now lives behind Keycloak, "does login work" is answerable by driving the flow against the provider rather than reading our own crypto.

Residual risk / what we're still watching

Deleting the hand-rolled module removed a class of self-inflicted crypto and rotation defects, but it relocated risk into configuration and into a dependency.

Access-token refresh is the next sensitive control. Keycloak's default access-token lifespan is five minutes, so refresh matters sooner than teams expect. Refreshing OAuth tokens inside the jwt/session callbacks — the Auth.js Refresh Token Rotation guide is the canonical approach — must handle a failed refresh by forcing re-authentication rather than silently serving a stale identity. This is the piece we are implementing most carefully.
Back-channel logout is a real gap until we close it. Today, signing out of Keycloak does not yet invalidate the Next.js session everywhere. Until back-channel logout is wired, a revoked or logged-out user can retain a valid app session until it expires — a bounded but genuine residual window we are actively tracking.
Redirect-URI and secret drift. A later loosening of the redirect allow-list, or a leaked/committed NEXTAUTH_SECRET or client secret, quietly reintroduces exposure. Both live in reviewed configuration and rotation.
Keycloak is now a dependency on the login path. We traded code we owned and distrusted for a system we must keep patched, monitored, and available. That is the right trade, but it makes Keycloak's own security posture part of ours, and we track its advisories.
Role-based access mapping. As we map Keycloak realm roles into the jwt/session callbacks for authorization, each mapping is a new place a claim can be misread into wrong access, so those callbacks get the same scrutiny the token logic does.

The net effect is that the scariest authentication code we owned is gone, the parts that are hard to get right now live in a system built to get them right, and the remaining risks are named, bounded, and on a list — chiefly refresh handling and back-channel logout — rather than concentrated in one engineer's head.

Sources & further reading

Auth.js — Keycloak provider — the official provider config, issuer/realm rules, and client-authentication setup
Next.js — Route Handlers (route.js) — App Router file convention for GET/POST handlers
Auth.js — Refresh Token Rotation — persisting and refreshing OAuth tokens in the jwt/session callbacks
PKCE for OAuth 2.0 (RFC 7636) — why public clients need it
Keycloak — Securing applications and services with OpenID Connect — OIDC endpoint and integration reference
An end-to-end NextAuth + Keycloak writeup — a useful third-party account of the full client setup, env vars, the App Router route handler, and the hardening checklist.

Collapsing 60 Trust Boundaries Into One: A Keycloak Identity Federation Threat Model and 90-Day Rollout

Gaberial Sofie — Fri, 07 Aug 2026 08:53:16 +0000

The exposure

Through two acquisitions we had accumulated three separate Active Directories, a bolt-on SSO product, and roughly sixty applications each with its own login page. From a security standpoint the helpdesk ticket volume was a symptom; the disease was the identity architecture underneath it. Every application stored its own credentials, every user carried a dozen of them, and — because remembering a dozen is impossible — people reused the same weak password across systems that had no shared trust boundary.

That is a large, badly-shaped attack surface. Credential reuse means the blast radius of one phished password is not one application but every application that shares it. Offboarding was worse: with sixty independent credential stores, revoking a departed employee's access was a per-app, hours-long chore, which means at any given moment there were almost certainly valid credentials belonging to people who should have had none. MFA coverage was patchy and per-app, so the strength of authentication varied by whichever team had last touched a login page. This is keycloak identity federation territory: the fix is not more helpdesk agents, it is collapsing sixty trust boundaries into one you can actually reason about. Keycloak's Server Administration Guide is the reference for the mechanics; the risk framing and sequencing we had to work out ourselves.

Threat model

Naming what we were defending against is what justified the rollout order.

Credential-reuse blast radius. A single reused password turns one compromise into many. As long as each app authenticates independently, there is no way to bound how far a stolen credential reaches.
Orphaned access after offboarding. Sixty credential stores mean revocation is slow and error-prone. Every store you forget is a standing account an attacker or a disgruntled leaver can use. The control objective is single-point revocation.
Inconsistent authentication strength. Without a central policy, MFA and lockout behavior differ per app. The weakest login page sets the real security posture, because that is where an attacker will aim.
Big-bang migration risk. Federating sixty systems simultaneously with no staging is itself a threat — to availability. The recurring failure story in this space is exactly that: a simultaneous cutover with no fallback that earns a multi-day outage and an emergency rollback. Availability is a security property, so the rollout had to be defensible at every step.

The design decision that follows: put a broker in the middle that becomes the single authority for authentication, so credential reuse loses its reach, revocation becomes one action, and MFA becomes a policy rather than sixty separate implementations — and get there in phases so we never bet the whole estate on one cutover.

Controls we added

Control 1 — a broker so no application stores a password again

Federation does not share credentials — that misconception is itself dangerous, because engineers who believe it will build the wrong thing. Federation shares cryptographically signed assertions that someone was authenticated. Keycloak sits between the identity provider (our Active Directory, or Google — the systems that vouch for identity) and the service providers (the sixty downstream apps), acting as a broker: what the docs call integrating identity providers. It federates our existing AD as the source of truth over LDAP, speaks OIDC and SAML to every app, and means no downstream application holds a credential.

The security payoff is single-point revocation: disable an account once at the identity provider and access to all sixty apps evaporates. That property — not the convenience — was the security team's reason to approve the project. For the sequencing and the console mechanics I kept a thorough third-party walkthrough of a phased federation rollout → open alongside the official docs.

Control 2 — a phased rollout that always has a fallback

We ran it over 90 days, and the phasing was a risk control, not a project-management nicety.

Days 1-30, foundation. Stand up dev, staging, and a production cluster. Connect the primary AD as LDAP user federation so disabled accounts and group changes propagate automatically. Then pick three deliberately low-stakes pilot apps — an internal wiki, a dashboard, a staging tool — nothing that pages anyone at 2am if it breaks. For external IdPs, the setting that trips everyone is First Login Flow → first broker login, which determines whether a first-seen federated user links to an existing account or auto-provisions; a wrong choice here is an account-takeover-shaped hazard, so we set it deliberately.
Days 31-60, expansion. Add Google as an external IdP for contractors, turn on MFA policies org-wide, integrate the next dozen apps. Monitoring became non-negotiable here: alerting on failed authentications, token-renewal failures, and session timeouts, because incidents live at those boundaries, not the happy path. Delayed detection is reduced detection.
Days 61-90, production. Roll out to the rest, harden policy, and — critically — stand up and test backup and disaster recovery for the cluster before decommissioning the old SSO product. A broker with no tested DR is a single point of total authentication failure.

Control 3 — client configuration as an enforced allow-list

For OIDC apps, each got a client (the OIDC and SAML client management guide is the canonical field reference). Two settings carry security weight: Valid Redirect URIs configured exactly — protocol and trailing slash matching character for character, no wildcards, because a loose redirect URI is an open-redirect and token-exfiltration vector; and Authorization Code flow everywhere with PKCE required for public clients. PKCE (RFC 7636) exists specifically to stop authorization-code interception on clients that cannot hold a secret, so we banned the Implicit flow outright.

For SAML apps the work is metadata exchange plus attribute mapping via per-client protocol mappers. Attribute mapping is where SAML silently goes wrong: if the app expects email and Keycloak sends emailAddress, login "succeeds" but the user is provisioned as a stranger — which, depending on the app, can be an authorization failure that grants the wrong access. We standardized on the HTTP-POST binding and reviewed every mapper.

Verifying the controls, not just shipping them

Two verification habits earned their keep. First, clock synchronization: our second SAML integration passed staging and failed intermittently in production with signature errors. SAML assertions carry NotBefore / NotOnOrAfter conditions, so IdP and SP clocks must agree within a tolerance — as the clock-skew writeup explains, a few seconds of drift rejects a valid assertion, and allowedClockSkew only widens the window rather than fixing the cause. One node had drifted; we enforced NTP across every host in the auth path. Check clocks before certs.

Second, negative testing of the controls themselves. We tested redirect URIs with deliberately invalid values, not just the happy path, and caught a client that would have accepted a wildcard redirect — a textbook open-redirect and token-leak risk — precisely because the test matrix included a URL we expected Keycloak to reject. A control you never test against a hostile input is security theater; a five-minute negative test replaced a future incident review.

Residual risk / what we're still watching

Collapsing sixty trust boundaries into one removes a whole class of reuse and orphaned-access risk, but it concentrates risk rather than deleting it.

The broker is now a high-value single point. A Keycloak compromise, a signing-key leak, or a cluster outage is an authentication event for the entire estate. That trade is only correct if the concentrated point is hardened, patched, monitored, and backed by tested DR — which is why DR testing gated the decommission of the old system, and why we track Keycloak advisories directly.
Clock and NTP are now a security dependency. Time drift on any host in the auth path degrades SAML validation. NTP health is monitored as a security signal, not just an ops metric.
SAML attribute-mapping drift. A future mapper change that misroutes email or group claims can silently grant wrong access. Mappers live in reviewed configuration, and we watch for provisioning that produces unexpected identities.
First-broker-login and redirect-URI loosening. Both are one careless edit away from reintroducing account-takeover or open-redirect exposure. They stay in reviewed config, not ad-hoc changes.
Partner federation expands the trust perimeter. We are extending federation to a couple of partner organizations so their staff reach shared portals via IdP-to-IdP trust — which means trusting another organization's authentication. Each new trusted IdP is new attack surface and gets onboarded with scrutiny over what claims we accept and how far that trust reaches, not rubber-stamped.

The net effect is that a stolen credential no longer roams sixty systems, offboarding is a single revocation, and MFA is a policy rather than sixty implementations — provided we keep treating the broker as the critical asset it has now become.

Sources & further reading

Keycloak Server Administration Guide — LDAP/AD user federation, identity brokering, first login flow
Keycloak — Integrating identity providers (Server Admin Guide, Ch. 9)
Keycloak — Managing OpenID Connect and SAML Clients, incl. protocol mappers (Ch. 12)
PKCE for OAuth 2.0 — why public clients need it (oauth.net / RFC 7636)
SAML Clock Skew in Keycloak — allowedClockSkew, NotBefore/NotOnOrAfter, and NTP
A phase-by-phase writeup of a 90-day federation rollout — a useful third-party account of the IdP/SP model and the config details that live outside the primary docs.

Green Tests Aren't a Safe Supply Chain: A GitHub Actions Security Audit Gate That Blocks Vulnerable Deploys

Gaberial Sofie — Thu, 06 Aug 2026 09:29:17 +0000

The exposure

Consider the timeline. Friday: all checks pass, PR merged, deploy succeeds, everyone goes home. The following Thursday a critical advisory drops for one of the gems we were running in production, with exploits already circulating — and the way we learned our own app was vulnerable was a Hacker News thread, not our pipeline.

That is a detection gap, and it is worth naming precisely because CI actively disguises it. Green tests prove your code behaves; they say nothing about whether your dependencies are safe. A pipeline that treats "tests passed" as "safe to ship" is conflating two very different claims, and the space between them is exactly where a known-vulnerable dependency reaches production unopposed. A github actions security audit gate closes that space by making a red audit block the deploy. The design principle we anchored on is the one GitHub itself pushes in its security hardening guide for Actions: make the safe path the default path, enforced by the pipeline rather than remembered by a human. For the workflow shape and the first-month rollout I also worked from a thorough third-party writeup of this audit-gate pattern →.

Threat model

Known-vulnerable dependency reaching production. A gem with a published CVE flowing through a green build into prod is the primary exposure — and because the test suite cannot see it, nothing in the default pipeline stops it.
Post-merge advisory window. New advisories drop after code merges. A dependency that was clean at merge becomes vulnerable in place while sitting in production, and a merge-time-only check never revisits it. This is a mean-time-to-detect problem: ours was roughly seven days, measured from disclosure to a human noticing.
License / legal exposure. A copyleft-licensed transitive dependency slipping into a commercial codebase is a distinct risk that neither tests nor CVE scanners address, and it surfaces at the worst time — during due diligence, not development.
The gate itself as supply-chain surface. Adding third-party Actions to enforce security introduces its own risk: an unpinned marketplace action is mutable and can be re-pointed at hostile code. The control must not become the vulnerability.

Controls we added

Control 1 — a red audit blocks the deploy

The core control is one line. An audit job runs the scanners; the deploy job declares needs: [audit], so it will not start unless the audit is green. No audit pass, no deploy — the safe path stops being something a human remembers and becomes what the pipeline enforces.

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with: { ruby-version: .ruby-version, bundler-cache: true }
      - run: gem install bundler-audit license_finder
      - run: bundle audit check --update
      - run: license_finder --quiet
      - uses: google/osv-scanner-action@v1
        with: { scan-args: '--recursive --skip-git .' }

  deploy:
    needs: [audit]   # deploy won't start until audit is green
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... deploy steps

Control 2 — three scanners for defense in depth

We chose three complementary tools rather than one, because they cover different blind spots and layering them is cheap:

bundler-audit — RubySec's patch-level verification, scanning Gemfile.lock against the ruby-advisory-db for known CVEs, and flagging insecure http:///git:// gem sources. Deepest on Ruby advisory data.
license_finder — reports dependencies outside your approved-license policy, covering the legal risk the CVE scanners ignore entirely.
Google's OSV-Scanner, backed by OSV.dev across many ecosystems — broadest coverage, so it catches vulnerabilities in polyglot repos, not just Ruby.

Overlapping controls is deliberate: no single advisory source is complete, so two independent scanners plus a license check reduce the residual chance that a known issue slips past all of them.

Control 3 — trigger strategy that closes the post-merge window

The triggers are a control in their own right:

on:
  push: { branches: [ main ] }
  pull_request: { paths: ['Gemfile*', '*.gemspec'] }
  schedule:
    - cron: '0 8 * * MON'   # weekly re-audit

pull_request scoped to dependency files so the gate only fires when dependencies actually change — which keeps it off most PRs and prevents the resentment that gets security gates disabled.
The weekly schedule cron is the one that addresses the post-merge window directly: it re-audits already-deployed code against advisories that did not exist at merge, moving mean-time-to-detect from about a week to same-day or next-scheduled-run.
push to main as the last check before production.

Control 4 — hardening the gate itself

Because the gate depends on third-party Actions, we treated that dependency as supply-chain surface. GitHub's hardening guide is explicit that pinning to a full-length commit SHA "is currently the only way to use an action as an immutable release," so third-party actions are SHA-pinned rather than tag-referenced. When the tagged OSV action broke on a stale reference, we installed the scanner as a binary instead — vendor-neutral and not subject to a mutable tag:

curl -sSL https://raw.githubusercontent.com/google/osv-scanner/main/scripts/install.sh | sh -s -- -b /usr/local/bin
osv-scanner --recursive --skip-git .

We also adopted a few Bundler settings to reduce source-substitution surface (bundle config set disable_multisource true, cache_all true, clean 'true'), layered Trivy alongside OSV for further depth, and put a quarterly dependency-hygiene day on the calendar.

Verifying the control, not just shipping it

A gate is only real if it fails when it should and the team keeps it on. The first run failed — we had two existing advisories on merge — and that failure was the point: triaging them was the whole value, not something to route around by disabling the step. Two design choices kept the gate trusted rather than resented. Because the audit runs on pull_request only when Gemfile* changes, most PRs never see it. And because bundle audit prints the advisory URL and the fixed version, "unblock" almost always meant a one-line version bump rather than a research project. After the team saw the red build point at a real, patchable issue every time — no false-positive noise — trust went up, not down. By week four a red audit was simply "bump the gem and move on." A couple of operational notes that matter: bundle audit needs --update or it scans a stale advisory DB and gives false comfort, and license_finder will surprise people (ours flagged a transitive GPL dependency nobody had noticed), so set your allowed-license policy up front.

Residual risk / what we're still watching

The gate reliably keeps known-vulnerable dependencies out of production, but it is not a complete supply-chain defense, and being honest about the edges is the point.

Scanners find known vulnerabilities only. A zero-day or a brand-new typosquat with no advisory yet passes clean. The gate lowers mean-time-to-detect for disclosed issues; it does not detect the undisclosed. Human review of new dependencies still carries that load.
Gate-bypass is a cultural failure mode. The most likely way this control dies is a developer commenting out the audit step to unblock a feature. We defend it by keeping it fast, legible, and low-false-positive so bypassing feels unnecessary — but a determined bypass is always possible, so we also protect the workflow file in review.
Third-party action drift. SHA-pinning freezes an action, but a frozen action also stops receiving fixes; we periodically review pinned SHAs so "immutable" does not become "stale and unpatched."
Coverage gaps beyond gems. The gate audits dependencies, not our container base images or the runner environment. We are extending the same pattern to image scanning next, and rolling the gate out as a reusable workflow so every repo inherits it rather than reimplementing it.
The advisory databases are themselves a dependency. Our detection is only as current as OSV.dev and the ruby-advisory-db; a gap or delay in their data is a gap in ours. Running two independent sources mitigates but does not eliminate that.

The measurable outcome that mattered: mean-time-to-detect a known-vulnerable dependency dropped from about a week to same-day, and twice the Monday cron flagged a fresh advisory on already-deployed code before any customer or news cycle did. What we keep watching is the gap the gate cannot close — the undisclosed vulnerability and the human tempted to route around a red build.

Sources & further reading

bundler-audit and the ruby-advisory-db — RubySec's patch-level verification and the advisory data behind it
LicenseFinder — Pivotal's dependency-license auditor
OSV-Scanner and OSV.dev — Google's multi-ecosystem scanner and the open vulnerability database it queries
Security hardening for GitHub Actions — GitHub's official guide (SHA-pinning, minimal GITHUB_TOKEN permissions)
Events that trigger workflows — GitHub docs for workflow_run, schedule, pull_request filters
A writeup of this audit-gate rollout — a useful third-party account of both workflow variants, the first-month adoption story, and the Bundler hardening config.

Adopting Keycloak Without a Bulk-Credential Migration: A Keycloak Custom Provider (SPI) Threat Model

Gaberial Sofie — Wed, 05 Aug 2026 10:05:12 +0000

The exposure

Our Keycloak rollout stalled on two problems that were, underneath, the same kind of problem: both were about where trust and evidence actually live. We had 200,000 users in a legacy PostgreSQL table with a bespoke password-hashing scheme, and compliance required that every authentication event — successful logins, failed attempts, password changes — reach our SIEM in near real time.

The naive fix for the first problem is a big-bang migration: export the users, re-hash or force-reset every password, cut over in a maintenance window. That migration is itself the exposure. Bulk-handling 200k credential records is a concentrated moment of risk — a dump of password material in flight, a mapping bug that silently drops or duplicates accounts, and a forced-reset flow that is a textbook phishing pretext ("we've upgraded our login, please reset your password here"). The second problem is a detection gap: Keycloak's default event storage kept auth events, but not where our monitoring could see them within seconds, which means the window between a credential-stuffing burst and anyone noticing was measured in whatever our slowest batch job took.

So the real question was not "how do we migrate?" but "how do we adopt Keycloak without ever creating a bulk-credential event, and without accepting a blind spot in login telemetry?" Keycloak, the open-source identity and access management server, answers that through its Service Provider Interfaces (SPI) — the extension seam that lets you change where users are authenticated and where events are sent without forking the server.

Threat model

Being precise about what we were defending against is what kept the extension small and the review tractable.

Bulk credential exposure during migration. Any process that reads, transforms, and re-writes 200k password hashes in one operation is a high-value target and a single point of failure. The control objective is to never have that operation exist at all.
Account-lockout and phishing risk from forced resets. A mass password-reset email trains users to click reset links, which is exactly the behavior an attacker wants to exploit. It also risks locking out legitimate customers on a mapping error, which is an availability incident.
Detection gap on authentication. If failed logins and password changes are not streamed to the SIEM promptly, credential-stuffing, brute force, and account-takeover attempts go unobserved until after the fact. Delayed telemetry is reduced detection capability, which is a security control, not an operational nicety.
Extension-introduced attack surface. Writing custom Java that runs inside the identity server is itself a risk: a provider on the request path can add latency, leak connections, or — worst case — mishandle a credential. The extension must be scoped so its blast radius is understood.

The governing decision: replace a one-time, high-magnitude credential-handling event with a gradual, per-login re-hash where the legacy store stays authoritative until each user proves themselves, and close the detection gap by streaming events at the source. Both are SPI providers, and both are small.

Controls we added

Keycloak ships LDAP and Active Directory federation, but our legacy store was neither, and its default event handling did not stream where we needed. That is precisely where you stop configuring Keycloak and start extending it — deliberately, and with the smallest surface that satisfies the requirement. I cross-referenced the official developer guide against a thorough third-party walkthrough of building SPI providers →, which works each interface end to end.

There are four SPI provider types — authentication, user storage, event listener, and policy. We needed two, and deliberately no more than two.

Control 1 — federate the legacy users so no bulk migration ever happens

We implemented org.keycloak.storage.UserStorageProvider, the same User Storage SPI that Keycloak's own built-in LDAP and AD federation is written against. The two methods that carry the weight are getUserById(String id, RealmModel realm) and getUserByUsername(String username, RealmModel realm); Keycloak calls them to resolve a user, and our implementation queried the legacy table and mapped rows to Keycloak user models on demand.

The security-relevant property is that users authenticate against the old store without ever being migrated in bulk. We validated each login against the legacy hash and, only on a successful authentication, transparently re-hashed that one credential into Keycloak's format. The re-hash happens exactly when the user has just proven they hold the password — there is no batch, no dump, no reset email, and the legacy store remains authoritative for anyone who has not yet logged in. That converts one large, concentrated credential-handling risk into a stream of individually trivial ones. Baeldung's custom user providers walkthrough is a solid worked reference for the interface.

Control 2 — stream auth events to the SIEM at the source

For the detection gap we implemented org.keycloak.events.EventListenerProvider, registered through Keycloak's Events SPI. The method that matters is onEvent(Event event), invoked on every login, failed attempt, password change, and account update. Our implementation serialized each event and pushed it to the SIEM ingest endpoint. The monitoring team got the authentication telemetry they needed to detect credential-stuffing and takeover attempts promptly, in a few hundred lines rather than a platform change.

Control 3 — build and package the extension so it cannot silently misload

Both providers are Java built with Maven. The single most important build decision is scope: both Keycloak artifacts (keycloak-services, keycloak-core) use provided scope, because Keycloak supplies them at runtime. You compile against them and do not bundle them.

<dependency>
  <groupId>org.keycloak</groupId>
  <artifactId>keycloak-services</artifactId>
  <version>YOUR_KEYCLOAK_VERSION</version>
  <scope>provided</scope>
</dependency>

Every provider builds on org.keycloak.provider.Provider (whose lifecycle includes close() for releasing resources) plus a ProviderFactory that reads configuration via init(Config.Scope config). That factory-and-config pattern is a control in its own right: it let the same JAR point at the test database in CI and the production database in prod, so the only difference between environments was reviewed configuration, not code.

Verifying the control, not just shipping it

An extension that runs inside the identity server is not trustworthy until it is demonstrated to fail safely. Our loop: mvn clean install, drop the JAR into Keycloak's providers/ directory, run a build, and confirm the factory appears in the startup log — if it is not listed, the META-INF/services registration is wrong and Keycloak silently ignores the JAR, which is a fail-silent condition worth catching in CI rather than in production.

For the user-storage provider we wrote an integration test that spun Keycloak up in a container against a seeded legacy database, then asserted that a known legacy user could authenticate and that their credential was transparently re-hashed on first success. A mapping bug caught here is a failing test; the same bug found in production is a customer who cannot log in. We never pointed a new provider at the production realm until it had authenticated a test user and emitted a test event end to end against a throwaway realm.

Residual risk / what we're still watching

Extending Keycloak solved the migration and detection problems, but it moved risk rather than eliminating it, and naming where it went is the point.

The event listener runs in the request path. A synchronous, slow SIEM push adds latency to every login and, in the worst case, could be a denial-of-service lever against authentication itself. We made the emit asynchronous behind a bounded queue so a slow or unreachable SIEM never blocks auth — but a bounded queue also means events can be dropped under sustained backpressure, so we monitor queue saturation as a signal that we are losing telemetry.
Version lock-step is now a standing obligation. The internal SPI is not guaranteed stable across Keycloak majors, so the provider must be rebuilt and re-tested against the exact version it runs on. A forgotten rebuild after an upgrade is a provider that compiles against the wrong internals and fails in subtle ways — we pin the version and rebuild on every upgrade.
Connection hygiene under load. A storage provider that does not release database connections in close() leaks them, which is an availability failure mode. We assert cleanup in the lifecycle method and watch connection-pool metrics.
The legacy store stays authoritative longer than we would like. Gradual re-hash is safer than a big bang, but it means the old database — with its bespoke hashing — remains in the trust path until the long tail of inactive users either logs in or is aged out. We are tracking the migration curve and will set a hard cutoff date after which un-migrated accounts are disabled rather than left as indefinite legacy attack surface.
Any future authentication or policy provider is new surface. We are evaluating a policy provider for location- and device-aware access on sensitive resources, and a step-up MFA authentication provider. Each runs inside the identity server, so each gets the same throwaway-realm-first, integration-tested, version-pinned treatment — no rubber-stamping code that sits on the credential path.

The net effect is that we adopted Keycloak without ever creating a bulk-credential event and without accepting an authentication blind spot, in roughly 300 lines of Java. The lesson we took from it is narrow and deliberate: the SPI seam is powerful precisely because it lets you avoid the riskier alternative, but every line you run inside the identity server is code on your most sensitive path, and it earns the same scrutiny as the server itself.

Sources & further reading

Keycloak Server Developer Guide — Service Provider Interfaces (SPI) — the provider/factory model, META-INF/services registration, and packaging.
Keycloak Server Developer Guide — User Storage SPI — the interface behind federating an external user store without migration.
Keycloak Server Developer Guide — Events — the Event Listener SPI used to stream auth events.
Baeldung — Using Custom User Providers with Keycloak — a hands-on Java worked example of a UserStorageProvider.
A hands-on custom-provider writeup — a useful third-party field-notes version of the Maven setup and both providers to keep open while building.

Shrinking the Blast Radius of Our Identity Plane: Deploy Keycloak on AWS ECS Fargate with Terraform

Gaberial Sofie — Tue, 04 Aug 2026 08:24:43 +0000

The exposure

The riskiest asset in our environment was never a customer-facing app. It was the component every one of those apps delegated trust to. Our Keycloak deployment issued the OIDC tokens and enforced the login flow for every internal service, and it ran on a single, hand-configured EC2 instance, auth-prod-1, with PostgreSQL installed on the same host. No infrastructure-as-code, no reviewed change history, no standby. Roughly two years of realm, client, user, and role-mapping data lived on one EBS volume for which nobody had a tested restore path.

Framed in blast-radius terms, that box was the highest-value single point of failure we owned. Its loss did not degrade one service; it revoked the ability to authenticate to all of them. And because the identity store was co-located with the application, a host compromise was not merely a downtime event, it was a full credential-store disclosure: password hashes, client secrets, and active session state, all reachable from the same shell. The state had been built by clicking and SSH-ing over two years, which meant the exposure was also un-auditable: we could not enumerate, with confidence, what was actually reachable or how it was configured. That uncertainty is itself a security finding.

The goal of the rebuild was not "move to containers." It was to shrink the blast radius, make the configuration auditable, and put defensible boundaries between the parts of the system that currently fail together.

Threat model

Before touching Terraform I mapped out what we were actually defending, because "make it highly available" is not a threat model.

Assets, in priority order. The integrity of the token-issuance path (an attacker who can mint or alter tokens owns every downstream service); the confidentiality of the realm database (password hashes, client secrets, session data); and the availability of the identity provider itself.

Adversaries and failure modes we scoped in, roughly ordered by the residual risk they carried on the old design:

Host compromise escalating to credential-store access. Application and database shared a trust boundary, so any RCE in Keycloak, any dependency CVE, any stray SSH key was one hop from the full user store. The blast radius of an app compromise equaled the value of the data on the same box.
Loss-of-availability events. A single instance, a single AZ, a single disk: availability of the entire authentication plane depended on the least reliable component in the stack.
Un-auditable configuration drift. Hand-applied changes with no review gate mean no way to reason about the current attack surface, or to tell an unauthorized change from an authorized one.
Network-adjacent lateral movement. A flat network path to the database, secured by host-level assumptions rather than an explicit deny-by-default boundary.
Application-layer identity flaws in Keycloak itself. Open-redirect and redirect-URI validation classes (for example the path-normalization bypass tracked as CVE-2024-1132, and the SSRF-via-request_uri class of CVE-2020-10770) are exploitable regardless of where Keycloak runs, and they leak or misdirect tokens. Hosting choices do not fix these; a patch cadence and strict hostname/redirect config do.

Explicitly out of scope for this pass: realm-level policy hardening (MFA enforcement, password policy, token lifetimes) and a formal disaster-recovery game day. Naming them as out of scope is deliberate. They are known residual risk, tracked below, not oversights.

Controls we added

The design applies defense in depth: no single control is load-bearing, and each one narrows the blast radius of the layer above it. We rebuilt the plane as Keycloak on ECS Fargate, Aurora PostgreSQL in private subnets, an Application Load Balancer, and Route 53, defined entirely as reviewed Terraform. A third-party writeup on the full Terraform setup covers the complete VPC/ALB/Route 53 wiring; below I focus on the controls and why each one earns its place in the threat model.

Control 1 — Remove the host to shrink the compromise surface

Moving Keycloak onto ECS Fargate removes the long-lived, hand-patched host from the picture entirely. There is no SSH surface, no persistent OS to accrue drift, and the container is replaced rather than patched in place. The task definition runs the official image in production mode against the managed database, following Keycloak's own production configuration guidance:

resource "aws_ecs_task_definition" "keycloak" {
  family                   = "keycloak"
  network_mode             = "awsvpc"           # mandatory on Fargate; gives each task its own ENI
  requires_compatibilities = ["FARGATE"]
  cpu                      = "1024"
  memory                   = "2048"
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn  # least-privilege pull + secrets read
  task_role_arn            = aws_iam_role.keycloak_task_role.arn

  container_definitions = jsonencode([
    {
      name      = "keycloak"
      image     = "quay.io/keycloak/keycloak:24.0.5"
      essential = true
      portMappings = [{ containerPort = 8080, protocol = "tcp" }]
      environment = [
        { name = "KC_DB",            value = "postgres" },
        { name = "KC_HOSTNAME",      value = "auth.example.com" },
        { name = "KC_PROXY_HEADERS", value = "xforwarded" }
      ]
      secrets = [
        { name = "KC_DB_URL",              valueFrom = aws_secretsmanager_secret.kc_db_url.arn },
        { name = "KC_DB_USERNAME",         valueFrom = aws_secretsmanager_secret.kc_db_user.arn },
        { name = "KC_DB_PASSWORD",         valueFrom = aws_secretsmanager_secret.kc_db_pass.arn },
        { name = "KEYCLOAK_ADMIN",         valueFrom = aws_secretsmanager_secret.kc_admin_user.arn },
        { name = "KEYCLOAK_ADMIN_PASSWORD", valueFrom = aws_secretsmanager_secret.kc_admin_pass.arn }
      ]
      command = ["start", "--optimized"]
    }
  ])
}

Two decisions are security-relevant. command = ["start", "--optimized"] runs Keycloak in production mode against a pre-built configuration rather than start-dev, disabling the dev-mode conveniences that widen the attack surface. And every credential moves out of environment into secrets, sourced from Secrets Manager at runtime, so plaintext passwords never sit in the task definition, in Terraform state as a resolved value, or in the ECS console.

Control 2 — Isolate the credential store behind a deny-by-default boundary

The single most important boundary in the old design that did not exist: the database is no longer on the application host. It moves to managed Aurora PostgreSQL in private subnets, and its ingress is scoped to the Keycloak service's security group rather than a CIDR range. That is least privilege expressed at the network layer: the database accepts 5432 from the identity of the calling service, and nothing else.

resource "aws_security_group" "database" {
  name   = "keycloak-database-sg"
  vpc_id = module.vpc.vpc_id

  ingress {
    description     = "PostgreSQL from Keycloak tasks only"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.keycloak.id]   # SG-to-SG, not a CIDR
  }
}

resource "aws_rds_cluster" "keycloak" {
  cluster_identifier      = "keycloak-cluster"
  engine                  = "aurora-postgresql"
  engine_version          = "15.4"
  database_name           = "keycloak"
  master_username         = "keycloak"
  master_password         = var.database_password       # bootstrap only; rotated into Secrets Manager
  backup_retention_period = 14
  storage_encrypted       = true                        # encryption at rest for the user store
  db_subnet_group_name    = aws_db_subnet_group.keycloak.name
  vpc_security_group_ids   = [aws_security_group.database.id]
  skip_final_snapshot      = false
  final_snapshot_identifier = "keycloak-final"
}

storage_encrypted = true protects the user store at rest, the SG-to-SG rule collapses the reachable network path to exactly one source, and 14-day retention with a real final snapshot gives us a recovery point the old ad-hoc setup never had. The host-compromise-to-credential-store path from the threat model is now a two-boundary problem instead of a one-shell problem.

Control 3 — Terminate TLS at the ALB and trust the proxy correctly

The ALB terminates TLS and is the only component with a public interface; the tasks sit in private subnets, unreachable except through it. This matters for correctness as well as exposure: behind a terminating proxy, Keycloak must be told to trust forwarded headers or it builds redirect and issuer URLs from the wrong host, which is both a broken login flow and, misconfigured, an open-redirect risk. Per the reverse-proxy guide we set KC_PROXY_HEADERS=xforwarded (the older KC_PROXY flag was deprecated in Keycloak 24), and per the hostname guide we pin KC_HOSTNAME to the exact Route 53 domain so issuer and admin URLs cannot be coerced to an attacker-supplied host. Locking the hostname is the direct mitigation for the redirect-validation CVE class from the threat model.

Verification

Controls that are not verified are assumptions. Before cutting over we exercised each one against the threat model:

Network boundary. From a host outside the Keycloak security group, a connection to the Aurora endpoint on 5432 must time out; from a Keycloak task it must succeed. We confirmed both, so the SG-to-SG rule is enforcing, not merely declared.
Secret hygiene. We grepped the rendered task definition and the ECS console for plaintext credentials and confirmed only Secrets Manager ARNs appear, and that resolved secret values are absent from Terraform state.
Proxy and hostname correctness. We validated that the OIDC discovery document, issuer, and every client redirect URI resolve to the canonical Route 53 hostname over TLS, and that requests spoofing a Host or X-Forwarded-Host header do not change the issued URLs.
Migration integrity. We exported the realms with kc.sh export, imported into a fresh Aurora database, and checked every client and redirect URI against a checklist. The new stack ran in parallel on a temporary hostname and served real logins before we moved DNS. Because the old host stayed untouched, rollback was a one-line DNS revert, keeping the cutover's own blast radius small.

Residual risk / what we're still watching

Rebuilding the plane reduced the exposure; it did not eliminate risk, and it would be dishonest to present it that way. What we are still watching:

Patch cadence is now the primary risk owner for application-layer flaws. Fargate solved host patching but not Keycloak's own CVE stream. Redirect-URI and SSRF classes like CVE-2024-1132 and CVE-2020-10770 are exploitable regardless of hosting, so we are wiring image-tag currency and CVE alerting into the pipeline and treating a stale image tag as a finding.
Bootstrap credentials and rotation. The Aurora master password still enters the world as a Terraform variable before it is rotated into Secrets Manager, and rotation is not yet automated. That bootstrap window is real residual risk; automatic rotation is the next control.
Distributed session state across tasks. Multiple Keycloak tasks behind the ALB without a shared cache can drop sessions as requests bounce between instances. Until the Infinispan/JDBC-Ping cache is configured we lean on sticky sessions, which is a workaround, not a design.
Realm-level policy and a DR game day remain out of scope. MFA enforcement, token lifetimes, and a tested full-restore of the encrypted Aurora snapshot are named, tracked work. A backup you have never restored is a hypothesis.
Admin console exposure. The admin endpoints currently share the ALB with the login flow. Splitting them onto a restricted path or internal listener is queued to further shrink the surface an external attacker can reach.

The core exposure, an entire company's authentication resting on one un-auditable pet server, is gone. What remains is a set of named, bounded risks with owners, which is a defensible position to operate from rather than a fragile one to hope over.

Sources

Configuring Keycloak for production — production-mode baseline and hardening checklist.
Configuring a reverse proxy (Keycloak) — proxy header modes and why KC_PROXY_HEADERS replaced KC_PROXY.
Configuring the hostname (Keycloak) — pinning KC_HOSTNAME and issuer URLs behind a load balancer.
Amazon ECS task definition parameters for Fargate — the awsvpc requirement and per-task ENI behavior.
What is Amazon Aurora? (AWS docs) — the managed, encrypted HA database backing the user store.
A thorough third-party writeup on the full Terraform setup — the complete RDS/ALB/VPC/Route 53 Terraform for this architecture.

Attack Surface of Hand-Rolled Auth: Consolidating Keycloak FastAPI Role Based Access Control Across 40 Endpoints

Gaberial Sofie — Mon, 03 Aug 2026 09:25:00 +0000

The exposure

Our FastAPI service enforced authorization the way a lot of services quietly do: every route decoded its own JWT, pulled the role claim, and made a local if user.role == 'admin' decision. Across roughly 40 endpoints and three engineers, that is not one access-control mechanism — it is 40 independent enforcement points, each of which can drift, and each of which is a place where a control can simply be absent.

The failure mode showed up in review: an admin-only endpoint with no role check at all. It had shipped weeks earlier and passed every test, because the tests asserted the happy path and nobody wrote a test for a control that was never there. The important thing is not the individual mistake. The important thing is the property of the system — authorization with no single point of enforcement fails open by omission, and omission is invisible. "The check is supposed to be there" is a hope, not a control.

Threat model

Before changing anything, it helps to be precise about what we were actually defending against, because that framing dictates which controls are worth the effort.

Broken access control — the top item on the OWASP Top 10 (A01:2021) — is exactly the class of defect we had: an authenticated user reaching an endpoint their role should not permit. Distributed, hand-written checks maximize the surface for this.
Privilege escalation via a missing or wrong check. A guest or user token reaching an admin route because one of 40 checks was omitted, misspelled ("admin" vs "administrator"), or copied from the wrong template.
Inconsistent token validation. Every service hand-parsing JWTs means signature verification, aud, iss, and expiry handling can differ per route. Any route that validates loosely widens the blast radius of a leaked or forged token.
No audit trail. When someone asks "who can reach this?", the answer being a grep across the codebase is itself a finding — you cannot attest to a control you have to reconstruct by reading source.

The residual-risk question that drove the design: if one endpoint is written wrong, how much does that cost us? With per-route checks, one mistake is one breach. The goal was to make the default path safe so that the cost of a single mistake drops toward zero.

Controls we added

The governing principle is centralizing the authorization decision into a single, hardened authority and making the safe integration the easy one — defense in depth backed by least privilege, rather than 40 bespoke gates. Keycloak, the open-source identity and access management server, becomes that authority: it owns identity, roles, and token issuance, and every service is reduced to validating what Keycloak signed. It speaks OpenID Connect, the identity layer over OAuth 2.0, so the integration is standard rather than bespoke, and the Keycloak securing-applications guide is explicit about how a relying party should validate issued tokens. For the FastAPI wiring itself I cross-referenced the official docs against a thorough third-party walkthrough of the realm/client/role setup →, which screenshots each console step.

Control 1 — a realm as the trust boundary

A realm is an isolated space for users, roles, and clients. In the admin console (typically http://localhost:8080 locally; older builds used the /auth context path) you open the dropdown top-left, choose Add realm, and name it. Treat the realm as a security boundary, not a namespace: everything inside shares an issuer and a signing key, and nothing outside it is in scope. That boundary is what lets you reason about blast radius at all.

Control 2 — a confidential OIDC client, scoped deliberately

Under Clients → Create client, set a client ID and keep the type OpenID Connect. Two settings carry real security weight:

Client authentication. Off for public clients (browser/SPA) that cannot hold a secret; on for confidential backends that can. Our FastAPI backend is confidential, so this is on and it holds a client secret. Getting this wrong is a genuine exposure, not a preference.
Valid Redirect URIs. This is an allow-list, and it is a control, not a convenience field. An overly broad redirect URI is an open-redirect and token-exfiltration vector, so it is set to the exact callback the app uses — no wildcards, no trailing-slash ambiguity.

Control 3 — three roles, least privilege by construction

Under Roles → Add Role we defined the minimum viable set:

admin — full administrative access; the smallest possible population.
user — day-to-day application access, no configuration or user management.
guest — read-only access to non-protected data.

Roles are then assigned per user under Role Mappings. The model is: define once, assign narrowly, enforce centrally. The Keycloak Server Administration Guide is the canonical reference for the realm, client, and role screens if you want the exhaustive version.

Control 4 — a single enforcement point that fails closed

This is the control that actually retires the exposure. The fastapi-keycloak library wraps the OIDC handshake so that protecting a route is one dependency injection rather than hand-written parsing:

from fastapi import FastAPI, Depends
from fastapi_keycloak import FastAPIKeycloak, OIDCUser

app = FastAPI()
keycloak = FastAPIKeycloak(
    server_url="http://localhost:8080/auth",
    client_id="your-client-id",
    client_secret="your-client-secret",
    realm="your-realm",
    callback_uri="http://localhost:8000/callback"
)

@app.get("/protected")
def protected_route(user: OIDCUser = Depends(keycloak.get_current_user)):
    return {"message": f"Hello, {user.username}"}

The security-relevant property is that Depends(keycloak.get_current_user) validates the token signature and claims against Keycloak and, on failure, rejects the request — it fails closed. The endpoint receives a typed, already-validated OIDCUser; role gates read from that object. There is no route-local JWT parsing left to get subtly wrong, and the token-validation logic is one implementation rather than 40.

The migration: reducing risk without a big-bang cutover

Replacing authorization on a live service is itself a high-risk change, so we sequenced it to keep the system defensible at every step:

Run Keycloak in parallel. Both auth paths coexisted for a sprint. New endpoints used the Keycloak dependency; existing routes kept their checks. Because both sides spoke JWTs, a request carried a token either could interpret, so there was no flag-day.
Mirror existing roles first. We recreated admin, user, and guest to match the old names exactly, so claims mapped one-to-one and we changed the enforcement mechanism without simultaneously changing the policy — one variable at a time.
Migrate highest-risk routes first. The admin endpoints, where a missing control is a real incident, moved first. Each migration was a small, reviewable PR: delete the manual parsing, inject the dependency, gate on the role.
Delete the legacy path only after the last route moved. A dormant second auth system is standing attack surface. Once every route was on Keycloak, the home-grown decoder was removed entirely rather than left "just in case."

Verifying the control, not just shipping it

Centralizing authorization is only worth anything if you can demonstrate it holds. Because the user authenticates against Keycloak, Keycloak issues signed tokens, and the service only validates them, the service never sees a credential — the sensitive material lives in one hardened system and everything downstream verifies signatures and reads claims. That separation is what makes the control auditable: "who can reach this?" becomes a query against role mappings in the console instead of a code archaeology exercise. We paired that with negative tests — a user token against an admin route must return 403 — so the absence of a check is now something a test can catch rather than something that ships silently.

Residual risk / what we're still watching

Centralizing the authorization decision removes a whole class of omission defects, but it does not make the system risk-free — it relocates the risk, and honesty about where it went is the point.

Keycloak is now a high-value single point. The blast radius of a realm compromise or a signing-key leak is the whole platform. That trades many small risks for one concentrated one, which is the right trade only if the concentrated point is hardened, patched, and monitored. We track Keycloak security advisories directly rather than waiting for a dependency bump.
Redirect-URI and audience configuration remain security-sensitive. A later loosening of an allow-list, or a mismatched aud mapping, quietly reintroduces exposure. These live in reviewed configuration, not ad-hoc changes.
Public vs confidential client drift. A client accidentally set public that needs to hold a secret is a real weakness; we assert client type in configuration review.
Token lifetime and revocation. Short access-token TTLs bound the window a leaked token is useful; we are still tuning refresh-token handling and session invalidation on role change, since a demoted user holding a valid token is a genuine residual gap until the token expires.
Group and multi-service rollout. We are moving to Keycloak groups for team-level role assignment and extending the same realm to a second and third service. Each new relying party is new attack surface that has to be onboarded with the same scrutiny, not rubber-stamped.

The net effect is that a single wrong endpoint is no longer one breach away, and the cost of the average authorization mistake has dropped from "incident" to "failing test." For a service still hand-rolling role checks, consolidating onto an IAM authority is among the highest-leverage reductions in attack surface available — provided you then treat that authority as the critical asset it has become.

Sources & further reading

Keycloak Server Administration Guide — canonical reference for realms, clients, roles, and role mappings.
Keycloak — Securing applications and services with OpenID Connect — how a relying party should validate issued tokens, and confidential vs public client guidance.
OpenID Connect Core 1.0 specification — the standard underneath the flow, from the OpenID Foundation.
OWASP Top 10 — A01:2021 Broken Access Control — the risk category this work directly addresses.
fastapi-keycloak on GitHub — the library providing FastAPIKeycloak, OIDCUser, and the get_current_user dependency.
A screenshot-by-screenshot Keycloak + FastAPI setup writeup — a useful third-party walkthrough of the console steps to keep open while wiring the realm and client.

When Your Resilience Vendor Is Your Single Point of Failure: A Multi-CDN Failover Threat Model After the November 2025 Cloudflare Outage

Gaberial Sofie — Sun, 02 Aug 2026 08:39:12 +0000

The exposure

At 11:20 UTC on November 18, 2025, Cloudflare fell over, and our stack fell over with it. Every dashboard we owned threw HTTP 500, 522, and 530. The detail that reframed the whole morning: when we tried to log into Cloudflare to disable a WAF rule and route around the problem, the Cloudflare dashboard itself was down. We were locked out of the controls that could have saved us.

It is worth stating the exposure in security terms, not just operational ones, because "the vendor had a bad day" is the wrong lesson. Availability is a security property — it sits alongside confidentiality and integrity — and a multi-CDN failover posture is the control that protects it against a single-provider failure. What the outage actually revealed was that our "resilient" architecture had exactly one point of failure, and it was the vendor whose entire job was resilience. The event hit an estimated 20% of all HTTP requests, roughly 7 million websites, and 30% of the top 1,000 sites; per Cloudflare's own post-mortem, a database permissions change caused a Bot Management feature file to double in size and blow past a hardcoded limit, crashing the traffic-routing software across the network and dragging the control plane down with it. I kept that post-mortem in one tab and a thorough third-party writeup of how one team scrambled and rebuilt afterward → in another while reworking our own single points of failure.

Threat model

Blaming the vendor is comfortable and useless. The outage exposed our own decisions, and naming them as threats is what made them fixable.

Single-provider control-plane failure. Cloudflare was authoritative DNS for every zone and proxied every hostname. When its control plane died we had zero ability to re-point anything — the failure of one provider was a total loss of both traffic routing and our ability to intervene.
Cascading failure with the control plane as victim. The overload did not stay contained; it rippled outward until the very system needed to recover was itself down — the textbook pattern Google's SRE book describes in its chapter on cascading failures. A recovery mechanism that shares a fate with the thing it recovers is not a recovery mechanism.
Undersized origins behind a cache. Origins sized for cache-hit traffic collapse in minutes when the CDN stops absorbing load — turning a CDN outage into an origin outage, a second-order failure we had built in.
Responder lockout via SSO through the failing provider. Engineers authenticated to internal tooling through Cloudflare Access. With Access degraded, the people who needed to respond could not reach the tools to do it. Concentrating the response path behind the same provider as the incident is an access-control failure mode, not just an availability one.

Controls we added

We gave ourselves a six-week sprint, in priority order.

Control 1 — secondary DNS, so no one control plane can strand us

DNS is the first and cheapest line of defense. We moved to a dual-provider authoritative setup — Cloudflare DNS primary, AWS Route 53 secondary — with both providers' nameservers registered at the registrar and both zones synchronized from a single Terraform source of truth so they cannot drift. Health checks follow the active-passive pattern AWS documents for Route 53 DNS failover. The point is not that Route 53 is "better" — it is that no single provider's control plane can strand us anymore.

Control 2 — a warm CDN fallback, not a slide-deck one

Cloudflare stays primary; AWS CloudFront is a pre-configured, continuously smoke-tested secondary, so "activate fallback" is a DNS weight change rather than a two-day build under fire. A fallback you have never exercised is a hypothesis, not a control.

Control 3 — a direct-origin escape hatch

We created a permanent, un-proxied path to origin: origin.ourdomain.com with proxying disabled, a 24-hour TTL so the record is cached far and wide before we ever need it, and documented copy-paste commands in the runbook. The security-critical detail is the certificate: the escape hatch carries a real, publicly-trusted TLS cert, not a Cloudflare Origin CA cert. Cloudflare's own SSL/TLS docs note that visitors hit untrusted-certificate errors the moment you disable proxying — which is exactly useless when Cloudflare is the thing that is down. An Origin CA cert only chains behind Cloudflare, so a failover that relies on it fails on TLS at the worst possible moment.

Control 4 — monitoring that does not depend on the thing it watches

Monitoring services through the CDN is like testing a smoke detector by asking the fire. We moved to direct-IP health checks (HTTP and TCP) against origin, origin-domain monitoring separate from the public hostname, two independent providers (UptimeRobot and Pingdom) so a single monitoring vendor blip does not blind us, and synthetic checks on the real login and checkout paths. Lost visibility during an incident is lost detection, so the monitoring path is deliberately decoupled from the serving path.

Control 5 — break-glass authentication outside the failing provider

This is the access-control control, and the one most directly in my lane. We kept Cloudflare Access for day-to-day SSO but added backup OIDC providers (Google Workspace and GitHub) wired up ahead of time, break-glass accounts that live entirely outside Cloudflare Access, and a one-page emergency-access document every on-call engineer holds offline. The principle is that the response path must never share a single point of failure with the incident — if the way in to fix things depends on the thing that broke, you have no way in.

Verifying the controls, not just shipping them

Infrastructure changes are worthless if nobody knows the sequence at 11:20 on a Tuesday, so the controls are backed by a timed runbook we drill quarterly against a deliberately broken staging environment: minute 0-5 detect and verify origin health independently; minute 5-15 flip DNS weight, activate the fallback CDN, enable origin-direct for critical paths; minute 15-30 communicate over a channel that is not behind the failing CDN; minute 30+ stabilize and start the post-mortem timeline. The drills are the verification — our first one took 40 minutes because nobody remembered where the break-glass doc lived, which is precisely the kind of finding you want in a drill and not in an incident. A simulated CDN loss now recovers to redirected traffic in under four minutes instead of three-plus hours hard down, but that number is only true as long as the next drill confirms it.

Residual risk / what we're still watching

The rebuild removed the single-provider single point of failure, but resilience work relocates risk rather than eliminating it, and the honest edges matter.

The escape-hatch certificate and TTL cut both ways. A publicly-trusted cert on origin. must be renewed and monitored like any production cert — an expired escape-hatch cert is a silent failure that only surfaces during the incident it was meant to survive. The 24-hour TTL that makes the record widely cached also means a mistake on that record lingers, so it changes only through reviewed IaC.
Origin autoscaling lag. When the CDN stops shielding origin, direct traffic is spikier than models predict. Pre-warming capacity is in the runbook, but the residual is that a large enough spike still outruns the scaler; we are tuning headroom.
Drills decay. A runbook is only as good as the last rehearsal, and quarterly is a compromise — long enough that details rot between drills. We watch drill times as the leading indicator of whether the control still works.
Hidden third-party dependencies. The scariest dependencies are the ones you did not know you had until they fail. We are mapping every third-party dependency in the stack and moving toward light chaos engineering — deliberately black-holing the primary CDN in staging on a schedule — to surface them before an outage does.
Adding providers adds surface. A second DNS provider, a second CDN, and backup OIDC providers each expand the set of vendors whose compromise or misconfiguration could affect us. More redundancy is more attack surface, so each addition is hardened and its credentials scoped rather than assumed benign.

The mindset shift the outage forced is the durable part: design assuming every provider will fail, then make the failure boring. What we keep watching is the gap between "we rehearsed this" and "the fire came at a time we did not pick" — because you only ever control the second half of that.

Sources & further reading

Cloudflare outage on November 18, 2025 — official post-mortem — the authoritative timeline and root-cause analysis, straight from Cloudflare.
Google SRE Book — Addressing Cascading Failures — the mental model for why an overload takes down far more than the thing that was overloaded.
AWS — Active-active and active-passive Route 53 DNS failover — the reference for the secondary-DNS failover pattern we adopted.
Cloudflare SSL/TLS docs — Origin CA certificates — why an Origin CA cert can't back a direct-to-origin escape hatch.
A hands-on writeup of a multi-CDN failover rebuild — a useful third-party field-notes version of this failover work to keep next to a runbook.

k3d, kind, Helm, Kustomize, Tilt, Skaffold, k9s: a one-screen map of who does what

Gaberial Sofie — Thu, 30 Jul 2026 09:38:26 +0000

Drowning in the local-Kubernetes tool zoo? Chapter 3 of the series sorts them onto shelves so you can see that you probably only need two of them. It's reference material, deliberately opinionated-but-fair.

Key takeaways

Two fundamentally different approaches. (1) Run all of Kubernetes locally (k3d/kind/minikube). (2) Connect your local process to a remote cluster (Telepresence/mirrord/Gefyra). The series takes approach 1.
Foundation layer: Docker (local cluster nodes are just Docker containers with Kubernetes inside) + kubectl (the official CLI everything else calls under the hood; it has Kustomize built in via -k).
Local clusters: k3d (k3s in Docker — fastest start, lowest memory, built-in registry), kind (upstream K8s, CNCF-certified, what the K8s project uses for CI/conformance), minikube (VM/driver + rich addons, great for learning). Consensus for dev: kind or k3d.
Manifest format: Helm (package manager — charts, templating, releases, rollbacks; ~75% of orgs, Helm 4 shipped Nov 2025) vs Kustomize (template-free base + overlays, built into kubectl). They complement more than compete.
Inner-loop accelerators: Tilt (Live Update + rich web UI, Tiltfile in Starlark), Skaffold (Google, YAML, CLI), DevSpace (two-way sync, YAML, CLI). Okteto and Garden solve adjacent problems.
The remote tools differ meaningfully: only Telepresence needs root; Telepresence and mirrord can swap files/env (Gefyra can't); only mirrord handles several local processes at once.
Helpers: k9s (terminal UI / "visual kubectl", colour-code prod red so you don't nuke it), kubectx/kubens, stern, krew.
Why the series picks k3d + Tilt: real K8s API + fast/cheap recreate, plus a minutes→seconds loop and a beginner-friendly dashboard. Both OSS and local.

Full article: https://dorokhovich.com/blog/local-k8s-tooling-overview?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-tooling-overview