DEV Community: Gaberial Sofie

I built a production-like local Kubernetes setup in 16 parts — here's the recap and where to grow next

Gaberial Sofie — Sun, 20 Sep 2026 11:12:50 +0000

This is the capstone of a series on running a genuinely production-like local Kubernetes environment. The whole point was environment parity: catch manifest, RBAC, and service-discovery problems on your laptop instead of from a 3am alert.

What the setup is

k3d (k3s in Docker) cluster dev with a built-in registry -> a FastAPI image that's the same artifact prod ships -> real manifests (Deployment/Service/Ingress, not docker-compose) -> a fast inner loop via Tilt -> PostgreSQL alongside -> ConfigMap/Secret config -> liveness/readiness probes + requests/limits -> observed by hand via kubectl + k9s.

Three directions to grow (don't adopt all at once)

GitOps (Argo CD / Flux, both CNCF Graduated). Git becomes the single source of truth; an in-cluster controller reconciles reality toward it. Argo CD's edge for beginners is a rich web UI (the Application CRD as an object tree); Flux is the GitOps Toolkit (controllers, image automation, OCI) but has no native UI. Common pattern: Argo for apps, Flux for infra — but pick one.
Observability stack (kube-prometheus-stack, Prometheus Operator, OpenTelemetry). One Helm command gives you Prometheus + Grafana + Alertmanager + exporters; scrape your app with a ServiceMonitor CRD, no hand-edited Prometheus config. Honest caveat: this stack is heavy and can take down a weak k3d cluster — trim components or cap requests. OpenTelemetry unifies traces/metrics/logs (auto-instrumentation for FastAPI).
Remote-cluster tools (Telepresence, mirrord). For the hard cases where local can't hold all the dependencies. mirrord's key nuance is the incoming-traffic mode: mirror (your process gets a copy of traffic — safe on shared clusters) vs steal (your process intercepts the Pod's traffic — diverts others' requests, use carefully). Telepresence intercepts via a VPN/tun model (needs root + an in-cluster component).

The measured posture: here's the ladder, don't climb it all at once. Reach for remote-dev tools only when a local cluster genuinely can't handle the scenario.

Full article: https://dorokhovich.com/blog/local-k8s-conclusion-and-next-steps?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-conclusion-and-next-steps

Local Kubernetes Dev — Part 16: Conclusion and next steps

Gaberial Sofie — Fri, 18 Sep 2026 07:53:00 +0000

The local Kubernetes series is complete — 16 parts.

If you're new and just arrived — start with chapter one on the inner dev loop. This final one pulls it all together: what we built and where to grow next.

What we ended up with — a real (not emulated) cluster on your laptop: k3d + a FastAPI image + real manifests (Deployment/Service/Ingress) + a fast inner loop on Tilt + PostgreSQL alongside + config/secrets + probes and limits + k9s. The main payoff is environment parity: manifest, RBAC, and service discovery problems show up locally, not via alerts at 3 a.m.

And three honest directions to grow (don't grab all at once):
• GitOps (Argo CD / Flux) — when the team and the number of environments grow;
• an observability stack (kube-prometheus-stack, Prometheus Operator, OpenTelemetry) — with a warning: a heavy stack can take down a weak k3d;
• remote development (Telepresence, mirrord) — and the difference between mirror mode (a copy of traffic, safe) vs steal (interception).

Read the finale and jump back to any chapter: https://dorokhovich.com/blog/local-k8s-conclusion-and-next-steps?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-conclusion-and-next-steps

Your Ingress does nothing until you have an Ingress controller (and 3 more local-k8s networking traps)

Gaberial Sofie — Sun, 30 Aug 2026 23:43:09 +0000

The single most common local-Kubernetes frustration is "I applied an Ingress and nothing responds." The docs are blunt about why: "You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect."

Takeaways

An Ingress manifest is just rules. It needs a running controller (Traefik, NGINX...) that reads them and actually accepts traffic. Which controller serves it is set by ingressClassName.
k3d ships Traefik out of the box (k3d is built on k3s), plus a built-in ServiceLB (Klipper) so LoadBalancer services don't hang in pending. But Traefik listens inside the cluster — to reach it from the host you must forward a port at cluster-creation time:

  k3d cluster create dev --api-port 6550 -p "8081:80@loadbalancer" --agents 2

You cannot add this port map to a running cluster. Forgot it? Recreate. This is the #2 pain after "forgot the controller."

DNS inside the cluster is by name, not IP (CoreDNS). FQDN pattern: <service>.<namespace>.svc.cluster.local. Short names resolve only within the same namespace via search domains — cross-namespace you need postgres.myapp or the full FQDN.
*.localhost is DNS behaving to spec, not k3d magic. RFC 6761 reserves .localhost for loopback. Chrome and Firefox resolve any *.localhost to 127.0.0.1 with zero config — but Safari and non-browser tools (curl, HTTP clients) treat it as an ordinary domain and hit the OS resolver, giving could not resolve host. Fix: add it to /etc/hosts.

The full traffic chain to memorize: curl -> host port 8081 -> Traefik (port 80) -> Service myapp -> Pod.

Full article: https://dorokhovich.com/blog/local-k8s-networking-and-ingress?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-networking-and-ingress

Local Kubernetes Dev — Part 11: Networking — reaching your service and Ingress

Gaberial Sofie — Sat, 29 Aug 2026 16:47:12 +0000

"I applied an Ingress — and it doesn't respond. Why?"

A local Kubernetes classic: you write an Ingress manifest, kubectl apply, open the browser — and silence. The thing is, an Ingress by itself is just rules. Until there's an Ingress controller in the cluster (Traefik, NGINX) that reads those rules and accepts traffic, nothing works. Straight from the docs: "Only creating an Ingress resource has no effect."

And the second most popular k3d pain: the loadbalancer port mapping is set ONLY at cluster creation. Forgot -p "8081:80@loadbalancer"? You'll have to recreate the cluster.

In the new article (part 11) I break networking down layer by layer:
• how Pods find each other by name via CoreDNS (and why the short name only works inside its own namespace);
• three ways to reach in from outside: port-forward, NodePort, Ingress — and when to use which;
• Traefik out of the box in k3d and the whole chain: curl → port 8081 → Traefik → Service → Pod;
• why http://myapp.localhost opens in Chrome but curl says "could not resolve host" (spoiler: RFC 6761, and it's not a bug).

Read it and stop fighting your cluster's networking: https://dorokhovich.com/blog/local-k8s-networking-and-ingress?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-networking-and-ingress

Running Postgres, Redis and RabbitMQ inside your local k3d cluster (with the 2025 Bitnami trap)

Gaberial Sofie — Mon, 24 Aug 2026 15:50:55 +0000

Chapter 9 of a local-Kubernetes series: the stateful-dependencies chapter the earlier ones deliberately deferred. Bring up your backing services in the cluster so local structurally matches prod — and avoid a 2025 supply-chain landmine.

Key takeaways

Why in-cluster, not compose-on-the-side: one world, not two. In-cluster Postgres/Redis/RabbitMQ gets you unified DNS/service discovery (app finds the DB at postgres, not localhost:5432), shared Secrets/ConfigMaps, and you catch RBAC/limits/probe problems before rollout. (Fair: compose is a legit choice if the service isn't going to k8s.)
The Bitnami warning without which this chapter would be harmful: since 2025-09-29 most public Bitnami OCI charts/images moved behind a Broadcom subscription; the rest are bitnamilegacy (unsupported, no security patches). Don't blindly recommend bitnami/* in 2026. Use Chainguard's drop-in charts, official operators, or a deliberately-pinned legacy tag (dev only).
The most predictable dev path is a raw 3-object manifest: Deployment (official postgres:17) + Service (stable DNS postgres) + PVC — no chart repos, no subscription dependency. Includes a pg_isready readiness probe.
The big storage surprise: in k3d, data is ephemeral by default — local-path-provisioner writes inside the node container, so k3d cluster delete wipes it. To persist, map a host dir at create: --volume $HOME/k3d-storage:/var/lib/rancher/k3s/storage@all.
Wire into Tilt: feed the manifest to k8s_yaml, order startup with resource_deps (myapp waits for postgres's readiness), port_forwards=['5432:5432'] to reach it from your IDE.
Migrations: a separate Job (+ initContainer wait), ordered postgres → migrate → seed → myapp. Migrations in the app's own initContainer is an anti-pattern (races across replicas, probe kills). Note Tilt's built-in helm() skips hooks — use helm_resource for anything with init hooks.
Mock vs run: OSS deps (Postgres/Redis/RabbitMQ) → run as-is; proprietary managed (S3/SQS/DynamoDB) → emulate with LocalStack, but it's not 1:1 (IAM, consistency, edge cases).

Full article: https://dorokhovich.com/blog/local-k8s-dependencies-databases-queues?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-dependencies-databases-queues

Local Kubernetes Dev — Part 9: Dependencies — databases, queues, caches

Gaberial Sofie — Fri, 21 Aug 2026 08:42:31 +0000

OK, your service runs in the cluster locally — but what about the database, the queue, the cache? This is the chapter the previous ones deliberately put off.

Part nine of the series is about stateful dependencies. We stand up PostgreSQL, Redis, and RabbitMQ INSIDE the local cluster (Helm or a "raw" manifest), wire them into Tilt, and sort out PersistentVolumes, migrations, and seeding on startup. The point is parity: if the database lives in the same cluster with the same charts that ship to staging, you get unified DNS and service discovery for free (the app finds the DB by the name postgres, not localhost:5432), shared Secrets/ConfigMaps, and you catch manifest problems before prod — not on a Friday evening.

An honest warning, without which this chapter would do more harm than good: as of 2025-09-29 the public Bitnami images moved behind a paid Broadcom subscription, so blindly recommending bitnami/* in 2026 is no longer an option. What to do instead — Chainguard drop-in charts, official operators, or a deliberate pin to a legacy tag. For dev, the most predictable path is a "raw" manifest of three objects (Deployment + Service + PVC) with the official postgres:17 image.

Plus: the biggest surprise — in k3d data is ephemeral by default (fixed with --volume $HOME/...:/var/lib/rancher/k3s/storage@all), migrations as a separate Job (not in the app's initContainer — that's an antipattern), and when to mock a managed service (RDS/SQS) with LocalStack and when not to. https://dorokhovich.com/blog/local-k8s-dependencies-databases-queues?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-dependencies-databases-queues

One Realm for Every App and Tenant Was a Time Bomb: A Keycloak Realms Clients Roles Threat Model

Gaberial Sofie — Wed, 12 Aug 2026 12:32:06 +0000

The exposure

The first Keycloak rollout worked on the first try, which is the dangerous part. Under the hood it was one realm for every app and every tenant, wildcard redirect URIs, a client secret embedded in a single-page app, and roles assigned directly to individual users. None of it threw an error, and that is precisely why it festered for six months until we had three apps, two customer tenants, and a growing suspicion the whole thing was held together with tape.

Understanding keycloak realms clients roles correctly is a security exercise, not a modeling preference, because each of those four sins maps to a concrete class of vulnerability. A working login flow and a safe one look identical from the outside, and the console will happily let you build the former long before you have the latter. Before changing anything it was worth cataloguing exactly what each shortcut exposed us to. For the concept model and the remediation I worked from a thorough third-party guide to Keycloak's core concepts → alongside the official administration guide.

Threat model

No tenant isolation — shared blast radius. One realm carrying all apps and both tenants means a misconfiguration, a leaked admin credential, or a bad role change in one tenant's context has blast radius across every tenant. There is no boundary to contain a mistake, which is the opposite of what a realm is for.
Wildcard redirect URIs — open redirect and token exfiltration. http://localhost:3000/* was convenient in dev and quietly rode into prod. A wildcard redirect lets an attacker steer the authorization-code callback to a URL they control, turning a valid login into a token-leak primitive.
Client secret in a SPA — a secret that is not secret. The single-page app was registered as if it could hold a secret; the secret shipped in the browser bundle, readable by anyone. A confidential client whose secret is public is a confidential client in name only, and it undermines the entire client-authentication assumption.
Per-user role assignment — unauditable and unscalable. Assigning roles to individuals by hand means "who can do X?" is answerable only by enumerating every user, and every onboarding is a manual chance to over-grant. Least privilege is not enforceable if entitlements are scattered.

The organizing principle for the fix is the three-word model: realms isolate identities, clients integrate apps, roles authorize actions. Every defect above was a violation of one of those three boundaries, so the remediation is to restore each boundary deliberately.

Controls we added

Control 1 — realms as isolation boundaries

A realm is a self-contained boundary with its own users, groups, clients, and roles — Keycloak's definition is that "a realm manages a set of users, credentials, roles, and groups" (Server Administration Guide). We split by tenant, and at minimum by environment (dev/staging/prod), so a misconfiguration in one is contained to one. Users no longer span realms automatically, and that is the point: if identity genuinely needs to be shared, identity brokering or federation is the explicit, reviewable tool for it, rather than an implicit shared bucket.

There is a ceiling to this control worth stating honestly, because over-applying it creates its own risk. Realm-per-tenant is right for a handful of tenants but stops scaling around a couple dozen — past that you are operating an IAM platform instead of shipping a product, and a sprawl of realms becomes its own misconfiguration surface. Keycloak's Organizations feature (preview in 25, GA in 26 — announcement) provides first-class multi-tenancy inside a single realm for that case. The rule: separate realms to isolate genuinely different security domains (different admins, or dev/staging/prod); Organizations to partition many customers who share one app's trust boundary.

Control 2 — clients with an honest access type and PKCE

A client's access type is a security control, not a convenience setting:

public — SPAs and native apps; no secret, because anything shipped to a browser or device is readable.
confidential — server-side apps that can actually keep a secret.
bearer-only — APIs that verify tokens and never start a login flow.

The SPA became a public client, and the leaked secret was retired rather than rotated-in-place, because the fix is to stop pretending a browser bundle is private, not to ship a fresh secret into the same exposure. What replaces the secret is PKCE (S256): designed specifically for OAuth public clients that cannot keep a secret (RFC 7636), it stops an intercepted authorization code from being redeemed by an attacker (Auth0's PKCE explainer). Turning it on is non-negotiable for browser and native clients. Alongside it, the wildcards died: valid redirect URIs and web origins are now exact entries, no *, to close the open-redirect vector.

Control 3 — roles and groups for least privilege by construction

Roles model permissions, and we now use two flavors deliberately: realm roles for org-wide levels (admin, editor, viewer) and client roles for app-specific permissions (billing:read). Composite roles let one Admin assignment imply a bundle — though a composite is also where over-grant hides, so each composite is reviewed for exactly what it expands to. The biggest change was mapping roles to groups, not individuals: assigning to users does not scale and is not auditable; group-based mapping is both. A new hire joins a group and inherits precisely the intended roles, which makes least privilege the default path rather than a discipline someone has to remember.

Control 4 — authorize from the token, not from identity

The refactor only pays off if the application uses the structure. The early mistake was scattering permission checks keyed on usernames — a hard-coded allow-list is unauditable and drifts silently. The clean version reads roles straight from the token: in OIDC, Keycloak places realm roles under realm_access.roles and client roles under resource_access[client].roles. An authorization check becomes "does this token carry the editor role?" rather than "is this user in my admin list?" Because Admin is modeled as a composite, one role in the token expands to everything it should, consistently. Permissions become a configuration concern in Keycloak, not a deployment concern in the codebase — a new hire is added to a group, the roles land in their token, and the app authorizes them with no code change or redeploy.

Verifying the controls, not just shipping them

The rebuild started from a minimal, reviewable baseline rather than clicking around the console:

docker run -p 8080:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:25.0.2 start-dev

Then a five-step skeleton: a dedicated realm per security domain, each app as its own client with exact redirect URIs and the correct access type, realm roles like Admin and User, and assignment via groups. The verification that mattered most was that the controls are now demonstrable: "who can write billing?" used to mean grepping code and cross-referencing a user table; now it is "which groups carry a role that composes billing:write?", answerable in the admin console in seconds. An entitlement you can query is an entitlement you can audit, and negative-testing a redirect URI with a deliberately invalid value confirms the allow-list actually rejects what it should.

Residual risk / what we're still watching

Restoring the three boundaries removed the acute exposures, but each control carries its own residual risk.

Realm sprawl vs. Organizations. Separate realms isolate cleanly at small scale but become an operational and misconfiguration burden past a couple dozen tenants. We are watching the tenant count against the point where Organizations-in-one-realm is the safer trade, since the wrong choice in either direction reintroduces risk — sprawl on one side, a weaker isolation boundary on the other.
PKCE and redirect-URI drift. The public-client posture depends on PKCE staying enforced and redirect URIs staying exact. A future convenience edit that re-adds a wildcard, or a client that quietly disables PKCE, silently reopens the token-leak vector. Both live in reviewed configuration.
Composite-role over-grant. Composites are ergonomic but concentrate authority; a role added to a widely-assigned composite grants it to everyone downstream at once. We review composite membership as an entitlement change, not a convenience.
Group hygiene. Group-based assignment scales, but it moves the risk to group membership — a stale or over-broad group is now the thing to audit. We periodically reconcile group membership against intended access.
The leaked secret is retired but assume-compromised. Because the old SPA secret was public for months, we treat it as burned permanently and monitor for any client still configured to accept it.

The lesson that stuck is that the console lets you build something that works long before it is something that is safe, and the three-word model — isolate, integrate, authorize — is both the setup mantra and the operating model that keeps the system explainable, and auditable, a year later.

Sources & further reading

Keycloak — Server Administration Guide — realms, clients, roles, groups, and composite roles.
Keycloak — Organizations announcement — multi-tenancy inside a single realm (preview in 25, GA in 26).
RFC 7636 — Proof Key for Code Exchange — why public clients use PKCE instead of a secret.
Auth0 — Authorization Code Flow with PKCE — a clear walkthrough of the SPA/native login flow.
A Keycloak realms/clients/roles walkthrough — a useful third-party account of the pitfalls, the cross-realm FAQ, and a runnable five-step example.

A Typosquatted Gem Almost Shipped, Caught by Luck: A Ruby Supply Chain Security Playbook

Gaberial Sofie — Tue, 11 Aug 2026 19:26:01 +0000

The exposure

A teammate opened a PR adding a small utility gem. The name looked right at a glance — one transposed character from a popular package. It passed our tests. On a normal Friday it would have merged. Someone caught the typo in review by luck, not process, and the gem turned out to be a typosquat whose extconf.rb ran code at install time that had no business running.

"By luck, not process" is the phrase that reframes this from an anecdote into a security finding. Ruby supply chain security is the discipline of not depending on luck — or on someone else's controls — for the integrity of code you ship. We were trusting gem install blindly and leaning entirely on RubyGems.org to catch everything upstream. It usually does, but "usually" is a probability, not a guarantee, and a plan that works right up until it doesn't is not a control. RubyGems.org's own account of how it protects the ecosystem puts a number on it: its automated tooling catches roughly 70-80% of malicious packages before anyone reports them. That upstream layer is real and valuable, but the residual 20-30% is precisely the population your build has to assume it will meet. For the fuller hardening notes I worked from a thorough third-party writeup of a Ruby dependency-security playbook → alongside the registry's post.

Threat model

Typosquatting and look-alike names. A one-character-off gem name is a social-engineering primitive aimed at review and autocomplete. It is caught, if at all, by attention — an unreliable control at 2pm on a Friday.
Arbitrary code execution at install time. extconf.rb, native extension builds, and install hooks run on developer laptops and CI runners with whatever privileges those hosts hold. A malicious gem does not need to reach production to do damage; installing it is the exploit.
Lockfile injection and source substitution. A version number can be re-pointed at different bytes upstream, and a machine-generated Gemfile.lock that nobody reads during review can quietly point a dependency at a different set of bytes than the manifest implies. Mixed gem sources open a substitution path where a gem is pulled from an unexpected origin.
The post-disclosure window. A gem that is clean today gets a CVE tomorrow while sitting in your lockfile. Detection that only runs when code changes never revisits already-shipped dependencies.
Over-reliance on upstream controls. Delegating your production security entirely to the registry's 70-80% means the ecosystem's residual miss rate is your residual risk, unmediated.

The organizing principle is defense in depth: the registry is the first line, and the build must be a deliberate second line behind it, because malicious packages are not rare events — they are background noise of the modern software supply chain.

Controls we added

Control 1 — shrink the attack surface

Pin versions and platforms in Gemfile.lock and commit it. No floating ranges into production.
No git dependencies without a commit SHA. A branch can be rewritten; a SHA cannot.
Narrow ranges (~>) for critical gems, and prune ruthlessly — every unused and transitive gem is surface area, reviewed on a quarterly dependency-hygiene day.

Control 2 — checksum verification, the highest-leverage single addition

Pinning a version number is necessary but not sufficient, because the number can be re-pointed at different bytes. Bundler 2.6 makes checksum verification first-class (--add-checksums is a documented bundle lock flag):

bundle lock --add-checksums

This writes a CHECKSUMS section into Gemfile.lock, and from then on Bundler refuses to install a gem whose contents do not match the exact bytes you locked. It closes the nastiest gap in a pin-the-version-only strategy: a checksum cannot be quietly swapped the way a version can be re-pointed. If you adopt one new control this quarter, make it this one. Alongside it, a few Bundler settings harden source handling — disable_multisource true (the underrated one, blocking the source-substitution class), cache_all true, and clean 'true'.

Control 3 — make the human review a process, not a hero act

The near-miss was caught by a person, so we turned the person into a checklist. Adding or bumping a gem now requires a short "why this gem?" note and a CHANGELOG link in the PR, documented in the PR template where the work happens rather than a wiki nobody reads. Dependabot and GitHub alerts are enabled so advisories are not manually watched. The cultural shift — that a dependency is code we are adopting, not a freebie we are grabbing — mattered more than any single tool.

Control 4 — gate it in CI

A checklist people can forget is a suggestion; a CI gate is a rule. Two independent scanners catch different things, and layering them is cheap: bundler-audit (checking Gemfile.lock against the ruby-advisory-db) and Google's OSV-Scanner (fed by osv.dev). It runs on every dependency-touching PR and weekly on a schedule:

on:
  pull_request:
    paths: ['Gemfile', 'Gemfile.lock']
  schedule:
    - cron: '0 6 * * 1'   # weekly on Mondays

jobs:
  bundler-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with: { ruby-version: '3.3', bundler-cache: true }
      - run: gem install bundler-audit
      - run: bundle audit check --update
  osv-scanner:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google/osv-scanner-action@v1
        with: { scan-args: '-L -r .' }

The weekly cron addresses the post-disclosure window directly: a vulnerability disclosed on Wednesday fails the build even if nobody touches the code. For teams that want typosquat detection and SBOM generation on top, a Ruby-specific scanner like GemGuard — pulling from OSV.dev and the ruby-advisory-db — slots in beside bundler-audit rather than replacing it.

Verifying the control, not just shipping it

The fastest way to get a security process ignored is to drop a wall of red CI checks on a team mid-sprint, so we staged it and verified adoption as deliberately as coverage. Week one, the scanners ran in warn-only mode — reporting findings without failing the build. That surfaced the existing backlog of advisories, gave us an honest baseline instead of a nasty surprise, and let us triage and either fix or explicitly accept each item. Only then did we flip the gate to blocking, by which point there was nothing left for it to fail on. The second thing that kept adoption alive was making the checks fast and legible: bundler-audit and OSV both run in well under a minute with a warm cache, and their output points straight at the offending gem and advisory, so a failure has an obvious fix and nobody routes around it. A gate that takes ten minutes or emits inscrutable output gets bypassed, which is how a control becomes theater.

Residual risk / what we're still watching

The playbook meaningfully raised the cost of a supply-chain compromise, but it does not reduce the residual to zero, and the edges are where the honesty has to be.

Scanners detect known vulnerabilities only. A brand-new typosquat or an undisclosed backdoor has no advisory yet and passes clean. That is exactly why the human "why this gem?" review still carries load the automation cannot — the checklist is a control against the unknown-unknown, not redundant with the scanners.
Signatures are not a force field. Gem signing is one control among pinning, checksums, review, and CI gating — treating it as a silver bullet would be a false-confidence failure. We weight it accordingly.
The lockfile diff still needs human eyes. Checksum verification defuses most lockfile-injection risk, but a reviewer who reads only the Gemfile and skims the machine-generated Gemfile.lock can still miss a dependency quietly re-pointed. We review the lockfile diff on any dependency PR.
Patch-bump noise vs. coverage. A blanket manual-review policy drowns the team; letting CI and scans gate patch bumps while humans review major/minor keeps signal high, but it does accept that a malicious patch release is caught by scanners rather than eyes. We watch that trade.
We still depend on the registry and the advisory databases. Our second line assumes RubyGems.org's first line and assumes OSV.dev and the ruby-advisory-db are current; a gap or delay in any of them is a gap in us. Two independent scanners mitigate the advisory dependency, and — because that shared line of defense runs partly on a small group's effort — we treat a recurring sponsorship of the ecosystem as risk management, not charity: the CI gate protects our repo, sponsorship protects the line in front of it.

If a team ships Ruby, the near-miss described here is a matter of when, not if. The residual we keep watching is the undisclosed package and the lockfile change nobody reads closely — the two places where luck, not process, is still doing quiet work.

Sources & further reading

RubyGems.org — How RubyGems.org Protects Our Community's Critical OSS Infrastructure (Aug 25, 2025)
rubysec — bundler-audit and the ruby-advisory-db it checks against
Google — OSV-Scanner, backed by osv.dev
Bundler docs — bundle lock --add-checksums (Bundler 2.6 checksum verification)
wilburhimself — GemGuard: typosquat detection + SBOM for Ruby
A full Ruby supply-chain playbook — a useful third-party account of the Bundler hardening settings and legacy-app rollout FAQ.

Governance Risk Is Supply-Chain Risk: Auditing Every Dependency After the RubyGems Hostile Takeover

Gaberial Sofie — Mon, 10 Aug 2026 08:12:28 +0000

The exposure

In September 2025 the people who built and ran RubyGems and Bundler lost administrative access to the infrastructure they maintained, and several publicly quit. One side called it a hostile takeover; the other called it responsible stewardship. The verdict is not the point. The exposure is: infrastructure serving billions of downloads a month — the infrastructure every bundle install, every deploy, every CI run reaches into — demonstrated that control over the packages your builds depend on can change hands overnight.

Framed as a security problem, the rubygems hostile takeover is a governance-risk event, and governance risk is supply-chain risk that no vulnerability scanner will ever flag. A CVE scanner tells you a dependency has a known flaw; it says nothing about whether that dependency's maintainers just walked out or its publishing rights just moved to someone you did not vet. My company does not contribute to RubyGems — we only consume it, which means we had silently accepted a dependency on both the packages and the people, and the crisis made that dependency visible. The Register covered the control fight as it erupted, Ruby Central later published its own RubyGems Fracture Incident Report owning the execution mistakes, and I worked through a thorough third-party first-person timeline of what one engineer changed in response → while treating our own exposure as a fire drill rather than a hot take.

Threat model

The timeline matters only insofar as it names the threats. Between September 9 and 19, the GitHub org was renamed, maintainers were removed, administrative access to the RubyGems, Bundler, and RubyGems.org teams was revoked — including control of the bundler and rubygems-update gems themselves — and the creator of Bundler announced the team was effectively finished. From that, the concrete threats:

Publishing rights changing hands overnight. Control over the exact packages your builds pull can move to a different party without your consent or knowledge. If your security model assumes the right people hold the keys, it is not a model — it is a hope. Ruby Central's own incident report concedes the foundation "lacked the structural ability to make this change directly," i.e. an informal power structure that a single actor could reconfigure.
Maintainer-exodus as a failure mode. The people who understand a critical dependency's internals can all leave at once. A healthy maintainer team is itself a dependency, and its collapse is a real failure mode — a bus-factor risk at ecosystem scale.
Registry availability during the incident. If RubyGems.org had gone dark or a gem had been yanked mid-fracture, builds that fetch from upstream at deploy time would have failed. Reachability of a third party you do not control sits on your critical path.
Funding-pressure-driven consolidation. Reporting since made clear how much funding pressure sat underneath the fight — The Register relayed developer Joel Drapper's allegation that Shopify pressed for consolidated control of the bundler and rubygems-update gems under threat of pulled funding. Commercial leverage over volunteer infrastructure is a governance attack surface in its own right.
Ecosystem split. The ousted maintainers did not disappear; Martin Emde announced The Gem Cooperative (gem.coop), a community-run mirror, which The Register covered as the maintainers' new home. "Which registry does our bundle install actually hit" went from a question I would have shrugged at to a line item.

The organizing principle: reproducibility is resilience. If a build needs the internet — and a particular governance arrangement — to be calm in order to succeed, it will fail when they are not.

Controls we added

I gave the drill one week and one goal: make sure a governance blowup or a bad actor upstream could neither stop us shipping nor poison our builds.

Control 1 — freeze the ground we stand on

Committed and audited Gemfile.lock so every version and platform is pinned, not floating into production.
Vendored the critical path. bundle cache copies every .gem into vendor/cache and, per the docs, makes future installs "use the gems in the cache in preference to the ones on rubygems.org"; bundle config set cache_all true keeps git and path gems there too. If RubyGems.org went dark or a gem was yanked mid-incident, deploys still work from the cache. This directly retires the registry-availability threat.
Pinned git dependencies to commit SHAs, never bare branch names. A branch can be force-pushed under new ownership; a SHA cannot be quietly swapped — which addresses the "control changed hands" threat at the integrity level.

Control 2 — inventory governance risk, not just CVE risk

Because scanners like bundler-audit (which checks Gemfile.lock against the ruby-advisory-db) are silent on governance, I built a checklist and ran our top dependencies through it:

Who actually controls publishing rights to this package today?
How many maintainers — is it a bus-factor of one?
Has ownership or org name changed recently?
Is there a funded, formal body behind it, or informal goodwill?
If it vanished tomorrow, what is our replacement path?

That last question turns a vague anxiety into a ranked list of things to fix, which is the difference between worrying about governance risk and actually managing it.

Control 3 — reduce the surface

Fewer upstreams means fewer governance fires can burn you. I pruned unused and transitive bloat and added a "why this dependency, and who is behind it?" note to code review for anything new — the same hygiene the ecosystem needs at the macro level, applied to our own repo. Least privilege for dependencies: adopt only what you can justify and account for.

Verifying the controls, not just shipping them

A control the team does not believe in gets removed the first time it is inconvenient, so the verification here was as much organizational as technical. I made the risk concrete with three questions borrowed from the crisis: if the maintainers of our single most critical dependency all quit tomorrow, what breaks and how fast; who can publish a new version of that package right now, and do we trust the process that grants them that power; and if that package got hijacked and shipped a malicious release, would our pipeline catch it before production. For most dependencies the honest first answer was "we don't know," and "we don't know" is the actual risk — so the checklist and the vendoring exist to convert those unknowns into documented, tested answers. The vendoring in particular is verifiable in the most direct way: a build run with upstream network access blocked either succeeds from vendor/cache or it does not, and ours does. Unexpectedly, "we are hardening against a real, recent, widely-reported failure in our supply chain" turned out to be the easiest resilience work I have ever had to justify to leadership.

Residual risk / what we're still watching

The drill meaningfully reduced our exposure to an upstream governance shock, but it did not make us independent of the ecosystem, and the edges deserve honesty.

Vendored gems drift and stale. vendor/cache protects availability, but a frozen cache also stops receiving security updates; a vulnerability disclosed in a vendored gem is now on us to notice and refresh. Vendoring trades an availability risk for a patch-currency obligation, so we pair it with scheduled re-audits rather than treating the cache as done.
Scanners still cannot see governance. The checklist is a manual control against a threat no tool flags, which means it decays unless it is re-run. We re-inventory the top dependencies periodically, because ownership and maintainer health change quietly between reviews.
The ecosystem split is unresolved. With gem.coop now standing alongside RubyGems.org, "which registry are we actually trusting, and what is its provenance" is an ongoing decision, not a settled one. We are watching how it resolves without letting our builds depend on it resolving well.
Transitive dependencies we do not control. We can inventory our direct dependencies' governance, but the long tail of transitive gems carries the same risk with far less visibility. That surface is the hardest residual to close, and it is where we assume the next surprise lives.
Our second line still assumes a first line. Vendoring and pinning protect us from upstream disappearing, not from a malicious release we vendor before anyone notices it is malicious — that gap is covered by scanning and review, which are themselves only as current as their data.

The durable lesson is that you depend on people, not just packages, and that a small group quietly holds up infrastructure the whole ecosystem leans on. When that group fractures, everyone downstream discovers exactly how much they were depending on trust they never examined. What we keep watching is that trust — deliberately, on calm afternoons, rather than during the next incident.

Sources & further reading

The Register — Open source to closed doors: RubyGems control fight erupts (Sep 25, 2025)
Ruby Central — RubyGems Fracture Incident Report
Martin Emde — Announcing The Gem Cooperative (gem.coop) (Oct 5, 2025)
The Register — Kicked from RubyGems, maintainers forge new home at Gem Cooperative (Oct 6, 2025)
Bundler docs — bundle cache: vendoring gems into vendor/cache
rubysec — bundler-audit: scan Gemfile.lock against the ruby-advisory-db
A field-notes dependency-audit checklist from the week of the crisis — a useful third-party account of the same governance-risk drill applied to a real repo.

When One Keycloak Node Is a Single Point of Failure: A Keycloak Cluster Nginx Threat Model for Zero-Downtime Upgrades

Gaberial Sofie — Sun, 09 Aug 2026 08:50:41 +0000

The exposure

We ran Keycloak the way most teams start: one node, one VM, "we'll cluster it later." Later arrived as a 40-minute outage. A routine security patch required a restart, the node came back with a config typo, and for the duration nobody in the company could authenticate to anything — not the admin tools, not the customer portal, not the internal apps that all trusted it.

It is worth being precise about what class of exposure that is, because "make it not crash" is the wrong framing. Availability is a security property — it sits alongside confidentiality and integrity in the CIA triad — and a single identity provider is a single point of failure whose blast radius is every system that delegates authentication to it. When that one node is down, the failure is not "one service is slow"; it is that the entire organization loses the ability to prove who anyone is. A keycloak cluster nginx design is the control that bounds that blast radius: the postmortem action item was blunt — Keycloak must survive a node loss and support rolling upgrades so that a routine patch is never again an org-wide authentication outage. I did not want to invent an HA design from first principles, so I worked from a thorough third-party guide to running Keycloak behind Nginx as a real cluster → alongside the official docs.

Threat model

Framing the outage as a security problem dictated what we actually had to build.

Total authentication loss on node failure. With one node, any crash, bad config, or patch takes down authentication for every dependent system simultaneously. The control objective is that a single node loss is survivable and invisible to users.
Patch and upgrade as forced outage. If the only way to apply a security patch is to restart the sole node, then staying patched and staying available are in direct conflict — which pressures teams to defer security updates. Rolling upgrades remove that conflict, so the cluster is also a patch-hygiene control.
Session loss as a re-authentication storm. If sessions live only on the failed node, a failover forces every active user to log in again at once. Beyond the user impact, a synchronized re-auth burst is itself a load event against the surviving node. Session state must survive a node loss.
Proxy misconfiguration as an integrity risk. Putting a reverse proxy in front means Keycloak now trusts forwarded headers to build redirect URLs and evaluate the request origin. Mishandled X-Forwarded-* headers produce wrong issuer/redirect behavior — a correctness-and-trust problem, not just a cosmetic one.

Controls we added

Control 1 — two nodes with replicated session state

Clustering adds redundancy, and Infinispan provides the distributed caches that replicate sessions and tokens across nodes, so a user logged in on node 1 stays logged in if node 1 disappears (Keycloak's distributed-cache guide documents the session, token, and authentication-session caches). Discovery is the first real decision: on VMs, JGroups with JDBC_PING (nodes find each other through a shared database table — now Keycloak's default discovery mechanism); on Kubernetes, KUBE_PING via the API with a StatefulSet and headless Service. We were on VMs with a shared Postgres, so we started on JGroups/JDBC_PING and planned a later migration to KUBE_PING.

Control 2 — Nginx as the boundary, configured deliberately

Nginx terminates TLS, hides the internal cluster topology, and load-balances across the nodes — the reverse-proxy pattern documented for HTTP load balancing. The Keycloak side uses the modern Quarkus proxy flags, which are the crux of running behind a proxy safely:

command: >
  start --http-enabled=true
        --hostname-strict=false
        --proxy-headers=xforwarded
        --metrics-enabled=true

--proxy-headers=xforwarded is the line people miss; the legacy PROXY_ADDRESS_FORWARDING env var is gone. Keycloak's reverse-proxy guide is explicit that xforwarded enables parsing of the X-Forwarded-* headers and that you relax --hostname-strict (or pin --hostname) so redirect URLs are built correctly. Getting this wrong produces "invalid issuer" and redirect errors — and, more subtly, means Keycloak may be trusting client-supplied headers it should not, which is why the proxy must be the only thing setting them.

The Nginx config matches those flags — TLS termination, forwarded headers, passive health checks, and sticky sessions:

upstream keycloak_http {
  ip_hash;                      # simple session affinity
  server keycloak-1:8080 max_fails=3 fail_timeout=10s;
  server keycloak-2:8080 max_fails=3 fail_timeout=10s;
}

server {
  listen 443 ssl http2;
  ssl_protocols TLSv1.2 TLSv1.3;
  location / {
    proxy_pass http://keycloak_http;
    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Port  $server_port;
  }
}

Passive health checks (max_fails=3 fail_timeout=10s) evict a bad node after a few failures. On affinity: ip_hash hashes on the first three octets of the client's IPv4 address (nginx ngx_http_upstream_module), which is coarse — everyone behind one corporate NAT lands on the same node. Cookie-based stickiness on the AUTH_SESSION_ID cookie pins each browser session to the node that owns it; we started with ip_hash for simplicity and moved to AUTH_SESSION_ID affinity as traffic grew.

Control 3 — hardening the now-critical identity tier

Concentrating all authentication behind a cluster makes that cluster the highest-value asset on the network, so it earns defense in depth beyond mere redundancy:

Pin --hostname so external URLs are consistent across nodes and cannot be influenced by a spoofed Host header.
Enable --metrics-enabled=true, scrape with Prometheus, and alert on 5xx, login errors, and session spikes — a login-error spike is a security signal (credential stuffing, brute force), not just an ops one, so lost telemetry is lost detection.
Run managed HA Postgres with DB TLS and tuned pools.
Lock the admin console behind IP/VPN, rotate admin credentials, and require 2FA for admins — the admin console is a full realm-compromise surface and does not belong on the open internet.
On Kubernetes, use a StatefulSet, a headless Service, and NGINX Ingress with correct timeout/header annotations.

Verifying the control, not just shipping it

An HA design is a hypothesis until failure is demonstrated on demand — assuming HA works because the diagram says so is how you end up with a more expensive single point of failure. Before pointing production DNS at the cluster we ran deliberate failure drills: with a colleague mid-session in the admin console, I killed keycloak-1 outright; Nginx's passive health check marked it down within fail_timeout, traffic shifted to keycloak-2, and because Infinispan had replicated the session, the colleague never saw a login prompt. We repeated it in reverse, then killed a node during an active token refresh. Only after all three drills passed did we cut over. Two lessons from that exercise are worth keeping: session affinity and replication are different controls (without stickiness, users bounced between nodes mid-login and hit failures in the brief window before a session propagated; replication is the safety net if a node actually dies — you want both), and JGroups is fussy about networking (our first attempt only half-formed a cluster because a firewall rule silently dropped discovery traffic, so confirm the cluster actually forms rather than assuming it did).

Residual risk / what we're still watching

Clustering bounded the blast radius of a node loss, but it introduced new failure modes and a more valuable target, and naming them is the point.

The cluster is now the highest-value asset. Redundancy protects availability, but it does not protect against a compromise of the identity tier itself — a stolen admin credential or a Keycloak CVE is now a whole-org event. That is why the admin console is VPN-gated with mandatory 2FA, and why we track Keycloak advisories directly rather than waiting for a bump.
Cluster-formation and discovery are silent failure modes. JGroups needs clean ports and coherent discovery, and inter-node latency degrades replication. A cluster that silently fails to form looks healthy until a failover drops sessions. We monitor cluster membership as a first-class signal.
Forwarded-header trust boundary. Because Keycloak now trusts X-Forwarded-* from the proxy, the proxy must be the only source of those headers; a path that lets a client inject them would let it influence issuer/redirect behavior. We treat the proxy configuration as security-sensitive and reviewed.
Session-affinity coarseness under NAT. ip_hash concentrates large NATed populations onto one node — an availability and load hot-spot; AUTH_SESSION_ID affinity mitigates it, and we watch per-node session distribution.
Multi-AZ is still open. Two nodes in one availability zone still share an AZ-failure fate. We are finishing the migration to KUBE_PING on Kubernetes and spreading nodes multi-AZ so a zone loss is as boring as a node restart.

The morning after go-live we patched a node in the middle of the workday and nobody noticed — which was the entire point. The residual work is to keep proving that property holds as the topology changes, and to keep treating the identity tier as the critical asset a whole organization now depends on.

Sources & further reading

Keycloak — Configuring a reverse proxy — the proxy-headers values and hostname flags, straight from the source.
Keycloak — Configuring distributed caches — how Infinispan replicates sessions and how nodes discover each other.
NGINX — HTTP Load Balancing — upstreams, passive health checks, and session persistence.
nginx ngx_http_upstream_module — the ip_hash directive and its IPv4 hashing behavior.
A Keycloak + Nginx clustering walkthrough — a useful third-party account of the Docker lab, production nginx.conf, and hardening checklist this analysis draws on.

Deleting Hand-Rolled Auth From a Next.js App: A Keycloak Nextjs Threat Model and NextAuth Cutover

Gaberial Sofie — Sat, 08 Aug 2026 17:16:55 +0000

The exposure

Every codebase has one module nobody wants their name on. Ours was lib/auth.ts — a homegrown JWT system in our Next.js app that had been "good enough for the MVP" three years earlier. It signed its own tokens, hand-rolled refresh-token rotation, and stored sessions in a way two engineers actively disagreed about. Exactly one person understood the rotation logic, and when he took vacation we froze all auth-adjacent work out of fear.

That is a security exposure before it is an engineering one. Hand-rolled authentication concentrates several of the highest-consequence failure modes in software into code that is rarely reviewed and understood by one person. A keycloak nextjs integration was attractive not because Keycloak is fashionable but because delegating authentication moves password storage, MFA, token signing, and refresh — the parts where a subtle bug is a breach, not a bug — out of our codebase entirely. The trigger was concrete: a security review flagged four separate issues in that one file. Rather than patch a system we did not trust, we ripped it out and stood on an identity provider we already ran, leaning on the official Auth.js Keycloak provider and Next.js App Router route handlers. The result was about forty lines.

Threat model

Being specific about what a hand-rolled auth module gets you exposed to is what justified deleting it rather than patching it.

Self-implemented token cryptography. A module that signs and verifies its own JWTs is one weak-algorithm choice or one missing signature check away from token forgery. This is the class of defect where "it works" and "it is secure" look identical from the outside.
Refresh-token rotation errors. Refresh logic understood by a single engineer is fragile in the worst place: a bug can either lock users out (availability) or fail to invalidate a stolen refresh token (a persistent-access foothold). The four review findings clustered here.
Ambiguous session handling. Two engineers disagreeing on how sessions are stored is a session-fixation and session-invalidation risk waiting to surface — you cannot invalidate a session cleanly if you cannot agree on where it lives.
Knowledge concentration as a security risk. Auth that only one person understands cannot be safely reviewed, patched, or incident-handled. Bus factor one on the credential path is itself a finding.

The decision: reduce our attack surface by owning as little authentication code as possible, and let Keycloak — whose entire job is to get password storage, MFA, and token signing right — own the security-critical parts.

Controls we added

Control 1 — delegate authentication, own almost none of it

NextAuth.js (branded Auth.js in v5; the next-auth package and Keycloak provider are the same) ships a first-party Keycloak provider. You declare a provider, point it at your realm's issuer URL, and it handles the Authorization Code flow, callbacks, and session cookies. The security-relevant property is subtraction: the token exchange, password handling, and MFA all move behind Keycloak, so the surface where our own code can be wrong shrinks to configuration. I cross-referenced the official provider docs against a thorough third-party walkthrough of the NextAuth-Keycloak wiring → while doing this.

One trap that had burned us: do not mix Pages Router examples into an App Router app. Most stale tutorials use NextApiRequest and pages/api/auth; in App Router you export GET/POST from a route.ts, per the Next.js file convention. Getting that straight up front is a correctness control — a subtly wrong handler on the auth path is not a place to be copy-pasting.

The whole integration lives at app/api/auth/[...nextauth]/route.ts:

import NextAuth from 'next-auth'
import KeycloakProvider from 'next-auth/providers/keycloak'

const handler = NextAuth({
  providers: [
    KeycloakProvider({
      clientId: process.env.KEYCLOAK_CLIENT_ID!,
      clientSecret: process.env.KEYCLOAK_CLIENT_SECRET || '',
      issuer: `${process.env.KEYCLOAK_URL}/realms/${process.env.KEYCLOAK_REALM}`,
    }),
  ],
  secret: process.env.NEXTAUTH_SECRET,
  session: { strategy: 'jwt' },
})

export { handler as GET, handler as POST }

The issuer must include the realm (/realms/<name>) — the provider docs are blunt about it, and a wrong issuer path is a common and security-relevant misconfiguration because it changes which authority you are actually trusting.

Control 2 — client configuration as an enforced allow-list

In Keycloak we created a nextjs-client (confidential, with a client secret) and treated two fields as controls, not preferences:

Valid Redirect URIs: http://localhost:3000/api/auth/callback/*, matched character for character including protocol and trailing slash. A loose or wildcard redirect is an open-redirect and token-exfiltration vector, so it is scoped to exactly where NextAuth listens.
Access type: confidential for the server-side surface, holding a secret; public clients get PKCE instead. Getting this classification wrong is a genuine weakness, not a style choice.

Control 3 — hardening we adopted deliberately

PKCE for public clients, always — RFC 7636 exists precisely to stop authorization-code interception on clients that cannot hold a secret; modern Auth.js enables it for OIDC providers by default.
KEYCLOAK_CLIENT_SECRET and NEXTAUTH_SECRET generated per environment (openssl rand -base64 32), pushed through the secrets pipeline, never committed, and rotated on a cadence.
HTTPS end-to-end with secure cookies in production.
Explicit session strategy (jwt, chosen on purpose) rather than silently mixing jwt and database sessions, which is a classic footgun with real invalidation consequences.

Control 4 — a cutover that never locked anyone out

Swapping authentication on a live product is a high-risk change, so we staged it to stay defensible throughout. We shipped the Keycloak route behind a feature flag and ran it in parallel with the old system for a week, dogfooding internally. We migrated user identities into Keycloak ahead of time and mapped them by email so nobody had to re-register — a re-registration flow is itself a phishing pretext we did not want to create. We kept the old lib/auth.ts in the tree but unreferenced for one release as a rollback escape hatch, then deleted it once the dashboards stayed green — a dormant second auth system is standing attack surface, so it did not linger. The cutover was a single flag toggle at low traffic, and the rollback plan was toggling it back.

Verifying the control, not just shipping it

The failure that validated the approach was a redirect loop that appeared the instant we deployed to staging: no useful error. The cause was a mismatched redirect URI down to a trailing slash the Keycloak Valid Redirect URI did not have. That is worth internalizing as a security habit, not just a debugging tip — redirect-URI matching is the control that keeps authorization codes from going to the wrong place, so when it is strict enough to break on a trailing slash it is also strict enough to reject an attacker's callback. We aligned it and added the check to our deploy runbook. Because authentication now lives behind Keycloak, "does login work" is answerable by driving the flow against the provider rather than reading our own crypto.

Residual risk / what we're still watching

Deleting the hand-rolled module removed a class of self-inflicted crypto and rotation defects, but it relocated risk into configuration and into a dependency.

Access-token refresh is the next sensitive control. Keycloak's default access-token lifespan is five minutes, so refresh matters sooner than teams expect. Refreshing OAuth tokens inside the jwt/session callbacks — the Auth.js Refresh Token Rotation guide is the canonical approach — must handle a failed refresh by forcing re-authentication rather than silently serving a stale identity. This is the piece we are implementing most carefully.
Back-channel logout is a real gap until we close it. Today, signing out of Keycloak does not yet invalidate the Next.js session everywhere. Until back-channel logout is wired, a revoked or logged-out user can retain a valid app session until it expires — a bounded but genuine residual window we are actively tracking.
Redirect-URI and secret drift. A later loosening of the redirect allow-list, or a leaked/committed NEXTAUTH_SECRET or client secret, quietly reintroduces exposure. Both live in reviewed configuration and rotation.
Keycloak is now a dependency on the login path. We traded code we owned and distrusted for a system we must keep patched, monitored, and available. That is the right trade, but it makes Keycloak's own security posture part of ours, and we track its advisories.
Role-based access mapping. As we map Keycloak realm roles into the jwt/session callbacks for authorization, each mapping is a new place a claim can be misread into wrong access, so those callbacks get the same scrutiny the token logic does.

The net effect is that the scariest authentication code we owned is gone, the parts that are hard to get right now live in a system built to get them right, and the remaining risks are named, bounded, and on a list — chiefly refresh handling and back-channel logout — rather than concentrated in one engineer's head.

Sources & further reading

Auth.js — Keycloak provider — the official provider config, issuer/realm rules, and client-authentication setup
Next.js — Route Handlers (route.js) — App Router file convention for GET/POST handlers
Auth.js — Refresh Token Rotation — persisting and refreshing OAuth tokens in the jwt/session callbacks
PKCE for OAuth 2.0 (RFC 7636) — why public clients need it
Keycloak — Securing applications and services with OpenID Connect — OIDC endpoint and integration reference
An end-to-end NextAuth + Keycloak writeup — a useful third-party account of the full client setup, env vars, the App Router route handler, and the hardening checklist.

Collapsing 60 Trust Boundaries Into One: A Keycloak Identity Federation Threat Model and 90-Day Rollout

Gaberial Sofie — Fri, 07 Aug 2026 08:53:16 +0000

The exposure

Through two acquisitions we had accumulated three separate Active Directories, a bolt-on SSO product, and roughly sixty applications each with its own login page. From a security standpoint the helpdesk ticket volume was a symptom; the disease was the identity architecture underneath it. Every application stored its own credentials, every user carried a dozen of them, and — because remembering a dozen is impossible — people reused the same weak password across systems that had no shared trust boundary.

That is a large, badly-shaped attack surface. Credential reuse means the blast radius of one phished password is not one application but every application that shares it. Offboarding was worse: with sixty independent credential stores, revoking a departed employee's access was a per-app, hours-long chore, which means at any given moment there were almost certainly valid credentials belonging to people who should have had none. MFA coverage was patchy and per-app, so the strength of authentication varied by whichever team had last touched a login page. This is keycloak identity federation territory: the fix is not more helpdesk agents, it is collapsing sixty trust boundaries into one you can actually reason about. Keycloak's Server Administration Guide is the reference for the mechanics; the risk framing and sequencing we had to work out ourselves.

Threat model

Naming what we were defending against is what justified the rollout order.

Credential-reuse blast radius. A single reused password turns one compromise into many. As long as each app authenticates independently, there is no way to bound how far a stolen credential reaches.
Orphaned access after offboarding. Sixty credential stores mean revocation is slow and error-prone. Every store you forget is a standing account an attacker or a disgruntled leaver can use. The control objective is single-point revocation.
Inconsistent authentication strength. Without a central policy, MFA and lockout behavior differ per app. The weakest login page sets the real security posture, because that is where an attacker will aim.
Big-bang migration risk. Federating sixty systems simultaneously with no staging is itself a threat — to availability. The recurring failure story in this space is exactly that: a simultaneous cutover with no fallback that earns a multi-day outage and an emergency rollback. Availability is a security property, so the rollout had to be defensible at every step.

The design decision that follows: put a broker in the middle that becomes the single authority for authentication, so credential reuse loses its reach, revocation becomes one action, and MFA becomes a policy rather than sixty separate implementations — and get there in phases so we never bet the whole estate on one cutover.

Controls we added

Control 1 — a broker so no application stores a password again

Federation does not share credentials — that misconception is itself dangerous, because engineers who believe it will build the wrong thing. Federation shares cryptographically signed assertions that someone was authenticated. Keycloak sits between the identity provider (our Active Directory, or Google — the systems that vouch for identity) and the service providers (the sixty downstream apps), acting as a broker: what the docs call integrating identity providers. It federates our existing AD as the source of truth over LDAP, speaks OIDC and SAML to every app, and means no downstream application holds a credential.

The security payoff is single-point revocation: disable an account once at the identity provider and access to all sixty apps evaporates. That property — not the convenience — was the security team's reason to approve the project. For the sequencing and the console mechanics I kept a thorough third-party walkthrough of a phased federation rollout → open alongside the official docs.

Control 2 — a phased rollout that always has a fallback

We ran it over 90 days, and the phasing was a risk control, not a project-management nicety.

Days 1-30, foundation. Stand up dev, staging, and a production cluster. Connect the primary AD as LDAP user federation so disabled accounts and group changes propagate automatically. Then pick three deliberately low-stakes pilot apps — an internal wiki, a dashboard, a staging tool — nothing that pages anyone at 2am if it breaks. For external IdPs, the setting that trips everyone is First Login Flow → first broker login, which determines whether a first-seen federated user links to an existing account or auto-provisions; a wrong choice here is an account-takeover-shaped hazard, so we set it deliberately.
Days 31-60, expansion. Add Google as an external IdP for contractors, turn on MFA policies org-wide, integrate the next dozen apps. Monitoring became non-negotiable here: alerting on failed authentications, token-renewal failures, and session timeouts, because incidents live at those boundaries, not the happy path. Delayed detection is reduced detection.
Days 61-90, production. Roll out to the rest, harden policy, and — critically — stand up and test backup and disaster recovery for the cluster before decommissioning the old SSO product. A broker with no tested DR is a single point of total authentication failure.

Control 3 — client configuration as an enforced allow-list

For OIDC apps, each got a client (the OIDC and SAML client management guide is the canonical field reference). Two settings carry security weight: Valid Redirect URIs configured exactly — protocol and trailing slash matching character for character, no wildcards, because a loose redirect URI is an open-redirect and token-exfiltration vector; and Authorization Code flow everywhere with PKCE required for public clients. PKCE (RFC 7636) exists specifically to stop authorization-code interception on clients that cannot hold a secret, so we banned the Implicit flow outright.

For SAML apps the work is metadata exchange plus attribute mapping via per-client protocol mappers. Attribute mapping is where SAML silently goes wrong: if the app expects email and Keycloak sends emailAddress, login "succeeds" but the user is provisioned as a stranger — which, depending on the app, can be an authorization failure that grants the wrong access. We standardized on the HTTP-POST binding and reviewed every mapper.

Verifying the controls, not just shipping them

Two verification habits earned their keep. First, clock synchronization: our second SAML integration passed staging and failed intermittently in production with signature errors. SAML assertions carry NotBefore / NotOnOrAfter conditions, so IdP and SP clocks must agree within a tolerance — as the clock-skew writeup explains, a few seconds of drift rejects a valid assertion, and allowedClockSkew only widens the window rather than fixing the cause. One node had drifted; we enforced NTP across every host in the auth path. Check clocks before certs.

Second, negative testing of the controls themselves. We tested redirect URIs with deliberately invalid values, not just the happy path, and caught a client that would have accepted a wildcard redirect — a textbook open-redirect and token-leak risk — precisely because the test matrix included a URL we expected Keycloak to reject. A control you never test against a hostile input is security theater; a five-minute negative test replaced a future incident review.

Residual risk / what we're still watching

Collapsing sixty trust boundaries into one removes a whole class of reuse and orphaned-access risk, but it concentrates risk rather than deleting it.

The broker is now a high-value single point. A Keycloak compromise, a signing-key leak, or a cluster outage is an authentication event for the entire estate. That trade is only correct if the concentrated point is hardened, patched, monitored, and backed by tested DR — which is why DR testing gated the decommission of the old system, and why we track Keycloak advisories directly.
Clock and NTP are now a security dependency. Time drift on any host in the auth path degrades SAML validation. NTP health is monitored as a security signal, not just an ops metric.
SAML attribute-mapping drift. A future mapper change that misroutes email or group claims can silently grant wrong access. Mappers live in reviewed configuration, and we watch for provisioning that produces unexpected identities.
First-broker-login and redirect-URI loosening. Both are one careless edit away from reintroducing account-takeover or open-redirect exposure. They stay in reviewed config, not ad-hoc changes.
Partner federation expands the trust perimeter. We are extending federation to a couple of partner organizations so their staff reach shared portals via IdP-to-IdP trust — which means trusting another organization's authentication. Each new trusted IdP is new attack surface and gets onboarded with scrutiny over what claims we accept and how far that trust reaches, not rubber-stamped.

The net effect is that a stolen credential no longer roams sixty systems, offboarding is a single revocation, and MFA is a policy rather than sixty implementations — provided we keep treating the broker as the critical asset it has now become.

Sources & further reading

Keycloak Server Administration Guide — LDAP/AD user federation, identity brokering, first login flow
Keycloak — Integrating identity providers (Server Admin Guide, Ch. 9)
Keycloak — Managing OpenID Connect and SAML Clients, incl. protocol mappers (Ch. 12)
PKCE for OAuth 2.0 — why public clients need it (oauth.net / RFC 7636)
SAML Clock Skew in Keycloak — allowedClockSkew, NotBefore/NotOnOrAfter, and NTP
A phase-by-phase writeup of a 90-day federation rollout — a useful third-party account of the IdP/SP model and the config details that live outside the primary docs.