DEV Community: Gaberial Sofie

Your Ingress does nothing until you have an Ingress controller (and 3 more local-k8s networking traps)

Gaberial Sofie — Sun, 30 Aug 2026 23:43:09 +0000

The single most common local-Kubernetes frustration is "I applied an Ingress and nothing responds." The docs are blunt about why: "You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect."

Takeaways

An Ingress manifest is just rules. It needs a running controller (Traefik, NGINX...) that reads them and actually accepts traffic. Which controller serves it is set by ingressClassName.
k3d ships Traefik out of the box (k3d is built on k3s), plus a built-in ServiceLB (Klipper) so LoadBalancer services don't hang in pending. But Traefik listens inside the cluster — to reach it from the host you must forward a port at cluster-creation time:

  k3d cluster create dev --api-port 6550 -p "8081:80@loadbalancer" --agents 2

You cannot add this port map to a running cluster. Forgot it? Recreate. This is the #2 pain after "forgot the controller."

DNS inside the cluster is by name, not IP (CoreDNS). FQDN pattern: <service>.<namespace>.svc.cluster.local. Short names resolve only within the same namespace via search domains — cross-namespace you need postgres.myapp or the full FQDN.
*.localhost is DNS behaving to spec, not k3d magic. RFC 6761 reserves .localhost for loopback. Chrome and Firefox resolve any *.localhost to 127.0.0.1 with zero config — but Safari and non-browser tools (curl, HTTP clients) treat it as an ordinary domain and hit the OS resolver, giving could not resolve host. Fix: add it to /etc/hosts.

The full traffic chain to memorize: curl -> host port 8081 -> Traefik (port 80) -> Service myapp -> Pod.

Full article: https://dorokhovich.com/blog/local-k8s-networking-and-ingress?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-networking-and-ingress

Local Kubernetes Dev — Part 11: Networking — reaching your service and Ingress

Gaberial Sofie — Sat, 29 Aug 2026 16:47:12 +0000

"I applied an Ingress — and it doesn't respond. Why?"

A local Kubernetes classic: you write an Ingress manifest, kubectl apply, open the browser — and silence. The thing is, an Ingress by itself is just rules. Until there's an Ingress controller in the cluster (Traefik, NGINX) that reads those rules and accepts traffic, nothing works. Straight from the docs: "Only creating an Ingress resource has no effect."

And the second most popular k3d pain: the loadbalancer port mapping is set ONLY at cluster creation. Forgot -p "8081:80@loadbalancer"? You'll have to recreate the cluster.

In the new article (part 11) I break networking down layer by layer:
• how Pods find each other by name via CoreDNS (and why the short name only works inside its own namespace);
• three ways to reach in from outside: port-forward, NodePort, Ingress — and when to use which;
• Traefik out of the box in k3d and the whole chain: curl → port 8081 → Traefik → Service → Pod;
• why http://myapp.localhost opens in Chrome but curl says "could not resolve host" (spoiler: RFC 6761, and it's not a bug).

Read it and stop fighting your cluster's networking: https://dorokhovich.com/blog/local-k8s-networking-and-ingress?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-networking-and-ingress

Running Postgres, Redis and RabbitMQ inside your local k3d cluster (with the 2025 Bitnami trap)

Gaberial Sofie — Mon, 24 Aug 2026 15:50:55 +0000

Chapter 9 of a local-Kubernetes series: the stateful-dependencies chapter the earlier ones deliberately deferred. Bring up your backing services in the cluster so local structurally matches prod — and avoid a 2025 supply-chain landmine.

Key takeaways

Why in-cluster, not compose-on-the-side: one world, not two. In-cluster Postgres/Redis/RabbitMQ gets you unified DNS/service discovery (app finds the DB at postgres, not localhost:5432), shared Secrets/ConfigMaps, and you catch RBAC/limits/probe problems before rollout. (Fair: compose is a legit choice if the service isn't going to k8s.)
The Bitnami warning without which this chapter would be harmful: since 2025-09-29 most public Bitnami OCI charts/images moved behind a Broadcom subscription; the rest are bitnamilegacy (unsupported, no security patches). Don't blindly recommend bitnami/* in 2026. Use Chainguard's drop-in charts, official operators, or a deliberately-pinned legacy tag (dev only).
The most predictable dev path is a raw 3-object manifest: Deployment (official postgres:17) + Service (stable DNS postgres) + PVC — no chart repos, no subscription dependency. Includes a pg_isready readiness probe.
The big storage surprise: in k3d, data is ephemeral by default — local-path-provisioner writes inside the node container, so k3d cluster delete wipes it. To persist, map a host dir at create: --volume $HOME/k3d-storage:/var/lib/rancher/k3s/storage@all.
Wire into Tilt: feed the manifest to k8s_yaml, order startup with resource_deps (myapp waits for postgres's readiness), port_forwards=['5432:5432'] to reach it from your IDE.
Migrations: a separate Job (+ initContainer wait), ordered postgres → migrate → seed → myapp. Migrations in the app's own initContainer is an anti-pattern (races across replicas, probe kills). Note Tilt's built-in helm() skips hooks — use helm_resource for anything with init hooks.
Mock vs run: OSS deps (Postgres/Redis/RabbitMQ) → run as-is; proprietary managed (S3/SQS/DynamoDB) → emulate with LocalStack, but it's not 1:1 (IAM, consistency, edge cases).

Full article: https://dorokhovich.com/blog/local-k8s-dependencies-databases-queues?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-dependencies-databases-queues

Local Kubernetes Dev — Part 9: Dependencies — databases, queues, caches

Gaberial Sofie — Fri, 21 Aug 2026 08:42:31 +0000

OK, your service runs in the cluster locally — but what about the database, the queue, the cache? This is the chapter the previous ones deliberately put off.

Part nine of the series is about stateful dependencies. We stand up PostgreSQL, Redis, and RabbitMQ INSIDE the local cluster (Helm or a "raw" manifest), wire them into Tilt, and sort out PersistentVolumes, migrations, and seeding on startup. The point is parity: if the database lives in the same cluster with the same charts that ship to staging, you get unified DNS and service discovery for free (the app finds the DB by the name postgres, not localhost:5432), shared Secrets/ConfigMaps, and you catch manifest problems before prod — not on a Friday evening.

An honest warning, without which this chapter would do more harm than good: as of 2025-09-29 the public Bitnami images moved behind a paid Broadcom subscription, so blindly recommending bitnami/* in 2026 is no longer an option. What to do instead — Chainguard drop-in charts, official operators, or a deliberate pin to a legacy tag. For dev, the most predictable path is a "raw" manifest of three objects (Deployment + Service + PVC) with the official postgres:17 image.

Plus: the biggest surprise — in k3d data is ephemeral by default (fixed with --volume $HOME/...:/var/lib/rancher/k3s/storage@all), migrations as a separate Job (not in the app's initContainer — that's an antipattern), and when to mock a managed service (RDS/SQS) with LocalStack and when not to. https://dorokhovich.com/blog/local-k8s-dependencies-databases-queues?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-dependencies-databases-queues

One Realm for Every App and Tenant Was a Time Bomb: A Keycloak Realms Clients Roles Threat Model

Gaberial Sofie — Wed, 12 Aug 2026 12:32:06 +0000

The exposure

The first Keycloak rollout worked on the first try, which is the dangerous part. Under the hood it was one realm for every app and every tenant, wildcard redirect URIs, a client secret embedded in a single-page app, and roles assigned directly to individual users. None of it threw an error, and that is precisely why it festered for six months until we had three apps, two customer tenants, and a growing suspicion the whole thing was held together with tape.

Understanding keycloak realms clients roles correctly is a security exercise, not a modeling preference, because each of those four sins maps to a concrete class of vulnerability. A working login flow and a safe one look identical from the outside, and the console will happily let you build the former long before you have the latter. Before changing anything it was worth cataloguing exactly what each shortcut exposed us to. For the concept model and the remediation I worked from a thorough third-party guide to Keycloak's core concepts → alongside the official administration guide.

Threat model

No tenant isolation — shared blast radius. One realm carrying all apps and both tenants means a misconfiguration, a leaked admin credential, or a bad role change in one tenant's context has blast radius across every tenant. There is no boundary to contain a mistake, which is the opposite of what a realm is for.
Wildcard redirect URIs — open redirect and token exfiltration. http://localhost:3000/* was convenient in dev and quietly rode into prod. A wildcard redirect lets an attacker steer the authorization-code callback to a URL they control, turning a valid login into a token-leak primitive.
Client secret in a SPA — a secret that is not secret. The single-page app was registered as if it could hold a secret; the secret shipped in the browser bundle, readable by anyone. A confidential client whose secret is public is a confidential client in name only, and it undermines the entire client-authentication assumption.
Per-user role assignment — unauditable and unscalable. Assigning roles to individuals by hand means "who can do X?" is answerable only by enumerating every user, and every onboarding is a manual chance to over-grant. Least privilege is not enforceable if entitlements are scattered.

The organizing principle for the fix is the three-word model: realms isolate identities, clients integrate apps, roles authorize actions. Every defect above was a violation of one of those three boundaries, so the remediation is to restore each boundary deliberately.

Controls we added

Control 1 — realms as isolation boundaries

A realm is a self-contained boundary with its own users, groups, clients, and roles — Keycloak's definition is that "a realm manages a set of users, credentials, roles, and groups" (Server Administration Guide). We split by tenant, and at minimum by environment (dev/staging/prod), so a misconfiguration in one is contained to one. Users no longer span realms automatically, and that is the point: if identity genuinely needs to be shared, identity brokering or federation is the explicit, reviewable tool for it, rather than an implicit shared bucket.

There is a ceiling to this control worth stating honestly, because over-applying it creates its own risk. Realm-per-tenant is right for a handful of tenants but stops scaling around a couple dozen — past that you are operating an IAM platform instead of shipping a product, and a sprawl of realms becomes its own misconfiguration surface. Keycloak's Organizations feature (preview in 25, GA in 26 — announcement) provides first-class multi-tenancy inside a single realm for that case. The rule: separate realms to isolate genuinely different security domains (different admins, or dev/staging/prod); Organizations to partition many customers who share one app's trust boundary.

Control 2 — clients with an honest access type and PKCE

A client's access type is a security control, not a convenience setting:

public — SPAs and native apps; no secret, because anything shipped to a browser or device is readable.
confidential — server-side apps that can actually keep a secret.
bearer-only — APIs that verify tokens and never start a login flow.

The SPA became a public client, and the leaked secret was retired rather than rotated-in-place, because the fix is to stop pretending a browser bundle is private, not to ship a fresh secret into the same exposure. What replaces the secret is PKCE (S256): designed specifically for OAuth public clients that cannot keep a secret (RFC 7636), it stops an intercepted authorization code from being redeemed by an attacker (Auth0's PKCE explainer). Turning it on is non-negotiable for browser and native clients. Alongside it, the wildcards died: valid redirect URIs and web origins are now exact entries, no *, to close the open-redirect vector.

Control 3 — roles and groups for least privilege by construction

Roles model permissions, and we now use two flavors deliberately: realm roles for org-wide levels (admin, editor, viewer) and client roles for app-specific permissions (billing:read). Composite roles let one Admin assignment imply a bundle — though a composite is also where over-grant hides, so each composite is reviewed for exactly what it expands to. The biggest change was mapping roles to groups, not individuals: assigning to users does not scale and is not auditable; group-based mapping is both. A new hire joins a group and inherits precisely the intended roles, which makes least privilege the default path rather than a discipline someone has to remember.

Control 4 — authorize from the token, not from identity

The refactor only pays off if the application uses the structure. The early mistake was scattering permission checks keyed on usernames — a hard-coded allow-list is unauditable and drifts silently. The clean version reads roles straight from the token: in OIDC, Keycloak places realm roles under realm_access.roles and client roles under resource_access[client].roles. An authorization check becomes "does this token carry the editor role?" rather than "is this user in my admin list?" Because Admin is modeled as a composite, one role in the token expands to everything it should, consistently. Permissions become a configuration concern in Keycloak, not a deployment concern in the codebase — a new hire is added to a group, the roles land in their token, and the app authorizes them with no code change or redeploy.

Verifying the controls, not just shipping them

The rebuild started from a minimal, reviewable baseline rather than clicking around the console:

docker run -p 8080:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:25.0.2 start-dev

Then a five-step skeleton: a dedicated realm per security domain, each app as its own client with exact redirect URIs and the correct access type, realm roles like Admin and User, and assignment via groups. The verification that mattered most was that the controls are now demonstrable: "who can write billing?" used to mean grepping code and cross-referencing a user table; now it is "which groups carry a role that composes billing:write?", answerable in the admin console in seconds. An entitlement you can query is an entitlement you can audit, and negative-testing a redirect URI with a deliberately invalid value confirms the allow-list actually rejects what it should.

Residual risk / what we're still watching

Restoring the three boundaries removed the acute exposures, but each control carries its own residual risk.

Realm sprawl vs. Organizations. Separate realms isolate cleanly at small scale but become an operational and misconfiguration burden past a couple dozen tenants. We are watching the tenant count against the point where Organizations-in-one-realm is the safer trade, since the wrong choice in either direction reintroduces risk — sprawl on one side, a weaker isolation boundary on the other.
PKCE and redirect-URI drift. The public-client posture depends on PKCE staying enforced and redirect URIs staying exact. A future convenience edit that re-adds a wildcard, or a client that quietly disables PKCE, silently reopens the token-leak vector. Both live in reviewed configuration.
Composite-role over-grant. Composites are ergonomic but concentrate authority; a role added to a widely-assigned composite grants it to everyone downstream at once. We review composite membership as an entitlement change, not a convenience.
Group hygiene. Group-based assignment scales, but it moves the risk to group membership — a stale or over-broad group is now the thing to audit. We periodically reconcile group membership against intended access.
The leaked secret is retired but assume-compromised. Because the old SPA secret was public for months, we treat it as burned permanently and monitor for any client still configured to accept it.

The lesson that stuck is that the console lets you build something that works long before it is something that is safe, and the three-word model — isolate, integrate, authorize — is both the setup mantra and the operating model that keeps the system explainable, and auditable, a year later.

Sources & further reading

Keycloak — Server Administration Guide — realms, clients, roles, groups, and composite roles.
Keycloak — Organizations announcement — multi-tenancy inside a single realm (preview in 25, GA in 26).
RFC 7636 — Proof Key for Code Exchange — why public clients use PKCE instead of a secret.
Auth0 — Authorization Code Flow with PKCE — a clear walkthrough of the SPA/native login flow.
A Keycloak realms/clients/roles walkthrough — a useful third-party account of the pitfalls, the cross-realm FAQ, and a runnable five-step example.

A Typosquatted Gem Almost Shipped, Caught by Luck: A Ruby Supply Chain Security Playbook

Gaberial Sofie — Tue, 11 Aug 2026 19:26:01 +0000

The exposure

A teammate opened a PR adding a small utility gem. The name looked right at a glance — one transposed character from a popular package. It passed our tests. On a normal Friday it would have merged. Someone caught the typo in review by luck, not process, and the gem turned out to be a typosquat whose extconf.rb ran code at install time that had no business running.

"By luck, not process" is the phrase that reframes this from an anecdote into a security finding. Ruby supply chain security is the discipline of not depending on luck — or on someone else's controls — for the integrity of code you ship. We were trusting gem install blindly and leaning entirely on RubyGems.org to catch everything upstream. It usually does, but "usually" is a probability, not a guarantee, and a plan that works right up until it doesn't is not a control. RubyGems.org's own account of how it protects the ecosystem puts a number on it: its automated tooling catches roughly 70-80% of malicious packages before anyone reports them. That upstream layer is real and valuable, but the residual 20-30% is precisely the population your build has to assume it will meet. For the fuller hardening notes I worked from a thorough third-party writeup of a Ruby dependency-security playbook → alongside the registry's post.

Threat model

Typosquatting and look-alike names. A one-character-off gem name is a social-engineering primitive aimed at review and autocomplete. It is caught, if at all, by attention — an unreliable control at 2pm on a Friday.
Arbitrary code execution at install time. extconf.rb, native extension builds, and install hooks run on developer laptops and CI runners with whatever privileges those hosts hold. A malicious gem does not need to reach production to do damage; installing it is the exploit.
Lockfile injection and source substitution. A version number can be re-pointed at different bytes upstream, and a machine-generated Gemfile.lock that nobody reads during review can quietly point a dependency at a different set of bytes than the manifest implies. Mixed gem sources open a substitution path where a gem is pulled from an unexpected origin.
The post-disclosure window. A gem that is clean today gets a CVE tomorrow while sitting in your lockfile. Detection that only runs when code changes never revisits already-shipped dependencies.
Over-reliance on upstream controls. Delegating your production security entirely to the registry's 70-80% means the ecosystem's residual miss rate is your residual risk, unmediated.

The organizing principle is defense in depth: the registry is the first line, and the build must be a deliberate second line behind it, because malicious packages are not rare events — they are background noise of the modern software supply chain.

Controls we added

Control 1 — shrink the attack surface

Pin versions and platforms in Gemfile.lock and commit it. No floating ranges into production.
No git dependencies without a commit SHA. A branch can be rewritten; a SHA cannot.
Narrow ranges (~>) for critical gems, and prune ruthlessly — every unused and transitive gem is surface area, reviewed on a quarterly dependency-hygiene day.

Control 2 — checksum verification, the highest-leverage single addition

Pinning a version number is necessary but not sufficient, because the number can be re-pointed at different bytes. Bundler 2.6 makes checksum verification first-class (--add-checksums is a documented bundle lock flag):

bundle lock --add-checksums

This writes a CHECKSUMS section into Gemfile.lock, and from then on Bundler refuses to install a gem whose contents do not match the exact bytes you locked. It closes the nastiest gap in a pin-the-version-only strategy: a checksum cannot be quietly swapped the way a version can be re-pointed. If you adopt one new control this quarter, make it this one. Alongside it, a few Bundler settings harden source handling — disable_multisource true (the underrated one, blocking the source-substitution class), cache_all true, and clean 'true'.

Control 3 — make the human review a process, not a hero act

The near-miss was caught by a person, so we turned the person into a checklist. Adding or bumping a gem now requires a short "why this gem?" note and a CHANGELOG link in the PR, documented in the PR template where the work happens rather than a wiki nobody reads. Dependabot and GitHub alerts are enabled so advisories are not manually watched. The cultural shift — that a dependency is code we are adopting, not a freebie we are grabbing — mattered more than any single tool.

Control 4 — gate it in CI

A checklist people can forget is a suggestion; a CI gate is a rule. Two independent scanners catch different things, and layering them is cheap: bundler-audit (checking Gemfile.lock against the ruby-advisory-db) and Google's OSV-Scanner (fed by osv.dev). It runs on every dependency-touching PR and weekly on a schedule:

on:
  pull_request:
    paths: ['Gemfile', 'Gemfile.lock']
  schedule:
    - cron: '0 6 * * 1'   # weekly on Mondays

jobs:
  bundler-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with: { ruby-version: '3.3', bundler-cache: true }
      - run: gem install bundler-audit
      - run: bundle audit check --update
  osv-scanner:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google/osv-scanner-action@v1
        with: { scan-args: '-L -r .' }

The weekly cron addresses the post-disclosure window directly: a vulnerability disclosed on Wednesday fails the build even if nobody touches the code. For teams that want typosquat detection and SBOM generation on top, a Ruby-specific scanner like GemGuard — pulling from OSV.dev and the ruby-advisory-db — slots in beside bundler-audit rather than replacing it.

Verifying the control, not just shipping it

The fastest way to get a security process ignored is to drop a wall of red CI checks on a team mid-sprint, so we staged it and verified adoption as deliberately as coverage. Week one, the scanners ran in warn-only mode — reporting findings without failing the build. That surfaced the existing backlog of advisories, gave us an honest baseline instead of a nasty surprise, and let us triage and either fix or explicitly accept each item. Only then did we flip the gate to blocking, by which point there was nothing left for it to fail on. The second thing that kept adoption alive was making the checks fast and legible: bundler-audit and OSV both run in well under a minute with a warm cache, and their output points straight at the offending gem and advisory, so a failure has an obvious fix and nobody routes around it. A gate that takes ten minutes or emits inscrutable output gets bypassed, which is how a control becomes theater.

Residual risk / what we're still watching

The playbook meaningfully raised the cost of a supply-chain compromise, but it does not reduce the residual to zero, and the edges are where the honesty has to be.

Scanners detect known vulnerabilities only. A brand-new typosquat or an undisclosed backdoor has no advisory yet and passes clean. That is exactly why the human "why this gem?" review still carries load the automation cannot — the checklist is a control against the unknown-unknown, not redundant with the scanners.
Signatures are not a force field. Gem signing is one control among pinning, checksums, review, and CI gating — treating it as a silver bullet would be a false-confidence failure. We weight it accordingly.
The lockfile diff still needs human eyes. Checksum verification defuses most lockfile-injection risk, but a reviewer who reads only the Gemfile and skims the machine-generated Gemfile.lock can still miss a dependency quietly re-pointed. We review the lockfile diff on any dependency PR.
Patch-bump noise vs. coverage. A blanket manual-review policy drowns the team; letting CI and scans gate patch bumps while humans review major/minor keeps signal high, but it does accept that a malicious patch release is caught by scanners rather than eyes. We watch that trade.
We still depend on the registry and the advisory databases. Our second line assumes RubyGems.org's first line and assumes OSV.dev and the ruby-advisory-db are current; a gap or delay in any of them is a gap in us. Two independent scanners mitigate the advisory dependency, and — because that shared line of defense runs partly on a small group's effort — we treat a recurring sponsorship of the ecosystem as risk management, not charity: the CI gate protects our repo, sponsorship protects the line in front of it.

If a team ships Ruby, the near-miss described here is a matter of when, not if. The residual we keep watching is the undisclosed package and the lockfile change nobody reads closely — the two places where luck, not process, is still doing quiet work.

Sources & further reading

RubyGems.org — How RubyGems.org Protects Our Community's Critical OSS Infrastructure (Aug 25, 2025)
rubysec — bundler-audit and the ruby-advisory-db it checks against
Google — OSV-Scanner, backed by osv.dev
Bundler docs — bundle lock --add-checksums (Bundler 2.6 checksum verification)
wilburhimself — GemGuard: typosquat detection + SBOM for Ruby
A full Ruby supply-chain playbook — a useful third-party account of the Bundler hardening settings and legacy-app rollout FAQ.

Governance Risk Is Supply-Chain Risk: Auditing Every Dependency After the RubyGems Hostile Takeover

Gaberial Sofie — Mon, 10 Aug 2026 08:12:28 +0000

The exposure

In September 2025 the people who built and ran RubyGems and Bundler lost administrative access to the infrastructure they maintained, and several publicly quit. One side called it a hostile takeover; the other called it responsible stewardship. The verdict is not the point. The exposure is: infrastructure serving billions of downloads a month — the infrastructure every bundle install, every deploy, every CI run reaches into — demonstrated that control over the packages your builds depend on can change hands overnight.

Framed as a security problem, the rubygems hostile takeover is a governance-risk event, and governance risk is supply-chain risk that no vulnerability scanner will ever flag. A CVE scanner tells you a dependency has a known flaw; it says nothing about whether that dependency's maintainers just walked out or its publishing rights just moved to someone you did not vet. My company does not contribute to RubyGems — we only consume it, which means we had silently accepted a dependency on both the packages and the people, and the crisis made that dependency visible. The Register covered the control fight as it erupted, Ruby Central later published its own RubyGems Fracture Incident Report owning the execution mistakes, and I worked through a thorough third-party first-person timeline of what one engineer changed in response → while treating our own exposure as a fire drill rather than a hot take.

Threat model

The timeline matters only insofar as it names the threats. Between September 9 and 19, the GitHub org was renamed, maintainers were removed, administrative access to the RubyGems, Bundler, and RubyGems.org teams was revoked — including control of the bundler and rubygems-update gems themselves — and the creator of Bundler announced the team was effectively finished. From that, the concrete threats:

Publishing rights changing hands overnight. Control over the exact packages your builds pull can move to a different party without your consent or knowledge. If your security model assumes the right people hold the keys, it is not a model — it is a hope. Ruby Central's own incident report concedes the foundation "lacked the structural ability to make this change directly," i.e. an informal power structure that a single actor could reconfigure.
Maintainer-exodus as a failure mode. The people who understand a critical dependency's internals can all leave at once. A healthy maintainer team is itself a dependency, and its collapse is a real failure mode — a bus-factor risk at ecosystem scale.
Registry availability during the incident. If RubyGems.org had gone dark or a gem had been yanked mid-fracture, builds that fetch from upstream at deploy time would have failed. Reachability of a third party you do not control sits on your critical path.
Funding-pressure-driven consolidation. Reporting since made clear how much funding pressure sat underneath the fight — The Register relayed developer Joel Drapper's allegation that Shopify pressed for consolidated control of the bundler and rubygems-update gems under threat of pulled funding. Commercial leverage over volunteer infrastructure is a governance attack surface in its own right.
Ecosystem split. The ousted maintainers did not disappear; Martin Emde announced The Gem Cooperative (gem.coop), a community-run mirror, which The Register covered as the maintainers' new home. "Which registry does our bundle install actually hit" went from a question I would have shrugged at to a line item.

The organizing principle: reproducibility is resilience. If a build needs the internet — and a particular governance arrangement — to be calm in order to succeed, it will fail when they are not.

Controls we added

I gave the drill one week and one goal: make sure a governance blowup or a bad actor upstream could neither stop us shipping nor poison our builds.

Control 1 — freeze the ground we stand on

Committed and audited Gemfile.lock so every version and platform is pinned, not floating into production.
Vendored the critical path. bundle cache copies every .gem into vendor/cache and, per the docs, makes future installs "use the gems in the cache in preference to the ones on rubygems.org"; bundle config set cache_all true keeps git and path gems there too. If RubyGems.org went dark or a gem was yanked mid-incident, deploys still work from the cache. This directly retires the registry-availability threat.
Pinned git dependencies to commit SHAs, never bare branch names. A branch can be force-pushed under new ownership; a SHA cannot be quietly swapped — which addresses the "control changed hands" threat at the integrity level.

Control 2 — inventory governance risk, not just CVE risk

Because scanners like bundler-audit (which checks Gemfile.lock against the ruby-advisory-db) are silent on governance, I built a checklist and ran our top dependencies through it:

Who actually controls publishing rights to this package today?
How many maintainers — is it a bus-factor of one?
Has ownership or org name changed recently?
Is there a funded, formal body behind it, or informal goodwill?
If it vanished tomorrow, what is our replacement path?

That last question turns a vague anxiety into a ranked list of things to fix, which is the difference between worrying about governance risk and actually managing it.

Control 3 — reduce the surface

Fewer upstreams means fewer governance fires can burn you. I pruned unused and transitive bloat and added a "why this dependency, and who is behind it?" note to code review for anything new — the same hygiene the ecosystem needs at the macro level, applied to our own repo. Least privilege for dependencies: adopt only what you can justify and account for.

Verifying the controls, not just shipping them

A control the team does not believe in gets removed the first time it is inconvenient, so the verification here was as much organizational as technical. I made the risk concrete with three questions borrowed from the crisis: if the maintainers of our single most critical dependency all quit tomorrow, what breaks and how fast; who can publish a new version of that package right now, and do we trust the process that grants them that power; and if that package got hijacked and shipped a malicious release, would our pipeline catch it before production. For most dependencies the honest first answer was "we don't know," and "we don't know" is the actual risk — so the checklist and the vendoring exist to convert those unknowns into documented, tested answers. The vendoring in particular is verifiable in the most direct way: a build run with upstream network access blocked either succeeds from vendor/cache or it does not, and ours does. Unexpectedly, "we are hardening against a real, recent, widely-reported failure in our supply chain" turned out to be the easiest resilience work I have ever had to justify to leadership.

Residual risk / what we're still watching

The drill meaningfully reduced our exposure to an upstream governance shock, but it did not make us independent of the ecosystem, and the edges deserve honesty.

Vendored gems drift and stale. vendor/cache protects availability, but a frozen cache also stops receiving security updates; a vulnerability disclosed in a vendored gem is now on us to notice and refresh. Vendoring trades an availability risk for a patch-currency obligation, so we pair it with scheduled re-audits rather than treating the cache as done.
Scanners still cannot see governance. The checklist is a manual control against a threat no tool flags, which means it decays unless it is re-run. We re-inventory the top dependencies periodically, because ownership and maintainer health change quietly between reviews.
The ecosystem split is unresolved. With gem.coop now standing alongside RubyGems.org, "which registry are we actually trusting, and what is its provenance" is an ongoing decision, not a settled one. We are watching how it resolves without letting our builds depend on it resolving well.
Transitive dependencies we do not control. We can inventory our direct dependencies' governance, but the long tail of transitive gems carries the same risk with far less visibility. That surface is the hardest residual to close, and it is where we assume the next surprise lives.
Our second line still assumes a first line. Vendoring and pinning protect us from upstream disappearing, not from a malicious release we vendor before anyone notices it is malicious — that gap is covered by scanning and review, which are themselves only as current as their data.

The durable lesson is that you depend on people, not just packages, and that a small group quietly holds up infrastructure the whole ecosystem leans on. When that group fractures, everyone downstream discovers exactly how much they were depending on trust they never examined. What we keep watching is that trust — deliberately, on calm afternoons, rather than during the next incident.

Sources & further reading

The Register — Open source to closed doors: RubyGems control fight erupts (Sep 25, 2025)
Ruby Central — RubyGems Fracture Incident Report
Martin Emde — Announcing The Gem Cooperative (gem.coop) (Oct 5, 2025)
The Register — Kicked from RubyGems, maintainers forge new home at Gem Cooperative (Oct 6, 2025)
Bundler docs — bundle cache: vendoring gems into vendor/cache
rubysec — bundler-audit: scan Gemfile.lock against the ruby-advisory-db
A field-notes dependency-audit checklist from the week of the crisis — a useful third-party account of the same governance-risk drill applied to a real repo.

When One Keycloak Node Is a Single Point of Failure: A Keycloak Cluster Nginx Threat Model for Zero-Downtime Upgrades

Gaberial Sofie — Sun, 09 Aug 2026 08:50:41 +0000

The exposure

We ran Keycloak the way most teams start: one node, one VM, "we'll cluster it later." Later arrived as a 40-minute outage. A routine security patch required a restart, the node came back with a config typo, and for the duration nobody in the company could authenticate to anything — not the admin tools, not the customer portal, not the internal apps that all trusted it.

It is worth being precise about what class of exposure that is, because "make it not crash" is the wrong framing. Availability is a security property — it sits alongside confidentiality and integrity in the CIA triad — and a single identity provider is a single point of failure whose blast radius is every system that delegates authentication to it. When that one node is down, the failure is not "one service is slow"; it is that the entire organization loses the ability to prove who anyone is. A keycloak cluster nginx design is the control that bounds that blast radius: the postmortem action item was blunt — Keycloak must survive a node loss and support rolling upgrades so that a routine patch is never again an org-wide authentication outage. I did not want to invent an HA design from first principles, so I worked from a thorough third-party guide to running Keycloak behind Nginx as a real cluster → alongside the official docs.

Threat model

Framing the outage as a security problem dictated what we actually had to build.

Total authentication loss on node failure. With one node, any crash, bad config, or patch takes down authentication for every dependent system simultaneously. The control objective is that a single node loss is survivable and invisible to users.
Patch and upgrade as forced outage. If the only way to apply a security patch is to restart the sole node, then staying patched and staying available are in direct conflict — which pressures teams to defer security updates. Rolling upgrades remove that conflict, so the cluster is also a patch-hygiene control.
Session loss as a re-authentication storm. If sessions live only on the failed node, a failover forces every active user to log in again at once. Beyond the user impact, a synchronized re-auth burst is itself a load event against the surviving node. Session state must survive a node loss.
Proxy misconfiguration as an integrity risk. Putting a reverse proxy in front means Keycloak now trusts forwarded headers to build redirect URLs and evaluate the request origin. Mishandled X-Forwarded-* headers produce wrong issuer/redirect behavior — a correctness-and-trust problem, not just a cosmetic one.

Controls we added

Control 1 — two nodes with replicated session state

Clustering adds redundancy, and Infinispan provides the distributed caches that replicate sessions and tokens across nodes, so a user logged in on node 1 stays logged in if node 1 disappears (Keycloak's distributed-cache guide documents the session, token, and authentication-session caches). Discovery is the first real decision: on VMs, JGroups with JDBC_PING (nodes find each other through a shared database table — now Keycloak's default discovery mechanism); on Kubernetes, KUBE_PING via the API with a StatefulSet and headless Service. We were on VMs with a shared Postgres, so we started on JGroups/JDBC_PING and planned a later migration to KUBE_PING.

Control 2 — Nginx as the boundary, configured deliberately

Nginx terminates TLS, hides the internal cluster topology, and load-balances across the nodes — the reverse-proxy pattern documented for HTTP load balancing. The Keycloak side uses the modern Quarkus proxy flags, which are the crux of running behind a proxy safely:

command: >
  start --http-enabled=true
        --hostname-strict=false
        --proxy-headers=xforwarded
        --metrics-enabled=true

--proxy-headers=xforwarded is the line people miss; the legacy PROXY_ADDRESS_FORWARDING env var is gone. Keycloak's reverse-proxy guide is explicit that xforwarded enables parsing of the X-Forwarded-* headers and that you relax --hostname-strict (or pin --hostname) so redirect URLs are built correctly. Getting this wrong produces "invalid issuer" and redirect errors — and, more subtly, means Keycloak may be trusting client-supplied headers it should not, which is why the proxy must be the only thing setting them.

The Nginx config matches those flags — TLS termination, forwarded headers, passive health checks, and sticky sessions:

upstream keycloak_http {
  ip_hash;                      # simple session affinity
  server keycloak-1:8080 max_fails=3 fail_timeout=10s;
  server keycloak-2:8080 max_fails=3 fail_timeout=10s;
}

server {
  listen 443 ssl http2;
  ssl_protocols TLSv1.2 TLSv1.3;
  location / {
    proxy_pass http://keycloak_http;
    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Port  $server_port;
  }
}

Passive health checks (max_fails=3 fail_timeout=10s) evict a bad node after a few failures. On affinity: ip_hash hashes on the first three octets of the client's IPv4 address (nginx ngx_http_upstream_module), which is coarse — everyone behind one corporate NAT lands on the same node. Cookie-based stickiness on the AUTH_SESSION_ID cookie pins each browser session to the node that owns it; we started with ip_hash for simplicity and moved to AUTH_SESSION_ID affinity as traffic grew.

Control 3 — hardening the now-critical identity tier

Concentrating all authentication behind a cluster makes that cluster the highest-value asset on the network, so it earns defense in depth beyond mere redundancy:

Pin --hostname so external URLs are consistent across nodes and cannot be influenced by a spoofed Host header.
Enable --metrics-enabled=true, scrape with Prometheus, and alert on 5xx, login errors, and session spikes — a login-error spike is a security signal (credential stuffing, brute force), not just an ops one, so lost telemetry is lost detection.
Run managed HA Postgres with DB TLS and tuned pools.
Lock the admin console behind IP/VPN, rotate admin credentials, and require 2FA for admins — the admin console is a full realm-compromise surface and does not belong on the open internet.
On Kubernetes, use a StatefulSet, a headless Service, and NGINX Ingress with correct timeout/header annotations.

Verifying the control, not just shipping it

An HA design is a hypothesis until failure is demonstrated on demand — assuming HA works because the diagram says so is how you end up with a more expensive single point of failure. Before pointing production DNS at the cluster we ran deliberate failure drills: with a colleague mid-session in the admin console, I killed keycloak-1 outright; Nginx's passive health check marked it down within fail_timeout, traffic shifted to keycloak-2, and because Infinispan had replicated the session, the colleague never saw a login prompt. We repeated it in reverse, then killed a node during an active token refresh. Only after all three drills passed did we cut over. Two lessons from that exercise are worth keeping: session affinity and replication are different controls (without stickiness, users bounced between nodes mid-login and hit failures in the brief window before a session propagated; replication is the safety net if a node actually dies — you want both), and JGroups is fussy about networking (our first attempt only half-formed a cluster because a firewall rule silently dropped discovery traffic, so confirm the cluster actually forms rather than assuming it did).

Residual risk / what we're still watching

Clustering bounded the blast radius of a node loss, but it introduced new failure modes and a more valuable target, and naming them is the point.

The cluster is now the highest-value asset. Redundancy protects availability, but it does not protect against a compromise of the identity tier itself — a stolen admin credential or a Keycloak CVE is now a whole-org event. That is why the admin console is VPN-gated with mandatory 2FA, and why we track Keycloak advisories directly rather than waiting for a bump.
Cluster-formation and discovery are silent failure modes. JGroups needs clean ports and coherent discovery, and inter-node latency degrades replication. A cluster that silently fails to form looks healthy until a failover drops sessions. We monitor cluster membership as a first-class signal.
Forwarded-header trust boundary. Because Keycloak now trusts X-Forwarded-* from the proxy, the proxy must be the only source of those headers; a path that lets a client inject them would let it influence issuer/redirect behavior. We treat the proxy configuration as security-sensitive and reviewed.
Session-affinity coarseness under NAT. ip_hash concentrates large NATed populations onto one node — an availability and load hot-spot; AUTH_SESSION_ID affinity mitigates it, and we watch per-node session distribution.
Multi-AZ is still open. Two nodes in one availability zone still share an AZ-failure fate. We are finishing the migration to KUBE_PING on Kubernetes and spreading nodes multi-AZ so a zone loss is as boring as a node restart.

The morning after go-live we patched a node in the middle of the workday and nobody noticed — which was the entire point. The residual work is to keep proving that property holds as the topology changes, and to keep treating the identity tier as the critical asset a whole organization now depends on.

Sources & further reading

Keycloak — Configuring a reverse proxy — the proxy-headers values and hostname flags, straight from the source.
Keycloak — Configuring distributed caches — how Infinispan replicates sessions and how nodes discover each other.
NGINX — HTTP Load Balancing — upstreams, passive health checks, and session persistence.
nginx ngx_http_upstream_module — the ip_hash directive and its IPv4 hashing behavior.
A Keycloak + Nginx clustering walkthrough — a useful third-party account of the Docker lab, production nginx.conf, and hardening checklist this analysis draws on.

Deleting Hand-Rolled Auth From a Next.js App: A Keycloak Nextjs Threat Model and NextAuth Cutover

Gaberial Sofie — Sat, 08 Aug 2026 17:16:55 +0000

The exposure

Every codebase has one module nobody wants their name on. Ours was lib/auth.ts — a homegrown JWT system in our Next.js app that had been "good enough for the MVP" three years earlier. It signed its own tokens, hand-rolled refresh-token rotation, and stored sessions in a way two engineers actively disagreed about. Exactly one person understood the rotation logic, and when he took vacation we froze all auth-adjacent work out of fear.

That is a security exposure before it is an engineering one. Hand-rolled authentication concentrates several of the highest-consequence failure modes in software into code that is rarely reviewed and understood by one person. A keycloak nextjs integration was attractive not because Keycloak is fashionable but because delegating authentication moves password storage, MFA, token signing, and refresh — the parts where a subtle bug is a breach, not a bug — out of our codebase entirely. The trigger was concrete: a security review flagged four separate issues in that one file. Rather than patch a system we did not trust, we ripped it out and stood on an identity provider we already ran, leaning on the official Auth.js Keycloak provider and Next.js App Router route handlers. The result was about forty lines.

Threat model

Being specific about what a hand-rolled auth module gets you exposed to is what justified deleting it rather than patching it.

Self-implemented token cryptography. A module that signs and verifies its own JWTs is one weak-algorithm choice or one missing signature check away from token forgery. This is the class of defect where "it works" and "it is secure" look identical from the outside.
Refresh-token rotation errors. Refresh logic understood by a single engineer is fragile in the worst place: a bug can either lock users out (availability) or fail to invalidate a stolen refresh token (a persistent-access foothold). The four review findings clustered here.
Ambiguous session handling. Two engineers disagreeing on how sessions are stored is a session-fixation and session-invalidation risk waiting to surface — you cannot invalidate a session cleanly if you cannot agree on where it lives.
Knowledge concentration as a security risk. Auth that only one person understands cannot be safely reviewed, patched, or incident-handled. Bus factor one on the credential path is itself a finding.

The decision: reduce our attack surface by owning as little authentication code as possible, and let Keycloak — whose entire job is to get password storage, MFA, and token signing right — own the security-critical parts.

Controls we added

Control 1 — delegate authentication, own almost none of it

NextAuth.js (branded Auth.js in v5; the next-auth package and Keycloak provider are the same) ships a first-party Keycloak provider. You declare a provider, point it at your realm's issuer URL, and it handles the Authorization Code flow, callbacks, and session cookies. The security-relevant property is subtraction: the token exchange, password handling, and MFA all move behind Keycloak, so the surface where our own code can be wrong shrinks to configuration. I cross-referenced the official provider docs against a thorough third-party walkthrough of the NextAuth-Keycloak wiring → while doing this.

One trap that had burned us: do not mix Pages Router examples into an App Router app. Most stale tutorials use NextApiRequest and pages/api/auth; in App Router you export GET/POST from a route.ts, per the Next.js file convention. Getting that straight up front is a correctness control — a subtly wrong handler on the auth path is not a place to be copy-pasting.

The whole integration lives at app/api/auth/[...nextauth]/route.ts:

import NextAuth from 'next-auth'
import KeycloakProvider from 'next-auth/providers/keycloak'

const handler = NextAuth({
  providers: [
    KeycloakProvider({
      clientId: process.env.KEYCLOAK_CLIENT_ID!,
      clientSecret: process.env.KEYCLOAK_CLIENT_SECRET || '',
      issuer: `${process.env.KEYCLOAK_URL}/realms/${process.env.KEYCLOAK_REALM}`,
    }),
  ],
  secret: process.env.NEXTAUTH_SECRET,
  session: { strategy: 'jwt' },
})

export { handler as GET, handler as POST }

The issuer must include the realm (/realms/<name>) — the provider docs are blunt about it, and a wrong issuer path is a common and security-relevant misconfiguration because it changes which authority you are actually trusting.

Control 2 — client configuration as an enforced allow-list

In Keycloak we created a nextjs-client (confidential, with a client secret) and treated two fields as controls, not preferences:

Valid Redirect URIs: http://localhost:3000/api/auth/callback/*, matched character for character including protocol and trailing slash. A loose or wildcard redirect is an open-redirect and token-exfiltration vector, so it is scoped to exactly where NextAuth listens.
Access type: confidential for the server-side surface, holding a secret; public clients get PKCE instead. Getting this classification wrong is a genuine weakness, not a style choice.

Control 3 — hardening we adopted deliberately

PKCE for public clients, always — RFC 7636 exists precisely to stop authorization-code interception on clients that cannot hold a secret; modern Auth.js enables it for OIDC providers by default.
KEYCLOAK_CLIENT_SECRET and NEXTAUTH_SECRET generated per environment (openssl rand -base64 32), pushed through the secrets pipeline, never committed, and rotated on a cadence.
HTTPS end-to-end with secure cookies in production.
Explicit session strategy (jwt, chosen on purpose) rather than silently mixing jwt and database sessions, which is a classic footgun with real invalidation consequences.

Control 4 — a cutover that never locked anyone out

Swapping authentication on a live product is a high-risk change, so we staged it to stay defensible throughout. We shipped the Keycloak route behind a feature flag and ran it in parallel with the old system for a week, dogfooding internally. We migrated user identities into Keycloak ahead of time and mapped them by email so nobody had to re-register — a re-registration flow is itself a phishing pretext we did not want to create. We kept the old lib/auth.ts in the tree but unreferenced for one release as a rollback escape hatch, then deleted it once the dashboards stayed green — a dormant second auth system is standing attack surface, so it did not linger. The cutover was a single flag toggle at low traffic, and the rollback plan was toggling it back.

Verifying the control, not just shipping it

The failure that validated the approach was a redirect loop that appeared the instant we deployed to staging: no useful error. The cause was a mismatched redirect URI down to a trailing slash the Keycloak Valid Redirect URI did not have. That is worth internalizing as a security habit, not just a debugging tip — redirect-URI matching is the control that keeps authorization codes from going to the wrong place, so when it is strict enough to break on a trailing slash it is also strict enough to reject an attacker's callback. We aligned it and added the check to our deploy runbook. Because authentication now lives behind Keycloak, "does login work" is answerable by driving the flow against the provider rather than reading our own crypto.

Residual risk / what we're still watching

Deleting the hand-rolled module removed a class of self-inflicted crypto and rotation defects, but it relocated risk into configuration and into a dependency.

Access-token refresh is the next sensitive control. Keycloak's default access-token lifespan is five minutes, so refresh matters sooner than teams expect. Refreshing OAuth tokens inside the jwt/session callbacks — the Auth.js Refresh Token Rotation guide is the canonical approach — must handle a failed refresh by forcing re-authentication rather than silently serving a stale identity. This is the piece we are implementing most carefully.
Back-channel logout is a real gap until we close it. Today, signing out of Keycloak does not yet invalidate the Next.js session everywhere. Until back-channel logout is wired, a revoked or logged-out user can retain a valid app session until it expires — a bounded but genuine residual window we are actively tracking.
Redirect-URI and secret drift. A later loosening of the redirect allow-list, or a leaked/committed NEXTAUTH_SECRET or client secret, quietly reintroduces exposure. Both live in reviewed configuration and rotation.
Keycloak is now a dependency on the login path. We traded code we owned and distrusted for a system we must keep patched, monitored, and available. That is the right trade, but it makes Keycloak's own security posture part of ours, and we track its advisories.
Role-based access mapping. As we map Keycloak realm roles into the jwt/session callbacks for authorization, each mapping is a new place a claim can be misread into wrong access, so those callbacks get the same scrutiny the token logic does.

The net effect is that the scariest authentication code we owned is gone, the parts that are hard to get right now live in a system built to get them right, and the remaining risks are named, bounded, and on a list — chiefly refresh handling and back-channel logout — rather than concentrated in one engineer's head.

Sources & further reading

Auth.js — Keycloak provider — the official provider config, issuer/realm rules, and client-authentication setup
Next.js — Route Handlers (route.js) — App Router file convention for GET/POST handlers
Auth.js — Refresh Token Rotation — persisting and refreshing OAuth tokens in the jwt/session callbacks
PKCE for OAuth 2.0 (RFC 7636) — why public clients need it
Keycloak — Securing applications and services with OpenID Connect — OIDC endpoint and integration reference
An end-to-end NextAuth + Keycloak writeup — a useful third-party account of the full client setup, env vars, the App Router route handler, and the hardening checklist.

Collapsing 60 Trust Boundaries Into One: A Keycloak Identity Federation Threat Model and 90-Day Rollout

Gaberial Sofie — Fri, 07 Aug 2026 08:53:16 +0000

The exposure

Through two acquisitions we had accumulated three separate Active Directories, a bolt-on SSO product, and roughly sixty applications each with its own login page. From a security standpoint the helpdesk ticket volume was a symptom; the disease was the identity architecture underneath it. Every application stored its own credentials, every user carried a dozen of them, and — because remembering a dozen is impossible — people reused the same weak password across systems that had no shared trust boundary.

That is a large, badly-shaped attack surface. Credential reuse means the blast radius of one phished password is not one application but every application that shares it. Offboarding was worse: with sixty independent credential stores, revoking a departed employee's access was a per-app, hours-long chore, which means at any given moment there were almost certainly valid credentials belonging to people who should have had none. MFA coverage was patchy and per-app, so the strength of authentication varied by whichever team had last touched a login page. This is keycloak identity federation territory: the fix is not more helpdesk agents, it is collapsing sixty trust boundaries into one you can actually reason about. Keycloak's Server Administration Guide is the reference for the mechanics; the risk framing and sequencing we had to work out ourselves.

Threat model

Naming what we were defending against is what justified the rollout order.

Credential-reuse blast radius. A single reused password turns one compromise into many. As long as each app authenticates independently, there is no way to bound how far a stolen credential reaches.
Orphaned access after offboarding. Sixty credential stores mean revocation is slow and error-prone. Every store you forget is a standing account an attacker or a disgruntled leaver can use. The control objective is single-point revocation.
Inconsistent authentication strength. Without a central policy, MFA and lockout behavior differ per app. The weakest login page sets the real security posture, because that is where an attacker will aim.
Big-bang migration risk. Federating sixty systems simultaneously with no staging is itself a threat — to availability. The recurring failure story in this space is exactly that: a simultaneous cutover with no fallback that earns a multi-day outage and an emergency rollback. Availability is a security property, so the rollout had to be defensible at every step.

The design decision that follows: put a broker in the middle that becomes the single authority for authentication, so credential reuse loses its reach, revocation becomes one action, and MFA becomes a policy rather than sixty separate implementations — and get there in phases so we never bet the whole estate on one cutover.

Controls we added

Control 1 — a broker so no application stores a password again

Federation does not share credentials — that misconception is itself dangerous, because engineers who believe it will build the wrong thing. Federation shares cryptographically signed assertions that someone was authenticated. Keycloak sits between the identity provider (our Active Directory, or Google — the systems that vouch for identity) and the service providers (the sixty downstream apps), acting as a broker: what the docs call integrating identity providers. It federates our existing AD as the source of truth over LDAP, speaks OIDC and SAML to every app, and means no downstream application holds a credential.

The security payoff is single-point revocation: disable an account once at the identity provider and access to all sixty apps evaporates. That property — not the convenience — was the security team's reason to approve the project. For the sequencing and the console mechanics I kept a thorough third-party walkthrough of a phased federation rollout → open alongside the official docs.

Control 2 — a phased rollout that always has a fallback

We ran it over 90 days, and the phasing was a risk control, not a project-management nicety.

Days 1-30, foundation. Stand up dev, staging, and a production cluster. Connect the primary AD as LDAP user federation so disabled accounts and group changes propagate automatically. Then pick three deliberately low-stakes pilot apps — an internal wiki, a dashboard, a staging tool — nothing that pages anyone at 2am if it breaks. For external IdPs, the setting that trips everyone is First Login Flow → first broker login, which determines whether a first-seen federated user links to an existing account or auto-provisions; a wrong choice here is an account-takeover-shaped hazard, so we set it deliberately.
Days 31-60, expansion. Add Google as an external IdP for contractors, turn on MFA policies org-wide, integrate the next dozen apps. Monitoring became non-negotiable here: alerting on failed authentications, token-renewal failures, and session timeouts, because incidents live at those boundaries, not the happy path. Delayed detection is reduced detection.
Days 61-90, production. Roll out to the rest, harden policy, and — critically — stand up and test backup and disaster recovery for the cluster before decommissioning the old SSO product. A broker with no tested DR is a single point of total authentication failure.

Control 3 — client configuration as an enforced allow-list

For OIDC apps, each got a client (the OIDC and SAML client management guide is the canonical field reference). Two settings carry security weight: Valid Redirect URIs configured exactly — protocol and trailing slash matching character for character, no wildcards, because a loose redirect URI is an open-redirect and token-exfiltration vector; and Authorization Code flow everywhere with PKCE required for public clients. PKCE (RFC 7636) exists specifically to stop authorization-code interception on clients that cannot hold a secret, so we banned the Implicit flow outright.

For SAML apps the work is metadata exchange plus attribute mapping via per-client protocol mappers. Attribute mapping is where SAML silently goes wrong: if the app expects email and Keycloak sends emailAddress, login "succeeds" but the user is provisioned as a stranger — which, depending on the app, can be an authorization failure that grants the wrong access. We standardized on the HTTP-POST binding and reviewed every mapper.

Verifying the controls, not just shipping them

Two verification habits earned their keep. First, clock synchronization: our second SAML integration passed staging and failed intermittently in production with signature errors. SAML assertions carry NotBefore / NotOnOrAfter conditions, so IdP and SP clocks must agree within a tolerance — as the clock-skew writeup explains, a few seconds of drift rejects a valid assertion, and allowedClockSkew only widens the window rather than fixing the cause. One node had drifted; we enforced NTP across every host in the auth path. Check clocks before certs.

Second, negative testing of the controls themselves. We tested redirect URIs with deliberately invalid values, not just the happy path, and caught a client that would have accepted a wildcard redirect — a textbook open-redirect and token-leak risk — precisely because the test matrix included a URL we expected Keycloak to reject. A control you never test against a hostile input is security theater; a five-minute negative test replaced a future incident review.

Residual risk / what we're still watching

Collapsing sixty trust boundaries into one removes a whole class of reuse and orphaned-access risk, but it concentrates risk rather than deleting it.

The broker is now a high-value single point. A Keycloak compromise, a signing-key leak, or a cluster outage is an authentication event for the entire estate. That trade is only correct if the concentrated point is hardened, patched, monitored, and backed by tested DR — which is why DR testing gated the decommission of the old system, and why we track Keycloak advisories directly.
Clock and NTP are now a security dependency. Time drift on any host in the auth path degrades SAML validation. NTP health is monitored as a security signal, not just an ops metric.
SAML attribute-mapping drift. A future mapper change that misroutes email or group claims can silently grant wrong access. Mappers live in reviewed configuration, and we watch for provisioning that produces unexpected identities.
First-broker-login and redirect-URI loosening. Both are one careless edit away from reintroducing account-takeover or open-redirect exposure. They stay in reviewed config, not ad-hoc changes.
Partner federation expands the trust perimeter. We are extending federation to a couple of partner organizations so their staff reach shared portals via IdP-to-IdP trust — which means trusting another organization's authentication. Each new trusted IdP is new attack surface and gets onboarded with scrutiny over what claims we accept and how far that trust reaches, not rubber-stamped.

The net effect is that a stolen credential no longer roams sixty systems, offboarding is a single revocation, and MFA is a policy rather than sixty implementations — provided we keep treating the broker as the critical asset it has now become.

Sources & further reading

Keycloak Server Administration Guide — LDAP/AD user federation, identity brokering, first login flow
Keycloak — Integrating identity providers (Server Admin Guide, Ch. 9)
Keycloak — Managing OpenID Connect and SAML Clients, incl. protocol mappers (Ch. 12)
PKCE for OAuth 2.0 — why public clients need it (oauth.net / RFC 7636)
SAML Clock Skew in Keycloak — allowedClockSkew, NotBefore/NotOnOrAfter, and NTP
A phase-by-phase writeup of a 90-day federation rollout — a useful third-party account of the IdP/SP model and the config details that live outside the primary docs.

Green Tests Aren't a Safe Supply Chain: A GitHub Actions Security Audit Gate That Blocks Vulnerable Deploys

Gaberial Sofie — Thu, 06 Aug 2026 09:29:17 +0000

The exposure

Consider the timeline. Friday: all checks pass, PR merged, deploy succeeds, everyone goes home. The following Thursday a critical advisory drops for one of the gems we were running in production, with exploits already circulating — and the way we learned our own app was vulnerable was a Hacker News thread, not our pipeline.

That is a detection gap, and it is worth naming precisely because CI actively disguises it. Green tests prove your code behaves; they say nothing about whether your dependencies are safe. A pipeline that treats "tests passed" as "safe to ship" is conflating two very different claims, and the space between them is exactly where a known-vulnerable dependency reaches production unopposed. A github actions security audit gate closes that space by making a red audit block the deploy. The design principle we anchored on is the one GitHub itself pushes in its security hardening guide for Actions: make the safe path the default path, enforced by the pipeline rather than remembered by a human. For the workflow shape and the first-month rollout I also worked from a thorough third-party writeup of this audit-gate pattern →.

Threat model

Known-vulnerable dependency reaching production. A gem with a published CVE flowing through a green build into prod is the primary exposure — and because the test suite cannot see it, nothing in the default pipeline stops it.
Post-merge advisory window. New advisories drop after code merges. A dependency that was clean at merge becomes vulnerable in place while sitting in production, and a merge-time-only check never revisits it. This is a mean-time-to-detect problem: ours was roughly seven days, measured from disclosure to a human noticing.
License / legal exposure. A copyleft-licensed transitive dependency slipping into a commercial codebase is a distinct risk that neither tests nor CVE scanners address, and it surfaces at the worst time — during due diligence, not development.
The gate itself as supply-chain surface. Adding third-party Actions to enforce security introduces its own risk: an unpinned marketplace action is mutable and can be re-pointed at hostile code. The control must not become the vulnerability.

Controls we added

Control 1 — a red audit blocks the deploy

The core control is one line. An audit job runs the scanners; the deploy job declares needs: [audit], so it will not start unless the audit is green. No audit pass, no deploy — the safe path stops being something a human remembers and becomes what the pipeline enforces.

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with: { ruby-version: .ruby-version, bundler-cache: true }
      - run: gem install bundler-audit license_finder
      - run: bundle audit check --update
      - run: license_finder --quiet
      - uses: google/osv-scanner-action@v1
        with: { scan-args: '--recursive --skip-git .' }

  deploy:
    needs: [audit]   # deploy won't start until audit is green
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... deploy steps

Control 2 — three scanners for defense in depth

We chose three complementary tools rather than one, because they cover different blind spots and layering them is cheap:

bundler-audit — RubySec's patch-level verification, scanning Gemfile.lock against the ruby-advisory-db for known CVEs, and flagging insecure http:///git:// gem sources. Deepest on Ruby advisory data.
license_finder — reports dependencies outside your approved-license policy, covering the legal risk the CVE scanners ignore entirely.
Google's OSV-Scanner, backed by OSV.dev across many ecosystems — broadest coverage, so it catches vulnerabilities in polyglot repos, not just Ruby.

Overlapping controls is deliberate: no single advisory source is complete, so two independent scanners plus a license check reduce the residual chance that a known issue slips past all of them.

Control 3 — trigger strategy that closes the post-merge window

The triggers are a control in their own right:

on:
  push: { branches: [ main ] }
  pull_request: { paths: ['Gemfile*', '*.gemspec'] }
  schedule:
    - cron: '0 8 * * MON'   # weekly re-audit

pull_request scoped to dependency files so the gate only fires when dependencies actually change — which keeps it off most PRs and prevents the resentment that gets security gates disabled.
The weekly schedule cron is the one that addresses the post-merge window directly: it re-audits already-deployed code against advisories that did not exist at merge, moving mean-time-to-detect from about a week to same-day or next-scheduled-run.
push to main as the last check before production.

Control 4 — hardening the gate itself

Because the gate depends on third-party Actions, we treated that dependency as supply-chain surface. GitHub's hardening guide is explicit that pinning to a full-length commit SHA "is currently the only way to use an action as an immutable release," so third-party actions are SHA-pinned rather than tag-referenced. When the tagged OSV action broke on a stale reference, we installed the scanner as a binary instead — vendor-neutral and not subject to a mutable tag:

curl -sSL https://raw.githubusercontent.com/google/osv-scanner/main/scripts/install.sh | sh -s -- -b /usr/local/bin
osv-scanner --recursive --skip-git .

We also adopted a few Bundler settings to reduce source-substitution surface (bundle config set disable_multisource true, cache_all true, clean 'true'), layered Trivy alongside OSV for further depth, and put a quarterly dependency-hygiene day on the calendar.

Verifying the control, not just shipping it

A gate is only real if it fails when it should and the team keeps it on. The first run failed — we had two existing advisories on merge — and that failure was the point: triaging them was the whole value, not something to route around by disabling the step. Two design choices kept the gate trusted rather than resented. Because the audit runs on pull_request only when Gemfile* changes, most PRs never see it. And because bundle audit prints the advisory URL and the fixed version, "unblock" almost always meant a one-line version bump rather than a research project. After the team saw the red build point at a real, patchable issue every time — no false-positive noise — trust went up, not down. By week four a red audit was simply "bump the gem and move on." A couple of operational notes that matter: bundle audit needs --update or it scans a stale advisory DB and gives false comfort, and license_finder will surprise people (ours flagged a transitive GPL dependency nobody had noticed), so set your allowed-license policy up front.

Residual risk / what we're still watching

The gate reliably keeps known-vulnerable dependencies out of production, but it is not a complete supply-chain defense, and being honest about the edges is the point.

Scanners find known vulnerabilities only. A zero-day or a brand-new typosquat with no advisory yet passes clean. The gate lowers mean-time-to-detect for disclosed issues; it does not detect the undisclosed. Human review of new dependencies still carries that load.
Gate-bypass is a cultural failure mode. The most likely way this control dies is a developer commenting out the audit step to unblock a feature. We defend it by keeping it fast, legible, and low-false-positive so bypassing feels unnecessary — but a determined bypass is always possible, so we also protect the workflow file in review.
Third-party action drift. SHA-pinning freezes an action, but a frozen action also stops receiving fixes; we periodically review pinned SHAs so "immutable" does not become "stale and unpatched."
Coverage gaps beyond gems. The gate audits dependencies, not our container base images or the runner environment. We are extending the same pattern to image scanning next, and rolling the gate out as a reusable workflow so every repo inherits it rather than reimplementing it.
The advisory databases are themselves a dependency. Our detection is only as current as OSV.dev and the ruby-advisory-db; a gap or delay in their data is a gap in ours. Running two independent sources mitigates but does not eliminate that.

The measurable outcome that mattered: mean-time-to-detect a known-vulnerable dependency dropped from about a week to same-day, and twice the Monday cron flagged a fresh advisory on already-deployed code before any customer or news cycle did. What we keep watching is the gap the gate cannot close — the undisclosed vulnerability and the human tempted to route around a red build.

Sources & further reading

bundler-audit and the ruby-advisory-db — RubySec's patch-level verification and the advisory data behind it
LicenseFinder — Pivotal's dependency-license auditor
OSV-Scanner and OSV.dev — Google's multi-ecosystem scanner and the open vulnerability database it queries
Security hardening for GitHub Actions — GitHub's official guide (SHA-pinning, minimal GITHUB_TOKEN permissions)
Events that trigger workflows — GitHub docs for workflow_run, schedule, pull_request filters
A writeup of this audit-gate rollout — a useful third-party account of both workflow variants, the first-month adoption story, and the Bundler hardening config.

Adopting Keycloak Without a Bulk-Credential Migration: A Keycloak Custom Provider (SPI) Threat Model

Gaberial Sofie — Wed, 05 Aug 2026 10:05:12 +0000

The exposure

Our Keycloak rollout stalled on two problems that were, underneath, the same kind of problem: both were about where trust and evidence actually live. We had 200,000 users in a legacy PostgreSQL table with a bespoke password-hashing scheme, and compliance required that every authentication event — successful logins, failed attempts, password changes — reach our SIEM in near real time.

The naive fix for the first problem is a big-bang migration: export the users, re-hash or force-reset every password, cut over in a maintenance window. That migration is itself the exposure. Bulk-handling 200k credential records is a concentrated moment of risk — a dump of password material in flight, a mapping bug that silently drops or duplicates accounts, and a forced-reset flow that is a textbook phishing pretext ("we've upgraded our login, please reset your password here"). The second problem is a detection gap: Keycloak's default event storage kept auth events, but not where our monitoring could see them within seconds, which means the window between a credential-stuffing burst and anyone noticing was measured in whatever our slowest batch job took.

So the real question was not "how do we migrate?" but "how do we adopt Keycloak without ever creating a bulk-credential event, and without accepting a blind spot in login telemetry?" Keycloak, the open-source identity and access management server, answers that through its Service Provider Interfaces (SPI) — the extension seam that lets you change where users are authenticated and where events are sent without forking the server.

Threat model

Being precise about what we were defending against is what kept the extension small and the review tractable.

Bulk credential exposure during migration. Any process that reads, transforms, and re-writes 200k password hashes in one operation is a high-value target and a single point of failure. The control objective is to never have that operation exist at all.
Account-lockout and phishing risk from forced resets. A mass password-reset email trains users to click reset links, which is exactly the behavior an attacker wants to exploit. It also risks locking out legitimate customers on a mapping error, which is an availability incident.
Detection gap on authentication. If failed logins and password changes are not streamed to the SIEM promptly, credential-stuffing, brute force, and account-takeover attempts go unobserved until after the fact. Delayed telemetry is reduced detection capability, which is a security control, not an operational nicety.
Extension-introduced attack surface. Writing custom Java that runs inside the identity server is itself a risk: a provider on the request path can add latency, leak connections, or — worst case — mishandle a credential. The extension must be scoped so its blast radius is understood.

The governing decision: replace a one-time, high-magnitude credential-handling event with a gradual, per-login re-hash where the legacy store stays authoritative until each user proves themselves, and close the detection gap by streaming events at the source. Both are SPI providers, and both are small.

Controls we added

Keycloak ships LDAP and Active Directory federation, but our legacy store was neither, and its default event handling did not stream where we needed. That is precisely where you stop configuring Keycloak and start extending it — deliberately, and with the smallest surface that satisfies the requirement. I cross-referenced the official developer guide against a thorough third-party walkthrough of building SPI providers →, which works each interface end to end.

There are four SPI provider types — authentication, user storage, event listener, and policy. We needed two, and deliberately no more than two.

Control 1 — federate the legacy users so no bulk migration ever happens

We implemented org.keycloak.storage.UserStorageProvider, the same User Storage SPI that Keycloak's own built-in LDAP and AD federation is written against. The two methods that carry the weight are getUserById(String id, RealmModel realm) and getUserByUsername(String username, RealmModel realm); Keycloak calls them to resolve a user, and our implementation queried the legacy table and mapped rows to Keycloak user models on demand.

The security-relevant property is that users authenticate against the old store without ever being migrated in bulk. We validated each login against the legacy hash and, only on a successful authentication, transparently re-hashed that one credential into Keycloak's format. The re-hash happens exactly when the user has just proven they hold the password — there is no batch, no dump, no reset email, and the legacy store remains authoritative for anyone who has not yet logged in. That converts one large, concentrated credential-handling risk into a stream of individually trivial ones. Baeldung's custom user providers walkthrough is a solid worked reference for the interface.

Control 2 — stream auth events to the SIEM at the source

For the detection gap we implemented org.keycloak.events.EventListenerProvider, registered through Keycloak's Events SPI. The method that matters is onEvent(Event event), invoked on every login, failed attempt, password change, and account update. Our implementation serialized each event and pushed it to the SIEM ingest endpoint. The monitoring team got the authentication telemetry they needed to detect credential-stuffing and takeover attempts promptly, in a few hundred lines rather than a platform change.

Control 3 — build and package the extension so it cannot silently misload

Both providers are Java built with Maven. The single most important build decision is scope: both Keycloak artifacts (keycloak-services, keycloak-core) use provided scope, because Keycloak supplies them at runtime. You compile against them and do not bundle them.

<dependency>
  <groupId>org.keycloak</groupId>
  <artifactId>keycloak-services</artifactId>
  <version>YOUR_KEYCLOAK_VERSION</version>
  <scope>provided</scope>
</dependency>

Every provider builds on org.keycloak.provider.Provider (whose lifecycle includes close() for releasing resources) plus a ProviderFactory that reads configuration via init(Config.Scope config). That factory-and-config pattern is a control in its own right: it let the same JAR point at the test database in CI and the production database in prod, so the only difference between environments was reviewed configuration, not code.

Verifying the control, not just shipping it

An extension that runs inside the identity server is not trustworthy until it is demonstrated to fail safely. Our loop: mvn clean install, drop the JAR into Keycloak's providers/ directory, run a build, and confirm the factory appears in the startup log — if it is not listed, the META-INF/services registration is wrong and Keycloak silently ignores the JAR, which is a fail-silent condition worth catching in CI rather than in production.

For the user-storage provider we wrote an integration test that spun Keycloak up in a container against a seeded legacy database, then asserted that a known legacy user could authenticate and that their credential was transparently re-hashed on first success. A mapping bug caught here is a failing test; the same bug found in production is a customer who cannot log in. We never pointed a new provider at the production realm until it had authenticated a test user and emitted a test event end to end against a throwaway realm.

Residual risk / what we're still watching

Extending Keycloak solved the migration and detection problems, but it moved risk rather than eliminating it, and naming where it went is the point.

The event listener runs in the request path. A synchronous, slow SIEM push adds latency to every login and, in the worst case, could be a denial-of-service lever against authentication itself. We made the emit asynchronous behind a bounded queue so a slow or unreachable SIEM never blocks auth — but a bounded queue also means events can be dropped under sustained backpressure, so we monitor queue saturation as a signal that we are losing telemetry.
Version lock-step is now a standing obligation. The internal SPI is not guaranteed stable across Keycloak majors, so the provider must be rebuilt and re-tested against the exact version it runs on. A forgotten rebuild after an upgrade is a provider that compiles against the wrong internals and fails in subtle ways — we pin the version and rebuild on every upgrade.
Connection hygiene under load. A storage provider that does not release database connections in close() leaks them, which is an availability failure mode. We assert cleanup in the lifecycle method and watch connection-pool metrics.
The legacy store stays authoritative longer than we would like. Gradual re-hash is safer than a big bang, but it means the old database — with its bespoke hashing — remains in the trust path until the long tail of inactive users either logs in or is aged out. We are tracking the migration curve and will set a hard cutoff date after which un-migrated accounts are disabled rather than left as indefinite legacy attack surface.
Any future authentication or policy provider is new surface. We are evaluating a policy provider for location- and device-aware access on sensitive resources, and a step-up MFA authentication provider. Each runs inside the identity server, so each gets the same throwaway-realm-first, integration-tested, version-pinned treatment — no rubber-stamping code that sits on the credential path.

The net effect is that we adopted Keycloak without ever creating a bulk-credential event and without accepting an authentication blind spot, in roughly 300 lines of Java. The lesson we took from it is narrow and deliberate: the SPI seam is powerful precisely because it lets you avoid the riskier alternative, but every line you run inside the identity server is code on your most sensitive path, and it earns the same scrutiny as the server itself.

Sources & further reading

Keycloak Server Developer Guide — Service Provider Interfaces (SPI) — the provider/factory model, META-INF/services registration, and packaging.
Keycloak Server Developer Guide — User Storage SPI — the interface behind federating an external user store without migration.
Keycloak Server Developer Guide — Events — the Event Listener SPI used to stream auth events.
Baeldung — Using Custom User Providers with Keycloak — a hands-on Java worked example of a UserStorageProvider.
A hands-on custom-provider writeup — a useful third-party field-notes version of the Maven setup and both providers to keep open while building.