Collapsing 60 Trust Boundaries Into One: A Keycloak Identity Federation Threat Model and 90-Day Rollout

#architecture #cybersecurity #security

The exposure

Through two acquisitions we had accumulated three separate Active Directories, a bolt-on SSO product, and roughly sixty applications each with its own login page. From a security standpoint the helpdesk ticket volume was a symptom; the disease was the identity architecture underneath it. Every application stored its own credentials, every user carried a dozen of them, and — because remembering a dozen is impossible — people reused the same weak password across systems that had no shared trust boundary.

That is a large, badly-shaped attack surface. Credential reuse means the blast radius of one phished password is not one application but every application that shares it. Offboarding was worse: with sixty independent credential stores, revoking a departed employee's access was a per-app, hours-long chore, which means at any given moment there were almost certainly valid credentials belonging to people who should have had none. MFA coverage was patchy and per-app, so the strength of authentication varied by whichever team had last touched a login page. This is keycloak identity federation territory: the fix is not more helpdesk agents, it is collapsing sixty trust boundaries into one you can actually reason about. Keycloak's Server Administration Guide is the reference for the mechanics; the risk framing and sequencing we had to work out ourselves.

Threat model

Naming what we were defending against is what justified the rollout order.

Credential-reuse blast radius. A single reused password turns one compromise into many. As long as each app authenticates independently, there is no way to bound how far a stolen credential reaches.
Orphaned access after offboarding. Sixty credential stores mean revocation is slow and error-prone. Every store you forget is a standing account an attacker or a disgruntled leaver can use. The control objective is single-point revocation.
Inconsistent authentication strength. Without a central policy, MFA and lockout behavior differ per app. The weakest login page sets the real security posture, because that is where an attacker will aim.
Big-bang migration risk. Federating sixty systems simultaneously with no staging is itself a threat — to availability. The recurring failure story in this space is exactly that: a simultaneous cutover with no fallback that earns a multi-day outage and an emergency rollback. Availability is a security property, so the rollout had to be defensible at every step.

The design decision that follows: put a broker in the middle that becomes the single authority for authentication, so credential reuse loses its reach, revocation becomes one action, and MFA becomes a policy rather than sixty separate implementations — and get there in phases so we never bet the whole estate on one cutover.

Controls we added

Control 1 — a broker so no application stores a password again

Federation does not share credentials — that misconception is itself dangerous, because engineers who believe it will build the wrong thing. Federation shares cryptographically signed assertions that someone was authenticated. Keycloak sits between the identity provider (our Active Directory, or Google — the systems that vouch for identity) and the service providers (the sixty downstream apps), acting as a broker: what the docs call integrating identity providers. It federates our existing AD as the source of truth over LDAP, speaks OIDC and SAML to every app, and means no downstream application holds a credential.

The security payoff is single-point revocation: disable an account once at the identity provider and access to all sixty apps evaporates. That property — not the convenience — was the security team's reason to approve the project. For the sequencing and the console mechanics I kept a thorough third-party walkthrough of a phased federation rollout → open alongside the official docs.

Control 2 — a phased rollout that always has a fallback

We ran it over 90 days, and the phasing was a risk control, not a project-management nicety.

Days 1-30, foundation. Stand up dev, staging, and a production cluster. Connect the primary AD as LDAP user federation so disabled accounts and group changes propagate automatically. Then pick three deliberately low-stakes pilot apps — an internal wiki, a dashboard, a staging tool — nothing that pages anyone at 2am if it breaks. For external IdPs, the setting that trips everyone is First Login Flow → first broker login, which determines whether a first-seen federated user links to an existing account or auto-provisions; a wrong choice here is an account-takeover-shaped hazard, so we set it deliberately.
Days 31-60, expansion. Add Google as an external IdP for contractors, turn on MFA policies org-wide, integrate the next dozen apps. Monitoring became non-negotiable here: alerting on failed authentications, token-renewal failures, and session timeouts, because incidents live at those boundaries, not the happy path. Delayed detection is reduced detection.
Days 61-90, production. Roll out to the rest, harden policy, and — critically — stand up and test backup and disaster recovery for the cluster before decommissioning the old SSO product. A broker with no tested DR is a single point of total authentication failure.

Control 3 — client configuration as an enforced allow-list

For OIDC apps, each got a client (the OIDC and SAML client management guide is the canonical field reference). Two settings carry security weight: Valid Redirect URIs configured exactly — protocol and trailing slash matching character for character, no wildcards, because a loose redirect URI is an open-redirect and token-exfiltration vector; and Authorization Code flow everywhere with PKCE required for public clients. PKCE (RFC 7636) exists specifically to stop authorization-code interception on clients that cannot hold a secret, so we banned the Implicit flow outright.

For SAML apps the work is metadata exchange plus attribute mapping via per-client protocol mappers. Attribute mapping is where SAML silently goes wrong: if the app expects email and Keycloak sends emailAddress, login "succeeds" but the user is provisioned as a stranger — which, depending on the app, can be an authorization failure that grants the wrong access. We standardized on the HTTP-POST binding and reviewed every mapper.

Verifying the controls, not just shipping them

Two verification habits earned their keep. First, clock synchronization: our second SAML integration passed staging and failed intermittently in production with signature errors. SAML assertions carry NotBefore / NotOnOrAfter conditions, so IdP and SP clocks must agree within a tolerance — as the clock-skew writeup explains, a few seconds of drift rejects a valid assertion, and allowedClockSkew only widens the window rather than fixing the cause. One node had drifted; we enforced NTP across every host in the auth path. Check clocks before certs.

Second, negative testing of the controls themselves. We tested redirect URIs with deliberately invalid values, not just the happy path, and caught a client that would have accepted a wildcard redirect — a textbook open-redirect and token-leak risk — precisely because the test matrix included a URL we expected Keycloak to reject. A control you never test against a hostile input is security theater; a five-minute negative test replaced a future incident review.

Residual risk / what we're still watching

Collapsing sixty trust boundaries into one removes a whole class of reuse and orphaned-access risk, but it concentrates risk rather than deleting it.

The broker is now a high-value single point. A Keycloak compromise, a signing-key leak, or a cluster outage is an authentication event for the entire estate. That trade is only correct if the concentrated point is hardened, patched, monitored, and backed by tested DR — which is why DR testing gated the decommission of the old system, and why we track Keycloak advisories directly.
Clock and NTP are now a security dependency. Time drift on any host in the auth path degrades SAML validation. NTP health is monitored as a security signal, not just an ops metric.
SAML attribute-mapping drift. A future mapper change that misroutes email or group claims can silently grant wrong access. Mappers live in reviewed configuration, and we watch for provisioning that produces unexpected identities.
First-broker-login and redirect-URI loosening. Both are one careless edit away from reintroducing account-takeover or open-redirect exposure. They stay in reviewed config, not ad-hoc changes.
Partner federation expands the trust perimeter. We are extending federation to a couple of partner organizations so their staff reach shared portals via IdP-to-IdP trust — which means trusting another organization's authentication. Each new trusted IdP is new attack surface and gets onboarded with scrutiny over what claims we accept and how far that trust reaches, not rubber-stamped.

The net effect is that a stolen credential no longer roams sixty systems, offboarding is a single revocation, and MFA is a policy rather than sixty implementations — provided we keep treating the broker as the critical asset it has now become.