Every SSO engineer has had this week. An integration is configured, tested and signed off in a lower environment. It gets promoted and within hours of go-live, tickets start arriving: users who can't reach the application, an application team that insists nothing on their side has changed, and a business stakeholder asking why a working integration broke.
Nothing broke. The integration was never tested against the conditions it now runs in.
I spent close to two years configuring and supporting enterprise SSO leveraging protocols such as SAML 2.0, OAuth 2.0 and OIDC through PingFederate and PingAccess in a large regulated banking environment. What struck me over that time is how few post-go-live failures were actually protocol failures. The assertions were well-formed. The flows were correct. The failures came from somewhere else: real user data, and configuration that arrived from other people.
This is a walk through the failure modes I saw repeatedly, with the symptom, the cause, and how to confirm it.
1. The directory holds something other than what you tested against
This is the least interesting cause and the most common one, which is why it belongs first.
The symptom. A handful of users report that SSO is broken. The application team reports that everyone else is fine. That combination sends everyone straight to the integration configuration, which is exactly the wrong place to look.
What's actually happening. The user has an account in the application, so everybody assumes they have a corresponding identity in the identity provider's directory. Sometimes they don't and sometimes they do, and its status is expired or disabled. Sometimes the identity exists and is active, but the address the user is signing in with isn't the address held against their record: they're typing an alias, an old address from before a name change, or a personal variant that was never the authoritative one.
The same problem shows up a level down, in attribute fulfilment. Attribute contracts are straightforward when the source data is clean, and test directories are clean. Production directories carry nulls in attributes you assumed were always populated, duplicates, inconsistent formatting, and records created by two or three different provisioning processes over the years. When a source attribute is empty or holds an unexpected value, the contract can't be fulfilled as intended, and what the service provider receives either fails validation or fails to match a user on their side. Case sensitivity is worth checking specifically where matching is done on an email address, and it's worth confirming the NameID format and value are what the service provider expects rather than what seems reasonable.
Why testing never catches it. Your test user was created for the purpose of testing. It is correctly populated, correctly named and permanently active. Production users arrive through joiner-mover-leaver processes that have been running for years, with all the residue that implies: dormant accounts, records from historical migrations, and users whose lifecycle status changed while nobody was watching.
**
How to confirm it.** Query the directory for the specific user before you look at anything else. Confirm the record exists, confirm the status, and confirm the attribute the integration matches on holds the value the user is actually presenting.
This presents slightly differently depending on where the enforcement sits. In a browser-based SAML flow, the user typically reaches the identity provider, fails at or immediately after authentication, and never gets an assertion. Behind a PingAccess agent, the same underlying problem can surface as a redirect loop or a generic access-denied page from the protected application's host, because the failure happens before the request is ever proxied onward which sends the application team looking at their own web server logs and finding nothing.
2. Authentication succeeds and access is still denied
This one is worth understanding properly, because it breaks the assumption that a successful login means a working integration.
I configured coarse-grained authorisation on a SAML connection using a centralised entitlement service. The application team supplied the namespace their entitlements were held under, I added it to the connection configuration, and it worked correctly in testing. After go-live, a set of users couldn't reach the application.
They were authenticating without any problem. An assertion was being generated and consumed. What they didn't have was an entitlement: the application team had populated the namespace with some of their user population and not all of it.
Why this is hard to spot. There is nothing wrong with the integration. Nothing in the authentication path is misconfigured. The failure lives in a handoff between two teams: one team owns the connection, another owns the entitlement data, and no test performed by either team in isolation would have caught it. The application team tested with users they'd added. I tested the connection, which worked.
The symptom that gives it away is that the failure is consistent per user rather than intermittent, and it happens after a successful authentication rather than during one. If a user can authenticate and still can't get in, stop looking at the assertion and start looking at what happens to it downstream.
How to prevent it. Before go-live, agree explicitly who owns entitlement population and get confirmation that the full intended user population has been loaded, not a sample. And test with at least one user drawn from the real population rather than from the list of users someone provisioned for the test.
The lesson generalises well beyond entitlement services: any point where authorisation is evaluated separately from authentication is a place where the integration can be perfect and the user can still be locked out.
3. Configuration you were given, rather than configuration you derived
Two of my more memorable production failures had the same root cause: a value that arrived from another party by document rather than being exchanged or verified.
The ACS URL. A vendor supplied the assertion consumer service endpoint for their production tenant. We configured it, promoted, and the flow failed at the point of assertion delivery. The URL was wrong. In the lower environment it had worked, because the vendor had actually tested that endpoint themselves; the production value had been transcribed from a specification and never exercised.
A browser trace makes this immediately obvious as you can see the assertion being POSTed and see exactly where it's going. Take the endpoint out of the trace and compare it character by character against what the service provider expects. It's usually a path difference, a tenant identifier, or a protocol scheme, and it is not the kind of thing anyone spots by reading a configuration screen.
While you have the trace open, check both flow directions. Integration testing is usually done IdP-initiated, from a portal or a test link. Real users arrive at bookmarked deep links inside the application, which is an SP-initiated flow and a different code path as it can fail while the tested route works perfectly, and it usually gets reported as "SSO is fine but the link is broken", which sends it to the application team rather than to you.
OAuth client parameters. The equivalent on the OAuth side is an application team whose code sends something the client configuration doesn't permit: a redirect URI that doesn't match what's registered, a grant type the client isn't configured for, a missing PKCE parameter, or a scope that was never granted. The error comes back from the authorisation server, the application team reports that SSO is failing, and the actual defect is in code you can't see.
The general rule. Consume metadata wherever the service provider publishes it, rather than transcribing endpoints by hand. Where you can't, treat every manually supplied value as an untested assumption and verify each one against a real transaction before sign-off. Ask for production values to be confirmed by someone who has used them, not by someone who has documented them.
4. Time
SAML assertions are valid for a window. The Conditions element carries NotBefore and NotOnOrAfter, subject confirmation data carries its own expiry, and the service provider validates all of it against its own clock.
In a lower environment the identity provider and the service provider are often on the same network, synchronised to the same time source, with negligible latency between them and sometimes on the same host. Production separates them: different datacentres, different regions, or a vendor-hosted service provider on the other side of the world with its own NTP arrangements. Any drift between the two clocks eats into the validity window, and a tight window plus modest drift produces rejections.
The symptom is distinctive. The service provider rejects the assertion as expired or not yet valid, and it can be intermittent, failing under load or at particular times of day when drift is at its worst. That makes it one of the harder failures to pin down from the identity provider side, because from where you're standing the assertion generated successfully.
Check NTP synchronisation on both sides, confirm the service provider's tolerance for clock skew, and don't tighten validity windows below what the deployment topology can actually support. Timezone and daylight-saving handling is worth verifying too, on anything not working in UTC.
5. Certificates
Every article about SAML failures leads with certificate expiry, which is roughly why this one doesn't. It is a real failure mode, it does cause outages, and it is also the best-understood problem in federation.
Two things are worth saying about it. First, the failure is abrupt and total rather than partial: if a signing certificate has expired or been rotated without the other party updating their trust, everything on that connection stops at once. That pattern is diagnostic in itself. Second, the underlying problem is almost never technical. Rollover mechanisms exist. What's usually missing is a named owner holding a renewal calendar for every connection, on both sides of it.
The order I check things in:
Most of my time on these incidents went into establishing where in the flow the failure happened. Once that's known, the cause is usually quick to find. A workable sequence:
1. Establish scope. One user, a defined group of users, or everybody? One user points at data. A group points at entitlement or attribute population. Everybody points at configuration, certificates or time.
2. Establish the path. Which flow? IdP-initiated or SP-initiated, portal or deep link? A failure that only occurs on one path is telling you something before you've read a single log.
3. Check the identity. Does the record exist, is it active, does the matching attribute hold the value being presented? This is the cheapest check and it resolves a surprising proportion of tickets.
4. Was an assertion generated? The identity provider's audit and transaction logs will tell you whether a transaction was attempted and what happened to it. Capture the tracking identifier for the transaction so you can follow it across log files.
5. Did the assertion reach the right place? A browser trace shows you the POST and its destination. This is where endpoint errors surface.
6. Did the service provider accept it? If it arrived and was rejected, you need the service provider's error: signature, audience, conditions, timing, or a failure to match the subject to a user.
7. Did anything happen after acceptance? Authorisation checks, entitlement lookups and application-side session handling all sit downstream of a successful assertion and can all deny access on their own.
Working the flow in order beats working from a hypothesis. The temptation with a live incident and an anxious stakeholder is to jump straight to the cause you saw last time, and that instinct is wrong often enough to be expensive.
A pre-production checklist
Things I'd want confirmed before promoting an integration, drawn from the above:
• At least one test performed with a user drawn from the real production population, not a purpose-made test account.
• A user with an inactive or expired status tested deliberately, so the failure mode is known rather than discovered.
• Directory attributes used by the contract profiled for null and inconsistent values across the real population.
• Every endpoint taken from published metadata where possible, and every manually supplied value confirmed by someone who has transacted against it.
• Production ACS URL verified in a browser trace rather than by reading configuration.
• Entitlement or authorisation data confirmed as fully populated, with a named owner.
• Both IdP-initiated and SP-initiated flows tested, the latter from a genuine deep link.
• Clock synchronisation confirmed on both sides, and validity windows sized for the real topology.
• Certificate expiry dates recorded with a named owner and a renewal date on both sides of the connection.
None of this is exotic. What it amounts to is the observation that a lower environment tests the configuration, and production tests the assumptions. Most of the assumptions belong to somebody else.
I work on enterprise identity and access management using PingFederate, PingAccess, SAML 2.0, OAuth 2.0 and OIDC. I write about federation, SSO and access governance. Based in the UK.
Top comments (1)
One thing this reminded me of is that SSO failures are often integration failures without protocol failures. SAML, OAuth or OIDC can all be working exactly as designed while the system still fails because the assumptions around identity, authorization, ownership or lifecycle don’t hold. That distinction changes how you troubleshoot incidents: instead of asking “Which protocol step failed?” you start asking “Which assumption stopped being true in production?”