This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
You Know What's Worse Than Not Being Able to Log In?
Being told everything worked right up until you try to actually use your account.
Yes, that was a real bug.
And, somehow, I ended up being pulled into another authentication mystery. At this point, Iām starting to think authentication bugs have a personal grudge against me. š
In my previous Smash Story, I wrote about a bug where users simply couldn't log in. This time, the problem was sneakier because most of the flow looked completely healthy. The user was approved, the background task ran, the email and SMS arrived, and Cognito had a user.
Then the user actually tried to use their account.
And everything fell apart.
It Started With Two User Pools
The authentication setup was fairly large and had evolved over time, so there wasn't one shiny User Pool doing everything.
We had an older Cognito User Pool supporting existing authentication flows, including mobile-based signup, while a newer User Pool handled a newer flow where users received an email containing their PIN. Both pools were intentional because they supported different parts of the authentication journey.
That wasn't the problem.
The interesting part was that the application database had its own representation of a user, while Cognito had another. On top of that, some of the work connecting those two systems happened asynchronously.
As long as everyone agreed about who the user was, nobody cared.
The moment they disagreed, authentication became very interested.
The Tiny Timing Window
The problem appeared in the partner and dependant journey.
A member could create a partner or dependant during signup or later from the member details area. A relevant non-member user would then approve the account, which scheduled an asynchronous task called SendingEmailsAfterApprovalBot in a TaskList database table.
That task ran every 15 minutes, and once it executed, the partner or dependant received an email and SMS containing a code. They could then confirm their account, set a password and sign in.
Sounds reasonable, right?
Now imagine this.
A partner gets approved at 10:01 AM, so the background task is scheduled and everyone moves on with their lives. At 10:05 AM, before the task has run, the partner changes their name.
That tiny change matters because the username for partner and dependant accounts is derived from their details. The database now has the new username, while Cognito may still know about the old username.
Nothing has visibly broken yet.
The task is still waiting patiently in the database, completely unaware that the identity it is about to process has changed underneath it.
Then, 15 minutes later, it wakes up.
Cognito Says, "I've Seen This Email Before"
The background task tries to create the Cognito user using the latest username stored in the database.
Cognito rejects the creation because the email address is already associated with an existing user, but that user has a different username.
And there is an important detail here: the Cognito username is immutable.
So we weren't dealing with a simple "just update the username" situation. We had two systems describing the same person with different usernames, and Cognito was quite reasonably refusing to create another user with the same email.
The database was saying:
"This is the current username for this person."
Cognito was effectively saying:
"I already know this email, and it belongs to someone with another username."
Same human.
Different digital identity.
And that mismatch was enough to break the journey.
Everything Looked Fine... Until It Didn't
What made this bug particularly sneaky was what happened next.
The user could still receive the expected email and SMS. There was no obvious failure screaming for attention. The asynchronous task had run, notifications had gone out and Cognito already had a user associated with the email.
So the user did what any reasonable person would do: they followed the instructions.
They entered their code, tried to confirm their account and attempted to set their password.
Except the identity represented in Cognito didn't match the latest identity represented in the database.
The user wasn't looking at a database synchronisation problem. They weren't thinking about immutable Cognito usernames or asynchronous jobs. They had received a perfectly legitimate code and were simply trying to finish creating their account.
From their perspective, the system had basically said:
"Here's your code. Everything is ready!"
Then:
"Actually... never mind."
That was the bug we had to chase.
Enter Sentry
By this point, we knew something was going wrong somewhere between the database, the background task and Cognito, but knowing where it broke was another story.
This is where Sentry became extremely useful. The error context gave us the details around the Cognito failure, while the breadcrumbs helped reconstruct the events leading up to it. Instead of seeing a Cognito error in isolation and assuming we simply had a duplicate-user problem, we could look at what happened before the failure and connect it back to the user's journey.
The important clue was that Cognito was rejecting the user creation because the email was already associated with an existing user. That told us what Cognito didn't like, but the surrounding context helped explain why we had reached that point in the first place.
We could see the relevant user and operation context around the failure, follow the activity leading up to the Cognito call, and use the correlation ID to connect the same operation across services and logs. That made it much easier to piece together the timeline rather than chasing individual log entries and hoping they belonged to the same request.
And once the timeline was visible, the problem suddenly made sense:
approval ā delayed task ā name changed ā username changed ā Cognito already has the email ā creation rejected ā identities drift apart ā confirmation fails
That was the moment the bug stopped looking mysterious.
The code told us what should happen. Sentry's error context, breadcrumbs and correlated logs helped us see what actually happened.
And honestly, that's one of the best parts of having good observability during an authentication investigation. The error tells you where the system complained; the breadcrumbs help you understand the story that led it there.
The Fix Was Simpler Than the Investigation
Once we understood the problem, the fix was surprisingly straightforward.
We already checked whether a Cognito user existed for the email, but we were treating that as enough. Instead, we changed the logic to ask: "Does the correct Cognito user for the current database record exist?"
If the email existed in Cognito but belonged to an outdated username, we deleted that user and recreated it with the latest username from the database. Since Cognito usernames are immutable, recreating the user was necessary to bring the two systems back into alignment.
The important change wasn't the delete-and-recreate itself. It was recognising that the database was the source of truth for the current username, while Cognito could still be holding an older representation of the same person.
The Weird Scenario Was the Important One
The happy path already worked, so the interesting tests were the ones where state changed between steps.
We tested the normal signup flows, partner and dependant creation, existing Cognito users, missing Cognito users and, most importantly, the scenario that exposed the bug: a partner or dependant was approved, changed their name before SendingEmailsAfterApprovalBot ran, and then the background process tried to create their Cognito user using the latest database state.
We then verified the part that actually mattered: could they receive their PIN, confirm their account, set their password and successfully sign in?
Because nobody cares about our beautifully synchronised identity records. They just want to get into their account.
The Interesting Part Wasn't Actually Cognito
Looking back, this wasn't really a Cognito bug. The database wasn't broken, and neither was the scheduled task. Each component was behaving according to its own rules.
The problem was the assumption connecting them: that the identity we started processing would still be the identity we needed when the asynchronous process eventually ran.
Then someone changed their name.
That small change was enough to make the database username different from the immutable username already held by Cognito. Since the email was already taken, Cognito rejected the new user creation and the two systems drifted apart.
That's the lesson I took from it: whenever identity exists in multiple systems, you need to be explicit about which system is the source of truth and what happens when those representations disagree.
Because eventually, they will.
Someone will change their name. A background job will run later. A legacy system will already have a record. And Cognito will quite reasonably tell you that it has seen that email before.
And That's What Made This Bug So Sneaky
The approval worked, the task ran, the notification was sent, Cognito had a user and the user followed the instructions. Yet somewhere between those perfectly reasonable steps, the identity had drifted.
From the user's perspective, none of the underlying details mattered. They didn't know there were two User Pools, that the old pool handled mobile signup while the newer one sent the email PIN, or that a background task ran every 15 minutes.
They simply knew they had done everything they were asked to do and still couldn't get into their account.
Which brings me back to the opening question.
What's worse than not being able to log in?
Being told everything worked right up until you try to actually use your account.
At least a failed login is honest. This bug was more like:
"Congratulations! Everything worked!"
"Really? Because I'm still locked out."
And that's probably the biggest lesson I took away from it: when debugging authentication, don't just look at the login screen. Follow the identity through the entire system, use your observability tools to reconstruct what actually happened, and find the moment when the different pieces stopped agreeing about who that user was.
Because sometimes the authentication bug isn't hiding at login.
Sometimes it's hiding in a background job, quietly waiting for someone to change their name.
Auth strikes again. šš
Top comments (0)