This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
This happened a while ago at one of the corporations where I worked.
I was working on a web application as a full-stack developer, so I was responsible for both the frontend and the backend.
Because the corporation was huge, there were several libraries and services shared across multiple teams and maintained by dedicated developers. It was basically open source inside a corporation. ๐
And of course, we used some of those libraries and services, because why reinvent the wheel, right?
So, after we released the first version of our web application, an interesting bug appeared.
When a user opened the application in multiple browser tabs, authentication would sometimes randomly break in one of them. Once that happened, the tab could not recover until the user pressed F5 or closed it completely.
Interesting.
As the senior developer on the team, I took the bug and started investigating.
After some time and a lot of attempts, I found that when the frontend tried to refresh the authentication token, it received a 403 Forbidden HTTP response, and the authentication flow broke completely.
So I played around with session storage and local storage in Chrome and tried changing the authentication configuration. It did not help much, but one thing was clear: the bug appeared to be inside the shared frontend authentication library.
Okay.
I collected all the logs and everything I had learned about the bug and went to the frontend team responsible for maintaining the authentication library.
After an hour-long discussion, they concluded that it was not their fault because the backend team maintaining the authentication service was not supposed to return that particular response code.
Yes, the frontend library and the backend authentication service were maintained by completely different teams.
I was not fully convinced, but there were more of them. ๐
So I collected all the logs again and went to the backend team.
We had another discussion and found that they were returning a valid response code because the frontend authentication library was sending a refresh token that had already been invalidated. The underlying problem was a race condition between multiple tabs. One tab refreshed the token and invalidated the previous refresh token while another tab was still trying to use it.
Okay...
So, once again, I grabbed all the logs and went back to the frontend team.
And in case you are wondering why the two teams did not simply communicate with each other, both teams gave me approximately the same answer:
This is a problem in your application, so your team has to solve it.
Back to the frontend team.
I presented the backend teamโs conclusion, but they were still not convinced.
At that point, I was starting to get a little angry, so I created a single ticket, added both teams to it, and included all the logs, requests, responses, timestamps, and everything else I had collected.
Weeks passed.
Conversations continued in the ticket and in chat.
But the bug resolution still looked far, far away.
So I decided to arrange a meeting for everyone involved.
And finally, something happened.
During the meeting, we agreed that the problem was in the frontend authentication library and that the frontend team would fix it.
Success!
Or so I thought.
Two weeks later, while testers and users were getting increasingly angry because the application randomly failed to refresh authentication tokens, the frontend team came back with their final conclusion:
They were unable to reproduce the problem.
And users should not open the application in multiple tabs.
Until that moment, opening multiple tabs had been considered a perfectly normal user flow...
At that moment, I closed the ticket, completely removed the corporate frontend authentication library from our project, and reimplemented the entire client-side authentication flow directly against the existing corporate authentication service.
The main changes were in the Angular HTTP interceptor and the token refresh flow. I updated the interceptor to handle authentication-related responses, including 403 Forbidden, without leaving the entire tab stuck in a broken state.
I also reworked how refreshed tokens were handled and stored in session storage. Invalidated tokens and failed refresh attempts were now handled explicitly instead of causing the authentication flow to stop completely.
Yes, it took two days and one night. And yes, I took a few ideas from the corporate authentication library, because otherwise it would have taken much longer. ๐
But I only have one set of nerves.
And you know what?
From that moment on, it worked like a charm. And whenever another bug appeared, it was fixed immediately because, this time, we actually owned the code.
And that is my Bug Smash story where corporate life can sometimes be slightly ridiculous, and fixing one bug can mean rewriting the entire feature.
Top comments (18)
Hi Daniel, great article (and awesome cover image ๐)! One thing that stopped me:
I know I'm probably living in a dream world, but actually this is what should happen every time when testing a new feature. How did it look in your case? You did not have QAs test the feature? Or did nobody consider such a case? Now that I think about it, I probably wouldn't think of opening so many tabs ๐ค Interesting.
Thanks! I hope I understood your question correctly. ๐
The feature was tested by our QAs, but this particular case was not discovered before the release. It required multiple tabs to refresh their tokens at just the right or wrong time, so it was an intermittent race condition rather than a consistently reproducible scenario.
One tab refreshed the token and invalidated the previous refresh token, while another tab was still trying to use it. The bigger problem was that the shared frontend library could not recover from the resulting 403 Forbidden response and left that tab completely stuck.
So yes, opening the application in multiple tabs was considered a normal use case, but nobody anticipated this exact token-refresh race condition during the original testing. Even after we identified it, the team maintaining the shared library was unable to reproduce it.. ๐
When I reimplemented the authentication flow(as part of the bug fix ๐), it went through QA again. They tested the original scenario directly, but the fix was also exercised indirectly during other test cases where opening multiple tabs was part of the testing process.
Ah, I see now. Thank you for the explanation. So tricky! ๐
Hahaha, that's a beautiful story! ๐คฃ Corporate "helpful" shared services at their finest.
We have something very similar in my company, too. The only difference is that ours isn't a collection of separate building blocks, but it's one huge framework. ๐
The people maintaining it seem to believe nobody will ever have to upgrade it, because every new version comes with breaking changes as if there were no tomorrow. They'll rename classes, methods... sometimes something goes from close() to closeDialog(), only to become close() again two versions later. Truly groundbreaking changes. ๐
At this point, the more we work around it and build our own implementations where it makes sense, the healthier our project tends to be. ๐
Thanks!
Thatโs hilarious. ๐ It looks like the lifecycle of these โhelpfulโ shared services is the same in every corporation.๐
Sometimes I feel like the maintainers develop these libraries mainly for their own test applications. At first, every team tries to use them to save time, but after enough struggles, many teams just reimplement the features themselves. Constant breaking changes combined with little willingness to fix bugs is just... corporate life. ๐
Good Luck๏ผ
You mean good luck surviving the ridiculousness of corporate life? ๐
Haha, speaking as a QA, you're exactly my kind of developer.And good luck with DEV's Summer Bug Smash: Smash Stories โ hope you bring it home! ๐
Oh, thanks! โค๏ธ But I suppose there will be much better submissions. I did not really think of this as a winning one. It was just one of the โfunnyโ stories I have from fixing bugs in the corporate world. ๐
The world is tattered and torn, but there's always someone quietly stitching it back together.
Here's to everyone who gives more than they take.๐
The cover image looks good! haha
Agree! ๐ฏ Looks awesome! ๐
Thanks to both of you @technogamerz, @klaudiagrz ! I played around with it a lot. Did you notice the LOTR theme? ๐ Especially the hobbit and Smaug. ๐
Ahahah, that was my guess! ๐ It gives LOTR or WoW vibes!
You have an excellent eye! ๐ฎ Iโm pretty sure my first prompt included both LOTR and WoW (World of Warcraft). Iโm a big fan of both btw. ๐
I spent so much time on both that my eye couldn't have missed it ๐ Damn, now I have the mood to play WoW again ๐ฅน
Great topic! Shared auth libraries are such a critical single point of failure. Iโm curious โ was this bug related to edge cases in token validation or race conditions?
Thanks! It was a race condition in the token refresh flow. One tab refreshed the token and invalidated the previous refresh token while another tab was still trying to use it. The shared authentication library then failed to recover from the resulting 403 Forbidden response.