Hello Everyone! 👋🏻
I’m Shreyash Srivastava, a 2nd year CSE student at DSCE and yes it's time for another blog. A lot of things happened in last few months, So before I begin I will just share this image here:
A Quick Catch-Up
I was awarded LiFT Scholarship from Linux foundation, was working on a project for our club (check it out!) also I recently got an internship at an early stage startup and got the opportunity to work with one of my seniors at Pointblank.
The company basically provides a platform for educators and students. Teachers can create assignments, activities, and other learning materials using interesting AI features like image generation, case study analysis, and presentation makers. Students can then use these materials to practice, learn, and receive personalized feedback. We also have a community page where teachers can share their creations with a wider audience.
So, One of the first tasks that I was assigned was to implement a credit system to limit the AI usage in our product. Until then, users on any plan could use our AI features an unlimited number of times. So my job was to introduce limits to manage the usage.
My Approach
In order to implement this feature what I thought of was to create a tokens collections(as we were using firebase) which will consists of userId, plan and the number of credits/tokens a user has.
And created a custom hook which will check if the user has tokens to make a request and it will also help in deducting the tokens after every successful call.
Now what I did was, wherever the call to our AI service was made I added a check to see the number of tokens the user has and deducted the credits after every successful call, and wherever the function was called i made sure that it will pass the userId from frontend and will authenticate the user at backend with help of that userId so I can check if the user exists or has the credits or not.
I made the changes and raised the PR to our development environment it was merged and everything was working, sounds too good right? Lol here comes the fun part....
The Chaos
It was 1:09 AM in night, I was going to sleep and then I received some messages,
Yes, The production was down. (Remember the T-shirt xD)
So, what happened was as my senior (who was in a hackathon at that time) was busy, I myself pushed some changes to both repositories and I didn't know that We DO NOT have development/staging branch for backend repository and we were using this for both production and development environment T-T.
The Issue
As I mentioned earlier what I did was pass the userId to backend in order to verify if the user exists in our database or not, right? But now as the frontend changes were not in production environment and I pushed my changes in backend it was expecting userId to be passed and due to this it was unable to find the user and the entire service was down T-T
Aftermath
In order to fix this, we immediately reverted the backend changes to get the platform back online. And after a proper round of testing with both the new frontend and backend on a staging environment, we successfully rolled out the credit system.
We also created a separate staging branch to ensure this thing never happens again. It was also a great learning experience for me as I'll never forget: Always Ask, Never Assume. I should have confirmed once before pushing the changes.
And after that everything went fine, it was all good and happy.
(I again crashed prod after two days as I forgot to create token documents for existing users in production database)
Top comments (0)