The most dangerous failures don't look like failures. They return 200, they pass every check, and they do the wrong thing perfectly.
Over the past few months, I have been privileged to work with a friend on a telegram bot that helps SIWES students with their logs, and this being my first time working on an application that has gone into production, I was very careful with everything. Double checking each new feature, deliberately triggering and trying to break it to make sure it was solid.
But believe me no matter how much I try to monitor each failure log, I have learnt that the most dangerous failures don’t look like your typical 401/403 error, but the ones that return 200 OK but silently fails, and then you assume everything is perfect, ready to be pushed into production.
Let me share a scenario with you, we recently worked on the subscription feature for the bot, where instead of manually paying every month, it tokenizes your card and auto debits you every month, I worked on this feature personally, and being aware of the risk involved in money movement and my fear of being sent to prison, I worked with utmost care on that feature.
My partner is an overly motivated individual so he gave me three days to work on this, so I got to work, first by reading a bunch of “how to set up a subscription engine in your application” articles, then went to Claude and we drafted the whole architecture together, and it was time to get to work, I did everything right (based on my knowledge at least), finished on the second day and was proud of myself, was ready to test and here came the bottleneck because to properly test that a monthly subscription engine works you need a span of thirty days, and we did not have thirty days, as it needed to be out there as soon as possible, now I tested the engine to make sure it only renders the payment by card option and tokenizes that card, and I was certain that after thirty days it was going to debit that card.
So we took a leap of faith, after my partner asked if I was certain I did it right, I went back to check and monitor the flow and was certain I did everything right by the books. Claude even double assured me, telling me to go rest, I got this one in the bag.
My pr was successfully merged without any deployment error, and I thought that was the end, all I had to do was monitor the logs for any error and fix it, day in day out, all I saw was 200 OK, I was feeling too confident.
On a random day I received a feedback from one of our valued user, where the bot had automatically debited him but is refusing him access, I was eating dinner when I saw the notification, I want to say I lost my appetite, but that would be a lie, because I finished my food before checking to see what was wrong, well not that I didn’t care enough but because I was certain it was a misunderstanding from his end because the logs were perfect from my end. I was about to be humbled.
I’m about to get technical here but I will try to make the terms really explanatory.
When a student onboards on the bot, the bot handles the signup. So when they manually trigger a payment, it quietly tucks the student's Telegram ID into the payment just like writing their name on the receipt. So when the payment goes through, the bot reads the name off the receipt and goes "ah, this is student X," and unlocks their access.
And with the Subscription engine, a month later, Paystack automatically renews the subscription. But this time, the bot didn't start the payment, Paystack did it on its own, automatically. And Paystack doesn't know to write the student's name on the receipt, that’s the bot’s job.
So when the student gets debited automatically and Paystack sends the webhook, the bot goes looking for the name like it always does... and finds nothing. It has no idea who this payment belongs to. So it fails to unlock the student's access.
But Paystack did its job, it told me the student had paid, and my overconfident code tells Paystack “oh, really? alright I got it”, even though it did not get anything. The whole thing looked good on the outside and my logs confirmed it too. Money was moved, Paystack saw it and told the bot, the bot confirmed that Paystack told it, but didn’t unblock the student.
First thing I did, and this is the part I'm weirdly proud of, I didn't touch anything. No panicking, no "let me just try re-running it." I went read-only. When a payment system is misbehaving in production, the fastest way to make it worse is to start changing things before you understand them. So I just read, I opened the webhook handler and asked one specific question, how does this code decide who a payment belongs to? And the answer was, it looks for the telegramId. That's it. That's the only way it knew who paid.
So the next question basically asked itself, is that field even on a renewal payment?
This is where I was humbled. On the first manual payment, the bot starts the payment, so the bot attaches the telegramId. But a renewal isn't started by the bot, Paystack does it automatically a month later. So I compared the two, a first-payment payload versus a renewal payload. And the renewal came in with no telegramId. The field the entire handler depended on simply wasn't there.
From there I just followed the obvious with that one fact in hand, no telegramId → the lookup has nothing real to search for → it finds no user → the database write blows up → and that big try/catch wrapped around everything quietly swallows the error and returns "200, all good." The renewal gets dropped.
I didn’t want to be confident, I wanted to be sure. So I checked it against reality instead of trusting my own reasoning:
- I confirmed on the Paystack side that the renewal event was genuinely delivered, and got a 200 back. That was the smoking gun, Paystack handed the message over successfully, so the failure wasn’t Paystack’s, it was mine.
- Then my partner, pulled the real transaction records, and the pattern matched exactly, the charged, locked-out users had a charge with no matching record on my side. That’s the fingerprint of a silently-dropped event. Money in, nothing written down.
My next instinct was to go check the users this had affected, I found 6 of them, and the immediate thing I did was to manually unblock them, and send them an apology message. So I got to work, Instead of depending on the telegramId, which only shows up when the bot starts the payment, I taught the handler to fall back to the customer's email, which rides along on every single charge, renewals included. If the telegramId is there, great, use it. If it's not, find the student by their email instead. And I killed the lying try/catch, so that when the handler genuinely can't figure out who paid, it stops whispering "200, all good" and actually screams, instead of dropping the student in silence.
I was reminiscing about this earlier, and realized that this isn't a telegram bot problem or even a payments problem. It's what happens anytime two systems pass data back and forth and you assume the second message carries what the first one did. Some places to go check the moment you finish reading this:
- Any webhook, any provider. Paystack, Stripe, GitHub, Twilio, Shopify, the payload on “created” is not the payload on “updated” or “renewed” or “deleted”. Don’t identify records by a field that only rides along on some of the events. Find the one that’s on all of them, or store your own stable ID the first time you see it and use that forever.
- Any “success no matter what” response. Every blanket return 200 , every empty catch { }, every bare except : pass is a room where a failure can hide. Before you silence something, make sure you can tell “I handled it” apart from “I gave up quietly.” Those two deserve opposite responses, one says thanks, the other should scream and alert you.
- Any retry you’re accidentally disabling. A lot of systems will retry a failure for you if you just let them know it failed. Paystack would’ve knocked on my door five more times. Swallowing the error didn’t just hide the bug, it threw away the safety net that would’ve caught it. Look at what your “success” responses are quietly turning off.
The takeaway I’d hand younger me:
Identify things by what’s guaranteed, not by what’s convenient. And never, ever let a failure return “success”, because the silent killers aren’t the loud ones that crash. They return 200, they pass every check, and they do the wrong thing perfectly.
If you read this to the end, I’m so proud of you!
Until my next article,
Gloria💕
Top comments (0)