Welcome to Code Chatter, your go-to series for conversational coding insights. What makes this series of questions different from all the others? Well, truth be told, not much, but they're still thought-provoking and fun. Join us as we explore the coding world, one witty question at a time.
Could you share a recent 'panic moment' during development, and how did you handle it?
Follow the DEVteam for more discussions and online camaraderie!
Top comments (6)
A few years ago, I built an A/B test for an enterprise level client. The test was making modifications to their SPA on one page, however, we were not aware that their checkout shared some of the same structure and classnames and we inadvertently removed their "Checkout" button, so user's couldn't complete their orders.
We quickly turned off the test, and I was able to fix it, but that was definitely one of the bigger panic moments because I had made a mistake and probably cost the company a good chunk of money...
Afterward, we had a long conversation about what had happened. There were many things that lead up to this, like:
Moral of the story, there is a process for a reason, and if you rush that process, or skip steps, eventually something will go wrong.
Most recent turned out to not be something I needed to sweat. Two of our most major clients called in reporting server not found errors on their site. Pull up both, sure enough, nothing. Start going back through all the changes we’d done recently, nothing would have caused this, much less in two unrelated sites.
Check the DNS records, and there’s NOTHING for either site…no A records, no MX, no cnames, nada…both sites completely wiped off the net. Panic mode sets in HARD through the entire department. Phone calls were made, no one has any clue what’s going on.
Turns out, the two “unrelated” sites were in face related. Both sites’ DNS were managed by the IT department of the provincial government…which was suffering a “network incident”, later determined to be a DDoS against their network, that took out the DNS servers.
We all went directly to the bar after work…
Most recent? Just now.
Facing one right now. Deployed project to QA. Without getting approval from the team members.
Now only now half of the features are moved. Dev team will be getting some amazing and interesting defects 😂😂
I pushed the EPO button in the server room, thinking it was the light switch...
Pushed a silly typo to prod.