So I brought down the site for a little while this morning. Now I'm interested in hearing about when you messed everything up!
For further actions, you may consider blocking this person and/or reporting abuse
So I brought down the site for a little while this morning. Now I'm interested in hearing about when you messed everything up!
For further actions, you may consider blocking this person and/or reporting abuse
Luciano Jung -
dev.to staff -
dev.to staff -
Michael Tharrington -
Latest comments (56)
I use to be a game developer.
One time I corrupted the external hard drive by mistake. It ruined months of work that was unrecoverable.
This was a four person team project, we all failed the final.
The mistake was not backing it up. I also had dropped the hard drive which caused the corruption. Obviously this was a accident, and it was my personal drive. But I also was very bad about sharing the work in process, thinking I could wow people. This caused team mates not having a copy that was nearly as close.
We recently made a switch from environment foo to environment bar. After the migration to bar was completed, I assumed we could take foo down. For reasons, foo should have stayed up and I had to spin up foo within 15 minutes. I managed to do it, but that was not a fun time.
Takeaway was the situation could have been avoided if I had just asked "Hey, do we still need foo?".
The worst thing I've done so far was: uploading staging environment in production and production in staging. Damn CI was lacking by then
A long time ago as a tech support guy I did an rm -rf on a customer site and forgot the "." before the "/". Whoopsie.
I was working on a quite easy and basic feature of an Economy system and the only advanced part was a really old legacy module that had two functions with really similar and bad names, something like
cr_EDC_use_ETM20andcr_EDC_use_ET20. My ide auto spelled me the wrong function and I managed to send several invoices to over 60k clients. Invoices with an infinite amount to pay. Luckily we had rollback systems, so clients never noticed. Later our old-school legacy programmer refactored the module with names that made sense to the younger generation :PI currently work in road design. Most of my career has been electrical building system design. So when I was new to road design I labeled a bunch of intersections with 20' radii. By a bunch I mean the whole project of 20+ intersections so over 80 corners, roughly. The problem is that they weren't actually drawn at 20', they were all drawn differently because I didn't know any better. Some were only 7', 13' etc. So when it was time to construct the curbs on each corner, nothing lined up. So now instead of a nice new consistent result these intersections all have random and different radii. It was embarrassing to say the least and I never was invited to visit the project during construction which I was thankful for!
I've brought down live sites numerously, messed up the master branch, ended up in merge conflict hell, grossly underestimate things, sent out e-mails to thousands of customers because i was using the production DB for developing, and all kinds of other sheit.
I've only messed up once of twice so bad a backup had to be used at the start of my career tho.
Developers should never test their own stuff or have access to production directly.
It always ends in chaos at some point in my experience. :)
Usually fixable chaos, but still.
In my internship while copy pasting code from other component I also copy pasted google analytics code and one of the variable was undefined.
It went unnoticed (since it was just part of analytics so it didn't exactly break anything π€·) and then they had bunch of undefined hits on their production analytics!
New job. First time monitoring a Unix shell scripting job. Script errors out with memory issues. Go in the respective folder, find a couple of files ~ 2GB big, delete them. Re-run the script, it fails again. This time looking for the files yours truly deleted. Had to wake up the SME to fix the issue.
In my first job I managed a firewall configuration app for a large Telco. It allowed Fortune 500 companies to make firewall changes on their network. Canβt remember the exact problem but long story short I changed something which stopped anyone logging in at all. No-one could make any changes in for 2 days until we found the issue. π³
This was particularly bad because it affected bonuses that year, whooops! ππ
I was learning JavaScript. I broke a website. The company I was consulting for (a Google Ads reseller) lost that client.
That was back in 2011 and I'm still learning JavaScript (and still consulting for the same company.)
Forgiveness is hard to find, but boy, when you find it!!!
Well, I stopped keeping count of it. But the best would be me dropping a critical production database(Yes, I didn't take a backup) while running a migration script and then reconstructing missing data from logs. Before you judge me please note that I was working 36 hr straight to meet a production deadline and this is why no one should work like that.
Years ago I moved from being a tech to a sys admin ... and one of my first responsibilities was swapping out the backup tapes in the pop. There's a bunch of story in the middle but the punchline ends with accidentally dropping a few important tapes down several flights of stairs.
Many years ago, I was a systems admin who was tasked with maintaining (among other things) an email server. The server was running Debian Linux for its OS, and Exim 3 for mail. Like an idiot, I figured that upgrading to Exim 4 would be no biggie, despite the many warnings that accompanied the release, and blithely went and apt-got it.
A couple hours of panicked hackery later and I'd finally managed to restore service back with Exim 3. Eventually, I got a very nice setup going on the cheap with virtual users in a mysql database,on Exim 4. But I never skipped past a warning like that again without doing my due diligence.
I was using PowerCLI to revert snapshots for a block of VMs that I was developing automation for. I use TextExpander to do that snippet, with fill-in fields for the hostname. Wellllll... I got in a rush and executed the snippet with the hostname field blank. That had the effect of running the action on EVERY. SINGLE. VM. that contained a snapshot. I caught it quickly... but not before I had reverted the snapshot for the VM that contained our production instance of Confluence. π³
Thankfully, the new Atlassian architect had things configured well. We lost about an hour's worth of data, but that's it. We were back online in about 15 minutes.