For further actions, you may consider blocking this person and/or reporting abuse
For further actions, you may consider blocking this person and/or reporting abuse
anna -
Sukhpinder Singh -
Joaquin Diaz -
Shafayet Hossain -
Top comments (17)
Ever deleted production data?
😬
drop database in production 😂
Restarting a service which was reporting unhealthy, to then notice the restart caused a cascade of problems right through the stack. Cold sweat and panic, especially as end users started throwing ticket after ticket around x y z not working.
A literal, turn it all off and back on again until we could work through to find the issue. Turns out a memory leak caused by an update to a library was the problem with the unhealthy service, but by restarting we then had a lot of traffic hitting a couple of other areas of the stack which were unable to keep up, normally this would have been handled perfectly fine, but they also suffered from the same leak and the extra traffic sent them over the edge much quicker.
We once had a bug on a financial application, where transaction amounts were calculated incorrectly. People were receiving more money than they should, hopefully it was a quick fix and no money was lost, but had me panicking!!!
I'm a tmux user and I always have 3 windows open: "local, production and random".
Thinking I was on "local", I was spamming some stuff (bash/apt installs) only to realize to my horror that I was on production and actually connected to a server.
Luckily nothing really happend but it was a very scary 20-minute confirmation check.
Back in 2000, fixing the Y2K bug on a big bank while the country was adopting the Euro, we had a migration project. The goal was to fix the Y2K and the project managers decided to do the migration to the Euro at the same time since they had the resources in the field. What can go wrong, right? So we had this huge cobol scripts that was suposed to do the exchange. 3 step operation, collect all money from all accounts, exchange the values, and deposit afterwards. So it collects fine, exchanges fine and it breaks. For 4 days no one with account in the bank (and this was a big bank) had available funds to use.
My story: I was told to update a system in production, we do it via SCP or SSH (we didn't know about git at the time), but this was the only client whose server was not local and was uploading files from our server to their server external, it took 1h: 40min (from local to local it was less than 5min) so they could not work during that time 😬. To workaround I started uploading to a different folder and then doing a local to local copy in production which took seconds without interrupting, then I learned git and thought "this could help a lot"
A story from a friend: He started at the same job as me, he was making a copy of a production database on our local test server, so he made a dump file and then ran the file. One day he did not check which server he was logged in to and inadvertently ran the file that dropped the entire production database 😳 fortunately it was a recent backup and the client lost only 1 hour of work, we called and informed them about the problem, so they had to re-record the missing information.
First time deploying to production, first time fixing an issue with prod without my more experienced colleagues available. Basically anything on prod makes me nervous still.
And realising the script you wrote is the reason the servers are down. Not a fun time but at least I learnt a lot from it.
Not exactly a developer story but it's about being scared. I was scared everyday for a long time that I was "leaving myself behind" if that makes any sense. Everyone's just better, more educated, smarter, etc. Then you wake up one day and realize you are your only competition and, just like all those "better" people did, you got to sit down at your keyboard and fall in love with learning and knowledge.
I'll still probably be really scared if I mess with production environments by mistake though :)
I used to be scared of any task, until I learned to enjoy it. Now if it does not work I know Im going to love it when it does :D
When you come back from your honeymoon break to the news that the lambda service you hooked up for the legal department has been erring out for, at least, a month, and no one noticed.