DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’» is a community of 966,155 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Sherry Day
Sherry Day

Posted on

What's the most scared you've ever been as a developer?

Top comments (17)

Collapse
 
ben profile image
Ben Halpern

Ever deleted production data?

Collapse
 
sherrydays profile image
Sherry Day Author

😬

Collapse
 
ariprayoga66 profile image
Ari Dwi Prayogo

drop database in production πŸ˜‚

Collapse
 
imthedeveloper profile image
ImTheDeveloper • Edited on

Restarting a service which was reporting unhealthy, to then notice the restart caused a cascade of problems right through the stack. Cold sweat and panic, especially as end users started throwing ticket after ticket around x y z not working.

A literal, turn it all off and back on again until we could work through to find the issue. Turns out a memory leak caused by an update to a library was the problem with the unhealthy service, but by restarting we then had a lot of traffic hitting a couple of other areas of the stack which were unable to keep up, normally this would have been handled perfectly fine, but they also suffered from the same leak and the extra traffic sent them over the edge much quicker.

Collapse
 
nombrekeff profile image
Keff

We once had a bug on a financial application, where transaction amounts were calculated incorrectly. People were receiving more money than they should, hopefully it was a quick fix and no money was lost, but had me panicking!!!

Collapse
 
ricardovfreixo_84 profile image
Ricardo

Back in 2000, fixing the Y2K bug on a big bank while the country was adopting the Euro, we had a migration project. The goal was to fix the Y2K and the project managers decided to do the migration to the Euro at the same time since they had the resources in the field. What can go wrong, right? So we had this huge cobol scripts that was suposed to do the exchange. 3 step operation, collect all money from all accounts, exchange the values, and deposit afterwards. So it collects fine, exchanges fine and it breaks. For 4 days no one with account in the bank (and this was a big bank) had available funds to use.

Collapse
 
linaran profile image
Deni

I'm a tmux user and I always have 3 windows open: "local, production and random".
Thinking I was on "local", I was spamming some stuff (bash/apt installs) only to realize to my horror that I was on production and actually connected to a server.

Luckily nothing really happend but it was a very scary 20-minute confirmation check.

Collapse
 
heatherw profile image
HeatherW

First time deploying to production, first time fixing an issue with prod without my more experienced colleagues available. Basically anything on prod makes me nervous still.
And realising the script you wrote is the reason the servers are down. Not a fun time but at least I learnt a lot from it.

Collapse
 
gregorgonzalez profile image
Gregor Gonzalez

My story: I was told to update a system in production, we do it via SCP or SSH (we didn't know about git at the time), but this was the only client whose server was not local and was uploading files from our server to their server external, it took 1h: 40min (from local to local it was less than 5min) so they could not work during that time 😬. To workaround I started uploading to a different folder and then doing a local to local copy in production which took seconds without interrupting, then I learned git and thought "this could help a lot"

A story from a friend: He started at the same job as me, he was making a copy of a production database on our local test server, so he made a dump file and then ran the file. One day he did not check which server he was logged in to and inadvertently ran the file that dropped the entire production database 😳 fortunately it was a recent backup and the client lost only 1 hour of work, we called and informed them about the problem, so they had to re-record the missing information.

Collapse
 
xarop_pa_toss profile image
Ricardo Giro

Not exactly a developer story but it's about being scared. I was scared everyday for a long time that I was "leaving myself behind" if that makes any sense. Everyone's just better, more educated, smarter, etc. Then you wake up one day and realize you are your only competition and, just like all those "better" people did, you got to sit down at your keyboard and fall in love with learning and knowledge.
I'll still probably be really scared if I mess with production environments by mistake though :)

Collapse
 
patracio profile image
Patracio

Deploy an ECS in production ... with develop properties.

Collapse
 
alumezi profile image
Arbnor Lumezi • Edited on

I used to be scared of any task, until I learned to enjoy it. Now if it does not work I know Im going to love it when it does :D

Collapse
 
xanagandr profile image
Guillermo Saldivar

When you come back from your honeymoon break to the news that the lambda service you hooked up for the legal department has been erring out for, at least, a month, and no one noticed.

Collapse
 
arham_abiyan profile image
Arham Awal Abiyan

Changing staging config to production config

Collapse
 
weaponsforge profile image
Weapons Forge

Days of extreme mental block: the days when my hands refuse to code what my mind wants.

Collapse
 
mrdulin profile image
official_dulin • Edited on

996 ICU + 251

Collapse
 
riadhoq profile image
Riadul Hoque

AWS terminating the admin server where your build server lives for no apparent reason, and you're the sole developer/devops! That was a sleepless night.

🌚 Life is too short to browse without dark mode