DEV Community

Discussion on: What's the most scared you've ever been as a developer?

Collapse
 
imthedeveloper profile image
ImTheDeveloper • Edited

Restarting a service which was reporting unhealthy, to then notice the restart caused a cascade of problems right through the stack. Cold sweat and panic, especially as end users started throwing ticket after ticket around x y z not working.

A literal, turn it all off and back on again until we could work through to find the issue. Turns out a memory leak caused by an update to a library was the problem with the unhealthy service, but by restarting we then had a lot of traffic hitting a couple of other areas of the stack which were unable to keep up, normally this would have been handled perfectly fine, but they also suffered from the same leak and the extra traffic sent them over the edge much quicker.