Before we start - I'm working on https://cloudash.dev, a brand new way of monitoring serverless apps 🚀. Check it our if you're tired of switching between 50 CloudWatch tabs when debugging a production incident.
(This post has been originally published on cult.honeypot.io
Building software is what we - developers - are paid for.
Unfortunately, more often than not we're also paid to break stuff and we get to enjoy an "amazing" opportunity to fix what we've broken.
We're not talking enough about those stories.
Same as your Instagram feed is full of absolute highlights of other people's lives, the same goes when it comes to stories we hear from other developers - few people are willing to admit that they've broken production third time this week.
On the other hand, I strongly believe that there's a lesson to be learned from every 'fuckup'. Not to mention that behind every rule in your company there might be a story.
"Why do we have a code freeze before major holidays?" - because Mike and Jenny had to spend their entire Christmas Eve migrating the database after a yolo-merge.
"Why can't I push directly to master? I know what I'm doing!" - sure, but one time Andrew basically deleted two weeks of work from the repo by accidentally force pushing an old commit to master (btw, I am not making this up, this actually happened in my career).
"Why there's a warning on my shirt telling me not to iron it while wearing it? Who does this?" - you know the deal, someone did, hence the warning.
With that being said, let me tell you a story of my biggest fuckup from the times I was a junior engineer.
Did anyone order fried motherboards?!
A bit of a background before we move along - at the very beginning of my career in tech I was a Junior Software Engineer at Samsung R&D Center Poland. I was lucky to be paid to build rather unique apps - my team was creating JavaScript applications for ... SmartTVs.
Side note: building apps for TVs is wonderful because there's literally a single resolution you need to care about and we could style entire apps with position: absolute; because why not!
You probably were not aware but SmartTVs have an entire motherboard in them (with a surprisingly good hardware on it - we're talking about multiple core processors and gigabytes of RAM! In a freaking TV!). At this point (back in 2013/2014) hardware was cheaper than software [citation needed].
Sometime in 2013 due to my vast experience with C++ (2 semesters in a university which apparently was enough) was moved to a brand new exciting project - Tizen.
To quote Wikipedia:
Tizen is a Linux-based mobile operating system backed by the Linux Foundation but developed and used primarily by Samsung Electronics.
At this point Tizen was really cutting edge (operating systems under heavy development tend to break all the time) but one day we got a present from HQ.
Three brand new shiny motherboards with the newest Tizen firmware.
🥳🥳🥳
I fried two of them to the point of no return in under an hour
Yes, literally fried the 💩 out of them.
Why?
Well, I was told to perform a system update on those motherboards and to follow the instructions I was given.
Unfortunately the instructions did not take into the account a quirk in the newest system version and performing those steps resulted in a SmartTV motherboard becoming a rather expensive useless piece of plastic and silicon.
After performing a system update on the first board I knew that something funky happened. Did I make a mistake? I had to, crap, what do I do now?
Since I didn't have a lot of experience I decided to simply repeat the steps one more time, this time making sure that I follow the instructions 100%.
Turns out that I did follow them to the letter both times.
And now - I could pretend that I didn't touch those board yet, that they came in broken and honestly everyone would believe me
After all, this was cutting edge stuff, things were supposed to break.
Luckily I decided to tell my team leader that:
- we have a problem
- I was following instructions to the letter
- 2/3 of our shiny new boards are bricked
- the manual needs to be updated as soon as asap because it may affect other departments
Luckily he just chuckled and asked me what exactly caused me to fry a second board immediately after I broke the first one 😂
Lessons learned:
- Take ownership - admit when you've made a mistake, don't try to blame it on others. Acknowledge the failure and try to become a better person/engineer having learned a valuable lesson.
- Raise issues early and clearly. - it's even better to raise an early/false alarm than to be silent when something is clearly broken.
- Follow instructions and documentation, but within reason - documentation ~can~ will get outdated and a software engineer needs to be able to deal with that. Not to mention that it's worthwhile to take some time to make sure that docs are up to date.
- Don't try to hide things that are broken (or, well, suboptimal). Being open with others takes you a long way and positions you as a trustworthy member of the team.
Top comments (1)
Very interesting and "funny" article. Yes, everybody should be more open and tell stories about their fuckups.
We all apparently learn from failures, but we all try to hide them and never share them. (I have a draft since ages about one that happened in my team, but I always wonder if I should, if I can, if it helps... if if if....)
Thanks for sharing