Over the years, when someone crashes something at work I (semi-jokingly) say:
If you don't crash something every so often, you aren't working hard enough.
And while that might not be 💯 percent correct, I feel it helps put the person at ease, especially if the person is new to the company. The truth is, it has and will happen to everyone.
So, have you 💥 crashed 💥 something important? Or brought the network to its knees? I'd ❤️ to hear about it.
Oldest comments (22)
In my early days, I learned (...eventually...) that complicated SQL queries with incomplete JOIN conditions on large data sets in production = non-responsive server = angry customers.
Fast forward to today, I insist that the IT department provide a dev/staging server which mimics production knowing full well that a runaway query (or never-ending loop, or recursive function with a misplaced terminal condition, or ...) can happen to anyone from time to time. Even the best of us! :)
We've got an old records system on an AS/400 that needed the data migrated. Short of writing a fully custom app to allow editing (due to the legal need to provide redaction capability forever), we needed some kind of front end. We have a robust document management system that provides all of the retention, security, search, redaction, and managed access features that we need. We looked at a ton of options, and this was the best fit.
So, I set about turning the data into a series of reports and feeding them into the system using their SDK. The first iteration just about killed the report server. At that point I learned how to generate reports locally on my batch machine, and how to do it in parallel. The new process resulted in the queues on the server being filled up, and the resources being saturated. Everything ground to a halt. My next lesson was in tuning the import process to not be so hard on the servers. The next hurdle was, in importing a couple of hundred million files, the SAN hosting the application's storage ran out of inodes. Every server hitting that storage location had a fit. There wasn't a lot I could do about that, but after the operations team cleared the issue, add that to the list of things I know to think about for the future.
I learned about a lot of neat things during that project. It gave me a ton of great lessons that I wouldn't have learned if stuff hadn't have broken.
My body
Brought down a couple warehouses for companies you've definitely heard of because our deployment strategy was to TeamViewer to customer site > Drop patched dlls. Not much QA there.
Filling the disk! Logging to a file in a tight loop will generate more output than you might think. I once logged to my home directory which was network mounted, next thing you know the server was offline and no one could login in. Doh!
Yeah...about 5 minutes before a demo at my company, I thought I could fix a simple SQL call that was causing so invalid data. It worked on my machine fine. But when I went to push it to the server, something went wrong and it broke the demo. The code was fine but the deployment failed. Took 15 minutes into the meeting to figure out what went wrong and fixed it. Whoops. haha.
I know that temptation, trying to squeeze in as much as possible into the demo.
I was fixing a friends laptop and bricked it... twice. I have the same thing to my computer (with my computer the power went out during an update only for a second but it still bricked it, it went out again when i was reinstalling linux).
I once messed up the key used for a particular cache. The system worked fine with 1-2 users testing it. The first morning after going to production everything went haywire. The users were getting error messages on every screen, regardless of whether they were using the affected feature or not. Productivity ground to a halt. That was Friday. I was out of town Thursday through the following Wednesday.
I was trying top delete a set of scripts in a subdirectory and typing faster than I thought led me to delete all of the scripts in our build server. I stood up immediately after realizing and walked over to the most senior developer on our team to tell him, and he said that it was no big deal. It was rsync'd to the secondary server and he could bring the scripts back with a vintage of just last night.
Many 💥, way too often.
Working directly in production databases is rare (and still a terrible idea) these days, but it used to be status quo.
Let's just say that I still tend to write my "WHERE" clauses first when writing SQL.
:)
The service that Baylor hospital systems use to estimate cost of procedures, some storage arrays, the custom hardware that my company makes (frequently, but not in the field), a motorcycle.
I am guilty of the old Update without a "Where" statement on a accounting production database (there was no staging environment at all).
And working previously on IT (more hardware stuff), I plugged a computer to the network, and somehow it caused an IP conflict between the computer and a software license server, which crashed the software on every computer in the network. That one caused lots of mayhem all through the building.
Seeing as my primary area of coding is vulnerability scan automation I tend to crash a lot of things. Central web services, DNS servers, Document Repo's, Data taking systems. I've killed them all. Each time I do it I have to spend a lot of time trying to figure out how to be kinder and more gentle to the systems on the network. My Favorite is scanning for anonymously writable FTP daemons. I get a lot of complaints from users that their printers are spitting out lovely notes from me.
I learned myself Entity Framework half a year ago because I really got tired of the way we did queries in my workplace's main product. Seems easy enough, but... Lots of fun when a seemingly innocent LINQ query refuses to do a simple join and instead fire of one query for every related item. And it doesn't slow the application down enough to be noticeable. Well, until our main products peak time in the week.. Server memory usage:📈 Our app:💥 Me that day:😓
But the occasional EF weird-stuff beats having to deal with stored procedures and stringbuilder-made query strings.
I saw this earlier and thought I was just going to read along and nod to some relatable content. Then visual studio hiccuped and I watched the entire bin folder start throwing errors because vs thought half the files were deleted. Somehow the project builds without them but it won't publish and I wasted enough time trying to fix it that I could have recreated the whole project