King Yemma and KY are pseudonyms. I have changed or generalized names and a few identifying details to keep internal information private.
Big Howdy,
I want to write about a project that, on paper, went pretty well.
The Project
Our server fleet had a major problem with dirty reboots. Think “Blue Screen of Death,” but on a much more expensive computer. Sometimes it happened when testing servers before putting them into production, which was annoying but manageable. Other times it happened in production while a customer was using the server, which was considerably less manageable and considerably more expensive for everyone involved.
The fleet health team was already investigating these failures, but there were a lot of them and the logs were a pain to work through. The servers are wildly complex, meaning there could be hundreds of reasons for one to crash like this.
To scale to the amount of work, I was assigned to onboard our servers to an internal tool called King Yemma. Dragon Ball reference, anyone? From here on out, I am calling it KY.
I was the only person from my team assigned to this work, so I worked directly with both the team that owned KY and the fleet health team.
The idea was simple enough: KY would pull the relevant logs for a dirty reboot, look for known failure patterns, and categorize the ticket. If it found a known issue, it could close the ticket automatically. If it found something new, it could help point the investigation in the right direction by tagging the ticket based on other rules in KY. The original team that built it was already using it with great success.
There was one problem: KY had not been built for our servers.
Getting KY Working
I worked with the team that owned KY to make it understand our systems. I added our log groups, pointed it at the right resources, and taught it where to find our dirty reboot tickets. I also added a feature to parse the logs and pull out the component that appeared to have failed, which was one of the requests that came down while I was working on it.
After that, I started working with fleet health to add fingerprints.
A fingerprint was basically a known pattern in the logs that mapped to a known failure. Generally, this meant doing a regex or pattern match on one or more log lines. The goal was to get enough of these into KY that it could automatically close tickets for problems that had already been seen and understood. If it did not have an exact match, KY could at least add tags to the ticket to point engineers in the right direction.
This meant I had to spend a lot of time looking through dirty reboot logs. Don’t get me wrong, the fleet health team worked with me and provided fingerprints when they found things too. It is not like they simply expected a software engineer to suddenly start doing a fleet health or hardware engineer’s job overnight. However, I still needed to do my part to speed up development.
I would find something in the logs that looked suspicious, ask someone from the fleet health team whether it was actually important (it rarely was), and then repeat. Turns out there are a lot of weird warnings and error messages that pop up during runtime that are apparently... completely meaningless?
Still, I got KY working on our systems. Together with the fleet health team, I added numerous fingerprints that could automatically close tickets. It also helped identify new buckets of failures for the team to investigate.
KY had another useful feature too. Someone could give it a server and a time range, and it would pull the relevant logs into one place on their local machine. Then they could use grep, awk, sed, or whatever else they wanted without waiting on CloudWatch or manually jumping between log groups. This meant KY could be useful for more than just fleet health. Everyone in the org could use it for their own debugging.
I cannot take credit for implementing that feature, but I CAN take credit for exposing it in the CLI for a normal user to call. I even gave a short presentation about it to the org.
At this point, it sounds like a smashing success, right?
If only it actually was...
So Much Work, So Little Value...
The actual goal for this whole boondoggle was to get the fleet health team to adopt KY as part of their normal investigation process for dirty reboots. The goal was to reduce the time they spent digging through logs and make the whole investigation process less painful and more automated. After all, there were only a handful of fleet health engineers and literally thousands of servers.
Unfortunately, they thought it was only slightly better than useless and decided manually searching through CloudWatch was faster.
And, honestly... they were right.
KY ended up being useful mostly after a problem had already been diagnosed. It could automatically close tickets for known failures, which was helpful, but it was not the investigation tool we all hoped it would become.
I had built something that worked well, but I had not built something my customer wanted to use.
What I Got Wrong
Looking back, my mindset, along with my lack of awareness, was a major part of the problem.
First, I did not understand our system very well at the time, and I did not understand the customer use case or how the fleet health team operated nearly as well as I thought I did.
Second, I was under a lot of organizational pressure at the time. I thought I had made a mistake that severely affected my standing, and was worried that KY was a test of whether I could recover. That fear turned out to be mistaken, but it changed how I approached the work.
As a result, I honestly hated this project. I hated reading logs that I did not understand. I hated programming Python when my normal language was Lua, since the languages are just similar enough that I would constantly confuse the syntax (this was before the days of AI). It frustrated the hell out of me every day.
But I wanted to perform well, show progress, and get to something I could point at and say, “look, I did the work.” Mostly, I just didn't want to get fired, which I was convinced would happen if I so much as sneezed too loudly.
The easiest way to do that was to add fingerprints. More fingerprints meant more tickets could be categorized automatically. More fingerprints meant more pull requests for the codebase. More fingerprints gave me a metric.
The problem was that the metric did not matter.
The fingerprints the fleet health team actually needed were often more complicated than simple log matching. Sometimes they depended on values that had to be decoded or interpreted in a particular way. Sometimes the decoded value had to come from a very specific component to have meaning. Sometimes specific values needed to come from multiple specific components within a specific time range.
The more useful work would have been making KY flexible enough for fleet health to create and manage those fingerprints themselves. I did some of that, but it was clearly secondary to adding fingerprints.
I was trying to do their job for them instead of building the thing that would help them do their job better.
I also did not understand the fleet health team’s workflow well enough. I talked to them, got feedback, and worked with them on fingerprints, but I was still mostly seeing their work from the outside. I understood the inputs and outputs more than I understood the actual process, or the system itself.
I should have spent more time sitting with them while they investigated failures. I should have asked my manager to loan me to the team for a bit so I could understand how they debugged, what information they needed first, and where the existing process was actually slow.
Instead, I focused on the work that felt safest.
What I Learned
There was also a more personal lesson in it for me. At the time, I was still learning how to operate in an environment where I had no solid footing. There was a lot of information, a lot of system complexity, and no obvious answer. These servers were complicated enough that a single machine was basically its own distributed system. There were hardware blocks, firmware, software, logs from different sources, and a bunch of interactions I did not understand yet, many of which I never figured out.
When it came to logs, I would sometimes feel so overwhelmed that I would shut down and not see the obvious line saying, “Error! I’m literally the root cause!”
When I felt overloaded, I had a tendency to narrow my focus until I found something concrete that I could make progress on. In this case, that became fingerprints.
Fingerprints were real work. They were useful. But they were also safer than going deeper into the uncomfortable question of whether KY itself was solving the right problem.
The project taught me a lot about our systems, and that knowledge was extremely useful later when I was writing tests and assisting with, and eventually driving, future investigations. It also gave me better relationships with both the KY team and the fleet health team, so I did gain a lot from it.
However, it was still a failure. After two months of work, I had nothing enduring to show for it because it was quickly forgotten. Even the log pulling feature was never really adopted after I gave that presentation on it.
KY failed to accomplish its most important goal because I optimized for what I could measure instead of what the customer actually needed. I was so worried about proving that I was doing good work that I stopped asking whether I was doing the right work.
Top comments (0)