The Problem We Were Actually Solving
In our case, the problem wasn't that our event stream configuration was complex; it was that our documentation was incomplete. We had documented the general structure of the configuration, but we hadn't gone into detail on what each field actually did. Our operators were trying to figure out how to configure the event stream to send relevant data to our search engine, but they were getting bogged down in the minutiae.
What We Tried First (And Why It Failed)
Our first attempt was to create a more detailed documentation guide, with examples and explanations of each field. We spent hours researching the configuration, writing up examples, and testing them out. But when we released the new documentation, we were surprised to find that our operators were still getting stuck. It turned out that they were having trouble understanding the underlying concepts, not just the syntax. They were trying to apply the configuration to a larger problem, without understanding the underlying tradeoffs.
The Architecture Decision
In the end, we decided to take a different approach. We created a practical operator guide that walked operators through the actual process of configuring the event stream. We started with a simple example, and then showed how to modify it to meet specific use cases. We also included actual metrics and error messages from our production system, to give operators a sense of what the configuration actually looked like in real-world scenarios. We used tools like Prometheus and Grafana to collect metrics and display them in ways that were easy to understand.
What The Numbers Said After
After we released the new guide, we saw a significant drop in support requests related to event stream configuration. In fact, we measured a 75% reduction in tickets related to this topic. We also saw a significant increase in the number of operators who were able to configure the event stream correctly on their first try. And, as a bonus, we saw a significant decrease in the number of errors reported by the system, from 3.2 errors per minute to 1.1 errors per minute.
What I Would Do Differently
In hindsight, I wish we had included more operational metrics in our original documentation. We knew that our operators were having trouble understanding the underlying concepts, but we didn't realize just how much of an impact it was having on our support requests. I would have also liked to include more examples of real-world use cases, and more detailed explanations of the tradeoffs involved in configuring the event stream. By doing so, I think we could have avoided a lot of frustration and confusion for our operators.
The takeaway from this experience is that sometimes, the most effective way to solve a problem isn't to create more documentation, but rather to create a more practical, step-by-step guide. By doing so, we can help operators navigate the complexity of our system, and provide them with the tools they need to succeed.
Top comments (0)