The Problem We Were Actually Solving
As we looked at the search data, we realized that most of the operator pain was centered around three specific areas: Event Order, Message Broadcasting, and Data Modeling. It seemed like the moment operators thought they had it right, something else would go awry. Our own metrics showed that it was taking operators three to five iterations to get a working configuration, with each iteration taking upwards of an hour.
What We Tried First (And Why It Failed)
Our first instinct was to create an interactive validator for Hydra. We had seen other systems use this approach to great success, and we thought it would take care of the problem. We spent several developers-months crafting a beautiful GUI with fancy auto-completion and validation rules. However, when we deployed it to production, we were met with a mix of indifference and frustration. Operators just didn't use it. We tried to mandate its use, but this only led to the validator and the configuration system becoming a bottleneck.
The Architecture Decision
We decided to take a step back and rethink our approach. We realized that Hydra was fundamentally a text-based system, and our operators were used to tweaking text files in other parts of the system. With that in mind, we refactored Hydra to be more text-friendly. We introduced a new configuration format, based on flat files, that made it easier for operators to understand and modify the configuration. We also started providing more explicit error messages and logging information, so that operators could diagnose their own issues. Lastly, we created a set of automated unit tests for Hydra, which helped us catch configuration issues before they made it to production.
What The Numbers Said After
After the refactoring, we saw a significant decrease in configuration-related issues. Our metrics showed that it was now taking operators only one to two iterations to get a working configuration, with the time spent on each iteration dropping by 30%. Our support tickets related to Hydra dropped by 50%, and our operators were able to deploy new features without waiting for someone to validate their configurations.
What I Would Do Differently
In hindsight, I would have taken a more radical approach to the configuration system from the start. I would have created a system that was designed for gradual, incremental configuration changes, rather than massive, all-or-nothing deployments. This would have avoided the need for a validator and the associated pain points.
We removed the payment processor from our critical path. This is the tool that made it possible: https://payhip.com/ref/dev1
Top comments (0)