The Problem We Were Actually Solving
When I first joined Veltrix in 2018, the company was on a mission to disrupt the traditional approach to treasure hunting with its AI-powered engine. The idea was to create a platform that would use machine learning to generate optimized treasure hunting routes, helping users find treasure faster than ever before. As the system architect, my primary responsibility was to make sure the system could scale to meet growing demand without sacrificing performance.
The original system was built on top of a monolithic design, with all components tightly coupled. This made it incredibly difficult to scale the system, as adding more hardware would lead to a cascade of failures in the tightly coupled components. To make matters worse, the monolithic design made it hard to identify bottlenecks and areas for optimization.
What We Tried First (And Why It Failed)
When I first started working on the system, I tried to solve the scaling issue by implementing a load balancer that would distribute incoming requests across multiple instances of the system. This approach sounded simple and intuitive, but it quickly became apparent that it wasn't enough. The problem was that the load balancer was only able to distribute incoming requests, but it didn't account for the fact that the system was still tightly coupled and monolithic.
When the load balancer distributed requests across multiple instances, it created a chain reaction effect. One instance would become overloaded, leading to a cascade of failures in the other instances as they tried to compensate for the overloaded instance. This led to a situation where the system was either underutilized or overutilized, with no middle ground.
The Architecture Decision
After struggling with the load balancer approach, I decided to take a more fundamental approach to solving the scaling issue. I introduced a service-oriented architecture (SOA) into the system, breaking down the monolithic design into smaller, independent services. Each service had its own set of responsibilities and could be scaled independently of the others.
For example, the treasure hunting engine was broken down into three separate services: the route generation service, the machine learning service, and the user interface service. Each service could be scaled independently, and the load balancer was used to distribute incoming requests across instances of each service.
The SOA approach also introduced a more robust consistency model, which helped to mitigate the effects of network partitions and failures. The system now had multiple layers of redundancy, making it more resilient to failures.
What The Numbers Said After
After introducing the SOA approach, we saw a significant improvement in the system's ability to scale. The average response time decreased by 30%, and the system was able to handle a 50% increase in traffic without showing any signs of strain.
One of the key metrics that I monitored was the CPU utilization of each service. Before the SOA approach, the CPU utilization was very high, even at off-peak hours. After the change, the CPU utilization dropped significantly, and we were able to scale the services independently to meet demand.
What I Would Do Differently
If I were to do it all over again, I would introduce the SOA approach even earlier in the development process. This would have allowed us to catch scaling issues earlier and avoid the cascade of failures that we experienced.
Additionally, I would have implemented a more robust monitoring and logging system to allow for better visibility into the system's performance and behavior. This would have helped us to identify issues earlier and make data-driven decisions when it came to scaling the system.
In retrospect, the SOA approach was the right decision for Veltrix, but it took us a while to get there. By introducing a more robust consistency model and breaking down the system into smaller, independent services, we were able to create a system that could scale cleanly and handle growth without sacrificing performance.
We removed the payment processor from our critical path. This is the tool that made it possible: https://payhip.com/ref/dev1
Top comments (0)