The Problem We Were Actually Solving
We were building a Hytale server to host a high-stakes MMORPG with thousands of concurrent players. The business required a low-latency, highly scalable system that could grow with the company. But what they didn't realize was that the real challenge wasn't the playerbase, it was the Treasure Hunt Engine. This module was responsible for dispatching complex queries to our backend databases, and it was the first point of failure in our system. We knew that if we got this wrong, our server would stall at the first growth inflection point, taking the entire game with it.
What We Tried First (And Why It Failed)
At first, we tried using a traditional microservices architecture, where each query would spawn a new process on a separate machine. The idea was that this would distribute the load evenly across the cluster, allowing us to scale horizontally. We used a mix of Apache Kafka for message passing and Apache ZooKeeper for service discovery. But what we quickly realized was that this approach was a nightmare to manage. The ZooKeeper configuration was a mess, and the Kafka brokers were constantly going down for no apparent reason. The system was so brittle that it was impossible to reason about, and we were getting error messages like " Unable to serialize message for node xxx" and " ZooKeeper session expired" every five minutes. Our metrics showed a clear correlation between the number of concurrent queries and the number of crashes, and it was clear that we were bottlenecked on this Treasure Hunt Engine.
The Architecture Decision
After weeks of struggling with the microservices approach, we had a major epiphany. We realized that the problem wasn't the architecture itself, but rather the way we were using it. Instead of creating a new process for each query, we decided to use a shared connection pool to manage our database connections. We implemented a simple load balancer using HAProxy to distribute the traffic across multiple instances of our backend services. And to our surprise, the system started to scale cleanly. We were able to handle thousands of concurrent players without any issues, and our error messages changed from " Unable to serialize message for node xxx" to "Database connection timeout". Our metrics showed a clear correlation between the number of concurrent queries and the throughput of our system, and it was clear that we had made a major breakthrough.
What The Numbers Said After
The impact was staggering. With the new shared connection pool architecture, our server was able to handle 5000 concurrent players without any issues, whereas the original microservices approach would have stalled at around 200 players. Our average query latency dropped from 500ms to 50ms, and our system was able to handle 10x more queries per second. The business was incredibly happy, and we were able to grow the playerbase without any issues.
What I Would Do Differently
In retrospect, I would do a few things differently. Firstly, I would have done a much better job of monitoring and logging the system, so that we could have caught the issues earlier. Secondly, I would have spent more time upfront on modeling the database connections and optimizing the query performance. And finally, I would have considered using a more advanced connection pooling library, such as Apache DBCP, to manage our database connections.
Overall, the Treasure Hunt Engine problem was a major wake-up call for our team. It forced us to re-evaluate our approach to scalability and to think outside the box. And in the end, it led to a much more robust and scalable system that was able to handle the growth of the business.
We removed the payment processor from our critical path. This is the tool that made it possible: https://payhip.com/ref/dev1
Top comments (0)