The Problem We Were Actually Solving
We were trying to build an engine that could handle thousands of concurrent users searching for treasures across 1000 acres of virtual landscape. Our initial tests showed a significant increase in latency as the number of users grew, which sparked a heated debate about optimisation.
What We Tried First (And Why It Failed)
Our first attempt was to shard our database across multiple instances, thinking that horizontal scaling would solve our problems. We spent weeks setting up a complex load balancer, routing traffic to separate database instances, and configuring distributed transactions to keep data consistent. However, our tests showed a 30% increase in latency due to the additional overhead of load balancing, connection pooling, and eventual consistency checks.
The most interesting metric to emerge from these tests was the "database latency ratio," which measured the average time it took for a query to complete relative to the number of concurrent users. The ratio skyrocketed from 0.1ms to 1.2ms, indicating that our sharding approach was not only failing to improve performance but also introducing new bottlenecks.
The Architecture Decision
After reviewing the results, we decided to take a different approach. We opted for a caching layer, using Redis to store frequently accessed data, such as treasure locations and player positions. This allowed us to offload a significant portion of the database load and reduce the number of queries needed to retrieve critical information.
We also implemented a consistency model that prioritised eventual consistency over strong consistency, trading off some accuracy for improved performance. This decision was not taken lightly, but our tests showed a 50% reduction in latency and a 20% increase in concurrent user capacity.
What The Numbers Said After
Our new approach was a resounding success. The treasure hunt engine was able to handle over 5000 concurrent users without significant latency spikes. The "database latency ratio" dropped from 1.2ms to 0.3ms, indicating that our caching and eventual consistency strategies were working as intended.
The most telling metric, however, was the "player drop-off rate," which measured the percentage of players who disconnected from the game due to lag. This rate plummeted from 12% to 4%, indicating that our optimisation efforts had significantly improved the player experience.
What I Would Do Differently
In retrospect, I wish we had spent more time understanding the root causes of our performance issues before diving into optimisation. Our initial tests were flawed, and we ended up optimising the wrong things, which ultimately made our problems worse.
If I were to redo our optimisation efforts, I would have started with a more thorough analysis of our system architecture, using tools like the Unix netstat command and the Linux perf tool to identify bottlenecks and hotspots. I would have also involved our database administrators earlier in the process to ensure that our caching and sharding strategies were aligned with their expertise.
Most importantly, I would have taken a more nuanced approach to optimisation, balancing performance gains with the added complexity of our system. Premature optimisation can be a slippery slope, and it's essential to strike a balance between speed and simplicity.
We removed the payment processor from our critical path. This is the tool that made it possible: https://payhip.com/ref/dev1
Top comments (0)