The Problem We Were Actually Solving
In our case, the problem was a classic example of the "thundering herd" problem - the Treasure Hunt engine was getting slammed with updates from all 500 players who opted-in for the event. This resulted in latency spikes, timeouts, and a poor overall user experience. We couldn't have Treasure Hunt events at scale without addressing this issue.
What We Tried First (And Why It Failed)
Initially, we thought a simple connection pooling mechanism would solve the problem. We implemented Apache DBCP, and as expected, the latency improved. However, we quickly hit a new bottleneck: the database was still getting hit with too many concurrent requests, even after the connection pool was implemented. The error messages pointed to connection timeouts, and the metrics showed that the Treasure Hunt engine was the culprit. We had to dig deeper.
As it turned out, our implementation of the connection pool was too simplistic, and the pool was getting exhausted after each event iteration. We realized that the connection pooling didn't address the root issue - the Treasure Hunt engine's request patterns were unpredictable and bursty.
The Architecture Decision
After months of experimenting with various solutions, we decided to switch to a message-driven architecture. We implemented RabbitMQ as the message broker and created a separate service, Treasure Hunter, to handle the updates. This service was designed to batch and process the updates asynchronously, reducing the load on the database.
We also implemented a more sophisticated connection pooling mechanism using HikariCP, which allowed us to scale the pool size dynamically based on the actual load. But more importantly, we addressed the root issue - the Treasure Hunt engine's request patterns - by introducing a message-driven architecture.
What The Numbers Said After
The results were astonishing. After deploying the new architecture, we observed a 30% reduction in latency and a 25% increase in overall performance. The metrics showed that the Treasure Hunter service was processing the updates in real-time, without impacting the main server. We could finally host Treasure Hunt events with 5,000 concurrent players without any issues.
What I Would Do Differently
If I were to redo the project, I would focus on designing the message-driven architecture from the start. While our initial implementation was correct, we wasted a significant amount of time and resources implementing a connection pooling mechanism that ultimately proved insufficient. A more modular design would have saved us months of trial and error.
Additionally, I would invest more time in monitoring and analyzing the Treasure Hunt engine's request patterns. Understanding the actual load and patterns would have allowed us to design a more efficient solution from the outset.
We removed the payment processor from our critical path. This is the tool that made it possible: https://payhip.com/ref/dev1
Top comments (0)