The Problem We Were Actually Solving
We had just launched a new live scoring feature for an online sports tournament, and the feedback was overwhelming. Thousands of concurrent users were flooding our system, causing lag, and we couldn't handle the load. The root cause was a classic – our event processing service was running out of CPU resources, bottlenecking the entire pipeline. We knew we needed to scale, but our current configuration was holding us back.
What We Tried First (And Why It Failed)
During the initial design phase, we opted for a simple, batch-oriented approach. We thought it would be easy to scale by just adding more instances, but we were wrong. Our batch window was 10 seconds, which meant our system would only process events in batches, leading to a constant queue buildup. When traffic surged, our system became overwhelmed, and the latency shot through the roof. We tried to boost the processing power, but it only led to wasted resources and higher costs. Our attempts to scale by replicating the service across multiple AZs only masked the problem temporarily.
The Architecture Decision
We decided to shift towards a more streaming-oriented architecture, using Apache Kafka as the message broker and Apache Flink as the event processing engine. We implemented a distributed architecture, where the service is running across multiple instances, each handling a portion of the load. We also introduced a rate limiter to prevent overloading the system during traffic surges. This change allowed us to scale more efficiently and handle the load without overwhelming our resources.
What The Numbers Said After
After the migration, our average pipeline latency decreased from 40 seconds to 5 seconds, and our query cost dropped by 75%. Our 95th percentile latency also improved significantly, from 120 seconds to 15 seconds. We achieved these improvements while keeping our costs relatively low, with a 25% reduction in resource utilization.
What I Would Do Differently
In hindsight, I would have prioritized a streaming-oriented architecture from the start, given the nature of our workload. If I were to do it again, I would also focus on designing a more robust monitoring and alerting system to catch these scalability issues earlier. Additionally, I would have implemented a more granular rate limiter to prevent overloading the system during traffic spikes. With these changes, we could have avoided the bottleneck and delivered a better experience for our users.
Top comments (0)