The Problem We Were Actually Solving
The real problem wasn't just the sheer volume of events - it was that we needed to route 80% of them to a specific service without incurring a massive latency penalty. The trouble was that our batch processing pipeline was a 10-minute job that kicked off once an hour, and it was causing all sorts of issues with our game state.
What We Tried First (And Why It Failed)
We tried building a batch-to-streaming ETL that would buffer up events from the past hour and then push them to the service in one big batch. Sounds great, right? But what we didn't realize was that the buffering was causing a 5-minute delay, and our service was complaining about events that were stale by the time they arrived. We also ended up with a massive 20 GB file that was taking ages to process - it was a recipe for disaster.
The Architecture Decision
We decided to go all-in on streaming and created a Kinesis stream that received all 10,000 events per minute. We then built a lambda function that consumed from that stream and immediately routed the events to the right service. It was a dramatic change from the batch approach, but it paid off - we reduced latency by 95% and events were now arriving at the service in real-time.
What The Numbers Said After
Our query cost on Athena decreased by 75% because we weren't trying to process 20 GB files hourly anymore. Our pipeline latency went from 10 minutes to just 100 milliseconds - a 99.95% reduction. And on top of that, our user satisfaction metrics skyrocketed - players were seeing game state updates happen in real-time, which made for a better user experience.
What I Would Do Differently
Next time, I'd definitely prioritize data quality at the ingestion boundary - we had a nasty case of event duplication that we only discovered weeks after the new system went live. It was a costly error that required some serious duct tape and prayer to fix. I'd also experiment with more advanced routing rules to minimize the number of events being sent to the service - it's still a 10 GB file every hour, after all.
Top comments (0)