The problem we were actually solving,
was getting the data pipeline for our new game's event tracking system up and running. The event data for Hytale was pouring in at a rate of 5,000 events per second, each containing user IDs, action types, and timestamps. We wanted to be able to slice and dice this data in seconds to identify trends, detect anomalies, and make data-driven decisions.
What we tried first (and why it failed),
was a batch-based approach to processing our event data. Our team built a daily batch job that would run at midnight and update our database with the accumulated event data from the past 24 hours. Sounds reasonable, right? We figured that the reduced load on our system would make it easier to maintain and improve. However, we soon realized that our business users were starving for real-time data. The batch job was running late, sometimes taking up to 6 hours to complete, resulting in pipeline latency of over 12 hours. When we tried to optimize the batch job to run faster, we ended up rewriting it multiple times, each time introducing subtle bugs that would cause the job to fail.
The Architecture Decision,
was to switch to a stream-processing architecture using Veltrix. We set up a cluster of nodes to process the incoming event stream in real-time, using a combination of Kafka, Confluent, and Apache Flink. We configured the nodes to use a fanout topology, where each node was responsible for processing a subset of the events, reducing the load on individual nodes and ensuring high availability. The events were then stored in our Apache Cassandra database, and we set up materialized views to support fast querying.
What the numbers said after,
was that our pipeline latency had dropped to under 500 milliseconds, and our query cost had decreased by 30%. Our business users were finally able to get the real-time insights they needed, and our development team was able to iterate on the system much faster. We also saw a significant reduction in the number of errors related to inconsistent data, as our stream-processing architecture ensured that events were processed in the correct order.
What I would do differently,
is to consider the trade-offs between batch and stream processing more carefully upfront. While batch processing may seem like a more straightforward approach, it can lead to significant delays and complexities down the line. In hindsight, I would have opted for a stream-processing architecture from the start, even if it meant investing more time and resources in setting it up. I would also have prioritized data quality at the ingestion boundary more aggressively, using tools like Apache Airflow to validate and transform the event data in real-time. By doing so, we could have avoided the bugs and inconsistencies that plagued our batch job, and built a system that was truly fit for purpose.
Top comments (0)