Designing a Treasure Hunt Engine for Event Data

The Problem We Were Actually Solving

When we first started designing the treasure hunt engine, our primary goal was to ensure that every event was processed within a 5-second latency window. This translated to an average pipeline latency of 3 seconds, with a maximum allowed latency of 10 seconds. We had a warehouse cost optimization goal of reducing our monthly expenses by 20% while maintaining the same level of query performance.

What We Tried First (And Why It Failed)

In our initial design, we opted for a batch processing approach, thinking it would simplify the architecture and reduce costs. We used Apache Beam to process events in 1-minute batches, and then loaded the processed data into a Redshift warehouse using Amazon Glue. While this approach did achieve our latency goal, it led to a few issues. First, the batch processing delayed our event processing by at least 5 minutes, causing our treasure hunt engine to respond sluggishly to real-time events. Second, the batch processing created a high volume of temporary data in our Redshift cluster, resulting in storage costs that skyrocketed by 30%. We were struggling to meet our warehouse cost optimization goal.

The Architecture Decision

After realizing the limitations of batch processing, we decided to switch to a streaming architecture using Apache Kafka and Apache Flink. We designed a pipeline that ingested event data in real-time, processed it using Flink, and then stored the results in a low-latency database like Amazon Aurora. This approach allowed us to meet our latency goals and significantly reduce our storage costs. We achieved an average pipeline latency of 1 second, with a maximum allowed latency of 5 seconds. Our monthly storage costs decreased by 40%, allowing us to meet our warehouse cost optimization goal.

What The Numbers Said After

After implementing the streaming architecture, we achieved the following numbers:

• Average pipeline latency: 1 second
• Maximum allowed latency: 5 seconds
• Monthly storage costs: $30,000 (down from $50,000)
• Query cost: $50 per hour (up from $40 per hour due to increased data volume)
• Freshness SLA: 99.99% (meeting our goal of processing all events within a 5-second latency window)

What I Would Do Differently

If I were to design the pipeline again, I would focus on data quality at the ingestion boundary. In our initial design, we relied on a simple schema validation to catch errors. However, this approach failed to catch a few critical errors that caused our pipeline to fail. I would implement a more robust data validation mechanism, using tools like Apache Airflow or Amazon Glue to catch errors before they reach the processing stage. This would help us avoid a costly rollback and reprocess of entire batches of data.

In conclusion, designing a treasure hunt engine for event data requires careful consideration of tradeoffs between latency, cost, and data quality. By learning from our mistakes and iterating on our design, we were able to meet our goals and create a high-performing system that's both efficient and robust.