The Problem We Were Actually Solving
What we were actually trying to solve was a mix of freshness SLAs (at least 95% of recommendations needed to be within 5 minutes of accuracy) and high query volumes (we expected a 5x increase in users during peak times). The engine had two main components: an offline ETL process that transformed our clickstream data into a database-friendly format, and an online query service that took user requests and used the data to make recommendations. We had recently switched from batch to streaming data ingestion, which helped reduce our data pipeline latency from 24 hours to 30 minutes. But the query service was still the bottleneck, and we couldn't figure out why.
What We Tried First (And Why It Failed)
Our initial attempt to scale the query service involved throwing more CPU and memory at it. We beefed up our cluster with 2x more nodes and upgraded our worker instances to 16GB RAM. But this only made things worse. Our query cost skyrocketed (up 300% from the previous month), and we started noticing query latency spikes during peak hours. It turned out that the increased resource pressure caused our query service to become even more chatty with the database, resulting in a feedback loop that further degraded performance. We were stuck in a vicious cycle.
The Architecture Decision
We decided to take a step back and re-evaluate our approach. We realized that the key issue was not the query service itself, but rather the data quality at the ingestion boundary. Our stream processing pipeline was producing a lot of noise - duplicate records, inconsistent formatting, and incorrect timestamps. These errors were propagating downstream and causing the query service to misbehave. We decided to add data validation and data cleansing stages to our pipeline, which helped reduce errors by 75%. We also optimized our query service to use a new query optimization technique that reduced query cost by 40%. With these changes, our query latency dropped below our target threshold of 50ms for 95% of queries.
What The Numbers Said After
After the changes, our query cost dropped by 60%, and our query latency averaged 35ms during peak hours. We also saw a significant decrease in query errors (down 90% from the previous month), which allowed us to simplify our error handling and debugging workflows. Overall, our changes reduced our total infrastructure cost by 25%, which helped us to scale our server more cost-effectively.
What I Would Do Differently
Looking back, I would have caught the data quality issue earlier by implementing more robust monitoring and logging around our stream processing pipeline. I would have also considered training-serving skew mitigation techniques to ensure our model performed well in both training and serving environments. Additionally, I would have evaluated more cost-effective options for our query service, such as using a caching layer or optimizing our data storage schema. These are lessons I'll keep in mind for future system design decisions.
Top comments (0)