The Problem We Were Actually Solving
Or, at least, I thought we were. In reality, our engineers were trying to solve a problem that was more nuanced than just "make it faster." We were trying to scale a single instance of our Treasure Hunt Engine to meet the growing demand of our product. The issue was that, under load, our instance would start to cache too many requests in memory, leading to eventual OOM (out of memory) errors. Our operators would then have to manually restart the instance, lose all the cached data, and restart from scratch.
What We Tried First (And Why It Failed)
Our first instinct was to simply add more instances of the Treasure Hunt Engine behind a load balancer. This seemed like an easy fix: just throw more servers at the problem. However, as we soon found out, our configuration files were not designed to handle this type of horizontal scaling. The default configuration of Treasure Hunt Engine was set to use a shared cache across all instances, which would lead to cache collisions and inconsistent data.
We tried to mitigate this by disabling the shared cache and using a local cache on each instance instead. However, this led to a new problem: our application would have to fetch the same data from the database multiple times, leading to excessive database queries and further slowing down the application.
The Architecture Decision
After weeks of experimentation and many late nights, we finally settled on a new architecture: we would use a Redis cluster as the primary cache store, and design our configuration files to use a token bucket algorithm to throttle the cache invalidation requests. This would ensure that our instances would only cache data that was relevant to their specific requests, and would prevent cache collisions.
We also decided to use a circuit breaker pattern to detect when the Redis cluster was under too much load, and would automatically route requests around it if necessary. This would prevent our application from becoming unresponsive when the Redis cluster was under pressure.
What The Numbers Said After
The metrics from our monitoring tools told the story. Our average response time dropped to 0.5 seconds, our CPU usage stayed under 20%, and our RAM usage stayed under 40%. Our operators could now scale our instances without fear of OOM errors or cache collisions. The Redis cluster was able to handle the load, and our circuit breaker pattern prevented our application from becoming unresponsive.
What I Would Do Differently
Looking back, I would have taken a more cautious approach to scaling our instance. Instead of throwing more instances at the problem, I would have worked on refactoring our configuration files to use a more distributed cache store from the start. This would have saved us weeks of experimentation and many late nights.
However, I would not have changed the decision to use a Redis cluster as our primary cache store. The benefits of using Redis far outweigh the costs, especially when used correctly. The key takeaway here is that, when scaling a distributed system, it's better to take a step back and re-evaluate your architecture before throwing more resources at the problem.
We removed the payment processor from our critical path. This is the tool that made it possible: https://payhip.com/ref/dev1
Top comments (0)