The Problem We Were Actually Solving
In our case, the issue wasn't about the treasure hunt engine itself but about its interaction with our event database, Eventstore. With each user move, the engine dispatched a flurry of events to Eventstore, creating a stream of data we couldn't afford to miss. However, our decision to use the in-memory data grid, JGroups, to cache these events allowed our system to scale within limits, but not without a catch: it stored each event in memory, leading to an ever-growing cache size. As we approached the server expansion milestone, our operators noticed the cache size had ballooned out of control, prompting them to question our architecture.
What We Tried First (And Why It Failed)
Initially, we opted for the standard approach to mitigate cache growth: setting a fixed maximum cache size and implementing a least-recently-used (LRU) eviction policy. However, as the treasure hunt engine continued to generate an avalanche of events, our cache would frequently reach its maximum size, forcing us to manually increase it and risking our system's memory becoming clogged. This expedient solution, coupled with the high overhead of the LRU implementation, led to a noticeable performance hit and increased latency. In a worst-case scenario, the LRU policy even dropped some events from the cache, causing our system to crash and lose precious user data.
The Architecture Decision
We made a significant change to our treasure hunt engine's configuration, trading off a more efficient caching strategy for a more scalable one. We switched from JGroups to a distributed caching solution, Hazelcast, which allowed us to store events across multiple nodes, reducing the memory footprint and load on each individual node. We also implemented a combination of a time-to-live (TTL) policy and a sliding window approach to dynamically manage our event cache, keeping relevant data cached while ensuring we didn't clog our system with aging events. Our new caching strategy significantly reduced cache growth while ensuring we didn't sacrifice performance and durability.
What The Numbers Said After
Before our caching overhaul, the median cache size had reached 2.5 GB with an average cache hit ratio of 75%. After our transition to Hazelcast and our new caching strategy, the median cache size decreased to 750 MB, and the cache hit ratio shot up to 95%. Moreover, the number of cache-related failures dropped by 90% during peak hours. Our new caching approach not only helped us scale but also gave us the flexibility to adapt to new business requirements without compromising performance.
What I Would Do Differently
Looking back, I recognize that we should have implemented load testing and a thorough cost-benefit analysis before settling on our initial caching solution. This would have helped us anticipate potential bottlenecks and make a more informed decision. Moreover, our caching strategy could have benefited from a more proactive and automated approach to cache management, incorporating strategies like adaptive TTLs and more sophisticated eviction policies. With these adjustments, we could have achieved even better results and potentially avoided some of the performance issues that arose during the chaos of server expansion.
Top comments (0)