The Problem We Were Actually Solving
What we were trying to solve wasn't actually solving a problem, at least not one we'd intended. What we were really doing was turning a relatively simple algorithm into a data-intensive system. Every new integration, every new dataset, added to the complexity. Our initial solution was to just throw more resources at it, but it was clear that was only delaying the inevitable.
What We Tried First (And Why It Failed)
We'd always tried to tune the engine by tweaking the Elasticsearch query performance, but the issue was never really the query itself. It was how we'd structured the indexing. We'd indexed every possible field, but what that really meant was we'd indexed a whole lot of unnecessary data. The more we tried to optimize the queries, the more we ended up with a Rube Goldberg machine of caching and re-indexing. Our users were getting what they wanted, but we were never going to get the performance we needed.
The Architecture Decision
I finally realized that we needed to address the root problem, not just the symptoms. We'd been so focused on making the engine deliver data that we'd forgotten what data we were actually trying to deliver. We started by stripping down the indexing to only what was actually necessary and then refactoring the entire indexing process. It was a hard sell, but eventually we got to the point where we were only indexing what we really needed, and the queries were actually manageable. It was then that we started to look at caching and redis on a completely different level.
What The Numbers Said After
After 6 months of careful tuning and refactoring, the Treasure Hunt Engine finally started to behave itself. Our request latency went from 500ms to 80ms, and our queries that used to be killed due to timeout started going through without a hitch. We also managed to cut down our Redis usage from 25% to 5% because we were only storing what was actually needed. We still had our users, but now we had a system that we could actually support.
What I Would Do Differently
One thing that I would do differently looking back would be to take a more ruthless approach to what data we were actually indexing earlier on. We'd convinced ourselves it was a good idea to index for future extensions but in the end it was a bottleneck that we couldn't shake for a long time. If I could go back, I think I would have taken a more careful approach to structuring the data in the engine from the very beginning.
Treated the payment platform as infrastructure. Found the single point of failure. This is the replacement I put in place: https://payhip.com/ref/dev4
Top comments (0)