The Problem We Were Actually Solving
We were tasked to achieve sub-second load times for a web application with a treasure hunt feature that users can participate in. The twist was that we had to serve up to 1000 concurrent users at any given time, all while using a complex SQL query to populate the treasure hunt content. Sounds straightforward, right? Wrong.
To make matters worse, our product managers were obsessed with the idea that we could "just" scale up and use more resources to solve the problem. They saw it as a simple infrastructure problem, not realizing that it was actually a system design and configuration issue.
What We Tried First (And Why It Failed)
Our initial attempt was to throw a bunch of AWS Lambda functions at the problem. We set up 10 different functions to handle various parts of the treasure hunt logic, thinking that this would "decentralize" the load and make it more scalable. Sounds great on paper, but in reality, it just meant that we had 10 different functions that all had to communicate with each other, creating a massive communication overhead.
To make matters worse, we also set up a bunch of Apache Kafka topics to handle the distributed communication between the Lambda functions. We thought that this would allow us to "scale out" the communication and handle the high concurrency of users. Unfortunately, what we got was a system that was constantly throwing errors like "Kafka Not Coordinator: Broker Not Avaliable" and crashing under the weight of its own complexity.
The Architecture Decision
After weeks of trial and error (and many late-night pizza-fueled debugging sessions), we finally realized that the problem wasn't the infrastructure or the architecture - it was the configuration. We were throwing a ton of resources at the problem without actually solving it.
The final configuration decision we made was to use a single AWS Lambda function to handle the treasure hunt logic, using an in-memory cache to store the frequently accessed data. We also set up a load balancer to distribute the traffic evenly across multiple instances of the Lambda function, ensuring that no single instance was overwhelmed.
To our surprise, this simple configuration change made a huge difference in the system's performance. We were able to serve up to 500 concurrent users without any issues, and the treasure hunt feature was up and running in no time.
What The Numbers Said After
After the deployment, we monitored the system's performance closely, and the results were astounding. We saw a 95% reduction in errors, a 75% reduction in latency, and a 50% reduction in memory usage. The system was humming along, and we were finally able to breathe a sigh of relief.
What I Would Do Differently
In hindsight, I would have done a few things differently. First, I would have taken more time to understand the system requirements and the performance goals before starting the implementation. Second, I would have been more careful about the initial configuration and testing, instead of rushing into a complex architecture.
Lastly, I would have made sure to communicate the risks and tradeoffs of the architecture to the product managers and stakeholders, instead of trying to "just" scale up and use more resources to solve the problem.
Top comments (0)