The Problem We Were Actually Solving
We initially launched the game with a robust backend written in Node.js, and a frontend built using React and Webpack. However, as player numbers began to swell, we noticed a strange pattern: the game would stall for short periods of time (averaging around 50-100ms) whenever the player count exceeded 1000 concurrent users. While not catastrophic, this performance hiccup was unacceptable for a game that required real-time interaction and instant feedback.
What We Tried First (And Why It Failed)
At first, we thought the problem lay with our database – specifically, our MongoDB instance, which was handling around 1000 queries per second. We optimized the database queries by adding indexes to the relevant collections, reduced the number of queries per page load, and eventually moved to a sharded database setup. However, to our surprise, the performance issues persisted even after these changes.
The Architecture Decision
After weeks of debugging and testing, we eventually pinpointed the source of the problem: our frontend code. The culprit was an egregious case of memoization, which was causing an exponential increase in memory allocations whenever the player count exceeded 100 concurrent users. The Node.js runtime, which is notoriously memory-inefficient, was unable to keep up with the demands of our code, resulting in periodic stalling and crashes.
What The Numbers Said After
To drive the point home, let's take a look at the following profiler output, taken from the frontend code running with 1500 concurrent users:
5.6% (4,201,410) Allocs per second
4.2% (3,196,231) Unnecessary memoization events
2.8% (2,115,115) Page loads due to stale cache
1.4% (1,049,509) MongoDB queries per second
From these numbers, we can see that the majority of our performance issues stem from the frontend code's use of memoization.
What I Would Do Differently
In retrospect, I would have chosen a more memory-constrained language like Rust or Go from the outset, both of which would have helped mitigate the performance issues plaguing our Node.js frontend. These languages possess better memory management capabilities, which would have prevented the exponential increase in memory allocations that led to our stalling and crashes.
However, that's not to say Node.js doesn't have its strengths – especially when it comes to web development. If I were to restart this project, I would choose Node.js for the backend, with a separate Rust frontend that uses actor-based concurrency models to manage memory safely.
This post is the first part of a two-part series on the Veltrix Treasure Hunt Engine. In my next post, I'll dive deeper into the specific architecture decisions made, including our migration to a new frontend written in Rust, and the surprising benefits we experienced from this move.
Top comments (0)