The Day Veltrix Became the Bottleneck Instead of the Treasure

The Problem We Were Actually Solving

Hytales treasure hunt engine needs to deliver a treasure packet to every player within 20 ms of the dig event, but our LuaJIT runtime was already spending 47 ms per dig site in GC pauses while the heap ballooned past 8 GB. The existing LuaJIT harness could not scale to 12 k concurrent players without violating the latency contract.

What We Tried First (And Why It Failed)

We moved the propagation layer to Go because we assumed the GC would be gentler. The switch removed the Lua allocator fragmentation, but the Go GCs mark-and-sweep introduced 4 ms pauses every 1.2 seconds, pushing p99 latency to 14 ms—still too high under load. Perf top showed the Go runtime itself allocating 1.8 GB RSS for less than 100 MB of actual treasure data, a 18× overhead we could not justify.

The Architecture Decision

We rewrote the resolver in Rust using Serde for zero-copy deserialization and tokio for async I/O. The schema stayed binary-compatible with the Lua version, so the game server could still deserialize the same 128-byte packet. We used mpsc channels with a work-stealing thread pool sized to the number of physical cores (32) to guarantee 20 ms p99. The Rust runtime itself ran in a single 4 MB stack, letting the OS page cache handle the rest.

What The Numbers Said After

After swapping the resolver in production, perf stat showed RSS drop from 8 GB to 1.3 GB and all-latency p99 at 8 ms under 12 k concurrent digs. Flame graphs showed the Rust resolver spent 0.4 % time in GC versus 18 % in the Go version. jemalloc stats confirmed zero allocator contention because Serde could read the packet directly from the socket buffer without copying. The only regret was the two-week learning curve for async Rust, especially pinning streams and managing unpin types correctly.

What I Would Do Differently

I would have prototyped the Rust resolver against the production Lua schema earlier, before committing to the Go rewrite. The Go version bought us breathing room but introduced latency cliffs we only discovered under load tests. Today we keep a Rust resolver in a separate crate with the same public API, which lets us hot-swap it into the Lua harness without changing the game client—a trick we wish we had discovered on day one.