DEV Community: Alex Silaev

Ten Years of Rollback Netcode in a Cross-Platform RTS, Without Fixed-Point

Alex Silaev — Sat, 31 Oct 2026 20:26:02 +0000

Mushroom Wars 2 is a real-time strategy game with thousands of units on screen. It has been live for about ten years, on Switch, PlayStation, Xbox, PC, iOS and Android, and at various points all of those platforms played against each other on the same servers. The netcode is deterministic lockstep with rollback. The simulation runs on plain float. There is no fixed-point math in it.

That last part usually starts an argument, so here is how the whole thing works, what actually breaks determinism in practice, and the one bug that taught me the most.

The setup: thin server, fat clients

There are a few classic multiplayer architectures: the server simulates and clients render, one client is the master, or every client simulates the same thing. RTS games and fighting games use the last one. Every client runs the full simulation, and only player inputs travel over the network. The server doesn't run game logic at all. It relays packets, and does a few small things like deciding the winner.

That makes the server absurdly cheap. In this setup a $50 server holds around 10,000 connected players. Traffic doesn't depend on the number of units either, which is the only reason thousands of units are possible at all.

Logic runs in fixed ticks. We use 33 ms, so 30 ticks per second. Classic PC RTS games used much longer network turns, around 200 ms in Age of Empires. Rendering and animation run at whatever frame rate the device can do; only the logic is tied to ticks.

Why rollback, and why StarCraft doesn't need it

Pure lockstep waits: a client doesn't compute tick N until it has every player's input for tick N. On PC that's fine. PC RTS games simply show "waiting for player" when someone lags, and the simulation only ever moves forward.

On mobile that doesn't work. Pings are high and jumpy, matches last three to five minutes, and nobody wants to stare at a waiting screen. So we do what fighting games do.

A client applies its own command immediately, so the game feels offline. The packet goes to the server, which forwards it to everyone else. When another client receives "player 1 did X at tick 1" while it is already at tick 5, it restores a saved snapshot from before tick 1, applies the input, and re-simulates ticks 1 to 5. Visual jumps are smoothed by interpolation, or not smoothed at all if units are slow enough that nobody notices.

Snapshots are taken every 10 to 20 ticks. Fighting games snapshot every tick, because they roll back two or three ticks constantly. We roll back further but less often.

Here is the skeleton. The state of the world is a set of flat arrays of structs, copied wholesale:

public sealed class RollbackSimulation {
    private const int SnapshotInterval = 10;
    private const int HistoryTicks = 600; // 20 seconds at 30 ticks per second

    private WorldState current;
    private readonly Dictionary<int, WorldState> snapshots = new Dictionary<int, WorldState>();
    private readonly SortedDictionary<int, List<PlayerInput>> inputs = new SortedDictionary<int, List<PlayerInput>>();
    private int tick;
    private int dirtyFromTick = int.MaxValue;

    public void AddInput(PlayerInput input) {
        if (input.tick <= this.tick - HistoryTicks) return; // too old: this is already a desync
        if (this.inputs.TryGetValue(input.tick, out var list) == false) this.inputs[input.tick] = list = new List<PlayerInput>();
        list.Add(input);
        list.Sort(PlayerInput.Comparer); // order within a tick: by playerId, then by seq
        if (input.tick < this.tick) this.dirtyFromTick = Math.Min(this.dirtyFromTick, input.tick);
    }

    public void Advance(int targetTick) {
        if (this.dirtyFromTick != int.MaxValue) {
            var restoreTick = this.dirtyFromTick - this.dirtyFromTick % SnapshotInterval;
            this.current.CopyFrom(this.snapshots[restoreTick]);
            this.tick = restoreTick;
            this.dirtyFromTick = int.MaxValue;
        }
        while (this.tick < targetTick) {
            if (this.tick % SnapshotInterval == 0) this.SaveSnapshot(this.tick);
            this.inputs.TryGetValue(this.tick, out var list);
            this.current.Step(list); // deterministic logic step
            ++this.tick;
        }
    }

    private void SaveSnapshot(int t) {
        if (this.snapshots.TryGetValue(t, out var s) == false) this.snapshots[t] = s = new WorldState();
        s.CopyFrom(this.current);
        this.snapshots.Remove(t - HistoryTicks);
    }
}

Two details in there matter more than they look.

The Sort inside a tick. If two players send commands for the same tick, every client must apply them in the same order. Packet arrival order is different on every client, so it can't be used. Sort by player id, then by sequence number.

The cost of CopyFrom. Re-simulating 10 to 20 ticks inside one frame is routine, so copying the world has to be close to a memcpy. If a snapshot means walking a thousand objects and allocating, rollback will kill any phone. On our weakest supported devices a logic tick takes about 6 ms out of the 33 ms budget, which leaves room to catch up two or three ticks per frame after a spike.

Free bonus: replays cost nothing. A match is just its inputs. You are always replaying a recording; it just happens to arrive over the network.

Who wins when two players do the same thing

Say there's one free spot left to build on, and two players click it 33 ms apart. Each one sees their own building first. Then player 2 receives player 1's packet, rolls back to tick 1, applies "player 1 builds here", reaches its own command at tick 2, and the command is now invalid. It stays in the history, it just doesn't do anything. Player 1 does the mirror image: rolls back, keeps their building, skips player 2's command.

Packet arrival order doesn't matter, only the ticks written in the packets. Once all packets have arrived, every client has exactly the same world.

Floats are deterministic. Mostly.

Here's the part people object to. IEEE 754 defines rounding for basic operations, so the same sequence of additions, multiplications and divisions produces the same bits on compliant hardware. If 10 / 100 comes out as 0.0999999 instead of 0.1, that's fine. What matters is that every client gets the same 0.0999999.

In practice, divergence comes from three places:

The compiler. Clients built by the same toolchain with the same settings agree. Build two clients differently and they can diverge. Fused multiply-add is the classic culprit: a * b + c with one rounding instead of two gives different low bits.
Math libraries. sin, cos, pow and friends are implemented differently on different platforms. Don't call engine or OS transcendental functions in simulation code. Write your own from basic operations.
A small tail of weird hardware. We saw divergence on roughly 1% of some Android devices from a few vendors. There are tens of thousands of device models out there, and you can't fix all of them. You can put those players in rooms with each other.

The real modern problem is vectorizing compilers. Unity's Burst, for example, may fuse operations into FMA, reorder them in fast-math modes, or use approximate instructions like rsqrt depending on the target CPU. As far as I know, its deterministic float mode was announced but never actually shipped. If you use Burst for simulation, you will probably end up needing something other than float.

On a recent project we went the textbook route and built fixed-point math so everything could go through Burst. Then we took it out again, and performance went up by about 40%. Fixed-point isn't free: every multiply needs a widening and a shift, and Sqrt, Sin and Atan2 become hand-written tables or iterations that eat exactly the time you were trying to save.

What I'd recommend today is neither general fixed-point nor float, but plain integers with a unit chosen per quantity. You know the ranges of your own data. If speed never exceeds 100 m/s and a tick never exceeds a second, millimeters and milliseconds are enough:

public struct Movement {
    public int positionMm;     // position in millimeters
    public int speedMmPerSec;  // at most 100,000 (100 m/s)

    // deltaMs is at most 1000
    public void Tick(int deltaMs) {
        // 100,000 × 1000 = 10^8, fits in int
        this.positionMm += this.speedMmPerSec * deltaMs / 1000;
    }
}

Integer math is identical on every CPU and costs nothing extra. Division truncates, which can lose precision for very slow objects, but it truncates the same way everywhere, so determinism is safe.

What actually causes desyncs

In ten years, floats were never our main problem. These were:

System.Random or UnityEngine.Random in logic. Randomness must be part of the world state, seeded, and rolled back with it.
Iterating a Dictionary or HashSet. Order is not guaranteed.
GetHashCode() on strings and objects. In .NET Core it is randomized per process.
The engine's physics. PhysX is not deterministic across machines.
Unstable sorts with equal keys. Always add a secondary key like the entity id.
Anything that touches deltaTime.

And one more category that no rule catches.

The forgotten field

Our snapshots used to be taken by CopyFrom methods written by hand, one per state class. A unit had about a hundred fields. Someone added a new one and forgot to copy it.

Nothing broke. Without a rollback, the field lives its normal life and the game is correct. It only breaks when a rollback lands exactly on a tick where that field matters: one client restores the snapshot and keeps the stale value from the discarded prediction, another client has the correct one. In our case, a building got captured on one client and stayed neutral on the other. Reproducing that by hand is close to impossible.

We caught it with hashes. Clients periodically hash the world state for a tick that is already confirmed (all inputs received) and send it to the server. The share of matches with mismatched hashes is a permanent metric for us. There is always some background noise from the hardware tail above, but one day the rate went up. That meant a new source of divergence, somewhere in code that touches simulation state.

The fix came in two steps. First, a reflection test that finds every type with a CopyFrom, fills an instance with random values, copies it, and compares every field, including private ones in base classes. It catches a forgotten field on the first run instead of a hundred matches later. Second, all blittable fields of a unit moved into one struct, so the copy became a single assignment. A new field can't be forgotten anymore, because the compiler copies it.

Hashes, cheaters, and the 1v1 problem

The same hashes handle cheating. If three clients report hash A and one reports hash B, the odd one out is told to roll back further and re-simulate. A transient glitch fixes itself. A modified client keeps producing B and eventually gets removed from the room. Nobody else notices anything.

In one-on-one games you can't tell who is right with two votes. For that, the server can run a headless client fed with all the match inputs. It renders nothing and re-simulates the whole match almost instantly; whoever it agrees with wins. It's also the best debugging tool we have for real desyncs.

Two practical notes on the hash itself. Don't use GetHashCode(). Walk the raw bytes of the state in a fixed order with something simple like FNV-1a, and zero your structs on creation, because padding bytes can contain garbage and break the hash for identical data. And only hash ticks for which all inputs are confirmed, or you'll chase "desyncs" that disappear after the next rollback.

A self-check worth stealing

The cheapest determinism test we have doesn't need a network. In the editor, a debug mode rolls back one tick on every tick, re-simulates it, and compares the hash. Any state that isn't fully captured by snapshots, any hidden dependency on time or order, shows up immediately, long before a real match.

Takeaways

Rollback lets a lockstep game feel offline on bad mobile connections, at the cost of re-simulating ticks. Make snapshots a memory copy, or don't do it.
Floats can be deterministic in production across consoles, PC and phones, if you control the compiler and stay away from platform math libraries and vectorizing fast-math. Fixed-point is a real cost, not a free safety net.
Most desyncs come from randomness, iteration order, hashing and hand-written copy code, not from floating point.
Hash confirmed ticks, track the desync rate as a metric, and keep a headless client on the server for disputes.

I'm Alex Silaev, CTO at Zillion Whales, the studio behind Mushroom Wars 2. This article is based on the networking chapter of my book on Unity performance, memory and architecture, available in English on Patreon and Boosty. The determinism ideas here also drive ME.BECS, my open-source deterministic ECS framework for Unity.

One Forgotten Field Out of a Hundred: Hunting a Desync in Mushroom Wars 2

Alex Silaev — Sat, 31 Oct 2026 14:06:02 +0000

Some bugs are invisible to the compiler, slip past gameplay tests, and can't be reproduced by hand. In a networked game with rollback, the classic example is a field someone forgot to copy. Here's how we found one in Mushroom Wars 2, and how we made sure this kind of bug can't happen again.

How our rollback works

Mushroom Wars 2 is an RTS where every player runs the same deterministic simulation and only inputs travel over the network. To avoid waiting for other players' commands, the client simulates ahead using a prediction. Before predicting, it saves a snapshot of the world. When the other players' inputs arrive, the client restores the snapshot and re-simulates those ticks with the correct data.

Snapshots were taken with a CopyFrom method on every state class, and each method listed its fields by hand. A unit has around a hundred of them. One day someone added a new field and forgot to add it to the unit's CopyFrom.

Why this bug is nearly invisible

As long as no rollback happens, the field lives its own life and the game behaves correctly. It only breaks when a rollback lands on a moment where that field matters. After the snapshot is restored, one client still holds the value from the discarded prediction, while another holds the correct one. The simulations diverge even though the inputs are identical.

For us it looked like this: on one client a player captured a building, while on the other client the same building stayed neutral. Reproducing that by hand is almost impossible, because you need a rollback on exactly the tick where the forgotten field changes.

How we found it: hashes

Clients send state hashes to the server, and the share of matches with mismatching hashes is one of our permanent metrics. There is always some background level of desyncs, but at some point the rate went above normal. That was the signal: a new source of divergence had appeared, and it had to be somewhere in code that touches simulation state.

Without that metric, a rare desync would have looked like a handful of odd player reports that nobody could reproduce.

Step 1: a test that finds the forgotten field

Before touching the code, I asked an AI model to write a test. It finds every class in the project that has a CopyFrom method, fills a source object with random values, copies it, and compares every field, private ones included. It's a mechanical task that is easy to verify by eye, which is exactly where AI does well.

public class CopyFromTests {

    private static IEnumerable<System.Type> CopyableTypes() {
        return typeof(Unit).Assembly.GetTypes()
            .Where(t => t.IsClass == true && t.IsAbstract == false && t.GetMethod("CopyFrom", new[] { t }) != null);
    }

    [TestCaseSource(nameof(CopyableTypes))]
    public void CopyFromCopiesEveryField(System.Type type) {
        var source = System.Activator.CreateInstance(type);
        var target = System.Activator.CreateInstance(type);
        TestData.FillRandom(source, seed: 42);
        type.GetMethod("CopyFrom", new[] { type }).Invoke(target, new[] { source });
        var flags = BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic;
        foreach (var field in type.GetFields(flags)) {
            var equal = TestData.DeepEquals(field.GetValue(source), field.GetValue(target));
            Assert.IsTrue(equal, $"{type.Name}.{field.Name} was not copied");
        }
    }

}

TestData.FillRandom and TestData.DeepEquals are recursive helpers that walk into nested structs, lists and arrays. One subtlety: GetFields doesn't return private fields declared in base classes, so the real test collects fields by walking up the BaseType chain.

A test like this catches a forgotten field on its first run instead of a hundred matches later.

Step 2: make the mistake impossible

A test is a safety net, but it's better when the mistake simply can't be made. I moved all of the unit's blittable fields into a single struct. Assigning a struct copies it in full, so CopyFrom shrank to one line for the data plus manual copying of a few reference fields:

public struct UnitData {
    public int ownerId;
    public int health;
    public float captureProgress;
    // ...and about a hundred more blittable fields
}

public class Unit {

    public UnitData data;
    public List<int> path = new List<int>();

    public void CopyFrom(Unit other) {
        this.data = other.data;
        this.path.Clear();
        this.path.AddRange(other.path);
    }

}

The field names here are illustrative. The point is that a hundred lines of assignments became one. A new field added to the struct is copied automatically, so there's no way to forget it anymore.

The result

This class of bugs is gone. The compiler copies the fields inside the struct, and the test guards the few reference fields that are still copied by hand. It runs on every change and immediately names the class and the field that was missed.

What I took away from this:

Listing fields by hand in copy, compare or serialization code is a time bomb. The more fields there are, the more certain it is that one of them will eventually be forgotten.
Keep simulation state in blittable structs. Copying, hashing and serialization then become operations on a single block of memory. My ECS framework ME.BECS is built on this idea: the whole world state there is one block of memory.
Compare state hashes between clients every few ticks. Then you see a desync on the tick it happened, not when a player complains.
Tests for mechanical invariants like "the copy equals the original" are a great job for AI. They're easy to review and pay off a lot.

I'm Alex Silaev, CTO at Zillion Whales, the studio behind Mushroom Wars 2, and I've been building games in Unity for more than 15 years. This is one of the production stories from my book on Unity performance, memory and architecture (currently in Russian). More of my work: GitHub, chromealex.com.

How the Bot in Mushroom Wars 2 Works: Forecasting, Openings and Honest Difficulty

Alex Silaev — Sat, 31 Oct 2026 14:05:56 +0000

Writing a bot that always wins is not the hard part of making an RTS. Writing a bot that's fun to play against is much harder. I've lost count of how many bots I've written. Here's what the bot in Mushroom Wars 2 is built on, what we had to teach it by hand, and how we tune difficulty so players don't feel the bot is throwing the game.

The core function: "how many will there be in T seconds"

Our bots are built on weights, an approach in the spirit of Utility AI. But weights alone don't get you far if the bot only looks at the current state.

An example. The map has enemy buildings with 50, 10 and 70 units. The bot has 100 units in its building and has to decide where to send them, and whether to send them at all. For simplicity, assume captures trade units one for one. The distances to the targets are 30, 60 and 80.

A human looks at this and thinks: I have 100, they have 50, easy. But every building produces units. So the most important method, and one of the first worth writing, is "how many will there be after some time." It's a universal tool, not just for units in buildings. Say an enemy regenerates HP and is running away from you. From your speed and theirs, you can work out when you'll catch them and how much HP they'll have regained by then. Maybe the chase isn't worth it. Or the other way round: with your damage output, maybe you can finish them off in time.

Most of the bot's decisions rest on this function. Plug it into Utility AI scoring and the bot immediately looks smarter, because it starts predicting: "I'm not going there."

How the bot picks a target

Back to the map. By the time our units cover 30 meters, the target building will have, say, 60 units. We can still take that: 100 against 60 leaves 40. Great? No.

As soon as we send units, the enemy sees where we're heading and can reinforce the target from nearby buildings. So there's a radius within which we account for the support the target can receive. To those 60 we add the 10 from one neighbor and the 70 from another, but not in full: we account for distance and how many of them can arrive in time. Now we're at, say, 110. Meanwhile, about 10 seconds pass, during which our own building produces another 20.

We add it all up and check whether we win. It can turn out that attacking here makes no sense, and it's better to go for buildings that are farther away but have no nearby point that could wipe us out after our attack. The bot picks what to defend and how to carry out attacks the same way.

The forecasting function is called hundreds of times per decision: for every target, every possible reinforcement, every attack source. So it should be pure and cheap. Here's a sketch for the "production plus reinforcements within a radius" mechanic:

public struct Building {
    public float2 pos;
    public float units;
    public float productionPerSec;
    public float capacity;    // limit after which production stops
    public int owner;
}

public static class Forecast {
    public static float UnitsAt(in Building b, float t)
        => math.min(b.units + b.productionPerSec * t, math.max(b.units, b.capacity));

    // How many defenders the target will have when the attack arrives.
    public static float DefendersOnArrival(in NativeArray<Building> all, int target,
                                           float arrivalTime, float unitSpeed,
                                           float supportRadius, float supportShare) {
        var tgt = all[target];
        float total = UnitsAt(tgt, arrivalTime);
        for (int i = 0; i < all.Length; ++i) {
            if (i == target || all[i].owner != tgt.owner) continue;
            float dist = math.distance(all[i].pos, tgt.pos);
            if (dist > supportRadius) continue;
            // The enemy only reacts once it sees our units leave,
            // and only helps if it arrives before we do.
            float travel = dist / unitSpeed;
            if (travel > arrivalTime) continue;
            total += UnitsAt(all[i], arrivalTime - travel) * supportShare;
        }
        return total;
    }
}

supportShare is your assumption about what fraction of a garrison the opponent will actually send to defend. Against a human, 0.5 to 0.7 is more reasonable than 1, and at high difficulty you can learn from replays how often players really reinforce. The resulting value is normalized and fed into the target's utility score along with the building's value and the risk of a counterattack.

Some games with similar mechanics have bots that reason like this: I have 100, there are 10 next door, so attack. And the bot just keeps feeding units away. In our game, many factors affect whether an attack succeeds. For example, a successful attack grants a kind of morale bonus that affects later attacks and defense. All of that has to be taken into account too.

Openings: what the bot couldn't learn on its own

We taught the bot to attack and defend, and it all worked. But it couldn't play the start of a match as efficiently as a human. At the beginning the map is empty, surrounded by neutral buildings that don't defend themselves; you just capture them. The bot captured everything, but much more slowly than a top player who has studied every map. And the start of a match shapes everything that follows.

So we recorded openings for it. We literally wrote down the initial moves in several variants, and at the start of a match the bot picks one at random. It develops like a player, and once it holds half the map it switches to its regular logic.

There's nothing wrong with openings, and they're useful in many genres, shooters included. You spawn at a specific location and almost always do one of a few standard things. After that comes micro, which any of the classic algorithms can handle. The key is to keep several variants and choose randomly, otherwise the bot becomes predictable.

Neural networks: two attempts

The first story isn't mine; a developer I know told it to me. In chess-like games, bots have a problem: they're either too strong or too dumb. Search many plies ahead and the bot is unbeatable. Search only a few, and a strong player calculates three moves ahead and humiliates it. That's exactly what happened to them: a good chess player showed up and tore the bot apart. The developer took a simple perceptron, three layers if I remember right, and fed it a few dozen games from a database. The network's output was a heatmap over the board's squares, and those coefficients were added to the evaluation in the main search algorithm. Even with shallow search, the bot started changing its strategy. It worked, but didn't ship, because of the computational cost of the search itself rather than the network.

My own attempt was in Mushroom Wars. I tried to teach the bot to use skills optimally, and we have a lot of them: speeding up units, burning units in an area, quickly moving units from one building to another, and so on. The bot uses skills at a basic level, and the rules are simple: for example, burning makes sense if the enemy has more units within a radius than some threshold. But when to use them was a big problem. We tried training a network on replays, weren't happy with the results, and dropped it. Most likely we just didn't train enough: there are a million parameters, and it all needs careful tuning.

Between a hard-coded rule and a neural network there are middle-ground options that usually get results faster:

Utility AI with forecasting. Evaluate not "how many enemies are in the radius right now" but "how much value will I destroy or save," using the same "what happens in T" function, and compare that with the cost of the cooldown.
Mini-simulation. If the game rules are deterministic (and in an RTS with lockstep networking they have to be anyway), you can run the simulation a few seconds ahead with and without the ability and compare the outcomes. It's expensive, so candidates are first filtered by simple rules and only the 2 or 3 best are simulated.
For chess-like bots, instead of limiting search depth, it works well to pick not the best move but a random one among moves whose evaluation is within a set margin of the best. That margin is your difficulty knob.

Making the bot more human

People often ask whether we try to make the bot's behavior more human. Especially in games where everything is easy to compute, like billiards, where the bot wrecks anyone the moment it gets a turn.

Yes, we do. We started by teaching the bot to evaluate mid-match whether it's worth playing to the end: how much territory it holds, how much the enemy has, and so on. The bot has ways to end a match, for example building towers and then surrendering.

In general, every bot probably starts the same way: you write the strongest bot you can. When it becomes uncomfortable to play against, you start handing it difficulty levels, which means making it weaker. The simplest way to weaken a bot in an RTS is a "thinking" delay. The bot starts to fumble: it sent units, should send support, and it's already too late. Things stop adding up, and the difficulty drops.

A couple of details that make such a delay feel natural:

The delay shouldn't be a constant. Draw it from a distribution that resembles human reaction time, for example log-normal with a mean of 300 to 600 ms on easy levels. Apply it not only to reactions but also to how often the bot reconsiders its decisions. Then the bot makes mistakes "like a human": not the same way every time, but occasionally.
Degrade perception, not decisions. A bot that sees worse or notices threats later looks natural. A bot that deliberately picks a worse option reads to players as "the bot is letting me win."

The easiest way to make a bot harder is to give it more at the start or tweak its coefficients: 10% faster production, faster units, more units at the start. But we only do that in the single-player campaign, where players are prepared for the bot to hold half the map from the beginning.

In multiplayer, early on we have bots that teach the player how to play. We gradually make them harder and then release the player to real people. Those bots play fair, one on one, with no handicaps, and the openings exist precisely so they behave like players.

Synchronized attacks

Here's what we did at the highest difficulty level. Say the bot holds three buildings and wants to attack a fourth. It doesn't send everyone at once. It first sends units from the farthest building; when they reach a certain line, it sends from the next one, then from the third. The whole crowd hits the target at the same moment. It's effective, it looks great, and players love doing it themselves.

Technically it's just planning backwards from the end: compute each group's travel time t_i = dist_i / speed, take the maximum T, and send group i with a delay of T - t_i.

Summary

Whatever you build on, whether FSM, Behaviour Tree, Utility AI or GOAP, two things work on top of it. A forecasting function, "what will happen in T," that makes the bot smarter. And an honest answer to the question of whether it's fun to play against.

I'm Alex Silaev, CTO at Zillion Whales, the studio behind Mushroom Wars 2, and I've been building games in Unity for more than 15 years. This article is adapted from the game AI chapter of my book on Unity performance, memory and architecture, available in English on Boosty. More of my work: GitHub, chromealex.com.

220,000 Units Instead of 4,000 and Other Production Stories from Mushroom Wars 2

Alex Silaev — Sat, 31 Oct 2026 14:05:46 +0000

Almost every story below started with code that worked until it didn't. These are real cases from Mushroom Wars 2, an RTS where thousands of units run around on a phone screen at once, and which we shipped on PS4, Xbox One, Switch, iOS, Android and PC. For each story: what broke, how we found it, what we did, and what the numbers were.

1. Garbage collection in the middle of a battle

The problem. Mushroom Wars 2 is a fast-paced strategy game, and any hitch ruins the feel. Garbage collection mid-battle caused noticeable stalls, especially on consoles (PS4, Switch).

How we found it. At first the freezes were visible with the naked eye. The Unity profiler and the console's native profiler showed that the stalls lined up with GC.Collect. On those platforms a collection could take up to a couple of seconds.

What we did.

Moved all gameplay code to object pools: not a single managed object is created during a battle.
Disabled the GC before a battle, then re-enabled it afterwards and collected manually on the results screen, where the pause goes unnoticed.
Back then this required hacks; today there's the built-in GarbageCollector.GCMode.

The numbers. Before: the heap grew noticeably during battles, because there were allocations mid-gameplay, and every one of them had to be found and removed. Collections took up to ~2 s on PS4 and Switch. After: the heap doesn't grow during a battle, and there are zero collections per battle.

The catch: a single forgotten allocation in battle starts growing the heap again, and with the GC disabled that's no longer a stall but memory running out.

2. The premature optimization that saved the release

When we wrote the unit renderer in 2015, Unity couldn't draw much in large numbers, so we optimized and micro-optimized everything we saw. The renderer needs to know the upper bound on the number of units in advance to allocate its buffers. We computed it as the sum of units in all buildings on the map, multiplied by the number of players (in theory, each one can use an ability that duplicates units). If we missed something, the extra units just don't get drawn, and that's fine: with a thousand units on screen you can't tell who's who anyway, let alone with four thousand.

That formula lived for seven years. Then it backfired.

3. 220,000 units instead of 4,000

The problem. A low-end device crashed while loading one specific map. Every other map worked fine.

How we found it. The behavior was typical of running out of memory, and the logs confirmed it.

The cause. The same formula from 2015. A level designer had placed 5,000 units in buildings "off the map" so they could be released by a trigger. As a result, instead of a maximum of ~4,000 units (and render buffers for 20,000), the game started reserving and processing ~220,000.

Why nobody noticed earlier. We render units with particles, and the extra particles didn't give themselves away. The map shipped like that for about three months: normal devices had enough memory, and only the low-end one crashed. In the end, we found the cause almost by accident while refactoring the unit renderer.

The fix. It turned out the particle system was being given a capacity based on that formula, multiplied by the number of layers on top. And it was drawing every particle in the buffer, not just the ones belonging to live units. We fixed the capacity calculation and started drawing only the particles actually in use.

The lesson. Code nobody has touched in seven years still enforces limits that everyone has forgotten about. Computed limits need to be checked against real content data. A build-time map validator that reports "this map needs a buffer for N units" would have caught the problem the day it appeared.

4. Unit paths that cost almost nothing

Units in Mushroom Wars 2 always move from building to building, and the map is static. That means every path can be built ahead of time in the editor, with no pathfinding at runtime at all. Paths are built in three stages:

A* on a grid. In the editor the whole field is a grid, and A* easily finds a path from building to building around obstacles. But a grid path is jagged and hugs obstacles.
A chain of colliders. Along that path we create a chain of box colliders connected by joints. Each collider is the size of a unit in the game. There are up to eight paths between buildings, so up to eight chains at once.
A physics simulation. We run physics 50 frames ahead. The chains push off each other and off obstacles, tighten and smooth out, like ropes. The result is clean paths through narrow passages, neat curves around buildings, and roads that don't overlap.

We save the collider positions as spline control points. At the start of a battle, a Catmull-Rom spline is built from them, and units simply move along the spline. A squad marches in neat rows and keeps formation, which looks good and has a big effect on how the game feels.

A battle can have up to 4,000 units, and building paths for them at runtime would be expensive; this way it's almost free. The limitation is obvious: the approach only works because the map and the set of routes are known in advance. A dynamic world needs hierarchical pathfinding with flow fields per chunk.

5. How a 2D game got real shadows

We built Mushroom Wars 2 for Steam first. Our experience and the hardware would have allowed 3D, but we deliberately chose 2D: any "beautiful 3D" looks dated within a few years, while good hand-drawn 2D ages slowly.

We drew the buildings and units in pixel-perfect 2D, but we wanted real shadows. The solution turned out to be fun. Around the outline of each building we made a low-poly mesh slightly larger than the building itself and placed the 2D sprite inside it, but didn't render the mesh. The mesh only took part in shadow calculations, casting and receiving shadows, while only the sprite was visible on screen. The result was pixel-perfect rendering with real-time shadows.

Later we dropped mesh shadows in favor of sprites and render textures, because we needed to support low-end devices. But the story stuck.

6. Players losing their accounts on iOS

The problem. Players who hadn't linked a social account complained about losing progress, mostly on iOS.

How it worked. We wanted people to be able to play without signing in. To do that, the client sent the server a unique string, and we used SystemInfo.deviceUniqueIdentifier. Once upon a time that was a stable hardware identifier; now it can change. On iOS it isn't the advertising IDFA (which requires ATT permission) but identifierForVendor, and it changes if the user deletes all of a developer's apps from the device. So that an OS-side ID change wouldn't affect the player, we stored the first value we received in PlayerPrefs. Of course we offered to link a social account to keep progress safe, but that's like asking people to read the readme.

How we found it. The complaints all came down to one scenario: the player deleted the game and reinstalled it. PlayerPrefs are deleted along with the app, and if there were no other apps of ours on the device, identifierForVendor changed too. New identifier, new account.

What we did. We started storing the key in the Keychain, which survives reinstalls. The complaints stopped. The Keychain has its own pitfall: if you transfer the app to a different developer account, you lose access to the data stored there.

Cheat sheet

Story	Lesson
GC mid-battle	Pool all gameplay objects, disable the GC during battles with `GarbageCollector.GCMode`, collect on the results screen
Premature optimization	An upper bound for the renderer, set early, saved the release, but its assumptions live on for years
220,000 units	Check computed limits against real maps with a build-time validator
Unit paths	If the map is static, build paths in the editor and leave only the spline for runtime
Shadows in 2D	An invisible low-poly mesh around the sprite only casts and receives shadows
Accounts on iOS	Store the guest account key in the Keychain, not in `PlayerPrefs`

I'm Alex Silaev, CTO at Zillion Whales, the studio behind Mushroom Wars 2, and I've been building games in Unity for more than 15 years. These stories come from the production chapter of my book on Unity performance, memory and architecture, available in English on Patreon and Boosty. More of my work: GitHub, chromealex.com.