Find Your First Desync in 15 Minutes

The bravest corner of game networking

One corner of game networking always struck me as braver than the rest: fighting games. Multiplayer architecture was my main interest for most of my career, before mobile gaming pulled me toward other problems, and I never picked a favorite between authoritative servers, relay servers, and peer-to-peer. Each fits some game and not the game next door. But fighting games ask for something the others do not.

A fighting game cannot wait for a server round trip before showing the local player's own punch. One extra frame between the button and the visible response and the game feels wrong, and players can tell. So that community built on deterministic simulation synchronized by inputs: every machine runs the same simulation from the same inputs and is expected to reach the same state. Two ways of handling latency sit on top. Delay-based netcode holds inputs for a few frames so everyone has them in time. Rollback netcode predicts the remote player's input, keeps going, and when the prediction turns out wrong it rewinds and replays those frames before you notice.

The more I read about this, the braver it looked. You are betting the match on two machines reaching exactly the same state, tick after tick, down to the bit. Floating-point behavior, iteration order, random-number generation: any of them can break the agreement.

A pattern kept appearing in the articles I read. The bet loses. A desync. Two players are suddenly in different realities, and the author describes a long hunt. The Factorio team put it best in an early Friday Facts post about their desync fixing: "You never know if the thing you just solved is the last one, or there are 1500 more waiting." Riot's series on making League of Legends deterministic describes an entire team built around the same fight.

I don't have that story. I have never shipped a lockstep or rollback game, and I have never lost that week. What I had was curiosity, and a suspicion: the studios doing this must have built some version of the same three tools for themselves. Record the inputs and a hash of every tick. Find the first tick where two recordings disagree. When the run reproduces, dump and diff the game state at that tick. I went looking for an open version of that toolkit that I could use with my own game and could not find one. So I built it, partly to learn Rust properly, and partly because writing the debugger for a problem is the most honest way I know to understand the problem.

It's called Tickwise. Here's what it does, and how to trace an injected desync with it in the next fifteen minutes.

Why a desync costs a week

Most bugs tell you where they are. A null reference points at a line. A crash may leave a useful stack trace. A desync can leave neither, and three reasons compound to make it expensive.

The symptom shows up far from the cause. Say one client rounds a physics value differently at tick 4,021. Nothing visible happens. The two simulations are a hair apart and stay that way until the difference reaches something that matters: a projectile that hits on one screen and misses on the other. By the time a human notices, the match is minutes past the moment that mattered, and nothing on screen points back to it.

Reproduction is most of the cost. Reproducing a desync means replaying the exact inputs, in the exact order, with the exact random seed. Without them the exact run is usually unrecoverable, and what reaches you is a report from a player in another country that says "it desynced around the fourth round, I think."

And the tooling, as far as I can see from the outside, tends to stay where it was born. I want to be careful here, because I have not worked inside these studios and most say little about their internals. What I can see is the public record: postmortems, conference talks, engine documentation. They describe the same three ideas again and again. But when I looked for those three things as a small package I could download and use with my own game, engine-agnostic, with replay verification and field-level diffs, I found the workflow inside one commercial engine and otherwise not at all. Excellent versions may well exist behind closed doors. In the public tools I reviewed, the ideas are shared and the code is not.

That gap is what made me want to try something. The game-specific parts of the problem look small to me: how to hash your state, and how to describe it as fields. If those two things are all a game has to provide, the recorder, the comparison, and the diff might live outside any engine and any project, built once, in the open.

So here is the belief I would like to break. Debugging a desync is not inherently a week of print statements. It is a week when the recording and comparison tooling is not there. Once the inputs and hashes are on disk, finding the first divergent tick is a comparison, not an investigation, and pointing at the field that diverged is one more command.

What Tickwise is, in one screen

Tickwise is a Rust library and a command line tool. The library sits inside your game loop and watches. It never runs your simulation and never touches your netcode. You call it once per tick, it writes a file, and the command line tool reads those files later.

The entire contract between your game and Tickwise is one trait with three methods:

pub trait DeterminismProbe {
    /// Called every tick. Must be cheap: a digest of the state most
    /// likely to reveal a desync, not the whole world.
    fn light_hash(&self) -> u64;

    /// Called every N ticks. Should cover everything that can
    /// influence a future tick.
    fn full_hash(&self) -> u64;

    /// Called only during replay, at the ticks you ask for. May be
    /// expensive. Returns the state as a flat list of named fields.
    fn state_dump(&self) -> StateDump;
}

The split between a light hash and a full hash is the design idea I would keep if everything else were thrown away. Hashing the entire state every tick can be too expensive in a large simulation, so you hash a small digest every tick and the whole thing every few hundred ticks. The light hash catches a desync the moment it touches something in the digest. The full hash catches divergences the light hash was not looking at, and in that case Tickwise reports the window rather than pretending to know the tick. A first divergent tick is exact for state the light hash covers, and a bounded interval for state only the full hash covers. The tool says which of the two you got.

If your state type already derives serde::Serialize, you do not write the trait at all. An automatic probe hashes the serialized bytes and turns the type into named fields by walking its structure. The hand-written trait is there for when you want control over the cost, and for the engine bridges coming later, since three plain functions cross a language boundary and a generic trait does not.

The workflow has two passes, and the command line tool has one command for each plus one for looking inside a file:

tickwise inspect session.rec     # what is in a recording
tickwise compare a.rec b.rec     # the first tick where two recordings disagree
tickwise diff a.dump b.dump      # which fields differ at that tick, and how

Pass one, while you play

• Every client records inputs and per-tick hashes into a .rec file.
• tickwise compare a.rec b.rec reports the first tick where the recorded hashes disagree, and which hash caught it.

Pass two, afterwards

• Replay each recording in your own loop, dumping the state at the reported tick into a .dump file.
• tickwise diff a.dump b.dump lists every field that differs, labeled structural, exact, or a float that drifted by less than your chosen epsilon.

The diff classifies rather than judges, because whether floats belong in your simulation is your decision.

Since "cheap" means nothing in a frame budget without a number, here are the measured ones, from the criterion benchmarks in the repository on a desktop machine. The recorder itself costs about 20 nanoseconds per tick with steady inputs and about 100 when inputs change every tick, because it encodes into a reused buffer and writes through a buffered file. The reference simulation's hand-written light hash takes 32 nanoseconds regardless of world size, which is just under one percent of that simulation's 3.3 microsecond tick at a thousand entities. Its full hash takes 14 microseconds and runs every few hundred ticks. On disk, the 6000 tick demo recording is 283 KiB, about 48 bytes per tick with inputs that change constantly, and far less when players hold a direction. The automatic serde probe is the one to measure yourself, since it serializes your whole state on every call and its cost scales with that; the repository's budget guide explains how.

What Tickwise is not, so nobody is disappointed later: it is not netcode, not a rollback engine, and it will not make your simulation deterministic for you. It tells you where determinism broke. Fixing it is still your job.

A note on how it was built, since I want to be upfront about it. I developed Tickwise with Claude, Anthropic's AI, as a programming partner. I wrote the design document, made the architectural and licensing decisions, and reviewed and committed every change. Claude produced much of the initial code and pushed back on several of my decisions, sometimes correctly. That workflow, including where it went wrong, deserves its own post, and it will get one.

Catch a bug with me

Enough description. Let's break something and find it.

The repository ships a small reference simulation: a few balls bouncing around an arena, two players nudging them, a score that goes up on every bounce. It is deliberately boring and deterministic to the bit. It also has a --chaos flag that sabotages it on purpose with a controlled model of one of four classic desync-producing bugs, starting at a tick you choose. Today's bug is stale-value: from the chosen tick on, the simulation reads a scratch value left over from the previous tick, one that should have been reset and was not, and folds it into the score. Safe Rust cannot read truly uninitialized memory, so this is a simulation of the stale-cache bug as it actually appears in production: the value is initialized, it is simply from the wrong moment. It is the kind of thing that passes code review because the code looks fine.

You need Rust installed and a few minutes of compile time on the first run.

cargo install tickwise-cli
git clone https://github.com/cosgunhalil/Tickwise.git
cd Tickwise

Record two sessions. Same seed, same inputs, six thousand ticks each. The second one carries the bug from tick 4021.

cargo run -q -p tickwise-refsim --example record_demo -- clean.rec
cargo run -q -p tickwise-refsim --example record_demo -- chaotic.rec --chaos stale-value 4021

Now pretend these came from two players' machines and you have no idea what happened. Ask:

tickwise compare clean.rec chaotic.rec

  first          6000 ticks, game tickwise-refsim, seed 0xddba11
  second         6000 ticks, game tickwise-refsim, seed 0xddba11

  verdict        first divergence at tick 4021, caught by the light hash, confirmed by the full hash at tick 4200, last agreement at tick 4020

  next           Pass 2: replay each recording in your own loop with
                 dump_at_ticks = [4021] to produce two .dump files, then run
                 tickwise diff a.dump b.dump

That is pass one, and it took a few milliseconds. The last line is deliberate: the output says what to do next, because nobody reads documentation with a desync open in another window.

Pass two. Replay both recordings and dump the state at tick 4021. The replayer checks every recorded hash against the live simulation as it goes, so if your simulation could not reproduce its own recording, you would find out here, before hunting anything.

cargo run -q -p tickwise-refsim --example replay_demo -- clean.rec clean.dump --dump-at 4021
cargo run -q -p tickwise-refsim --example replay_demo -- chaotic.rec chaotic.dump --dump-at 4021 --chaos stale-value 4021

wrote clean.dump with the state at tick 4021
replayed ticks 0 to 5999, every hash matched the recording

One thing to be clear about, because it is the hardest part of a real desync. This example is deliberately reproducible: passing the same chaos mode during replay recreates the faulty execution. A genuinely nondeterministic bug, a race, unrecorded external state, platform-specific float behavior, may fail that hash verification instead. In that case Tickwise still identifies the first recorded divergence, but it cannot reconstruct the original state from inputs alone, and the field-level diff is out of reach until you either reproduce the divergent execution or capture state during the original run. The tutorial shows exactly this happening with two of the other chaos modes. Recording periodic state dumps during play, so that two machines' dumps can be diffed without reproducing anything, is the natural next step for the recorder, and it is on the list.

And the question we actually care about: what is different?

tickwise diff clean.dump chaotic.dump

tick 4021       1 difference over 41 fields: 0 structural, 1 exact, 0 sub-epsilon float drift
  exact          score: 3317 versus 4811663725493808200

  verdict        1 difference across 1 compared tick
  next           an exact difference at the first divergent tick is your lead. Trace that field's last write backwards through the tick

One field out of forty-one. The score on one side is 3317. On the other it is a nineteen-digit number, which immediately suggests that an invalid or stale value entered the score calculation. The other forty fields agree. The first differing field is not automatically the root cause, since a corrupted random state or a reordered loop can surface somewhere downstream, but it is a sharply narrowed lead: the writes that can affect score during tick 4021. In this simulation that is one function.

Four commands. The full tutorial in the repository walks through this with more explanation, then through the other three chaos modes, because each teaches something different.

The one worth previewing is float-drift, which nudges one ball's velocity by a single bit every tick from the strike onward. Run the same steps and compare says something new:

  verdict        divergence caught by the full hash at tick 4200, while the light hash saw nothing: the light hash has a blind spot, and the real divergence happened at or before this tick, last agreement at tick 3900

The light hash in this simulation covers the score, the random state, and the player positions, not ball velocities, so a one-bit velocity change is invisible to it. The full hash fires at its next scheduled check, tick 4200, and Tickwise tells you plainly that the light hash missed something and the real divergence lies in the 300 ticks before. Dump just after the last agreement and the diff shows one field, one bit, classified as sub-epsilon drift; dump at 4200 and the same field has compounded into an exact difference. A light hash is only as good as what you put in it, and the repository has a checklist for exactly that question.

Three things I learned building it

This project was partly an excuse to learn Rust properly, and partly an excuse to make a few mistakes in public and write them down.

Design for the language you have not written yet. From the first day, the plan was that Tickwise would eventually reach Unity and other engines through a C ABI. That one constraint shaped everything. The probe trait has three plain methods and no generics, because three function pointers cross a language boundary and a generic trait does not. The state dump is a flat list of named fields built with insert calls, not a tree, because a sequence of push calls is far simpler to expose across a C ABI than a Rust-owned recursive tree. None of the FFI code exists yet, and the core is already shaped for it. Designing for a second language turned out to be a good way to design a clean API for the first one.

A file format that must never panic will find your bugs for you. Recordings will come from other people's machines, sometimes half-written when a game crashed. So the rule was that malformed input produces an error, never a panic, and the tests enforce it: one truncates a valid recording at every possible byte length, another flips every single byte, and a fuzzer runs on every push. The truncation sweep found a real bug on the day the format was written: a bounds check on the seek index could underflow on a crafted file and panic in debug builds. It never reached a user, because the test written to catch that class of mistake existed before the code it tests.

My favorite example did not exist. The design document opened with the dream output, players[2].velocity.x: 3.5 vs 3.5000001, a sub-epsilon float drift. When I wrote the test for it, the compiler's lint pointed out that 3.5000001 is not representable as an f32. It rounds to exactly 3.5. The nearest single-precision value above 3.5 is 3.5000002, one unit in the last place away. I had carried a number in my head for weeks that could not exist in the type I was writing about. The diff engine now compares float bits, never decimal strings, and the tests use the real neighbor.

Where it is going

Version 0.2.2 is on crates.io: the recorder, the two file formats, all three commands, the replayer, the serde layer, and two integrations in the repository, one with GGRS's example game and one with the Bones ECS framework. It is the first complete Rust-only release.

The next step is leaving Rust. First a C ABI: one crate, tickwise-ffi, built as a shared and a static library with a generated C header, exposing the recorder and the dump builder as plain functions, with prebuilt binaries for Windows, macOS, Linux, Android, and iOS published from CI. Every engine bridge after it will be a thin wrapper over that one surface, which is why it comes first and gets the most careful review.

Then Unity, as a package you add from the repository URL: a C# interface mirroring the three probe methods, a recorder that works in any loop, and a sample scene with a deterministic mini game and a chaos toggle. I have spent most of my career around Unity, so this is the bridge I care most about getting right, and since the Unity editor cannot run on a CI machine, it will be validated by hand in a real editor before anyone is asked to trust it. Each bridge will start with recording and compare, with replay and dumps following once recording is proven in that engine.

More engines follow on the same C ABI, in whatever order people actually ask for. No dates.

Keep reading

If this made you curious about the field rather than just the tool, these are the pieces I learned the most from, roughly in reading order.

• Deterministic Lockstep by Glenn Fiedler. The clearest explanation of the model, and the source of the standard: "Not close. Not near enough. Exactly the same."
• Fast-Paced Multiplayer by Gabriel Gambetta. The authoritative-server side: client-side prediction and server reconciliation. Read it to see what lockstep gives up and gets back.
• 1500 Archers on a 28.8 by Paul Bettner and Mark Terrano. The 2001 Age of Empires paper, still one of the best accounts of why lockstep exists.
• GGPO by Tony Cannon. The rollback library that changed fighting games.
• Explaining how fighting games use delay-based and rollback netcode by Ricky Pusch at Ars Technica. The explainer that fixed the vocabulary.
• Determinism in League of Legends by Riot Games. Making an existing engine deterministic after the fact.
• The Factorio team's Friday Facts posts on desyncs, starting with #63, The endless struggle. Search the archive for "desync"; few teams have written as candidly about hunting them.
• GGRS, the Rust rollback library, and its SyncTest session. Tickwise is designed to sit next to it, not replace it.

Try it

cargo add tickwise --features serde
cargo install tickwise-cli

The repository is at github.com/cosgunhalil/Tickwise, dual licensed MIT or Apache-2.0. The fifteen minute tutorial is the longer version of what you just read, and the hash coverage checklist answers the question every reader asks next: what to put in each hash.

If you have a desync story, or better, a recording of one, the issue tracker has a template for it. Real recordings from real games are the most valuable thing this project can receive right now, and I would rather learn from your bug than from another one of mine.