A user types a comment on a train, the tunnel cuts the WebSocket mid-send, and the app reconnects 40 seconds later. Did the comment arrive zero times, once or twice? In my issue tracker the answer has to be "once", every time, and two small rules make that true.
Each workspace lives in one Durable Object. Browsers keep a copy in IndexedDB, show edits immediately, and sync over a WebSocket. The server decides what actually happened; the browser only predicts. Here are the rules that hold this together.
Rule 1: every op gets the next version, with no gaps
Every write appends ops to a log (the oplog), and each op gets the next workspace version: 41, 42, 43, never 41, 43. Here is the core of it, simplified from log.ts:
class ChangeSet {
#next: number;
constructor(private readonly db: Db /* ... */) {
this.#next = currentVersion(db) + 1; // max(oplog.version), or the compaction floor
}
#append(op: Op): void {
this.db.insert(schema.oplog).values({ version: op.version /* ... */ }).run();
this.ops.push(op);
this.#next += 1;
}
}
It runs inside transactionSync, so a write that fails halfway rolls back, and the numbers it took go back with it. A test checks this: a duplicate issue id is rejected, and the next issue still gets number 2, not 3.
The number is the whole sync state of a client. A replica stores one integer, the highest version it has applied. To catch up, it sends hello with that integer, and the server sends every op after it:
const ops = opsSince(this.#db, cursor, OPS_PAGE_SIZE, state.userId);
send(ws, { type: "delta", from: cursor + 1, to, ops });
No diffing, no per-row timestamps. "Give me everything after 41."
Why a gap would be silent
Suppose a client at version 41 receives an op for version 43. If it applied it and moved to 43, version 42 would be lost for good. The next catch-up asks for ops after 43. Nothing errors. One issue is just missing a rename, or never got deleted, and nobody knows until someone compares two screens.
So the client refuses. Here is applyDelta, simplified:
applyDelta({ from, to, ops }: Delta): boolean {
if (to <= this.#version) return true; // already have it
if (from > this.#version + 1) return false; // a gap: apply nothing
const fresh = ops.filter((op) => op.version > this.#version);
// ...apply fresh, then:
this.#version = to;
return true;
}
Anything already applied is skipped, so a replay is harmless. A gap returns false, and the client throws the connection away and fetches a snapshot. A test covers it: ops that start at version 4 for a replica at 2 are refused, and not one of them is applied.
Note from and to. Some ops are private (a member's inbox), and a client never receives other people's. Their versions still exist, so a delta names the range it covers, which can be wider than the ops it carries. A delta with from: 6, to: 7 and no ops is valid and moves the client forward. Without the range, the client couldn't tell "nothing for me" from "I missed something".
The server also refuses when the numbers don't add up. A client claiming a version higher than the head (say, after the server was restored from an older state), or lower than the compaction floor, gets reset and starts from a snapshot.
Rule 2: a mutation id applies once
The client picks an id for every mutation and sends it with the change. The server records each applied id with the version it produced. From mutations.ts:
const applied = db.select().from(schema.appliedMutations)
.where(eq(schema.appliedMutations.id, mutationId)).get();
if (applied) {
if (applied.clientId !== actor.clientId) {
throw new MutationRejected("invalid", "Mutation id was already used by another client");
}
return { version: applied.version, ops: [] };
}
A repeat returns the original version and writes nothing. ops: [] means nothing is broadcast either. The client gets the same ack it would have got the first time.
Back to the tunnel. The client keeps a mutation in its pending list, in IndexedDB with strict durability, until an ack arrives. When the socket reconnects, it says hello, catches up, and on ready resends everything still pending. If the first attempt did reach the server, the resend is a no-op with the original ack. If it didn't, the resend is the first attempt. The client doesn't need to know which.
One test scripts this: the server closes the socket just as the first mutation arrives, and the client must send the identical message again after reconnecting. Another sends the same id twice and checks that the workspace holds one issue and both calls return the same version.
The clientId check is there so one client can't replay or collide with another's id.
The server decides, the client predicts
The UI doesn't wait. When you edit, the client runs a prediction (optimistic.ts) over its confirmed state plus the pending list. The comment at the top says what it can and can't do:
Mirrors the server's mutation logic closely enough to be convincing; the server's result always replaces the prediction.
The replica keeps two things apart: confirmed state, which only server ops change, and a pending list. What you see is confirmed state with every pending mutation replayed on top. When an op arrives, or an ack or a rejection, the view is rebuilt from scratch. Nothing is ever "fixed up".
Two consequences. First, a wrong prediction corrects itself. In one test, Alice and Bob each create an issue at the same moment and both see ENG-1. The server gives one ENG-2, and both screens converge on that. A rejected mutation (someone deleted the project your edit pointed at) just leaves the pending list, and the edit disappears.
Second, and this is what surprised me: the prediction runs again after the server has already applied the change. The server broadcasts the ops to every live socket, including the sender's, before it sends the ack. So for a moment the confirmed state already contains your change while your mutation is still pending. The comment on applyOptimistic says it outright: "each case must be safe to apply to state that already contains its effect." Cases check whether their result is already there; team.create begins with if (draft.get("team", id)) return;. Idempotence turned out to matter on the client too.
Where it isn't free
Idempotency needs memory. Compaction deletes the applied-mutation records older than 30 days, and drops ops that are both old and far behind the head (10,000 ops and 30 days, in my config). A resend more than 30 days late wouldn't be recognized. I don't rely on the oplog floor to catch that case, because the two limits are checked separately. A client that stays offline for a month with unsent edits is the case I'd test next.
Was it worth it?
Yes, and the cost was small: a counter, a table of applied ids, and a client that refuses to guess. Most of the value is what I never had to write: no merge code for missed ops, no dedupe on the client, and no "did it save?" state.
What do you use for this in your own sync code: version numbers like these, timestamps, CRDTs? And has a silent gap ever bitten you?
The code is open source if you want the details: packages/sync-server/src/log.ts in the tracker's repository.
I drafted this with an AI assistant (Claude), which also wrote much of the code it describes. I checked the facts and numbers against the code.
Top comments (2)
The from and to range is the detail I liked most. An empty delta that still advances the client is easy to miss.
One thing to pin down is how long applied mutation ids live. If a record is pruned while a client still holds that mutation as pending in IndexedDB, the resend after ready looks new and applies twice.
Is the retention of those ids tied to the compaction floor?
Good question, and no, they aren't tied to the floor. Applied ids are kept by age: compaction runs every 6 hours and forgets ids older than 30 days. Ops are dropped only when they are both older than 30 days and more than 10,000 versions behind the head, so the floor never passes anything younger than the ids.
That still leaves your case. A client sends a change, loses the ack, then stays away for more than 30 days in a quiet workspace. It's still above the floor, so it gets a delta instead of a reset, resends, and the server no longer knows the id. Creates survive that, because ids are made on the client and a second create with the same id is refused. Issue edits mostly do too: they carry the version the client edited, so a field someone else changed since is a conflict, not an overwrite. The rest are last-writer-wins (editing a comment, renaming a project, archiving an issue), and those would apply again over newer edits.
The client already stores when each pending mutation was made, so the fix is for it to stop resending anything older than the id retention and report it as not sent. I've opened an issue for it. Thanks for reading that closely.
Reply drafted with AI help (Claude) and checked against the code.