A failover test becomes trustworthy only when wall-clock timing is removed from the verdict. For a game studio's shared kanban board, model regional recovery as an ordered state transition, drive it with a logical clock, and prove that a card can create exactly one video room with a narrowly scoped token after ownership moves.
Short answer: test observable invariants, not reconnection speed: one accepted operation, one room identity, one scope, and monotonically advancing board state.
This is the key shift. Before: disconnect a region, sleep for five seconds, then hope every browser looks right. After: pause delivery, transfer an explicit epoch, replay operations, and advance a test-owned clock until each named condition is true. The second model tells you which contract failed. It doesn't confuse a slow laptop with a broken recovery path.
What should realtime regional failover testing prove for a shared kanban board?
Start with authority. Imagine a game studio board with a card named Review boss-fight capture. Moving that card into Live review asks the backend to spin up a video room. The browser may request the move, but it can't choose the room ID, mint a broader token, or declare which region owns the board. Treat every client field as a claim.
The test should prove four invariants:
- Every client operation has a stable operation ID, and retrying it doesn't create a second room.
- An epoch identifies the current regional owner; writes from an older epoch are rejected.
- The room token is bound to the intended room, participant, role, and expiry rather than to the whole account.
- A recovered replica applies each accepted board operation once and exposes a monotonic revision.
Those rules separate control-plane state from media transport. WebRTC defines browser-facing real-time communication APIs, but signaling is outside its scope. That leaves the application responsible for carrying offers, answers, room authorization, and failover ownership through a signaling design it can test. The board operation is therefore the durable intent; joining the media room is a consequence of accepted intent, not evidence that the write committed.
Keep that boundary sharp.
Replace sleeps with a logical clock and explicit events
A timing-based test usually says, in effect, wait 5000 ms and inspect the DOM. It can pass while duplicate work is still queued, or fail because CI was busy. A deterministic harness instead exposes the state machine's inputs: client operation, replication acknowledgement, lease expiry, ownership transfer, and delayed delivery. The harness controls their order.
Here is a compact TypeScript model. It isn't a production server; it is the executable contract that a production adapter must satisfy. Notice that the client supplies an operation ID and requested action, while the trusted side supplies epoch, revision, room identity, and token claims.
type Column = "backlog" | "live-review" | "done";
type Role = "viewer" | "presenter";
type Move = {
opId: string;
cardId: string;
target: Column;
participantId: string;
requestedRole: Role;
};
type TokenClaims = {
roomId: string;
participantId: string;
role: Role;
expiresAtTick: number;
};
type Result = { revision: number; roomId?: string; token?: TokenClaims };
class BoardRegion {
private seen = new Map<string, Result>();
private revision = 0;
constructor(
readonly name: string,
private epoch: number,
private tick: number,
) {}
transferOwnership(epoch: number): void {
if (epoch <= this.epoch) throw new Error("stale epoch");
this.epoch = epoch;
}
advanceTo(tick: number): void {
if (tick < this.tick) throw new Error("clock moved backward");
this.tick = tick;
}
apply(move: Move, epoch: number): Result {
if (epoch !== this.epoch) throw new Error("wrong owner epoch");
const previous = this.seen.get(move.opId);
if (previous) return previous;
this.revision += 1;
const roomId = move.target === "live-review"
? `room:${move.cardId}`
: undefined;
const token = roomId
? {
roomId,
participantId: move.participantId,
role: move.requestedRole,
expiresAtTick: this.tick + 30,
}
: undefined;
const result = { revision: this.revision, roomId, token };
this.seen.set(move.opId, result);
return result;
}
}
The 30 above is a test tick interval, not a claim about a suitable real-world token lifetime. Production expiry should come from a documented security policy and the threat model. I'm not sure there is one duration that fits both a moderated tournament broadcast and a two-person art review; the evidence needed is each workflow's join latency, reconnect behavior, and acceptable exposure window.
Now script the failure without timers. The first owner accepts the operation. Delivery of its result to the client is withheld. Ownership moves to the second region, and the client retries the same operation ID because it never saw the acknowledgement.
const move: Move = {
opId: "op-1042",
cardId: "capture-77",
target: "live-review",
participantId: "editor-12",
requestedRole: "presenter",
};
const primary = new BoardRegion("region-a", 8, 100);
const accepted = primary.apply(move, 8);
// The adapter copies committed operation IDs before ownership is announced.
const recovered = new BoardRegion("region-b", 8, 100);
recovered.transferOwnership(9);
const retried = recovered.apply(move, 9);
if (accepted.roomId !== retried.roomId) {
throw new Error("room identity changed across recovery");
}
if (retried.token?.participantId !== "editor-12") {
throw new Error("token escaped participant scope");
}
One detail is intentionally abstract: the sample doesn't pretend that two in-memory maps replicate. Your adapter must seed the recovered region with committed deduplication records before it announces epoch 9. Test that ordering at the storage boundary with a controllable message queue or a fake transport. The unit model gives fast counterexamples; an integration test proves the wiring preserves the same contract.
Make the assertions observable
A green UI is weak evidence. Emit structured events at the points where authority changes, and correlate them with boardId, opId, epoch, revision, cardId, and roomId. Never log the room token. A token belongs in a secret-bearing response path, while the safe claim dimensions can be represented as bounded labels or audited fields according to the team's telemetry policy.
A useful event sequence reads like a diagram in words: client proposes op-1042 to epoch 8; region A commits revision 311; acknowledgement is delayed; region B restores through revision 311; ownership advances to epoch 9; the client retries op-1042; region B returns the prior room identity without a second creation effect. That narrative is much easier to debug than expected 1, received 2 after an arbitrary timeout.
Use metrics for aggregates and logs or traces for individual proof. Count ownership transfers, stale-epoch rejections, deduplicated retries, and token-mint decisions. Alert on an invariant breach, such as two distinct room identities for one operation ID, rather than on a fixed recovery duration alone. Duration still matters operationally, but it should be measured after correctness is established.
This also makes the before/after crisp: before, the test observes elapsed time and screen state; after, it observes a causal chain and terminal revision. Much better.
Two objections: real networks and expiring tokens
The first objection is that a logical clock makes the test unrealistic. It does remove scheduler and network randomness from the core suite — deliberately. Keep a smaller end-to-end layer that exercises browsers, WebRTC negotiation, and real regional infrastructure, but make its pass condition event-based too. Wait for a named revision and ownership epoch with a generous outer deadline; treat that deadline as a test-abort guard, not as proof of recovery.
The catch is that this approach is not suitable when the failure comes from a browser's actual media stack, NAT traversal, or platform-specific device behavior. Use browser automation and controlled network impairment for those questions. Stick with the state-machine harness for ordering, replay, idempotency, and authorization, because adding real packet delay there expands the schedule space without clarifying the expected result.
The second objection is token expiry during failover. Don't freeze production policy merely to make a test pass. Advance the logical clock across the expiry boundary and assert the intended rule: an expired token cannot be reused, a refresh is authorized from trusted board state, and the replacement retains the same room, participant, and allowed role unless a server-side policy changed them. Test a viewer asking to become a presenter, too; the untrusted request must not grant authority absent trusted authorization.
Short-lived credentials introduce a real trade-off. Tighter expiry limits the useful window of a leaked credential, while frequent refresh adds control-plane traffic and can make recovery paths more active. Your mileage may vary. Choose the interval from the risk model, instrument refresh outcomes, and keep that choice out of the deduplication contract.
A practical release gate
Before deployment, run the same scenario across three layers: a pure state-machine suite over many event orderings, a storage-and-messaging integration suite with delivery manually paused, and a narrow browser test that joins the resulting room. In the first layer, enumerate acknowledgement-before-transfer, transfer-before-acknowledgement, duplicate retry, and expiry-before-retry. In the second, pause the transport after the durable commit and before the client acknowledgement, then restore the successor through the committed revision before exposing its new epoch. In the browser layer, keep the path narrow: move capture-77, observe the accepted revision, join room:capture-77, and verify that the participant receives only the presenter role authorized for that card. The release gate should require stable operation identity, a single side effect, monotonic revision, current epoch, and exact token scope at every layer.
No sleeps.
Don't require a magical five-second recovery. Record recovery duration as a service objective with a separately chosen threshold, then investigate its distribution without letting a fast but incorrect run pass. Correctness first. Speed second.
This method has a cost: adapters need explicit hooks for clocks, transport delivery, and ownership events, and the team must maintain a reference state model. For a low-value board where duplicate side effects are harmless, that machinery may be excessive; a simpler idempotency integration test can be enough. For a board action that creates a privileged video room, the reference model earns its keep because it turns client distrust and regional recovery into reviewable rules.
Top comments (0)