Short answer: realtime schema versioning needs explicit failure handling: keep the wire contract backward-compatible, persist an event sequence, and make reconnect replay from a client checkpoint instead of guessing the current screen state.
Replay first.
The page fires first. A quiz player reports that an answer vanished after their phone switched from Wi-Fi to cellular. The on-call view shows a reconnect spike, but the service is returning 200 responses. That is the dangerous kind of incident: transport health looks fine while two versions of the client disagree about what an answer event means.
In a multiplayer quiz, typing indicators and read receipts are small events beside the score, timer, and question state. They still cross the same unreliable boundary. Treat every event as a versioned contract, and failure handling becomes a replay problem with explicit rules rather than a collection of UI patches.
What should a reconnecting quiz client replay?
The client needs three pieces of durable context: a schema version it can decode, a monotonically increasing event sequence, and the last sequence it applied. A reconnect request carries that checkpoint. The server replays events after it, or sends a fresh snapshot followed by newer events when the retention window is gone. That choice needs to be visible in the protocol response, because an operator debugging a live match must be able to tell whether the client recovered by replay or by rebuilding state. Include the selected base sequence, the highest sequence in the response, and a reason such as retention_expired in internal telemetry. During a rolling deploy, preserve those fields even if the UI never displays them; they turn a vague report of “my answer disappeared” into a bounded interval that can be inspected.
Do not use a wall-clock timestamp as the checkpoint. Clocks jump, and two devices can produce the same millisecond. A sequence belongs to the room (or match) and is ordered by the authoritative writer. The client can then apply event 418, notice that 419 is missing, and ask for replay instead of silently displaying an incomplete scoreboard.
The replay path should be idempotent. Applying the same answer_submitted twice must not award points twice; applying an old typing_started event must not resurrect a typing badge after its expiry. Store an event ID or sequence in the reducer, and make the state transition conditional on whether that event has already been seen.
The alert-to-action trace
Start with the alert an operator actually receives: replay_gap_rate > 0.5% for five minutes, split by client build and room. The first action is to inspect the gap histogram, not to restart the gateway. A spike in gaps for one build points toward a contract change; gaps across all builds suggest retention, ordering, or a reconnect storm.
Work backwards to the signal that should have fired earlier. Emit counters for reconnect attempts, replay requests, snapshot fallbacks, duplicate event IDs, and decode failures. Attach the room sequence and schema version as structured fields. Never put answer text or player tokens in those fields; logs are part of the data boundary.
Here is a compact server-side envelope. The payload is intentionally opaque to the transport layer.
package realtime
type Envelope struct {
RoomID string `json:"room_id"`
Seq uint64 `json:"seq"`
EventID string `json:"event_id"`
Schema uint16 `json:"schema"`
Kind string `json:"kind"`
Payload []byte `json:"payload"`
}
type Checkpoint struct {
LastSeq uint64
SchemaMax uint16
}
func ReplaySince(log []Envelope, cp Checkpoint) []Envelope {
out := make([]Envelope, 0)
for _, event := range log {
if event.Seq > cp.LastSeq && event.Schema <= cp.SchemaMax {
out = append(out, event)
}
}
return out
}
The instrumentation change is simple but consequential: count a successful replay only after the reducer acknowledges the final sequence. Otherwise, the metric measures bytes sent, not state recovered. Add a trace that follows reconnect, checkpoint validation, replay or snapshot selection, and reducer acknowledgement.
Thresholds have a cost. A low decode-failure threshold pages someone for a single stale client; a high one lets a whole room miss the final question. Your mileage may vary because match duration and network behavior differ, so tune thresholds from a week of per-build data and document the decision in the runbook.
How do schema versions and failure handling protect a multiplayer quiz?
Use additive changes first. A new optional field can be ignored by an older reader, while a renamed field or changed enum value needs a migration plan. Keep the envelope stable and version the payload separately. During a rolling deploy, the writer must emit an event that every supported reader can either understand or safely skip.
For a breaking change, negotiate capabilities at handshake time. The client advertises the highest schema it can decode; the server chooses a compatible representation or asks for a snapshot. A server should not downgrade an already committed event in place, because that makes audit and replay disagree. Write a translation at the boundary and test it against recorded envelopes.
I once assumed a reconnect was successful when the socket reopened. It wasn't. The reducer had applied events 731 and 733, then rendered a winner banner before 732 arrived. The fix was an explicit contiguous-sequence check: rendering a terminal match state waits until the gap is filled or a snapshot establishes a new base. That extra state, awaiting_replay, is less exciting than a clever retry and far easier to reason about at 02:00.
Typing indicators deserve a different expiry rule from answers. They are ephemeral hints, so a snapshot may omit them and a reconnect can start with an empty set. Read receipts affect user-visible progress and should be represented by a durable event or a snapshot field. Mixing those lifetimes in one generic cache is how stale indicators survive while a receipt disappears.
Failure modes worth testing before production
Test the sequence, not just the happy-path message. Drop event 2 of 3, duplicate event 2, reorder events, and reconnect while a snapshot is being generated. Inject a client that understands schema 4 into a room whose oldest retained event is schema 2. The expected result is deterministic: replay when possible, snapshot plus tail when necessary, and a clear incompatibility response when neither representation is safe.
Use property tests for the reducer: applying an event twice yields the same state as applying it once; applying a valid log in order reaches the same state after a disconnect and replay. Keep golden envelope fixtures for every deployed schema. They catch accidental changes to field names and enum values that compile cleanly.
Chaos tests should include the false-positive path. If a monitoring rule pages on every reconnect, operators will mute it and miss the real gap. Alert on user-impacting symptoms such as unrecovered sequence gaps, decode failures by build, and terminal-state divergence. A green socket count is not proof of a green game.
Choosing an implementation boundary
An in-process room log is adequate for a single instance and short matches. Once rooms move between instances or need replay after a process restart, put the ordered event stream behind a durable log and define its retention explicitly. The choice is less about a fashionable transport than about who owns ordering, acknowledgement, and recovery.
WebRTC data channels can carry peer-facing messages, but the application still needs an authority for sequence assignment and backfill; the W3C recommendation describes the transport, not your match semantics. A WebSocket gateway has the same responsibility split. Keep the protocol adapter thin so you can change transport without rewriting the reducer.
The catch is operational scope. This design is not suitable when the product only needs best-effort presence and can discard state on refresh; a simpler request/response flow is easier to operate. Stick with a full event log when scores, receipts, or dispute review must survive reconnects, and budget for retention, migration tests, and dashboards.
Top comments (0)