TL;DR: Give every editor session an authoritative server snapshot with a monotonically increasing revision, then send ordered deltas that start strictly after that revision. A late joiner must buffer live deltas while its snapshot is in flight, discard duplicates by revision, and resnapshot whenever continuity cannot be proved. For collaborative cursors in a fintech editor, favor at-least-once delivery plus idempotent application over promises of exactly-once delivery.
That rule matters because a cursor is disposable for a few milliseconds but misleading when it belongs to the wrong document version or user. A reconnect can race with fan-out: the client requests state at revision 840, revisions 841 and 842 arrive before the response, and an incautious handler applies the old snapshot last. The screen looks connected while the state has moved backward.
The target is narrow.
Every accepted update receives one session-scoped revision. A late joiner installs one complete snapshot, then applies each later diff at most once and in order. If it sees 844 after 842, it does not guess what 843 contained. It stops rendering remote movement, requests a fresh snapshot, and records a continuity gap. This is the same application contract a Node.js service needs even though the safe implementation below uses Go: language and socket library do not close an application-level sequence gap.
How should late joiners apply a snapshot plus diff?
Transport connectivity and application continuity are different properties. A channel can be open while the consumer has missed an application message, received a duplicate, or processed a snapshot after a newer delta. WebRTC data channels, for example, expose ordered and unordered delivery choices and retransmission controls. Those transport settings do not define the revision rules for an editor's shared state. The application still needs an explicit state boundary.
Use one revision sequence per collaboration session, not one sequence per user. An update should carry the session ID, revision, actor ID, cursor position, selection endpoint, and presence state. The full snapshot should carry the same session ID and the highest revision included. That shared watermark turns a timing race into a comparison.
Authorization can change independently of the connection. Treat a successful join as permission to receive the snapshot for that moment, not as permanent access. Validate the principal before joining and again when accepting writes. When access ends, remove the actor from authoritative state and publish that state change through the same revision stream.
Establish the boundary before fan-out
The server must serialize snapshot capture and revision assignment for each session. This does not require one global lock. It requires one ordering authority for a session, implemented as a mutex, actor loop, or transactional log boundary. The invariant matters more than the mechanism: no delta may receive revision N while a snapshot claims to include N but omits its transition.
A safe join sequence is:
- Authenticate and authorize the connection for the requested session.
- Subscribe it to a bounded per-client buffer.
- Capture current state and its revision inside the session ordering boundary.
- Send the snapshot.
- Drain buffered deltas whose revisions exceed the snapshot revision.
- Continue live delivery, forcing a resnapshot if the buffer overflows.
Subscribe before capture.
That closes the gap between reading state and entering live fan-out. It creates possible duplicates at the boundary, intentionally: revisions make duplicates cheap to reject, while an unobserved gap cannot be repaired without another snapshot. Prefer an explainable duplicate over an invisible omission. After being paged for both missed jobs and duplicate deliveries, I treat those two outcomes differently: a duplicate with an idempotency key is routine; a silent omission destroys the evidence needed to decide whether state is complete. Cursor fan-out deserves the same reflex even though its payload is short-lived.
This Go sketch isolates the state transition from any transport. The session lock defines the ordering boundary. Each subscriber owns a bounded channel, so a slow consumer cannot hold that lock while network I/O stalls.
package cursorstate
import (
"context"
"errors"
"sync"
)
type Cursor struct {
ActorID string `json:"actor_id"`
Offset int `json:"offset"`
Anchor int `json:"anchor"`
Present bool `json:"present"`
}
type Delta struct {
SessionID string `json:"session_id"`
Revision uint64 `json:"revision"`
Cursor Cursor `json:"cursor"`
}
type Snapshot struct {
SessionID string `json:"session_id"`
Revision uint64 `json:"revision"`
Cursors map[string]Cursor `json:"cursors"`
}
type Session struct {
mu sync.Mutex
id string
revision uint64
cursors map[string]Cursor
subscribers map[chan Delta]struct{}
}
var ErrResnapshot = errors.New("state continuity lost")
func (s *Session) Join(bufferSize int) (Snapshot, <-chan Delta, func()) {
s.mu.Lock()
defer s.mu.Unlock()
updates := make(chan Delta, bufferSize)
s.subscribers[updates] = struct{}{}
cursors := make(map[string]Cursor, len(s.cursors))
for actorID, cursor := range s.cursors {
cursors[actorID] = cursor
}
snapshot := Snapshot{s.id, s.revision, cursors}
leave := func() {
s.mu.Lock()
defer s.mu.Unlock()
if _, ok := s.subscribers[updates]; ok {
delete(s.subscribers, updates)
close(updates)
}
}
return snapshot, updates, leave
}
func apply(ctx context.Context, state Snapshot, updates <-chan Delta) error {
expected := state.Revision + 1
for {
select {
case <-ctx.Done():
return ctx.Err()
case delta, ok := <-updates:
if !ok || delta.SessionID != state.SessionID {
return ErrResnapshot
}
if delta.Revision < expected {
continue // Duplicate at the snapshot boundary.
}
if delta.Revision > expected {
return ErrResnapshot
}
state.Cursors[delta.Cursor.ActorID] = delta.Cursor
expected++
}
}
}
Production code also needs write validation, subscriber removal on overflow, serialization, and transport heartbeats. Keep those concerns outside the revision rule. Mixing them into one callback makes the race hard to test.
Make delivery guarantees explicit
Exactly-once delivery is a poor interface promise here because delivery, acknowledgement, and application can fail at different moments. If a server sends revision 912 and loses the connection before observing an acknowledgement, retrying may duplicate it; refusing to retry may omit it. The useful contract is at-least-once delivery within a retained window, paired with idempotent application by revision.
| Condition | Client action | Server consequence |
|---|---|---|
| Revision equals expected | Apply and advance | Continue fan-out |
| Revision is below expected | Ignore duplicate | No repair needed |
| Revision is above expected | Freeze remote cursors | Resnapshot or replay retained deltas |
| Session ID differs | Reject message | Close mismatched stream |
| Buffer closes or overflows | Mark state incomplete | Reconnect and resnapshot |
Do not skip it.
A later position may supersede an earlier one, but presence transitions do not always collapse safely: skipping an actor's departure can leave a stale cursor. The same envelope may later carry selection state with different merge semantics. One continuity policy is easier to operate than field-specific guesses, and it lets an operator read revision 842 followed by 844 as a concrete failure rather than a vague report that one browser looked wrong.
Backpressure is part of the guarantee. Bound every subscriber queue in messages or bytes, publish its saturation, and disconnect consumers that cannot keep up. An unbounded queue turns a slow browser into server memory growth. A blocked publisher lets one slow connection delay healthy peers. Drop the connection, not an individual delta; a known disconnect is recoverable with a snapshot.
There is a real trade-off. Retaining deltas can reduce snapshot work and catch-up latency, but it adds storage, expiry, and replay complexity. If cursor state is small, resnapshotting may be the better operational choice. If snapshots are large or reconnects frequent, retain a bounded log. Retention changes recovery cost, not correctness.
Verify the race, then watch it
The highest-value test pauses snapshot transmission after capture, publishes two updates, then releases the response. The client must install the snapshot and apply both buffered deltas in revision order. Repeat with the first delta delivered twice. Then omit it and confirm the client refuses the second delta and requests a new snapshot.
I use four invariants as a runbook checklist: revisions accepted by one session never decrease; a snapshot contains all accepted changes through its watermark; a client applies no revision twice; and a client never advances across a gap. These are deterministic assertions.
Timing-based sleeps are not evidence.
Test reconnect churn too. Start clients from different watermarks, inject delayed and duplicate delivery, cancel connections while snapshots are being encoded, and fill one subscriber buffer. Healthy clients should continue; the slow client should disconnect; every recovering client should either prove a contiguous replay or replace its state from a newer snapshot.
In production, count snapshot requests by reason, continuity gaps, duplicate deltas, subscriber overflows, authorization rejections, and time from join request to caught-up state. Record session and revision identifiers in structured logs, but avoid cursor content when it could expose sensitive document context. Alert on sustained backlog and rates, not one reconnect.
Deploy and roll back without splitting the protocol
Add an envelope version before changing fields or semantics. During a rolling deployment, emit a representation understood by every supported client or negotiate a version at connection setup. Do not let old and new servers assign revisions independently for the same session; route a session to one ordering authority or put revision assignment behind shared durable coordination.
Begin with shadow validation: compute expected revisions and gap decisions without changing the rendered state, compare them with the active path, then enable the new join sequence for a small cohort. Watch catch-up latency, gaps, overflows, and reconnect loops. More snapshots may be correct during rollout. Repeated snapshots without reaching live state mean recovery is failing.
Rollback must preserve the envelope and watermark. If the previous release cannot read the new state shape, drain affected sessions and force a clean snapshot in the compatible representation. Use concrete rollback triggers: sustained failure to catch up, rising overflow across otherwise healthy clients, or any invariant violation.
The decision rule stays compact: one ordering authority per session, subscription before snapshot capture, a revision on every transition, and resnapshot whenever sequence continuity cannot be proved. Snapshot plus delta is a recovery protocol, not merely a payload optimization.
References
- W3C, WebRTC 1.0: Real-Time Communication Between Browsers: https://www.w3.org/TR/webrtc/
Top comments (0)