Short answer: send a small recipient-specific delta when a customer may see only their own queue position; send a whole-queue snapshot only to an authorized display that genuinely needs the entire ordered set. For a small SaaS, the least complex defensible design is usually a versioned delta plus a periodic or on-demand snapshot for repair. Every update carries a monotonically increasing queue version, clients reject older versions, and a detected gap triggers resynchronization.
The page says customer queue display stale, and the on-call sees an apparently healthy publisher beside a growing count of clients whose acknowledged version trails the current queue version. That is the useful symptom. A page based only on publish errors arrives too late, because a successful publish says nothing about whether the customer view converged.
The publisher was green.
This distinction drives the message-size comparison. A snapshot can repair state in one delivery but repeats data that most recipients neither need nor should receive; per-recipient deltas keep the hot path small but require explicit gap detection. Do the byte arithmetic with encoded messages, then choose the smallest disclosure boundary that still meets the recovery SLO.
Should a queue publish one message per waiter?
A logistics dispatch session is a state machine. A shipment request joins a queue, changes position, becomes ready, or leaves. The customer-facing display needs to converge on an authorized projection of that state. If one subscriber misses version 418, version 419 may still arrive, and a green publish counter can coexist with a wrong screen.
The signal that should fire earlier is version lag, measured at the receiving edge: current_queue_version - last_applied_version, grouped by session and bounded by the service-level objective. Transport acknowledgements help diagnose the path, but they do not prove that the UI applied the state. Define the SLO in terms of visible convergence, such as the proportion of active displays that apply the current authorized version within the chosen time window. The target and window must come from the product's tolerance for stale queue positions; there is no honest universal number.
Start with three counters: produced versions, applied versions, and resynchronizations. Add a histogram for convergence time and a gauge for connected displays by session. Do not put customer IDs into metric labels. High-cardinality identity belongs in sampled traces or structured logs with suitable access controls and retention.
One trap is paging on a single late client. A browser in the background, a device losing radio coverage, or a closed tab can all stop acknowledging progress without indicating a service-wide failure. Page on an SLO-burning cohort; retain per-connection lag for diagnosis. Otherwise the alarm measures client churn and trains the on-call to ignore it. The limitation of this cohort view is equally plain: aggregation can hide one customer who has been stale for a long time, so the dashboard still needs a longest-lag view even when that view does not page anyone.
Message size is multiplication, not a payload contest
Compare encoded bytes over the same observation window. Let S be encoded snapshot bytes, D encoded recipient-delta bytes, N connected recipients, C queue changes, and R snapshot repairs. Ignoring shared framing for a moment, whole-queue fan-out costs roughly C * N * S; recipient deltas cost roughly C * N * D + R * S. Add transport framing, authentication metadata, retries, and compression as measured terms rather than intuition.
For a capacity-planning example, suppose a test fixture has 80 waiting customers. Its encoded authorized snapshot is 6,400 bytes and its encoded per-recipient position update is 120 bytes. With 80 connected customers and 10 queue changes, the payload arithmetic is 5,120,000 bytes for repeated snapshots versus 96,000 bytes for deltas, before repairs and framing. These are illustrative inputs, not benchmarks. Serialize your actual schema because field names, identifiers, timestamps, and encoding choices change the result.
The ratio is informative; the absolute result decides capacity.
Eighty recipients matter.
| Decision concern | Recipient delta | Authorized snapshot |
|---|---|---|
| Normal-path bytes | Scales with the changed projection | Scales with the full authorized projection |
| Recovery | Needs gap detection and a snapshot fetch or push | Replaces prior state when accepted |
| Ordering | Version check is mandatory | Version check is still mandatory |
| Privacy boundary | Can disclose one customer's projection | Must be filtered for the display's authorization |
| Server work | More projections or routing keys | Larger serialization and filtering work |
| Client work | Apply a transition | Replace and render state |
Compression does not settle the architecture. It may reduce repeated fields in snapshots, but the sender still constructs the authorized payload and the receiver still handles it. Measure compressed and uncompressed sizes with representative data, then preserve hard limits for both. The trade-off is workload-dependent: a slowly changing, publicly visible depot board can make snapshots the simpler choice, while a private customer view with frequent position changes makes whole-queue publication inappropriate. Recipient deltas are not suitable when clients cannot preserve a version or request repair; snapshots are not suitable when their contents exceed the recipient's authorization boundary.
Instrument the version contract
The useful instrumentation sits beside the state transition. A message needs a session identifier, a queue version, a type, and the smallest authorized payload required for that recipient. This Go sketch shows application semantics, not a transport guarantee:
package queue
import (
"errors"
"fmt"
)
type Update struct {
SessionID string `json:"session_id"`
Version uint64 `json:"version"`
Kind string `json:"kind"`
Position int `json:"position,omitempty"`
}
type View struct {
SessionID string
Version uint64
Position int
}
var ErrVersionGap = errors.New("queue version gap")
func Apply(view View, update Update) (View, error) {
if update.SessionID != view.SessionID {
return view, errors.New("session mismatch")
}
if update.Version <= view.Version {
return view, nil // Duplicate or reordered old delivery.
}
if update.Version != view.Version+1 {
return view, fmt.Errorf("%w: have %d, received %d", ErrVersionGap, view.Version, update.Version)
}
if update.Kind != "position_changed" {
return view, errors.New("unsupported update kind")
}
view.Version = update.Version
view.Position = update.Position
return view, nil
}
Old duplicates become harmless, while a forward gap becomes observable and recoverable. The resync path returns an authorized snapshot with a version; the client replaces local state only when that version is newer. A reconnect begins the same way. The end-to-end claim should remain modest: the display converges under the retry and repair policy you test.
No version, no guarantee.
Persisting the state transition and publishing its notification in unrelated operations creates an ambiguity: either the state commits without an event, or an event escapes before the state does. A durable change record processed asynchronously can close that gap, provided its retry behavior and retention meet the recovery objective. Test the crash points around commit, publication, acknowledgement, and client application.
Test disorder deliberately. Duplicate version 418, deliver 419 before 418, disconnect after receipt but before acknowledgement, expire retained events, and reconnect from an older snapshot. Also test authorization changes during a session. A whole-queue payload cached under yesterday's access decision is a data exposure, not a harmless stale view.
Delivery guarantees belong in the buy-versus-build review
A managed transport can remove socket operations from the on-call queue; a self-hosted path can offer tighter control over retention, placement, and protocol behavior. Neither choice removes the application version contract. Ask where acknowledgement occurs, how long replay remains possible, what ordering scope exists, what happens under subscriber backpressure, and which side owns reconnect storms.
| Operating question | Managed path | Self-hosted path | Decision evidence |
|---|---|---|---|
| Who pages for saturation? | Service boundary may absorb part of it | Platform team owns the full path | Documented responsibility and drill results |
| Can retention meet repair time? | Constrained by the offered contract | Configurable within team capacity | Tested maximum disconnect window |
| How is lock-in contained? | Adapter and export paths matter | Protocol and storage choices still create coupling | Recovery exercise using the abstraction |
| What is the real cost? | Usage plus integration and incident work | Compute, storage, upgrades, and on-call load | Same traffic model and staffing assumptions |
WebRTC data channels are one standards-based option for peer-to-peer data exchange, and the W3C specification defines their behavior and API surface. That does not make a peer mesh the default for a customer queue: topology, authorization, discovery, and recovery remain system responsibilities. Apply the same encoded-byte and convergence tests to any transport.
The deployment gate should include a shadow calculation of both message shapes against production-like fixtures, a load test with synchronized reconnects, and a rollback rule tied to convergence rather than publisher throughput alone. Capacity must include headroom for repair traffic. If every client requests a snapshot after a brief network interruption, the recovery path can become the largest burst in the system.
Set the alert where action changes
Use recipient deltas for routine customer updates when authorization is narrow and changes are frequent. Use a versioned snapshot for initial load, verified gaps, and displays authorized to render the whole queue. If a client cannot implement version checks and repair, repeated snapshots may be simpler, but account for disclosure and fan-out before accepting that simplicity.
Then tune the page. An alert threshold below the normal tail of client behavior creates noise, interrupts sleep, and consumes the same engineering capacity needed to improve recovery. A threshold that waits until many displays are stale protects the pager while abandoning the SLO. Start from the error-budget burn that requires human action, validate it during disconnect and reconnect drills, and keep lower-severity signals on a dashboard for trend work.
False positives have a concrete cost: an engineer investigates transport health while the underlying event is a backgrounded browser or a session that has already ended. Filter inactive sessions, distinguish disconnected from connected-but-behind, and require enough affected traffic to justify a page. The final architecture is credible only when its alert points to an action the on-call can take.
References
- W3C, WebRTC 1.0: Real-Time Communication Between Browsers: https://www.w3.org/TR/webrtc/
Top comments (0)