When clients show different high bids, the live connection is rarely enough to explain which screen is wrong: a bid is authoritative only after the auction service commits it, while every client renders an incomplete, delayed view of that history. In a healthtech procurement editor, clinicians may be co-editing a formulary note while their cursors move ephemerally, but the adjacent auction panel must converge on the same accepted high bid after any reconnect.
TL;DR: assign a monotonic sequence at the authoritative ordering boundary, persist accepted bids before publishing them, and resume from each client's last contiguous sequence. Measure sequence gaps, recovery duration, consumer lag, and per-auction view divergence. Timestamps and connection state aid diagnosis; neither establishes bid order.
Why can healthy connections show different high bids?
A green connection indicator proves little. One browser may have missed sequence 184 during a network transition, another may have observed 185 before 184, and both can be connected now. The useful question is: which committed event prefix has each client applied?
Start at the authority boundary. The auction service validates a bid, serializes competing writes for one auction, commits the accepted event, and allocates its sequence in the same transaction or equivalent atomic operation. A browser timestamp cannot do this job. Clocks differ, equal timestamps occur, and network arrival order is not business order. Use timestamps for latency analysis and a server sequence for ordering.
The event needs enough identity to make retries harmless and enough position to expose a hole:
{
"auction_id": "rx-482",
"sequence": 185,
"event_id": "01J...",
"bid_id": "bid-7f31",
"amount_minor": 425000,
"currency": "USD",
"accepted_at": "2026-09-22T09:14:31.482Z"
}
Keep cursor movement out of that durable bid log. Collaborative cursors are high-rate presence signals: after a disconnect, the current position is useful and the old trail is not. Accepted bids are the opposite. Every committed transition affects the result and must be recoverable. Sharing a connection is possible, but sharing retention and recovery semantics creates avoidable failure modes.
This separation also limits sensitive data. Cursor messages need an editor-session pseudonym and position, not patient identifiers. Auction events need procurement identifiers, not clinical note contents. Narrow schemas reduce what can leak through logs, traces, and recovery tooling.
Put recovery ahead of transport
WebSocket, standardized by RFC 6455, provides a two-way channel over one TCP connection. TCP's current specification is RFC 9293. It preserves byte order within that connection, but a reconnect creates a new connection; it does not restore application messages the browser never received. WebRTC data channels expose ordered and retransmission settings, yet they do not define durable auction history or a resume contract. The application still needs a cursor into committed state.
That cursor should mean "last contiguous sequence applied," not "largest sequence observed." If a client receives 185 after applying through 183, recording 185 as its checkpoint hides the missing event. Buffer 185 briefly, request backfill from 184, and advance only after both have been applied.
from dataclasses import dataclass, field
from typing import Any
@dataclass
class OrderedView:
next_sequence: int
pending: dict[int, dict[str, Any]] = field(default_factory=dict)
def receive(self, event: dict[str, Any]) -> list[dict[str, Any]]:
sequence = int(event["sequence"])
if sequence < self.next_sequence:
return [] # Duplicate delivery after a retry or reconnect.
self.pending[sequence] = event
applied = []
while self.next_sequence in self.pending:
applied.append(self.pending.pop(self.next_sequence))
self.next_sequence += 1
return applied
def missing_from(self) -> int | None:
if self.pending and min(self.pending) > self.next_sequence:
return self.next_sequence
return None
For a long absence, return a snapshot with an explicit snapshot_sequence, then events greater than that boundary. Apply the snapshot atomically before replaying its tail. Without the boundary, an event racing with snapshot generation can be omitted or counted twice.
Backfill must be bounded. A client should not retain an unlimited pending map while waiting for a vanished sequence; cap it, pause live application at the cap, and fetch a newer snapshot. The trade-off is direct: a larger buffer absorbs longer reorder windows but consumes more memory and delays the point at which the client admits it needs a snapshot. The limits depend on measured event rates and retention, so they are capacity settings, not universal constants.
Cursor recovery remains simpler: send the newest position for each active editor after presence is re-established. Do not replay obsolete movements.
Order first.
Measure the sequence, not merely the socket
Connection counts, reconnects, and ping latency are useful transport signals. They cannot say whether two clients agree on the leader. Add metrics at ordering, publication, recovery, and rendering boundaries.
| Signal | Bounded dimensions | Diagnostic value |
|---|---|---|
| Published minus committed sequence | shard, process role | Publisher lag |
| Client contiguous sequence lag | app version, region | Slow consumers |
| Gap detections | app version, transport | Missed or reordered delivery |
| Backfill duration | result class, region | Recovery health |
| View digest mismatch | app version, cohort | Reducer defects or missing data |
Avoid auction_id, user_id, and event_id as metric labels. Their cardinality grows with traffic. Put those identifiers in sampled traces or structured logs, subject to retention and access controls; metrics should use bounded dimensions such as region, version, shard, and recovery result.
A view digest is especially useful. Have each client report its last contiguous sequence plus a hash of the canonical projection: leader bid ID, integer amount, currency, status, and sequence. Compare a client digest with the server digest only when both claim the same sequence. A mismatch at equal sequence implicates reducer logic or nondeterministic handling; unequal sequences implicate delivery or recovery.
Compare equal prefixes. Always.
Do not hash formatted currency, localized text, or cursor state. OpenTelemetry defines shared concepts for metrics and traces, but sampling remains an operational choice. Preserve an event ID and sequence in diagnostic records so an engineer can move from an alert to the committed event without putting high-cardinality values on every time series.
Alert on outcomes over a window: sustained sequence lag, recovery failures, or digest mismatches at equal prefixes. A reconnect spike may only reflect a mobile network transition with successful recovery. Paging on every reconnect hides the delivery gaps that actually matter.
Reconstruct one chain of custody
For a report, capture the auction ID, both contiguous sequences, application versions, regions, and time range. Suppose one screen displays sequence 183 while another displays 185. Do not begin by comparing their wall-clock timestamps. First confirm that 184 and 185 exist in the committed log, then find the publication record for each, then compare the two clients' contiguous checkpoints. If the lagging client reports 183 and has buffered 185, the gap is explicit: its recovery request must begin at 184. If it reports 185 but renders the projection for 183, delivery succeeded and the checkpoint contract or reducer is broken. Trace that same event through gateway delivery, client receipt, reducer application, and rendered digest. The first missing boundary narrows ownership, while the equal-sequence digest separates state-transfer trouble from rendering logic.
Check four cases in order:
- If accepted bids have an ambiguous per-auction position, fix serialization at the write boundary. Subscribers cannot repair undefined order.
- If sequence 184 is committed but absent from publication, inspect the transactional outbox or equivalent commit-to-publish handoff. Publishing before commit can expose rolled-back state; committing without a durable handoff can strand an event.
- If the gateway sent 184 but the checkpoint stops at 183, inspect disconnect timing and resume. A delivery attempt is not an acknowledgement of reducer application.
- If both clients applied through 185 but equal-prefix digests differ, replay the fixture through each application version. Check integer money handling, locale-sensitive parsing, mutable reducers, and tie-break rules.
Transmit money as an integer amount in minor units with an explicit currency, and compare integers everywhere. Cursor coordinates can tolerate visual interpolation. Bid values cannot tolerate client-specific rounding.
Test with deliberate cuts. Disconnect before publication, after gateway receipt but before local application, and while backfill overlaps new live events. Duplicate an event. Delay one sequence. Generate a snapshot while another bid commits. The invariant stays concise: clients at the same contiguous sequence must produce the same canonical projection.
Roll out the recovery contract
Introduce sequences and event IDs before enforcing resume behavior. During a shadow phase, retain the existing display path and compute the canonical projection beside it; compare digests only at equal sequence boundaries. This exposes reducer differences without changing the visible auction.
Then enable resumable backfill for a small cohort grouped by application version and region. Watch gap rate, completion, sequence lag, buffer saturation, and digest mismatches. Increase exposure only after forced-disconnect tests converge. Rollback should select a known snapshot path, not erase sequence metadata.
Finally, preserve the split in the healthtech editor: durable ordered recovery for accepted auction state, latest-value refresh for collaborative cursors. The transport may change. The correctness contract should not.
Top comments (0)