A logistics auction cannot tolerate presence that merely looks plausible. If two dispatchers see different high bids, treat the bid stream as an ordered, recoverable log and treat the video room as a separate, scoped collaboration surface. Do not use peer arrival, a data-channel callback, or wall-clock time as the authority for who won.
TL;DR: assign every accepted bid a server-side monotonic sequence within one auction, return the current sequence when a client joins or resumes, reject tokens whose auction and room scopes do not match, and measure sequence gaps, replay lag, stale-view duration, and presence reconciliation latency. Those four metrics distinguish an ordering defect from a missed delivery, a slow recovery, or a presence display that has drifted away from authoritative membership.
The bounded incident I would investigate is this: a carrier and a dispatcher enter the video room for auction A-1042; the carrier submits 18,400, then another carrier submits 18,650; one screen still highlights 18,400 while the other highlights 18,650. The tempting diagnosis is "WebRTC dropped the bid." That conclusion skips two boundaries. WebRTC defines peer connections and data channels, but the application still chooses its signaling, authorization, ordering, persistence, and recovery semantics. Presence is also a projection of session state, not proof that every bid event reached every browser.
The invariant is narrower and more useful: for a given auction, every authorized client that has recovered through sequence n must derive the same high bid from the same accepted event prefix. If that invariant holds, video may reconnect and presence may briefly reconcile without changing the commercial result. If it fails, a green participant dot is irrelevant.
How should clients debug different high bids and missed messages?
A connected media session answers the wrong question. It says nothing about whether two clients consumed the same application events in the same order. A bid can be accepted while a browser is suspended, a transport can reconnect after an intermediary times out, or two network paths can deliver application messages with different delay. A client that applies "last callback wins" then turns arrival order into auction truth.
Wall clocks make this worse. Device time can differ, and a timestamp does not establish a unique order when events share a resolution or traverse different paths. The accepting service should choose order once, inside the auction's consistency boundary, and publish that decision as an integer sequence. Clients may display timestamps, but they should reduce state by sequence. Missed messages leave a different fingerprint. Suppose a client has applied sequences 71 and 73. That is a gap, even when 73 carries the largest amount. The client must stop claiming it is current, request recovery beginning at 72, and only restore a current indicator after it has applied a contiguous prefix through the server's advertised head. Quiet corruption is worse than a visible stale state. Presence has its own race. A scoped room token can authorize entry to the collaboration room, while auction membership controls access to bid events; binding both claims to the same auction identifier prevents a token for one load from becoming an accidental subscription to another. Token expiry and peer departure should initiate reconciliation, but neither should delete accepted bids. Commercial state outlives a participant session. During triage, inspect the authoritative head before staring at packet timing: if the two clients report heads 73 and 73 but one last-applied value is 71, recovery is the suspect; if both applied 73 yet compute different winners, the reducer or snapshot is suspect; if bids agree while rosters differ, isolate presence reconciliation. Three comparisons replace a broad accusation against realtime with a bounded subsystem and a falsifiable next check.
No cursor, no proof.
The four measurements that separate the failure modes
Start with sequence-gap count, labeled by auction and client class rather than by individual user. Increment it whenever the next received sequence is greater than lastApplied + 1. A burst points to delivery or resume behavior; a persistent gap means recovery is not closing the prefix. Avoid user IDs and raw token identifiers in metric labels, because their cardinality grows without a capacity bound.
Replay lag is authoritativeHead - lastApplied. It converts a vague report such as "the tablet is behind" into a backlog measured in events. Pair it with replay duration in histograms, since a lag of 20 recovered in 40 milliseconds has a different operational meaning from the same lag persisting through the bidding window. Capacity planning follows naturally: test recovery at the largest credible disconnect backlog, not just steady-state fan-out.
Stale-view duration measures the time from detecting a gap or disconnect until the client again holds a contiguous prefix at the advertised head. This is the metric I would attach to the auction-view SLO. Availability alone is too generous: a page can respond, video can flow, and the decision surface can still be stale. Define the acceptable threshold from the business window, then budget the path across detection, fetch, apply, and render.
Presence reconciliation latency measures how long the displayed roster differs from the room authority after a join, leave, expiry, or reconnect is observed. It is intentionally separate from bid freshness. Combining them into one "realtime health" number hides the exact subsystem that is failing.
Measure both.
A useful trace carries auction_id, room_id, connection_id, event_seq, last_applied, and a generated resume attempt ID. It should not carry the scoped token. Logs can then answer whether sequence 72 was accepted, published, requested during replay, applied, and acknowledged without turning credentials into telemetry.
Make recovery part of the protocol
The smallest defensible protocol has three behaviors: snapshot with a head sequence, ordered events after that head, and replay from an explicit sequence. The snapshot and stream handoff must not create an uncovered interval. One practical pattern subscribes first, reads the snapshot and head, buffers newer events, then applies the buffer in order; another obtains both from a storage system with a consistent cursor. The technology matters less than proving the boundary under concurrency.
The Go example below shows the client-side reducer, where the important choice is to refuse a noncontiguous update. It does not pretend that an in-memory map is durable storage or a cross-process sequencer.
package auction
import (
"errors"
"fmt"
)
type Bid struct {
AuctionID string
Sequence uint64
Amount int64 // Minor currency units.
BidderID string
}
type View struct {
AuctionID string
LastApplied uint64
HighBid Bid
Current bool
}
var ErrGap = errors.New("bid sequence gap")
func (v *View) Apply(b Bid) error {
if b.AuctionID != v.AuctionID {
return fmt.Errorf("auction scope mismatch")
}
if b.Sequence <= v.LastApplied {
return nil // Duplicate or already covered by a snapshot.
}
if b.Sequence != v.LastApplied+1 {
v.Current = false
return fmt.Errorf("%w: have=%d received=%d", ErrGap, v.LastApplied, b.Sequence)
}
if b.Amount > v.HighBid.Amount {
v.HighBid = b
}
v.LastApplied = b.Sequence
v.Current = true
return nil
}
A production consumer should respond to ErrGap by coalescing concurrent recovery attempts, fetching from LastApplied + 1, verifying auction scope on every event, and limiting replay work. If the retained log no longer covers the requested sequence, fetch a fresh snapshot rather than guessing across the hole. Backpressure also needs a policy: bounded buffering, an observable overflow counter, and snapshot recovery are safer than unbounded memory growth during a slow client episode.
Test the protocol with permutations, not a single happy-path integration test. Duplicate 72, delay 72 until after 73, reconnect after 71, expire the room token during replay, and deliver an event carrying the wrong auction ID. Assert that both clients either converge on the same contiguous prefix or identify themselves as stale. Never assert that callback order happens to match send order across separate connections.
Recovery is load.
Buying the transport or building the state path
The managed-versus-self-hosted decision does not remove the auction invariant. It changes who operates connection fan-out, media traversal, retention, and recovery plumbing. I use presence accuracy as the first filter because an inexpensive system that cannot expose membership authority or resume position transfers its cost into on-call ambiguity.
The trade-off is operational ownership.
| Approach | Team operates | Lock-in surface | Presence and recovery question |
|---|---|---|---|
| Managed media plus managed messaging | Auction reducer, scopes, and verification | Connection, event, and identity APIs | Can membership changes and resume cursors be observed independently? |
| Managed media plus owned event log | Bid ordering, retention, replay, and scopes | Media session API | Can room identity be bound to the owned auction identity? |
| Self-hosted media and event path | Traversal, fan-out, ordering, storage, upgrades, and paging | Internal protocols and deployment model | Is the on-call team staffed to prove both roster and event convergence? |
The middle option often creates the clearest fault boundary for analysis, but it is not a universal recommendation. The right column is the procurement test. Demand evidence that a reconnect can resume from a known cursor, that room membership has an authoritative query or event model, and that exported telemetry supports the four measurements without high-cardinality labels. Then load-test the evidence. Documentation promises do not establish an SLO.
Capacity plans should include peak concurrent rooms, participants per room, bid-event burst rate, retained events per auction, replay amplification after a regional network interruption, and the CPU cost of rebuilding snapshots. Model the recovery storm. Steady-state throughput is usually the flattering number; simultaneous resumes reveal whether replay competes with new bids and stretches stale-view duration.
Where this design does not apply
This approach has real limitations. A sequence per auction is unnecessary for a purely social video room whose messages have no shared derived winner and can expire without recovery; choose an ephemeral room channel there. It is also insufficient when bids must coordinate across multiple auctions in one atomic rule; that requires a wider consistency boundary and a carefully defined global operation, not concatenated local sequences. The trade-off is extra storage, replay traffic, and reducer complexity in exchange for an auditable order. If the business permits best-effort messages and no shared winner, paying that cost is hard to defend.
Use the narrower tool.
For a single-authority live auction, though, the decision rule is firm: order accepted bids once, recover by cursor, and expose staleness until recovery completes. Keep room presence scoped and measurable, but never let it impersonate bid correctness. The outcome is operationally plain: two clients with the same contiguous prefix compute the same high bid, while any client without that prefix says it is behind.
Top comments (0)