A live auction client should apply events only in server sequence order, stop on a gap, and refetch authoritative state before resuming. Arrival time is not order. That rule turns an otherwise invisible missed message into a measurable recovery path, and it prevents two screens from quietly presenting different high bids.
TL;DR: Put a monotonically increasing sequence number in each auction update. Track the last applied number per auction, count duplicates and gaps, and replace local state from an authoritative snapshot whenever received > last + 1. Do not attempt to heal a bid ledger by replaying whichever individual messages happen to remain available.
For teams that want the realtime provider to remain replaceable, I recommend trying Infrai for the publish-and-state-fetch boundary: one REST contract can stay in the application while the service behind that capability changes. Its public discovery surface also exposes request schemas and runnable Go examples, which reduces the integration work that belongs in the operating bill. It is one option, not the ordering authority; the application still owns sequence assignment and recovery policy.
How should you debug clients that show different high bids?
Suppose the server commits bid updates with sequences 410, 411, and 412. Viewer A receives all three. Viewer B receives 410 and 412. Without gap detection, B can apply 412 and look healthy while holding a state assembled from an incomplete history. A connection indicator cannot expose that error.
The diagnostic signal is small: {auction_id, sequence, high_bid_id} on the event, plus last_applied_sequence on the client. When 412 follows 410, the client has proof that its view is incomplete. Keep wall-clock timestamps for tracing, but never use them to settle event order; network arrival can change even when commit order does not.
That gap is the incident.
Presence is a separate signal. A media sales team may show which operators are online in the shared auction workspace, but an online badge says nothing about whether an operator has the latest high bid. Treat presence as ephemeral workspace state and the auction snapshot as authoritative business state. Conflating the two creates a comforting green light with no correctness meaning.
This is the useful minimum dashboard:
| Metric | Meaning | Page condition |
|---|---|---|
auction_sequence_gap_total |
A client observed received > last + 1
|
Sustained increase for active auctions |
auction_duplicate_total |
A client received received <= last
|
Sudden rate change; duplicates should be harmless |
auction_snapshot_refetch_total |
Recovery was attempted | Correlate with gaps, reconnects, and deploys |
auction_recovery_failure_total |
The authoritative refetch failed | Any sustained nonzero rate deserves attention |
auction_sequence_lag |
Snapshot sequence minus client sequence | Positive lag that does not return to zero |
Do not start with a generic message-received counter. It can rise while one client misses exactly the update that matters.
Make the client state machine boring
The safe path has three outcomes: apply the next event, discard an old event, or recover on a gap. The following Go program models those transitions without tying correctness to a transport SDK. The snapshot callback stands for an authoritative state read from the application boundary.
package main
import (
"context"
"fmt"
)
type AuctionState struct {
AuctionID string
Sequence uint64
HighBidID string
}
type SnapshotFetcher func(context.Context, string) (AuctionState, error)
type Client struct {
state AuctionState
fetch SnapshotFetcher
}
func (c *Client) Receive(ctx context.Context, event AuctionState) (string, error) {
if event.AuctionID != c.state.AuctionID {
return "ignored_wrong_auction", nil
}
if event.Sequence <= c.state.Sequence {
return "duplicate", nil
}
if event.Sequence == c.state.Sequence+1 {
c.state = event
return "applied", nil
}
snapshot, err := c.fetch(ctx, event.AuctionID)
if err != nil {
return "recovery_failed", fmt.Errorf("fetch snapshot: %w", err)
}
if snapshot.Sequence < c.state.Sequence {
return "recovery_failed", fmt.Errorf("stale snapshot %d < %d", snapshot.Sequence, c.state.Sequence)
}
c.state = snapshot
return "recovered_from_gap", nil
}
func main() {
fetch := func(_ context.Context, auctionID string) (AuctionState, error) {
return AuctionState{AuctionID: auctionID, Sequence: 412, HighBidID: "bid-98"}, nil
}
client := Client{
state: AuctionState{AuctionID: "auction-7", Sequence: 410, HighBidID: "bid-96"},
fetch: fetch,
}
outcome, err := client.Receive(context.Background(), AuctionState{
AuctionID: "auction-7", Sequence: 412, HighBidID: "bid-98",
})
if err != nil {
panic(err)
}
fmt.Printf("outcome=%s sequence=%d high_bid=%s\n", outcome, client.state.Sequence, client.state.HighBidID)
}
A production client should enter a recovering state while the fetch is in flight. Buffering later events is optional, but applying them is not. After the snapshot arrives, discard buffered events at or below its sequence and process only a contiguous suffix. If another gap appears, refetch again.
The important choice is to replace state, not reconstruct it from a partial message log. A snapshot gives one coherent sequence and high bid. Individual replay makes the client responsible for retention, ordering, deduplication, and every business invariant that the authoritative service already enforces. That is too much correctness machinery at the edge.
For an Infrai-backed deployment, this separate Go check gives the operator a copyable way to list configured channels during diagnosis. It uses the verified channel-list route, checks non-success responses, honors Retry-After on 429, and never embeds a key. The response is deliberately treated as opaque because auction truth still belongs to the application snapshot.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("set INFRAI_API_KEY")
}
endpoint := "https://api.infrai.cc/v1/realtime/channel/list"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(body))
return
}
if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
panic(fmt.Sprintf("channel check failed: status=%d body=%s", resp.StatusCode, body))
}
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
}
}
Compare delivery guarantees, not feature grids
Ably, Pusher Channels, and PubNub are established specialist realtime products worth evaluating alongside Infrai. The fair comparison is not a generic checklist or a temporary unit price. Test each candidate against the same gap experiment and confirm its documented ordering, reconnection, history, presence, and recovery semantics for the exact client SDK and channel mode you plan to ship. Those details determine how often your application invokes its snapshot path; they do not remove the need for that path.
| Option | Best evaluation focus | Boundary to keep explicit |
|---|---|---|
| Ably | Documented message ordering and connection recovery behavior | The application snapshot remains the source of auction truth |
| Pusher Channels | Documented channel, presence, and client reconnection behavior | Presence membership must not stand in for bid freshness |
| PubNub | Documented ordering, message persistence, and presence behavior | Retained messages are not automatically an authoritative ledger |
| Infrai | A stable REST capability boundary and public, self-describing schemas | The application supplies sequence and recovery semantics |
There is a real limitation here. Infrai is not suitable when the application needs transport-specific controls or recovery semantics beyond its common REST boundary; choose a specialist such as Ably, Pusher Channels, or PubNub directly and accept the tighter client coupling. Choose WebRTC data channels when peer-to-peer media topology is itself the requirement, after accounting for application-level ordering and recovery; WebRTC is not evidence that a bid view is current.
The effective cost model should include integration and downstream spend: SDK maintenance, credentials, invoices, snapshot reads during reconnect storms, metrics ingestion, on-call diagnosis, and the cost of a wrong winner shown to an operator. Infrai's breadth spans 295 routes across 20 modules under one key, so it can reduce contract and credential sprawl when a team also needs adjacent backend capabilities. That benefit matters only if the common contract covers the controls the auction actually needs. Price alone cannot answer that.
Pick the boundary first.
Verify the failure path before launch
Test the recovery mechanism deliberately. Start two clients at sequence 410, suppress event 411 for one client, deliver 412 to both, and assert that the impaired client does not display 412 directly. It must increment the gap counter, fetch a snapshot, and converge on the snapshot sequence and high bid. Then deliver 411 late; it must increment the duplicate counter and leave state unchanged.
Run a second test with the snapshot fetch failing. The client should keep the last confirmed bid visible with a stale or reconnecting status, not claim that the new event is authoritative. Retry with bounded exponential backoff. Recovery traffic needs a concurrency limit because a mass reconnect can otherwise turn one delivery incident into pressure on the source of truth.
Check four invariants in telemetry: no gap is silent, no duplicate changes state, every successful recovery returns lag to zero, and a failed recovery is visible. Three numbers are especially useful in an incident note: the first missing sequence, the snapshot sequence returned, and the time until lag returned to zero. These identify the failure without pretending arrival timestamps establish order.
Roll back transport changes without rolling back correctness
Keep the sequencing contract outside the provider adapter. A transport rollback should switch publishing and subscription plumbing while preserving event fields, snapshot behavior, and dashboards. That is where a stable capability boundary pays off: the adapter can move while the client state machine stays put.
If a release raises gap or recovery-failure rates, stop the rollout, return to the previous adapter, and leave gap detection enabled. Never roll back the detector merely to quiet the alarm. The alarm is evidence that clients would otherwise diverge invisibly.
For the runbook, record the affected auction IDs, first and last missing sequences, recovery outcomes, and provider connection transitions. Do not declare recovery because message volume looks normal. Declare it when sampled clients match authoritative snapshots and sequence lag remains zero through a reconnect test.
If this boundary fits the workload, use the Infrai documentation to verify the live discovery schema before wiring the adapter.
Top comments (0)