Trkfpn392751

Posted on Oct 19

Live Auction Recovery Explained — Node.js Detects Missed Realtime Messages by Sequence

#node #realtime #sre

A page that says "notifications are delayed" is already late. The least complex reliable design is to put a monotonically increasing sequence number on every auction update, persist the client's last applied number, and refetch an authoritative snapshot whenever a reconnect reveals a gap. Do not replay blindly into UI state.

TL;DR: treat the realtime channel as a prompt to reconcile, not as the database. If a browser has applied sequence 1842 and next receives 1845, it must stop applying deltas, fetch current state through Express, then resume from the sequence returned with that snapshot. This gives an in-app notification feed a clean recovery path after sleep, Wi-Fi changes, deploys, and dropped connections.

The page the on-call sees should identify the damaged invariant: realtime_sequence_gap_total rose while snapshot recovery success fell. A generic disconnect count is weaker. Disconnects happen during ordinary navigation; an unrecovered gap means a user may be looking at a stale bid, status, or notification.

I have been paged by missed jobs and duplicate deliveries; both make a healthy transport graph look far more comforting than it should.

How can Node.js detect missed realtime messages by sequence?

Assume each live-auction event has an auction-scoped sequence, and the snapshot contains the latest sequence plus the complete client-visible state. Sequence 1845 proves only that at least two positions were not observed by this client. It does not prove which messages were lost, and it does not make 1845 safe to apply on top of state from 1842.

Freeze incremental application. Fetch the snapshot. Replace local state atomically, record the snapshot sequence, and ignore any buffered event at or below that value. Events above it can then be considered in order. If another gap appears, reconcile again.

Stop there.

That rule is deliberately conservative. A replay log can reduce snapshot traffic, but it introduces retention windows, cursor validation, compaction behavior, and a second recovery failure mode. For a B2B SaaS screen where correctness matters more than squeezing every byte from reconnects, a bounded snapshot is often the better first implementation.

Use one sequence domain consistently. An auction-scoped counter is easy to reason about for an auction view; a global counter will naturally contain gaps caused by unrelated auctions and will create false alarms. The server must assign the number in the same ordered commit path that makes the represented state authoritative. Assigning it earlier allows a failed write to leave a permanent hole.

This design has a real limitation: full refetch is unsuitable when authoritative state is too large to transfer within the recovery objective. In that case, use a retained change log with a validated cursor, a bounded replay window, and snapshot fallback after cursor expiry. The trade-off is operational weight. The team now owns log retention, cursor expiry, replay authorization, compaction, and the case where recovery falls off the retained window. A tiny state document favors refetch; a multi-megabyte state graph may justify replay or partitioned snapshots. Make that choice from measured state size and recovery latency, not from transport fashion.

A minimal recovery contract

The transport can be Server-Sent Events, WebSocket, or a WebRTC data channel. Recovery semantics sit above all three. SSE provides reconnection behavior and an event identifier field; WebSocket provides a two-way channel but does not define application replay; WebRTC data channels can be configured for ordered or unordered delivery. None removes the need to define what the application does after its local state is suspect.

WebRTC 1.0 also makes the transport choice visible: a data channel can use ordered delivery or bounded retransmission settings. Choosing unordered or partially reliable delivery can be valid for ephemeral telemetry, but auction state still needs the application sequence and reconciliation rule. Transport ordering cannot repair an event that was never delivered.

Keep the contract small:

Every delta carries auction_id, sequence, event_id, and a typed payload.
GET /auctions/:id/state returns the complete visible state and sequence from one consistent read.
The client applies only the exact successor to its last sequence.
Duplicate or older events are ignored by event_id and sequence; future events trigger recovery.

Here is the recovery core in Go. The same state machine belongs around a Node.js WebSocket or SSE client; keeping it transport-independent makes it testable without a socket.

package realtime

import (
    "context"
    "fmt"
)

type Event struct {
    AuctionID string
    EventID   string
    Sequence  uint64
    Kind      string
}

type Snapshot struct {
    AuctionID string
    Sequence  uint64
    Status    string
    HighBid   int64
}

type State struct {
    Snapshot
    Seen map[string]struct{}
}

type FetchSnapshot func(context.Context, string) (Snapshot, error)

func Apply(ctx context.Context, s *State, e Event, fetch FetchSnapshot) error {
    if _, duplicate := s.Seen[e.EventID]; duplicate || e.Sequence <= s.Sequence {
        return nil
    }

    if e.Sequence != s.Sequence+1 {
        snapshot, err := fetch(ctx, e.AuctionID)
        if err != nil {
            return fmt.Errorf("recover auction %s: %w", e.AuctionID, err)
        }
        s.Snapshot = snapshot
        s.Seen = make(map[string]struct{})
        return nil
    }

    // Apply the typed delta here, then advance the cursor.
    s.Sequence = e.Sequence
    s.Seen[e.EventID] = struct{}{}
    return nil
}

The Express snapshot handler should read state and sequence consistently, return both in one JSON response, and set cache policy appropriate to user-specific auction data. Authorization must be checked on the snapshot path just as it is on subscription. A reconnect is not an authorization bypass.

Work backward from the page

The alert should fire on failed recovery, sustained recovery latency, or clients that remain behind the server's published sequence after reconciliation. Start with counters for gaps detected, snapshot attempts, snapshot failures, and duplicate events. Add a histogram for recovery duration and a gauge or sampled distribution for sequence lag.

Correlate them with auction_id only in logs or traces, not as an unbounded metrics label. Event and user identifiers have the same cardinality problem. The useful trace spans reception of the gap event, snapshot request, consistent read, state replacement, and resubscription. It should record the old sequence, observed sequence, and snapshot sequence without recording private payloads.

The earlier signal is a divergence check: after recovery, the client's acknowledged sequence should converge on the snapshot sequence or a later contiguous event. Instrument that invariant before alerting on raw reconnects. This separates a noisy network from an actual stale-state risk.

One sharp edge deserves its own test: a delta can arrive while the snapshot request is in flight. Put incoming deltas in a bounded buffer, then discard entries at or below the snapshot sequence and process the rest in ascending order. If the buffer overflows, refetch instead of guessing. Recovery must also have a deadline and backoff; an endless retry loop can turn one backend slowdown into a request surge.

Test the failure, not the happy socket

Unit tests should feed 1841, 1842, 1845 into the state machine and assert that 1845 is not applied before replacement. Then test duplicate 1842, an out-of-order 1844 after 1845, a snapshot at 1846, and a snapshot failure. These cases are deterministic. Timers are unnecessary.

At the integration boundary, expose a controllable test stream that can omit, duplicate, and reorder events. Verify the Express response and the emitted event derive their sequence from the committed state. During deployment, keep old and new readers compatible with the same event envelope; changing sequence scope or reset behavior requires an explicit migration, because clients cannot infer a new domain.

Idempotency is the reflex here. The event ID protects side effects such as showing a notification once, while the sequence protects ordering within the chosen domain. They solve different failures, so collapsing them into one field makes incident analysis harder.

Two IDs. Two jobs.

Set thresholds after measuring recovery

Alerting on a single sequence gap will page on transient, self-healing behavior. Alerting only when users complain is worse. The practical threshold combines impact and duration: page when a meaningful population cannot reconcile within the service objective; create a lower-urgency signal for elevated gaps that recover successfully.

Measure the normal reconnect baseline first, then exercise dropped events and snapshot errors in a controlled environment. The exact threshold depends on traffic shape, auction duration, and the freshness promise made to users. Those inputs cannot be replaced by a universal number.

There is a cost to sensitivity. Set the threshold too low and routine laptop sleep or mobile handoff wakes the on-call, training the team to distrust the page. Set it too high and stale auction state persists behind a healthy-looking socket count. The alert should represent failed convergence, because that is the condition that requires human action.

DEV Community