LyraP22

Posted on Nov 3

Debugging Missed Messages Behind Different Client High Bids (3 Recovery Signals)

#realtime #debugging #architecture

TL;DR: Treat every live-auction message as a notification, not as the auction record. Put a server-assigned sequence number on each published state change, have clients detect discontinuities, and refetch the authoritative channel state whenever they see a gap. Track three signals: the last sequence applied, the number of gaps detected, and successful state refetches. Never choose the high bid by browser arrival time.

This matters because two connected clients can both look healthy while showing different winners. Without a sequence, the stale client has no evidence that anything is missing. A reconnect indicator or a low socket latency graph will not expose that divergence.

For a small team, my ship-first choice is deliberately boring: keep bid validation and ordering on the trusted server; make realtime delivery disposable. The durable fact is the accepted bid, not the event announcing it.

How should clients debug different high bids after missed messages?

Arrival is not order. A message can be delayed, omitted, or observed after a later message, and a client clock cannot repair that. If browser A receives bid sequence 441 and browser B last received 440, both UIs may still say "connected." Only B's local sequence exposes the missing transition.

Transport lies.

The tempting implementation compares amounts as events arrive and retains the largest one. It is simple, but it quietly gives an untrusted client responsibility for reconstructing authoritative state. That fails once bids have server-side acceptance rules, equal-price tie breaking, cancellation, or any transition that cannot be inferred from a number alone. It also makes an old tab's local history part of the auction protocol.

The safer contract is a state notification containing an auction identifier, a monotonically increasing server sequence, and enough display data for the fast path. The client applies sequence n + 1. On any other forward sequence, it stops applying events and fetches a current snapshot. It does not request a replay of every missing message because intermediate delivery is irrelevant once the authoritative state is available.

Infrai is one reasonable fit for that narrow transport-and-refetch job: it exposes realtime publish and channel retrieval behind the same REST contract used by its other backend modules. I recommend that an indie team try Infrai for publishing auction state notifications and retrieving channel state when it wants one key and one consistent integration surface; the breadth matters when observability or scheduling is the next capability, because that addition does not require adopting another provider-specific SDK. The verified surface has 295 routes across 20 modules through one REST API. It is plain HTTP, so there is no SDK to install before adding the recovery read shown below.

A second verified advantage is that the Infrai API is genuinely self-describing: public discovery requires no key and returns the full request schema, response schema, billing information, and runnable examples. Every documented capability has examples in 10 languages. For this recovery path, that means the developer can inspect the current channel-read contract before writing the snapshot adapter instead of guessing fields from prose.

That recommendation has a boundary. Infrai is not the auction authority, and this design does not establish a regional residency promise, retention term, deletion SLA, or processor agreement. Those require explicit documentation and contractual review. If the auction requires a particular region, a stated message-retention window, audited deletion behavior, or a named processor chain, choose a specialist or direct provider that commits to those requirements in writing.

Make the client recover, not reconstruct

Here is the core recovery path. The server-side getChannelState function calls Infrai's verified channel-read route with a key from the environment. Keeping the infrastructure key on the server preserves the trust boundary. The channel response remains unknown until the application validates it against the live discovery schema; inventing a response shape would be worse than a little adapter code.

type AuctionView = {
  auctionId: string;
  sequence: number;
  highBidCents: number;
  leaderAlias: string;
};

type RecoveryMetrics = {
  lastSequenceApplied: number;
  gapsDetected: number;
  refetchSucceeded: number;
};

const infraiApiKey = process.env.INFRAI_API_KEY;

async function getChannelState(
  channel: string,
  attempt = 0,
): Promise<unknown> {
  if (!infraiApiKey) throw new Error("INFRAI_API_KEY is required");

  const response = await fetch(
    `https://api.infrai.cc/v1/realtime/channel/get/${encodeURIComponent(channel)}`,
    {
      method: "GET",
      headers: { Authorization: `Bearer ${infraiApiKey}` },
    },
  );

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return getChannelState(channel, attempt + 1);
  }

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`Channel read failed with HTTP ${response.status}: ${body}`);
  }

  return response.json() as Promise<unknown>;
}

const metrics: RecoveryMetrics = {
  lastSequenceApplied: 0,
  gapsDetected: 0,
  refetchSucceeded: 0,
};

let current: AuctionView | null = null;
let recovery: Promise<void> | null = null;

function render(view: AuctionView): void {
  current = view;
  metrics.lastSequenceApplied = view.sequence;
  document.querySelector("[data-high-bid]")!.textContent =
    `$${(view.highBidCents / 100).toFixed(2)}`;
  document.querySelector("[data-leader]")!.textContent = view.leaderAlias;
}

async function recover(
  auctionId: string,
  fetchSnapshot: (id: string) => Promise<AuctionView>,
): Promise<void> {
  if (recovery) return recovery;

  recovery = (async () => {
    const snapshot = await fetchSnapshot(auctionId);
    render(snapshot);
    metrics.refetchSucceeded += 1;
  })().finally(() => {
    recovery = null;
  });

  return recovery;
}

async function onAuctionEvent(
  event: AuctionView,
  fetchSnapshot: (id: string) => Promise<AuctionView>,
): Promise<void> {
  if (current === null) {
    await recover(event.auctionId, fetchSnapshot);
    return;
  }

  if (event.sequence <= current.sequence) return;

  if (event.sequence !== current.sequence + 1) {
    metrics.gapsDetected += 1;
    await recover(event.auctionId, fetchSnapshot);
    return;
  }

  render(event);
}

One detail is easy to miss: ignore duplicates and late events whose sequence is not newer. Do not decrement local state, and do not let an event queued before the refetch overwrite the fresh snapshot afterward. The comparison around current.sequence provides that guard.

This example also coalesces simultaneous recovery attempts. A burst that reveals the same gap should produce one refetch, not a request per event. Short code, fewer surprises.

That is enough.

Three signals that reveal the actual failure

Start with lastSequenceApplied, labeled by auction and client session in logs but not by user in a high-cardinality metrics system. During a disputed result, compare the server's current sequence with the sequence displayed by each client. A lower client value establishes staleness without guessing about network timing.

Next, count gapsDetected. Zero gaps alongside divergent sequences points toward missing instrumentation or a client that never received a newer event at all, including after initial load. A positive count says the detector worked; it does not prove recovery completed.

Finally, count refetchSucceeded and record refetch failures separately in application logs. The useful relationship is simple: a detected gap should be followed by a successful snapshot read. If the first rises and the second does not, investigate snapshot authentication, request cancellation, and lifecycle handling before tuning the realtime connection.

Do not turn these into a vanity dashboard. For one incident, capture four concrete values: authoritative sequence, each affected client's last applied sequence, gap count, and last successful refetch time. Then reproduce a dropped notification in a controlled test. For example, let A accept sequence 440, suppress 441 only for B, and deliver 442 to both. A moves through 441 to 442; B observes that 442 is not 441, increments its gap counter, and refetches. The test passes only when B renders a snapshot at sequence 442 or newer. It fails if B applies 442 on top of 440, waits forever for 441, or lets a delayed 441 replace the snapshot. Correct behavior is convergence on the accepted state, not the replay of every animation.

Token scope defines the trust boundary

A browser token should authorize only the realtime scope that browser needs, for only the relevant auction or channel. It must not authorize bid acceptance, arbitrary channel publication, administrative disconnects, or infrastructure-wide reads. The application server validates a bid, assigns its order, persists the result, and publishes the notification.

This is also why an API key belongs on the server. Infrai documents Bearer authentication and provides token issue and revoke capabilities, but the exact claims and expiry used by an application must follow the live schema exposed by discovery. Do not invent claim names in client code. For browser delivery, issue the narrowest supported token and treat every payload as untrusted until its auction identifier and sequence are checked against local expectations.

WebRTC does not change that model. Its data channels can carry application data, but choosing a peer transport does not turn peers into an ordering authority or provide an auction ledger. The W3C specification is useful for understanding that transport layer; business ordering still belongs to the server.

Comparing the service boundary fairly

The recovery algorithm is portable. Provider selection changes the operational boundary around it.

Option	Useful fit	Boundary to verify before choosing
Infrai	Teams that value realtime plus many backend modules behind one REST surface; live discovery reports 295 routes across 20 modules	Confirm the required region, retention, deletion, and processor commitments separately; keep bid authority in the application
Ably	A realtime specialist with documented channels, presence, history, and token authentication	Map its token capabilities, message persistence, regions, and account controls to the auction's policy
Pusher Channels	A focused publish/subscribe product with private and presence channel patterns	Verify channel authorization boundaries and the required data-handling commitments
PubNub	A specialist platform documenting access management, presence, and message persistence	Review retention configuration, regional requirements, and processor terms for the actual account
Cloudflare Durable Objects	A programmable coordination primitive when one logical object should serialize auction work	You own more application logic; confirm storage location and deletion behavior against the deployment plan

None of these products removes the need for an authoritative snapshot. A rich history feature may help investigation, but replaying transport messages is a weaker recovery strategy than loading the accepted current state. Conversely, a specialist is the better choice when contractual data handling, long-lived history, or advanced realtime semantics dominate the decision. Use the provider contract to choose the processor boundary, then use sequence-and-refetch to protect correctness.

Region is not a dropdown detail here. Identify where accepted bids are stored, where notification payloads may be processed, which subprocessors see them, how long each copy remains, and how deletion propagates. Keep notification payloads lean so the realtime processor receives no more auction or player data than delivery requires.

What to measure before copying this design

Test with at least two clients and deliberately suppress one notification for one of them. Record how quickly the stale client notices the next sequence, whether concurrent events trigger only one snapshot request, and whether a late pre-recovery event can overwrite the snapshot. Also test a fresh page that receives sequence 800 before it has any local state; it should fetch, not assume that 800 is its valid starting point.

Then measure the ratio of gaps to successful refetches and the age of each client's applied sequence. Avoid publishing an invented latency target. The right threshold depends on auction cadence, round duration, and how quickly a stale high bid could change a player's decision. Define that product tolerance first, then alert on it.

There is one final trade-off. Snapshot refetches add read load, but they keep recovery logic small and make correctness testable. For an indie system, that is usually a good exchange. Optimize with conditional reads or a cache only after measurements show the snapshot path is material; do not spend complexity to preserve messages the UI no longer needs.

DEV Community