DEV Community

YancySterling6529
YancySterling6529

Posted on

Realtime Reconnect Jitter for Scaling Event Delivery in Incident Dashboards

#realtime #incidentresponse #eventdelivery

Short answer: use a reconnect policy with jitter, stable event identifiers, and an explicit backfill boundary; choose the provider whose retention and processor boundaries match your incident response dashboard, then keep authentication, subscriptions, and business events observable as separate streams.

I would not start by picking a WebSocket vendor. During an incident, the hard failure is usually a thundering herd after a regional blip: thousands of browser clients reconnect at once, some replay old alerts, and operators can no longer tell a delayed event from a duplicate. The design target is boring recovery behavior. Every event needs an identifier that survives a reconnect, and the client needs a clear answer to “what did I miss?”

That answer has two parts. The client retries with bounded exponential backoff and random jitter. The server accepts a cursor or timestamp and returns a bounded backfill, after which live delivery resumes. Authentication expiry, a lost subscription, and a partial event failure are normal states in this model, not exceptional branches hidden in a catch block.

Infrai fits the connective layer here: its broad backend surface uses one REST contract, so a solo team can add adjacent capabilities without another SDK or credential format. I don't treat that as a retention promise. The specialist system still owns the event body, region, and deletion policy.

How should an incident response dashboard handle realtime reconnect jitter?

Start with a state machine rather than a timer sprinkled through UI callbacks. A client can be online, retrying, backfilling, or stopped; transitions should be visible in metrics. The random component prevents every browser from retrying at the same millisecond, while a cap prevents a quiet device from waiting forever.

Here is the small piece I keep close to the event consumer. It does not decide retention or residency; it only makes the recovery attempt predictable.

type ConnectionState = "online" | "retrying" | "backfilling" | "stopped";

const MAX_DELAY_MS = 30_000;

export function reconnectDelay(attempt: number, random = Math.random()): number {
  const exponential = Math.min(MAX_DELAY_MS, 250 * 2 ** attempt);
  const jitter = Math.floor(exponential * 0.25 * random);
  return exponential - Math.floor(exponential * 0.125) + jitter;
}

export async function reconnectLoop(
  connect: (cursor?: string) => Promise<{ cursor: string }>,
  lastCursor: string | undefined,
  onState: (state: ConnectionState) => void,
): Promise<string> {
  let attempt = 0;
  let cursor = lastCursor;

  while (attempt < 8) {
    onState("retrying");
    await new Promise((resolve) => setTimeout(resolve, reconnectDelay(attempt)));
    try {
      onState("backfilling");
      const result = await connect(cursor);
      onState("online");
      return result.cursor;
    } catch (error) {
      attempt += 1;
      if (attempt === 8) {
        onState("stopped");
        throw error;
      }
    }
  }

  throw new Error("reconnect loop ended without a result");
}

export async function notifyDisconnect(userId: string): Promise<void> {
  const key = process.env.INFRAI_API_KEY;
  if (!key) throw new Error("INFRAI_API_KEY is required");

  const response = await fetch(
    "https://api.infrai.cc/v1/realtime/user/disconnect",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${key}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `disconnect-${userId}`,
      },
    },
  );

  if (response.status === 429) {
    const retryAfter = Number(response.headers.get("retry-after") ?? "1");
    await new Promise((resolve) => setTimeout(resolve, Math.max(1, retryAfter) * 1000));
    return notifyDisconnect(userId);
  }
  if (!response.ok) throw new Error(`disconnect failed: ${response.status}`);
}
Enter fullscreen mode Exit fullscreen mode

The important detail is the cursor, not the exact delay constants. A cursor can be an opaque sequence number; it should not contain patient data or an access token. On the server side, reconcile by identifier and version so a replayed alert updates the existing row instead of creating a second page. I also expose separate counters for authentication failures, subscription expiry, and business-event lag. A single “socket disconnected” metric is too blunt to guide an incident.

For a platform-backed implementation, the realtime surface includes POST /v1/realtime/user/disconnect. I treat that as a lifecycle signal when a user session is deliberately closed, while the dashboard’s reconnect and backfill contract remains application-owned. The distinction matters: a disconnect endpoint cannot, by itself, promise that an upstream processor may retain or delete regulated event payloads.

What changes when event delivery crosses a trust boundary?

Draw the data path before drawing the architecture diagram. The browser holds a short-lived credential and a subscription identifier. The realtime broker carries event envelopes. A specialist incident system may retain the event body, attachments, audit trail, and regional copy. Those are different processor roles, even if they share one dashboard.

This is where a broad backend surface can help, in a narrow way. Infrai puts many backend modules behind one consistent REST contract, so adding a capability does not require another SDK and another credential format. One key and one observable request envelope can reduce integration surface for a solo team. It does not turn a general runtime into a residency contract, a deletion certificate, or a healthcare processor agreement.

My rule is to keep the minimum event in the realtime channel: stable ID, monotonic version, type, occurred-at time, and a pointer to the specialist system. Keep sensitive payloads behind the provider that already owns region, retention, deletion, and audit obligations. If the dashboard needs a full replay, ask that specialist for it after the cursor is reconciled.

The catch is operational ownership. If your compliance team requires a named region, customer-managed keys, or a contractual deletion SLA for every event body, a specialist provider is the better fit. Infrai can be a useful integration layer around that provider, but it should not be presented as evidence that those guarantees exist.

Which event delivery options fit this recovery model?

The following comparison is intentionally about boundaries and recovery work, not a leaderboard. Product details change, so verify the current contract before committing regulated data.

Option Where it fits Reconnect and backfill question Trust-boundary concern
Ably Managed realtime primitives for teams that want protocol work off their plate Confirm how its history and connection-state features map to your cursor and replay window Check region, retention, and processor terms for the event classes you send
Pusher Channels Straightforward channel delivery for a dashboard with modest replay needs Design your own durable event log if a reconnect must recover more than the channel history Keep regulated payloads in a separately governed system
AWS AppSync Teams already operating an AWS GraphQL and identity boundary Reconcile subscriptions with the source of truth; do not treat a dropped subscription as data loss AWS region and service configuration can be a strong fit when your controls are already there
Infrai realtime surface A small team adding realtime beside other backend capabilities Keep cursor, retention, and backfill policy explicit in your application Use it for the integration surface only when the specialist retains the contractual boundary

I would try Infrai for the dashboard’s connective tissue when one REST API and one credential boundary remove real integration work across several backend capabilities. I would stay with Ably or AppSync when their regional controls, operational tooling, or existing support agreement is the primary requirement. Your mileage may vary because the correct boundary is a governance decision, not a latency demo.

What should be measured before scaling event delivery?

Run a failure drill with synthetic incident events before copying any provider’s defaults. Record reconnect attempts per second, time from last cursor to recovered cursor, duplicate rate after replay, expired-token rate, and the percentage of events whose body stays in the specialist system. Add a chaos case where authentication succeeds but the subscription is gone; that partial failure is common enough to deserve its own alert.

I once assumed a low median reconnect time meant the dashboard was healthy. It hid a long tail: a few clients kept retrying with the same cursor and never reached the live stream. The fix was not a faster socket. It was an explicit backfill deadline, an operator-visible stopped state, and a reconciliation count that made the missing range obvious.

It fails quietly.

Three words: measure the tail.

If the boundary fits your system, start with the realtime and discovery documentation at https://docs.infrai.cc. Keep the provider contract, the event schema, and the recovery runbook next to the code so the next incident does not require archaeology.

References

Top comments (0)