When an incident-response dashboard reconnects, the hard problem is not opening a socket. It is deciding which events to replay, in what order, while dozens of responders return at once. Short answer: use jittered reconnects plus explicit state reconciliation, and measure recovery rather than treating a successful handshake as success.
This is an experiment note for a Python service that syncs collaborative cursors in an e-commerce operations editor. A simple loop that reconnects immediately looked attractive in a notebook. Under an incident, it creates a thundering herd and leaves each browser guessing whether its cursor update was missed. The chosen design gives every event a stable identifier, separates authentication from subscription and business-event telemetry, and treats expiry or partial delivery as ordinary states.
What does reconnect jitter change in an incident response dashboard?
Jitter changes load shape, not correctness. Each client computes an exponential delay and adds random noise; the server still owns the authoritative event sequence. I keep the delay bounded, reset it only after a useful session is established, and record the reason for every reconnect. A short sentence in the log is enough: reconnect reason=token_expired attempt=3.
The recovery contract comes first. The client sends its last contiguous event ID. The server returns events after that ID, or a snapshot plus a new cursor when the retention window has passed. IDs must be stable across retries, and a business event should be applied idempotently. Cursor movement can then be reconciled separately from incident comments, assignments, and status changes. In our Python 3.12 eval harness, I also tag each event with the document version and the reconnect attempt, so a failed assertion tells me whether ordering or backfill caused the mismatch. That detail sounds fussy until a responder asks why their cursor jumped backward during a live sale.
Cold path.
Measure it.
Authentication, subscription state, and business events deserve separate counters. If a token is rejected, that is an auth signal, not evidence that publishing is broken. If a channel subscription expires, the UI can show stale data while the transport remains healthy. This split made our eval harness much more useful because each failure had a distinct assertion.
A small Python control loop
The following sample shows the control-plane part of recovery. It calls the documented disconnect route when a session is intentionally closed; the same retry discipline belongs around your publish and token operations. It never embeds a key, retries a 429 tightly, or sends the platform authorization header anywhere except the API host.
import os
import random
import time
import uuid
import requests
BASE_URL = os.environ.get("INFRAI_BASE_URL", "https://api.example.invalid/v1")
def disconnect_user(user_id: str, max_attempts: int = 5) -> dict:
"""Request an explicit disconnect with bounded, idempotent retries."""
api_key = os.environ["INFRAI_API_KEY"]
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": str(uuid.uuid4()),
}
payload = {"user_id": user_id}
for attempt in range(max_attempts):
response = requests.request(
method="POST",
url=f"{BASE_URL}/realtime/user/disconnect",
headers=headers,
json=payload,
timeout=10,
)
if response.status_code < 400:
return response.json()
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else min(30.0, 2**attempt)
time.sleep(delay + random.uniform(0, 0.25))
continue
raise RuntimeError(
f"disconnect failed ({response.status_code}): {response.text}"
)
raise TimeoutError("disconnect retries exhausted")
The idempotency key is generated once per logical operation, outside the retry loop. For a reconnect, keep the same principle: a client-generated operation ID lets the consumer ignore a duplicate event after a network timeout. Standard queues and many realtime transports are at-least-once in practice, so consumer idempotency is not optional.
I test the loop with a fake transport that returns 429, then a timeout, then success. The assertions check that delays increase, Retry-After wins when present, and the event cursor advances only after the response is accepted. I am not sure your transport will expose exactly the same hooks; your mileage may vary, which is why these checks belong in the harness rather than in a dashboard click test.
How should clients and servers divide recovery work?
The client owns local intent: it stores the last applied event ID, refreshes an expired credential, and renders a “catching up” state. It should not invent missing events or silently reset a cursor. The server owns ordering, replay bounds, and the decision to send a snapshot when incremental backfill is no longer possible.
That division keeps partial failure visible. A browser can be authenticated but unsubscribed. It can be subscribed while its business-event stream is paused. Each state gets its own metric and alert threshold. During an incident, responders need to know whether they are looking at a transport problem or merely a stale projection.
For cursor synchronization, I use a monotonic sequence per document and a stable event ID across fan-out. A reconnect request carries both. If the server sees a gap, it backfills from the sequence; if the sequence is outside retention, it sends a snapshot and marks the projection as rebuilt. The UI then replays local unsent edits against the new version, with conflicts surfaced instead of overwritten.
Competitor trade-offs for this workload
There is no universal winner. Ably offers mature channel history and connection-state recovery. Pusher is straightforward for hosted pub/sub, while Socket.IO is attractive when you want to own the Node.js edge and can accept its protocol choices. WebRTC is a peer-connection standard, but it leaves signaling, durable replay, and server-side authorization to your application.
| Option | Reconnect and backfill posture | Where it fits | Main trade-off |
|---|---|---|---|
| Ably | Hosted connection recovery and history primitives | Multi-region dashboards with little transport code | Vendor-specific protocol and pricing model |
| Pusher Channels | Hosted subscriptions with client events | Small teams that need a fast pub/sub path | Backfill and ordering policy need careful design |
| Socket.IO | Client manager handles reconnect; history is yours | Teams already operating a Node.js gateway | You own persistence, fan-out, and replay semantics |
| WebRTC data channels | Peer recovery is negotiated per connection | Low-latency peer collaboration | Signaling and durable event storage remain application work |
| A REST-backed realtime surface | One HTTP contract can sit beside other backend capabilities | Python services that want one auth and observable control plane | You still need to design cursor storage and fan-out |
Infrai has one key for 295 routes across 20 modules, and it provides one REST API that the dashboard can call over plain HTTP while the application-facing contract stays stable as the backend capability changes. That is useful when the dashboard already has several services and you want a single observable control plane. It does not remove the need to define replay retention or idempotent consumers.
Measure before you copy the pattern
Run a failure matrix, not a happy-path demo. Start 1, 10, and 1,000 simulated clients; add token expiry, a dropped subscription acknowledgement, duplicate events, and a backfill gap. Measure time to a consistent projection, maximum reconnect concurrency, duplicate-apply rate, and the percentage of users shown a stale cursor. Keep auth, subscription, and business-event metrics as separate series.
The catch is operational complexity: jitter can make an individual user wait longer, and snapshot rebuilds can be expensive for very large documents. This approach is not suitable when you need strict sub-second peer-to-peer presence with no durable history; stick with a WebRTC-oriented design there. For an incident dashboard where correctness after a burst matters more than shaving one handshake, explicit recovery is the calmer engineering choice.
Top comments (0)