Short answer: use a heartbeat for liveness, a durable event log for delivery, and an explicit replay path for operators who reconnect during an incident. A WebSocket can carry the live view, but it should not be your only copy of an alert.
In a logistics control room, a poll is running while a wave of trucks leaves a depot. The dashboard asks each session to report that it is alive, then fans out state changes to dispatchers. The hard part is not drawing a green dot. It is deciding what “delivered” means when one browser sleeps, a cellular link changes, or 400 people open the same incident.
Which delivery guarantee fits a live incident dashboard?
Start with the failure you can tolerate. This small table is my field guide.
| Option | Pick this when | Delivery contract | Main cost |
|---|---|---|---|
| Direct browser WebSocket | A stale view is acceptable and the source can be queried again | Best effort, ordered per connection | Reconnect storms and no built-in history |
| Durable log plus WebSocket | Every alert change must be recoverable | At-least-once with a cursor | Idempotency and storage operations |
| Queue per consumer group | Teams need independent processing (paging, audit, analytics) | At-least-once per group | More moving parts and lag to watch |
| Hybrid snapshot plus delta stream | The UI needs a fast first paint and exact catch-up | Snapshot, then ordered deltas | Snapshot/version coordination |
For an incident response dashboard, I usually choose the hybrid. The snapshot gives a reconnecting dispatcher a known version. The delta stream fills the gap. Exactly-once delivery is rarely worth promising to a browser; idempotent application is easier to explain and test.
How should realtime heartbeat monitoring scale event delivery?
Think in three lanes: presence, events, and replay. Presence is ephemeral. Events are facts. Replay is the safety net.
The following TypeScript sketch keeps those lanes separate. It uses an abstract broker so the same contract can sit over a self-hosted log, a managed queue, or a database-backed outbox. No SDK is required in the dashboard process.
type SessionId = string;
type Cursor = number;
type Heartbeat = {
sessionId: SessionId;
sentAtMs: number;
};
type PollEvent = {
id: string;
version: Cursor;
pollId: string;
kind: "started" | "vote" | "closed";
payload: Record<string, unknown>;
};
interface EventStore {
append(event: PollEvent): Promise<void>;
readAfter(pollId: string, cursor: Cursor, limit: number): Promise<PollEvent[]>;
}
interface Presence {
touch(heartbeat: Heartbeat, ttlMs: number): Promise<void>;
remove(sessionId: SessionId): Promise<void>;
}
interface Socket {
send(message: string): void;
close(code: number, reason: string): void;
}
const HEARTBEAT_TTL_MS = 15_000;
const MAX_REPLAY = 250;
export async function onHeartbeat(
presence: Presence,
heartbeat: Heartbeat,
): Promise<void> {
await presence.touch(heartbeat, HEARTBEAT_TTL_MS);
}
export async function replayFrom(
store: EventStore,
socket: Socket,
pollId: string,
lastSeenVersion: Cursor,
): Promise<Cursor> {
const events = await store.readAfter(pollId, lastSeenVersion, MAX_REPLAY);
for (const event of events) {
socket.send(JSON.stringify({ type: "poll.event", event }));
}
return events.at(-1)?.version ?? lastSeenVersion;
}
The broker consumer should persist an event before acknowledging it. The browser includes its last applied version when it reconnects; the server sends a bounded replay, then switches to live delivery. If the gap is larger than MAX_REPLAY, send a fresh snapshot and its version instead of flooding the socket.
That boundary matters. A 15-second presence TTL is a product decision, not a law of nature. A poll used for dispatch may need a shorter alert threshold, while a driver on an elevator connection needs a longer one. Measure heartbeat age, reconnect count, replay size, and event-application failures as separate metrics. A single “connected” gauge hides the interesting failure.
What fails first during fan-out?
The first failure is often a reconnect storm. When a node or Wi-Fi access point drops, thousands of clients retry at once. Add randomized exponential backoff, cap the retry rate per tenant, and make the handshake cheap. Do not perform a full historical query before you have accepted the connection. In a depot poll, that means accepting the session, recording its last cursor, and returning a small readiness response before you touch the replay store; otherwise every reconnect competes with the very database query needed to recover it. Watch the shape of the burst, too: a flat rate limit can punish a quiet depot and still let one large tenant consume every socket, so partition admission by tenant and reserve capacity for the operators who own the incident.
The second is duplicate work. At-least-once delivery means a consumer can see the same id twice after a timeout. Store the last applied event IDs or versions with the poll projection, and make vote application idempotent. “We have a queue” is not an idempotency strategy.
The third is head-of-line blocking. One slow browser must not hold the fan-out loop. Give each socket a bounded buffer; when it fills, pause live sends, close with a documented code, and let the reconnect path perform snapshot-plus-replay. This is a controlled loss of immediacy, not a loss of the event.
I once started with a single isAlive boolean and a green badge. It looked fine in a quiet test. Under a burst, the badge stayed green while the cursor lagged by hundreds of versions. The fix was embarrassingly concrete: expose lag and replay age next to presence. Small numbers tell the truth.
How do you test and operate the delivery contract?
Write the contract as scenarios before selecting infrastructure:
- Kill a client after version 42, reconnect, and verify it applies 43 through 50 exactly once to its projection.
- Delay acknowledgements, then confirm the producer retries without creating a second vote.
- Open a controlled reconnect burst and verify backoff, admission limits, and recovery time.
- Advance the cursor past the retained log and verify a snapshot is served with a matching version.
Instrument both sides of the boundary. Useful counters include heartbeat_expired_total, fanout_dropped_total, and event_duplicate_total. Histograms should cover publish-to-apply latency and replay duration. Sample payloads in logs only after removing driver names, phone numbers, and route details; incident telemetry is still operational data.
The WebRTC specification is a useful reminder that real-time systems have explicit connection and state transitions, even when the UI makes them look instantaneous. Model those transitions in runbooks: connecting, live, catching up, degraded, and closed. Alert on time spent in a state, not just on socket count.
This pattern is not suitable when the dashboard can safely poll a small dataset every few seconds, or when your team cannot operate a durable store and replay metrics. A plain WebSocket with a refresh button may be the right engineering choice. Stick with it when the business can tolerate a stale panel and operators have another source of truth.
Keep the choice boring when the risk is boring.
Your mileage may vary on retention. Keep enough history to cover the longest realistic outage, then test that assumption quarterly. The decision is successful when an operator can answer two questions quickly: “Which sessions are truly alive?” and “Which events did this session miss?”
Top comments (0)