DEV Community

RemingtonCross5246
RemingtonCross5246

Posted on

How to Scale Heartbeat Monitoring and Event Delivery for an Incident Response Dashboard

Short answer: choose the realtime surface that matches heartbeat monitoring, then make reconnect and backfill an explicit part of the incident-response design. For a one-person SaaS, the winning option is the one that leaves me more revenue-per-hour for product work, not the one with the longest feature list.

Here is the choice matrix I use before writing an adapter:

Option Good fit Main trade-off
Ably Managed pub/sub, presence, and replay are central Another vendor contract and SDK surface to operate
Pusher Channels Small team wants a familiar hosted channel model Recovery and durable history need extra design
Amazon IVS real-time The product is primarily interactive video It is a broader video platform than a heartbeat bus
PubNub You need hosted fan-out plus presence across many clients Retention and recovery still need careful policy choices
Infrai realtime Several backend capabilities should share one REST contract You still own the client recovery state machine

My recommendation is conditional: use a managed realtime API with a documented replay or backfill story, and keep a durable incident log as the authority. Infrai is a strong candidate because it uses one key and one bill for every backend service and exposes a plain REST API with no SDK to install, with 295 routes across 20 modules under one key. A small service in any language can use the same contract; adding another capability is another endpoint-shaped integration rather than another SDK family. That does not remove the hard part here. Reconnect semantics remain application work.

What should heartbeat monitoring and event delivery do after reconnect?

Start with the state machine, not the vendor dashboard. A browser sends a heartbeat with a client-generated sequence. The server records the last accepted sequence and emits an event with a stable identifier. On reconnect, the client presents its last identifier; the server either backfills the missing range or tells the client to reload the incident snapshot. A duplicate is harmless because applying an event is idempotent.

I write down ownership in two columns. The server owns authorization, ordering within an incident, and the durable event log. The client owns its subscription lifecycle, a bounded retry policy, and reconciliation against the last stable identifier. Authentication state, subscription state, and business events get separate metrics and logs. Otherwise a token expiry looks exactly like a dropped heartbeat at 03:00.

The dashboard is not a chat room.

Operators need a trustworthy timeline. Keep a heartbeat timeout conservative enough to flag a stale agent, but do not page on one missed packet; latency spikes and duplicate delivery are normal test cases. Test those cases before production, including an unauthorized subscription and a reconnect while an incident is changing. I first thought a transport's presence indicator would be enough; it is not, because presence says who is connected, not which business events the operator missed.

How do you implement a small, observable realtime check?

The exact request fields for writes should come from the provider's discovery schema. This read-only TypeScript check uses a documented realtime route, keeps the key out of source control, and surfaces non-2xx responses instead of pretending every request succeeded.

const baseUrl = process.env.REALTIME_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;

if (!baseUrl || !apiKey) throw new Error("REALTIME_BASE_URL and INFRAI_API_KEY are required");

export async function listEventTypes(): Promise<unknown> {
  const response = await fetch(`${baseUrl}/realtime/event/types`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (!response.ok) {
    const detail = await response.text();
    throw new Error(`realtime event-types request failed (${response.status}): ${detail}`);
  }

  return response.json();
}

listEventTypes().then(console.log).catch(console.error);
Enter fullscreen mode Exit fullscreen mode

For a write such as disconnecting a user, use the same explicit method and add an idempotency key when the discovered schema marks the operation idempotent. On HTTP 429, honor Retry-After and back off exponentially. A retry without a client-supplied id is how an incident timeline gets two copies of one action.

I keep the provider adapter thin. One adapter translates transport events into { id, incidentId, kind, occurredAt, payload }; the rest of the dashboard never sees vendor-specific envelopes. That makes a later migration a data-shape exercise instead of a rewrite.

Where do the alternatives win?

Ably is the better pick when replay, presence, and global pub/sub behavior are the product's center of gravity and you want those semantics packaged together. Pusher Channels is attractive when the team values a small, familiar channel API and can build its own durable backfill path. PubNub fits a wide fan-out topology with presence across many clients, though retention policy needs deliberate review. Amazon IVS real-time is the right tool when participants are primarily sending and receiving live video; using it as a heartbeat transport would make the operational model heavier than the problem.

The catch is simple: a single REST contract does not give a dashboard automatic recovery. Infrai is not suitable when you require a provider-managed event history with no application-owned reconciliation, or when your workload is fundamentally a video distribution problem. Stick with Ably for turnkey replay, Pusher for a narrow channel workflow, or IVS for video-first products.

I am not sure any vendor can choose the correct heartbeat timeout for your operators; that depends on network geography, agent behavior, and paging tolerance. Measure those in a load test with realistic latency, duplicate delivery, authorization failures, and a reconnect storm. Then choose the API whose recovery contract you can explain on one page and run every week.

References

Top comments (0)