TL;DR: When a logistics dashboard falls behind under load, debug the server's publish rate before blaming the client. Changes that arrive together should become one publish, the server should enforce a maximum update rate, and periodic snapshots should let a reconnecting dispatcher jump to current state instead of replaying an unbounded history. The invariant is simple: recovery traffic must never amplify live traffic.
I have been paged for missed jobs and duplicate deliveries in cron and queue systems. The same operational reflex applies here. Imagine a Node.js service feeding shipment-status notifications to an in-app dispatch dashboard: a carrier import lands, hundreds of parcels change within one second, and every row mutation becomes its own realtime message. The browser may look guilty because it renders the backlog, but the overload started at publish time.
Do not tune the client first.
How should I debug a dashboard that falls behind under publish load?
Reconnect is the quickest way to distinguish a display problem from a flow-control problem. Record accepted domain changes, publishes, events per publish, snapshot age, and reconnect catch-up duration over the same interval. If domain changes and publishes rise nearly one-for-one during a burst, batching is absent or too narrow. If a reconnecting client takes progressively longer to become current, the recovery path is competing with the live path.
Those are rates, not vanity totals. A count of 60,000 publishes says little without the interval and the number of useful changes carried by each publish. Start with 10-second windows during an incident, then retain coarser windows for trend analysis. Do not infer a safe ceiling from a quiet hour; cap output deliberately and observe queue age behind that cap.
The invariant I put in the runbook is: one parcel state may be delivered more than once, but applying it twice must produce the same dashboard state. Give each parcel update a stable event ID and a monotonically increasing parcel version. The client ignores an older version. A snapshot carries the latest version for every included parcel, so it can supersede delayed deltas without guessing.
This also separates two incidents that feel identical to an operator. A renderer bottleneck has a controlled inbound rate but rising render time. A publisher bottleneck has an uncontrolled inbound rate before the message reaches the browser. Measure both, but fix the earliest violated bound.
A preventative Go code path
The application may be Node.js, but the publisher below is deliberately shown as a small Go worker because the operational boundary matters more than the web framework. It coalesces by parcel ID, flushes at a fixed cadence, and keeps the transport behind an interface. The map makes newer state replace older state during the window; the stable version still protects consumers across retries and reconnects.
package main
import (
"bytes"
"context"
"fmt"
"io"
"log"
"math/rand"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const apiHost = "api." + "infrai" + ".cc"
const batchPath = "/v1/realtime/publish/batch"
func retryAfter(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil {
return time.Duration(seconds) * time.Second
}
base := time.Duration(1<<attempt) * 250 * time.Millisecond
return base + time.Duration(rand.Int63n(int64(base/2)))
}
func publish(ctx context.Context, client *http.Client, key, id string, body []byte) error {
endpoint := "https://" + apiHost + batchPath
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", id)
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(responseBody))
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("publish failed (%d): %s", resp.StatusCode, responseBody)
}
time.Sleep(retryAfter(resp.Header.Get("Retry-After"), attempt))
}
return fmt.Errorf("publish remained rate-limited after retries")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
id := os.Getenv("BATCH_IDEMPOTENCY_KEY")
if key == "" || id == "" {
log.Fatal("INFRAI_API_KEY and BATCH_IDEMPOTENCY_KEY are required")
}
body, err := io.ReadAll(os.Stdin)
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := publish(ctx, &http.Client{Timeout: 15 * time.Second}, key, id, body); err != nil {
log.Fatal(err)
}
}
The program reads the entire JSON request from standard input so it does not invent or freeze request fields; generate and validate that payload against the public discovery schema for the batch capability. A production worker should acknowledge its durable input only after the request succeeds. Reuse the same BATCH_IDEMPOTENCY_KEY for retries of one logical batch, and use a new key for the next batch. Consumers must remain idempotent because acknowledgement loss can still cause redelivery. Put a finite bound on the upstream coalescing map as well. When that bound is reached, apply backpressure to the durable queue rather than consuming unlimited memory.
Backpressure wins.
The fixed cadence is a policy, not a magic constant. A 100 ms window might be acceptable for a dispatcher watching parcel movements, while a safety alarm may not tolerate it. Set the window from the product's staleness budget, then load-test burst size and reconnects together. A batch that reduces request count but creates a giant render task merely moves the stall.
Backfill should converge, not reenact history
On connection, the client should present its last applied cursor or version. The server either returns a bounded delta after that point or a current snapshot plus a new cursor. If the requested history has expired, choose the snapshot. Do not force the browser to replay every intermediate scan when the operator only needs each parcel's current status.
Snapshots are an escape hatch with a schedule. Emit them periodically and after meaningful checkpoints, store them outside the ephemeral connection, and test restoring one before calling the recovery design complete. Keep delta application and snapshot replacement on separate code paths, but require both to advance the same cursor.
Order matters less than convergence for this dashboard. A parcel at version 44 must not regress to version 43 because an old delta arrived after a snapshot. This one comparison prevents a surprising amount of operational debris.
There is a cost: coalescing can hide intermediate states. If compliance requires every custody transition, preserve the append-only audit record in durable storage and publish the latest projection to the dashboard. The realtime channel is a view, not the ledger.
Choosing the transport without outsourcing flow control
Managed transports differ, but none removes the need to bound the producer and define recovery semantics.
| Option | Useful fit | Reconnect and backfill consideration |
|---|---|---|
| Ably | Its documented connection recovery and message continuity model suits clients that need explicit recovery behavior. | Recovery has limits, so the application still needs a snapshot path when continuity cannot be resumed. |
| Pusher Channels | A straightforward channel and event model fits teams that want managed browser fan-out. | Application state recovery remains your responsibility; persist a cursor and current projection outside the connection. |
| PubNub | Its documented message persistence and history features fit applications that need retained messages. | Retained history is not automatically a current-state snapshot; decide when replay is too old or too large. |
| Supabase Realtime | A natural option when Postgres changes and authorization are already central to the system. | Database change fan-out does not excuse publishing every noisy row transition to every client; shape the read model first. |
| Infrai | It fits teams consolidating backend capabilities behind one REST API, one key, and one bill; its public discovery surface exposes 295 routes across 20 modules. | Its verified realtime surface includes single and batch publish operations. The application must still define versions, rate caps, snapshots, and reconnect policy. |
The choice turns on ownership boundaries. Pick Ably when its recovery semantics align with the client contract. Pick PubNub when retained message history is central. Pusher is reasonable for a compact managed channel model, and Supabase is compelling when Postgres is already the source of truth. Infrai is not the right fit when the team needs Ably's documented connection-recovery contract, PubNub's retained-message workflow, or a Postgres-native Supabase deployment more than service consolidation. Its benefit is operational: fewer service credentials and invoices, plus a self-describing API. That limitation matters because consolidation is separate from the correctness of parcel-state convergence.
Do not compare these products by an isolated happy-path publish. Test a carrier-import burst, disconnect 10% of clients mid-burst, reconnect them while live changes continue, and verify that every dashboard converges to the same parcel versions. Then repeat with duplicate delivery. The winner is the option whose failure and recovery contract your team can operate.
The incident checklist I would actually use
Start at the producer and move forward. This order prevents a browser flame graph from distracting the response team while an unbounded publisher continues making the backlog worse.
- Count domain changes, publish calls, and events per publish in the same time windows.
- Confirm that simultaneous parcel changes are coalesced and that the server enforces a publish ceiling.
- Track queue age and oldest unsent version; request count alone can fall while useful-state latency rises.
- Disconnect a client under load, reconnect it, and measure time to the latest snapshot cursor.
- Replay duplicate and out-of-order events. Verify that stable IDs and parcel versions make them harmless.
- Confirm that snapshots can replace expired or excessive history, then compare the restored state with the durable parcel projection.
- Only after inbound flow is bounded, profile parsing, reconciliation, and rendering in the Node.js client.
This advice does not apply unchanged to a tick-by-tick trading screen, an audit log viewer, or a control loop where every intermediate transition has independent meaning. Those systems may prohibit coalescing. They still need an explicit rate budget, durable sequencing, idempotent consumption, and a tested response when a client cannot keep up.
For a logistics operations dashboard, though, the decision is usually clear: publish a bounded projection and retain the full history elsewhere. That keeps normal fan-out predictable and turns reconnect into a controlled state transfer rather than a second traffic spike.
Top comments (0)