The page says a Node.js live logistics poll is open, yet votes are barely moving: measure realtime delivery latency at the client, alongside connection health, before the session host has to ask whether drivers can see the question. A healthy API and successful publish calls do not prove observation. That is too late to start wondering what "delivered" means.
TL;DR: put a server timestamp and stable message ID on every poll event, have each connected client report the delay it actually observed, and aggregate those reports by poll, region, client version, and short time window. Alert on a sustained percentile trend plus a connection-health signal, never one slow sample. Keep an idempotent durable queue before fan-out so reconnecting clients do not turn a transient socket problem into a lost poll.
This is the useful split: the server records acceptance, the client records observation, and the queue records work that still must happen. A green publish response proves only the first of those.
How should Node.js measure realtime delivery latency and connection health?
Start at the page and work backward. The visible symptom is a poll with an abnormal response curve. The preceding signal should be a rise in client-observed delivery latency, segmented finely enough to distinguish one bad application release from broad fan-out trouble. Before that, connection health should show clients disconnecting, failing to acknowledge recent traffic, or reconnecting repeatedly.
The primary service-level indicator is elapsed time from published_at to client receipt. Use a wall-clock timestamp because the measurement crosses machines, but remember that it includes client clock error. The operational answer is not to discard it. Track impossible negative values and extreme skew separately, compare the distribution across many clients, and use monotonic time only for intervals measured within one process.
A single sample is weak evidence.
A rolling p95 can reveal a damaged tail, while report volume tells you whether that percentile represents the audience or a tiny surviving subset. I would page only when latency remains outside the session's objective for several windows and enough reports are arriving to make the signal credible. The exact threshold cannot be universal: a ten-second audience poll and a five-minute dispatch survey have different failure budgets. This is an explicit trade-off between early detection and interruption cost, not a magic percentile copied from another service.
Short silence matters too. If publish acceptance stays normal while client reports collapse, the absence is itself a signal. Separate that condition from high latency because the runbook actions differ: one points toward connection or client health, while the other may point toward congestion, fan-out backlog, or a slow client cohort.
Instrument the observation path
The event needs a stable ID, a poll ID, and a UTC publication time. The browser or mobile client calculates observed delay when it receives the event, then sends a small report asynchronously. Do not delay rendering the poll while reporting telemetry. Reports can be retried by message ID and client instance ID; the metrics receiver must deduplicate them.
Although the application in question is Node.js, the following Go probe is deliberately independent of a Node runtime. It gives the on-call team a small, runnable receiver for validating the metric contract from a browser, test client, or staging session. All production clients should emit the same fields.
package main
import (
"encoding/json"
"log"
"net/http"
"sync"
"time"
)
type DeliveryReport struct {
MessageID string `json:"message_id"`
PollID string `json:"poll_id"`
ClientID string `json:"client_id"`
ClientBuild string `json:"client_build"`
Region string `json:"region"`
PublishedAt time.Time `json:"published_at"`
ObservedAt time.Time `json:"observed_at"`
}
type sample struct {
PollID, ClientBuild, Region string
Delay time.Duration
}
var (
mu sync.Mutex
seen = map[string]struct{}{}
samples []sample
)
func report(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
defer r.Body.Close()
var v DeliveryReport
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 16<<10)).Decode(&v); err != nil {
http.Error(w, "invalid report", http.StatusBadRequest)
return
}
if v.MessageID == "" || v.PollID == "" || v.ClientID == "" || v.PublishedAt.IsZero() || v.ObservedAt.IsZero() {
http.Error(w, "missing required field", http.StatusBadRequest)
return
}
dedupeKey := v.MessageID + ":" + v.ClientID
mu.Lock()
defer mu.Unlock()
if _, ok := seen[dedupeKey]; ok {
w.WriteHeader(http.StatusNoContent)
return
}
seen[dedupeKey] = struct{}{}
samples = append(samples, sample{
PollID: v.PollID, ClientBuild: v.ClientBuild, Region: v.Region,
Delay: v.ObservedAt.Sub(v.PublishedAt),
})
w.WriteHeader(http.StatusAccepted)
}
func main() {
http.HandleFunc("/delivery-reports", report)
server := &http.Server{
Addr: ":8080",
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 60 * time.Second,
}
log.Fatal(server.ListenAndServe())
}
The in-memory map makes the example runnable, not production-ready. In production, put the deduplication key and samples in durable storage with bounded retention. Export histogram buckets rather than every raw measurement if cardinality or privacy makes raw client IDs inappropriate. Never use client_id as an unbounded metrics label.
The instrumentation change is small, but the semantic change is large: delivery is now something the receiving side can attest to. Server-side publish latency remains useful for diagnosis. It just cannot stand in for end-to-end observation.
For a real publish probe, the next program calls the verified realtime route. It accepts the capability's current JSON body through INFRAI_PUBLISH_JSON instead of freezing an undocumented schema into an example. Use the same stable message ID for IDEMPOTENCY_KEY and the client report. The program requires Go 1.20 or later, where errors.Join is available.
package main
import (
"bytes"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const (
apiScheme = "https://"
apiHost = "api." + "infrai" + ".cc"
apiVersion = "/v1"
publishPath = "/realtime/publish"
)
func retryDelay(h string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(h); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(h); err == nil && time.Until(at) > 0 {
return time.Until(at)
}
return time.Duration(1<<attempt) * time.Second
}
func publish(client *http.Client, key, idempotencyKey string, body []byte) error {
var last error
for attempt := 0; attempt < 5; attempt++ {
publishURL := apiScheme + apiHost + apiVersion + publishPath
req, err := http.NewRequest(http.MethodPost, publishURL, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
last = err
time.Sleep(retryDelay("", attempt))
continue
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
closeErr := resp.Body.Close()
if readErr != nil || closeErr != nil {
return errors.Join(readErr, closeErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
last = fmt.Errorf("rate limited: %s", responseBody)
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("publish failed (%s): %s", resp.Status, responseBody)
}
fmt.Println(string(responseBody))
return nil
}
return fmt.Errorf("publish retries exhausted: %w", last)
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
idempotencyKey := os.Getenv("IDEMPOTENCY_KEY")
body := []byte(os.Getenv("INFRAI_PUBLISH_JSON"))
if key == "" || idempotencyKey == "" || len(body) == 0 {
fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY, IDEMPOTENCY_KEY, and INFRAI_PUBLISH_JSON")
os.Exit(2)
}
client := &http.Client{Timeout: 15 * time.Second}
if err := publish(client, key, idempotencyKey, body); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Keep the response in the trace context for diagnosis, but do not reinterpret acceptance as delivery.
How does a durable queue change the guarantee?
A socket fan-out and a durable job solve different parts of the incident. The socket minimizes delay for connected clients. The queue preserves intent when a client is offline or a publish attempt must be retried. Standard queues are at-least-once, so a consumer must make message_id idempotent; otherwise a retry can display the same poll twice or count two acknowledgements.
The handoff is queued notification -> worker claims message -> realtime publish -> client reports observation. The queue record carries the exact event envelope, including message_id and published_at. The worker uses the same ID as its deduplication key and marks completion only after publish acceptance. A reconnect path may fetch outstanding events, but it must pass through the same client-side deduplication logic.
Infrai is one option when the operational goal is to keep that contract while changing the provider behind a capability. Its realtime and jobs/queues capabilities sit behind the same REST API and key, so an offline user can remain a durable message rather than a lost publish; its documented idempotency convention is also relevant to retrying the handoff. The supporting advantage is operational: one credential boundary covers both stages. This does not remove the need to define acknowledgement, retention, retry, and client deduplication semantics in your own application. Its limitation is equally concrete: it is not suitable when a team needs a vendor-specific client feature that the common contract does not expose, or when policy requires separate credentials and blast radii for queueing and realtime delivery. In those cases, choose the specialized product and accept the integration work.
With Pusher Channels plus Amazon SQS, the comparable design normally means two service signups, a Pusher credential set, AWS credentials or an IAM role, and glue that translates an SQS message into a Channels publish while preserving IDs and retry state. That can be a good trade when the team already operates AWS and wants SQS controls. It is still glue the team owns.
Do not claim exactly-once delivery.
The defensible promise is at-least-once processing around an idempotent event, plus measured observation for connected clients. Clean words prevent dirty incidents.
Which realtime stack fits this poll?
The decision turns on delivery guarantees at fan-out, not the prettiest client API.
| Option | Durable handoff | Connection and delivery evidence | Best fit and boundary |
|---|---|---|---|
| Pusher Channels + Amazon SQS | SQS supplies the durable stage; application glue publishes to Channels | Pusher exposes connection state, while end-to-end poll delay still needs the timestamp/report pattern | Strong fit for AWS teams willing to operate two credential and failure domains |
| Ably Pub/Sub | Ably provides realtime channels and connection recovery features | Connection-state changes help diagnosis; application-level observation still needs client reports | Good when recovery behavior and a mature realtime SDK surface matter |
| Socket.IO + BullMQ | BullMQ supplies Redis-backed jobs; Socket.IO handles application fan-out | Socket.IO exposes connection events; the team owns the metric pipeline and most operations | Good for teams that need code-level control and already run Node.js plus Redis |
| Infrai realtime + jobs/queues | Both capability groups share one API key and base service contract | Client-observed metrics remain an application responsibility | Good when provider interchangeability and fewer credential boundaries outweigh a specialized vendor SDK |
These are not interchangeable implementations. Socket.IO is a library and protocol ecosystem, not a hosted durable messaging service. BullMQ makes Redis part of the production dependency graph. SQS is durable queueing, not a browser fan-out product. Pusher and Ably concentrate more of the realtime connection work in a managed service. The table is a boundary map, not a winner board.
WebRTC is another real technology, but it is usually the wrong default for this particular control-plane event. Its peer connection model is valuable for media and direct data channels; a logistics poll still needs server authority, durable offline intent, and observable audience fan-out. Adding peer-to-peer topology does not answer those requirements by itself.
Tune the alert without training people to ignore it
After deploying the client report, run the alert in record-only mode through representative sessions. Inspect p50, p95, report count, negative-delay count, disconnect rate, and the ratio of observed messages to eligible connected clients. Break down by client build and region before widening the page. This is where a bad mobile release stops looking like a global provider incident.
There is a specific trap in alerting on the worst sample: backgrounded phones, clock jumps, and one saturated device can all produce ugly numbers while the poll remains healthy for the audience. Paging on that sample creates an expensive false positive and teaches the on-call engineer to distrust the alert. Requiring several degraded windows reduces noise, but it also spends detection time. That trade must be written into the service objective and tested against the poll duration.
The runbook should begin with three questions. Are publish acceptances normal? Are enough client reports arriving? Is the regression isolated to a build, region, or connection state? Those answers route the response toward the producer, fan-out layer, or client without pretending that one dashboard can prove delivery.
False positives have a delivery cost of their own. Every unnecessary page competes with the next real one, so set the threshold from observed session distributions and revisit it when audience shape or client behavior changes. Measure first. Then page.
Further reading
- Pusher Channels documentation: https://pusher.com/docs/channels/
- Amazon SQS delivery semantics: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html
- Ably connection state and recovery: https://ably.com/docs/connect/states
- Socket.IO connection state recovery: https://socket.io/docs/v4/connection-state-recovery
- BullMQ documentation: https://docs.bullmq.io/
- W3C WebRTC 1.0: https://www.w3.org/TR/webrtc/
Top comments (0)