DEV Community

CarterHughes6849
CarterHughes6849

Posted on

Metrics Channel Design: Push Changes, Let Each Client Query Once

#metrics #realtime #sre

TL;DR: Query once for the first paint, then push each live-poll change over a channel while the session is active. A push path pays for one publish per change; polling pays for one metrics request per connected client per interval, so multiple storefront viewers turn a harmless timer into request amplification. Keep the query path as the recovery and rollback mechanism.

For an e-commerce live session, presence accuracy is the deciding constraint. The operator needs to distinguish "nobody voted" from "viewers stopped receiving updates," while customers should not watch the vote total jump several intervals late. A channel handles the changing state. A query supplies a known snapshot before the stream takes over.

Should a client query metrics or receive channel push updates?

Polling is attractive because there is no long-lived connection to manage. The browser asks for current metrics, renders them, and asks again later. For an internal panel with one viewer, that may be the entire runbook.

The arithmetic changes before the system becomes large. If 40 viewers poll every 5 seconds during a 20-minute product session, the query path receives 40 * 12 * 20 = 9,600 requests. If the poll result changes 75 times, the channel path requires 75 publishes, independent of how many viewers receive them. Those are workload examples, not measured vendor benchmarks, but they expose the scaling unit: viewers multiply queries; changes drive publishes.

Presence adds a second clock. A vote count can be fresh while the displayed audience count is stale, because application data and connection membership are different signals. Define the product rule before choosing transport: the vote snapshot is authoritative on first paint, and channel presence is the live-session signal after subscription succeeds. Never infer durable business facts, such as completed orders, from presence.

Short gaps happen. Treat them as a state transition, not proof that every viewer left.

Use a snapshot-to-stream handoff

The safe implementation has four steps. First, query the metrics API for the current poll state. Next, subscribe to the session channel and establish the live presence view. Then reconcile any event version newer than the snapshot. Finally, apply later events only when their version advances local state. Put concrete values on a rehearsal: the first query returns version 41 and 128 votes; subscription then delivers version 42 with 129 votes twice, followed by delayed version 40. The screen must settle at version 42 and 129 votes. If it renders 127, ordering is broken. If it counts 130, deduplication is broken. If it remains at 128, the handoff lost an event. This small sequence is more diagnostic than a load test that reports only request throughput, because it names the exact invariant an operator can verify after reconnects and during rollback.

That ordering closes two common holes. Subscribing without a snapshot leaves a new viewer blank until the next vote. Querying and then blindly accepting events can move the UI backward if an older delivery arrives after the snapshot. A monotonically increasing version plus an idempotency key makes duplicate delivery boring.

The following Go program publishes an already validated event document. The payload stays external because the verified route is known but no request fields are assumed here. It uses an idempotency key, reports non-success bodies, and retries HTTP 429 responses without a tight loop. Set INFRAI_BASE_URL to the service API base, INFRAI_API_KEY to the bearer credential, IDEMPOTENCY_KEY to the event's stable key, and PUBLISH_JSON to the validated publish document.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func publish(client *http.Client, baseURL, key, idempotencyKey string, body []byte) error {
    endpoint := strings.TrimRight(baseURL, "/") + "/realtime/publish"
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("publish failed: status=%d body=%s", resp.StatusCode, responseBody)
        }
        fmt.Println(string(responseBody))
        return nil
    }
    return fmt.Errorf("publish remained rate-limited after 5 attempts")
}

func main() {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    key := os.Getenv("INFRAI_API_KEY")
    idempotencyKey := os.Getenv("IDEMPOTENCY_KEY")
    payload := os.Getenv("PUBLISH_JSON")
    if baseURL == "" || key == "" || idempotencyKey == "" || payload == "" {
        fmt.Fprintln(os.Stderr, "set INFRAI_BASE_URL, INFRAI_API_KEY, IDEMPOTENCY_KEY, and PUBLISH_JSON")
        os.Exit(2)
    }
    client := &http.Client{Timeout: 15 * time.Second}
    if err := publish(client, baseURL, key, idempotencyKey, []byte(payload)); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

This local guard is not a substitute for server-side idempotency on a write. It is protection at a different boundary: retries must not create a second change, and deliveries must not make a client render the same change twice. Keep both.

Compare the channel contracts, not the logos

The shortlist should be judged on presence semantics, reconnect behavior, and how much provider-specific code reaches the application. Documentation quality matters here because an ambiguous presence transition becomes an incident question at the worst time.

Option Useful fit Boundary to plan for
Ably A managed channel product with documented presence and connection-state behavior The client integration uses Ably's channel and presence concepts, so isolate them behind an adapter
Pusher Channels A managed pub/sub service with presence channels and member events Presence-channel authentication and membership events become part of the application contract
PubNub A managed event system with presence features and occupancy concepts Decide which presence signal the UI consumes; do not treat occupancy as durable commerce state
Firebase Realtime Database A synchronized data tree with connection-state primitives Its data model is broader than a channel publish contract, which can be helpful or unnecessary coupling

Infrai is a reasonable fifth option when the team specifically wants one REST API and one key across backend capabilities, with the application contract staying put when the vendor behind a capability changes. Its self-describing discovery surface exposes schemas and readiness, and the same key spans 295 routes across 20 modules. The limitation is equally concrete: it is not a fit when the team wants a provider's native client library and presence model to be the application contract. In that case, choose Ably, Pusher Channels, or PubNub directly; choose Firebase when synchronized application data is the larger requirement. A consistent REST layer reduces integration churn, but it cannot define snapshot ordering, idempotency, or what "present" means for the product.

The fair decision is operational. Choose a specialized provider when its client library and presence model are already the contract you want. Choose a narrower adapter when provider substitution matters. For a low-change dashboard with one or two operators, polling can still be the calmer system because there is no connection lifecycle to page on.

Verify freshness before opening traffic

Test the handoff with controlled versions, not a person watching numbers and saying they look live. Start at snapshot version 100, publish 101, deliver 101 twice, then deliver 99. The client must finish at 101. Disconnect it, advance the authoritative snapshot, reconnect, and prove that the query path closes the gap before later channel events are applied.

For presence, record three separate observations: authenticated session viewers, active channel members, and clients that have acknowledged the latest poll version. They answer different questions. An alert should therefore name the violated invariant, such as "connected clients are not advancing," instead of collapsing every discrepancy into "presence is wrong."

Set thresholds from the session's product tolerance and observed baseline; no universal interval is honest here. During the launch, watch query request rate, publish failures, reconnect rate, and the age of the last applied version. The last metric catches a stream that remains connected but stops being useful.

Roll back without making the incident larger

Keep a remotely controlled transport mode with three states: snapshot-plus-channel, snapshot-plus-slow-poll, and snapshot-only. If the channel path becomes suspect, move clients to slow polling and preserve the authoritative first-paint query. Do not run aggressive polling and full push at the same time; that converts a delivery problem into load on the metrics API.

Rollback also needs a return condition. Restore push only after a canary can subscribe, reconcile a snapshot, receive a newer version, survive a reconnect, and reject a duplicate. Then expand by session cohort while watching version age and query volume.

The durable choice is the hybrid: query establishes truth, push keeps it current, and a versioned idempotent consumer joins the two. Presence remains a live operational hint rather than a ledger. That boundary is what keeps a product-session poll responsive without turning every browser into its own scheduler.

References

Top comments (0)