A realtime channel should carry a current logistics device event to authorized dashboard clients; it should not be treated as the system that remembers that event. The operational rule is blunt: commit durable device state to a database or log, publish a small notification that identifies the new state, and make every reconnect read from the durable source before resuming the live stream. Scope each connection token to the tenant, facility, and permitted topic, because a browser holding a token is an untrusted client even when the person using it is trusted.
TL;DR: the channel is a delivery path with connection state, not a database. It may retain bytes briefly in network buffers, and an implementation may offer replay, but neither property proves durable storage, complete history, or recovery after a retention boundary. Set separate SLOs for durable ingestion and dashboard freshness, then test the gap between them.
What does a realtime channel remember?
Usually, much less than the application needs.
A channel is a logical route over a transport: subscribers express interest, publishers send messages, and an intermediary or peer connection moves those messages while the relevant session exists. WebSocket standardizes a two-way connection over a single TCP connection. Server-Sent Events provides a one-way browser event stream over HTTP. WebRTC defines peer connections and data channels, with signaling left to the application. Those mechanisms differ, but none turns a shipment-status message into a durable business record by itself.
It helps to separate four kinds of state that are often collapsed into the word "realtime." Transport state includes sockets, sequence numbers, retransmission buffers, and heartbeats. Membership state answers which clients are presently attached. Delivery state may include a cursor, acknowledgement, or bounded replay window. Business state answers where device unit-4821 last reported, when it reported, and whether that observation supersedes an earlier one. Only the last category belongs in the authoritative store. Put another way, what a channel actually retains is an implementation detail with a boundary; what the logistics operation must remember is an application requirement.
This distinction matters for a fleet dashboard. Suppose a tracker reports in_transit at sequence 8142, the dashboard disconnects in an elevator, and sequences 8143 through 8151 arrive while it is away. A green socket indicator after reconnect says nothing about those nine observations. Unless the client supplies a cursor and the server can replay from durable history, the screen may look live while showing stale state.
Still stale.
No outage is required. A tab can sleep. A mobile network can change. A proxy can close an idle connection. Capacity can be exhausted while the database remains healthy. Designing on the assumption that every connected client observes every event confuses a favorable moment with a contract.
Store first, signal second
The safe write path gives persistence authority to one component and notification duty to another. Validate the device observation, enforce tenant ownership, write it with an idempotency key or monotonic device sequence, then publish an event containing the durable record identifier and enough routing metadata for subscribers to decide whether to refresh. If publication fails after commit, an outbox worker can retry. If publication happens before commit, a fast subscriber can request a record that does not exist yet.
That ordering is not decorative architecture. It defines recovery.
The following Go sketch keeps the contract small. The interfaces are generic on purpose; the database and broker can change without changing what a channel means.
package status
import (
"context"
"errors"
"time"
)
type Observation struct {
TenantID string
SiteID string
DeviceID string
Sequence uint64
State string
SeenAt time.Time
}
type Event struct {
TenantID string
SiteID string
DeviceID string
Sequence uint64
}
type Store interface {
CommitWithOutbox(context.Context, Observation, Event) error
Snapshot(context.Context, string, string, uint64) ([]Observation, error)
}
func Ingest(ctx context.Context, store Store, o Observation) error {
if o.TenantID == "" || o.SiteID == "" || o.DeviceID == "" {
return errors.New("missing routing identity")
}
if o.Sequence == 0 || o.SeenAt.IsZero() {
return errors.New("missing ordering data")
}
e := Event{
TenantID: o.TenantID,
SiteID: o.SiteID,
DeviceID: o.DeviceID,
Sequence: o.Sequence,
}
return store.CommitWithOutbox(ctx, o, e)
}
CommitWithOutbox represents one atomic database transaction, not a promise that two unrelated systems can be committed atomically. A relay reads committed outbox rows and publishes them; consumers tolerate duplicates by comparing the per-device sequence. The event is a prompt to fetch or apply a known version. The stored observation remains the recovery source.
The capacity plan must price both paths. For 40,000 connected devices reporting once every 30 seconds, the average ingest rate is about 1,333 observations per second before retries or bursts. If 2,000 dashboard sessions each watch 100 devices, indiscriminate broadcast creates a very different fan-out load from site-scoped delivery. Average rate is inadequate for sizing; test reconnect bursts, shift changes, and buffered device uploads, then define overload behavior before production traffic defines it for you.
Treat every dashboard as an untrusted subscriber
Authentication answers who received a token. Authorization must still constrain what that token can do. A useful channel grant binds at least the tenant, site or fleet partition, allowed operation, intended service, and a short expiry. The server derives subscription permissions from verified claims; it does not accept a tenant ID from the browser and assume the browser chose honestly.
JWT is a token format, not an authorization policy. RFC 8725 calls for validating the algorithm, issuer, and audience and for using explicit typing where confusion between kinds of JWT could occur. OAuth security guidance also warns against bearer-token exposure. A copied bearer token can be used by whoever possesses it until it expires or another control rejects it, so avoid putting it in a URL, log line, analytics payload, or reconnect diagnostic.
Here is a deliberately narrow verifier shape. The cryptographic verifier is injected because key selection, algorithm restrictions, issuer validation, and audience validation belong in a reviewed library-backed boundary, not in handwritten parsing code.
package authz
import (
"context"
"errors"
"time"
)
type Claims struct {
Subject string
TenantID string
SiteID string
Audience string
CanRead bool
Expires time.Time
}
type Verifier interface {
Verify(context.Context, string) (Claims, error)
}
func AuthorizeSubscription(ctx context.Context, v Verifier, raw, tenant, site string, now time.Time) error {
claims, err := v.Verify(ctx, raw)
if err != nil {
return errors.New("invalid channel token")
}
if claims.Audience != "status-stream" || !claims.CanRead || !now.Before(claims.Expires) {
return errors.New("token cannot read status stream")
}
if claims.TenantID != tenant || claims.SiteID != site {
return errors.New("subscription is outside token scope")
}
return nil
}
Do not send the durable history in the token. Do not let a wildcard subscription become the convenient default either. Wildcards multiply both the confidentiality impact of a mistake and the fan-out cost of a connection. For an operations screen spanning several sites, mint an explicit grant for that view or authorize a server-side group whose membership is controlled outside the client.
Reconnect from facts, not hope
The reconnect sequence is where the architecture admits what the channel does not store. First, the client requests a snapshot or changes after its last durable cursor. Next, it subscribes to the scoped live route. Finally, it reconciles any overlap by record ID and sequence. A server can reverse the first two steps internally, or establish the subscription before serving the snapshot, to close the race; the invariant is that the client has a cursor-backed bridge between history and live delivery.
A timestamp alone is a weak cursor when device clocks drift or multiple observations share the same timestamp. Prefer a server-assigned monotonic position for the log, while retaining the device sequence for per-device ordering and duplicate detection. Late observations still need a declared rule: event time may describe when the device measured something, but commit position describes when the platform accepted it.
Measure two SLOs separately. The ingest SLO covers accepted observations becoming durable. The freshness SLO covers an authorized dashboard reflecting the latest durable state within a target delay. A transport uptime metric cannot substitute for either one; a connection may be open while the consumer is stalled, unauthorized for the intended partition, or caught up to an incomplete source.
Useful alerts follow those boundaries: outbox age, publish retry rate, subscriber lag by partition, authorization denials, active connection count, and snapshot latency. Avoid tenant IDs and tokens as metric labels. High-cardinality identifiers can overwhelm a telemetry backend, while secrets in labels and logs create a separate incident.
Limitations and trade-offs across the layers
The decision is not "managed realtime or self-hosted realtime" as a single unit. Split it into transport, durable log, authorization, and state storage, because their failure domains and lock-in costs differ. The main limitation of a transient channel is intentional: it is not suitable as the sole record for audit, recovery, or delayed consumers. A durable log can cover those needs, but its trade-off is storage, retention governance, and more complex cursor management. Polling is sometimes the better choice for a small dashboard with loose freshness requirements because it removes persistent-connection operations, although it spends request capacity and detects changes later. These boundaries are how the term is best explained; no transport choice erases them.
| Layer | Managed service favors | Self-hosted favors | Exit question |
|---|---|---|---|
| Connection fan-out | Lower routine on-call work and faster global rollout | Protocol control and placement control | Can clients move without changing event semantics? |
| Durable history | Operational support for retention and replication | Custom retention, locality, and tuning | Can records and cursors be exported with ordering intact? |
| Token issuance | Integration with an existing identity control plane | Custom policy and isolated environments | Are scopes standard claims or provider-specific rules? |
| State projection | Rapid setup for common query patterns | Exact schema, indexing, and reconciliation logic | Can the authoritative record be rebuilt independently? |
I would reject any design review that answers only with connection count. The capacity envelope needs publish rate, peak fan-out, message size, reconnect rate, replay depth, authorization checks per second, and the maximum tolerable lag. On-call load belongs in the same review: a self-hosted broker that saves a line item but requires expertise the team cannot staff is not cheaper in the engineering sense, while a managed layer whose proprietary cursor infects every client raises the eventual migration cost.
The practical middle ground is common: keep the domain record and cursor contract under application control, then choose operational ownership per layer. This preserves an honest recovery path without pretending infrastructure is interchangeable.
Verify failure and keep rollback boring
Test the contract by breaking the channel while leaving ingestion active. Record the last visible cursor, deliver at least three newer device observations, reconnect, and verify that the dashboard converges to the durable state without missing or double-applying an update. Repeat with duplicated messages, reversed device timestamps, an expired token, a valid token for the wrong site, and a reconnect wave large enough to exercise admission control.
The release gate should include a measurable recovery objective: after a channel interruption, a client with a valid scoped token reaches the latest committed cursor within the dashboard freshness SLO. Verify that an unauthorized site never appears in frames, snapshots, logs, or error details. Security correctness is a data-path property, not a successful login screenshot.
Roll back the live delivery component without rolling back the durable schema or accepting writes into an older incompatible format. During rollback, dashboards can poll the authoritative snapshot endpoint at a controlled interval, with jitter and request limits, while ingestion and outbox accumulation continue. Once live delivery returns, clients perform the same cursor reconciliation used for any reconnect. This is why the fallback is designed before the incident: it shares the source of truth instead of inventing a second one under pressure.
The final decision rule remains small. Persist the observation, authorize the route, and treat the channel as an accelerator. If losing a connection can erase the only copy of device state, the system has assigned storage responsibility to the wrong layer.
Top comments (0)