Realtime publish alone cannot reliably deliver notifications to a disconnected chat-room client. That is why the page says notification delivery has fallen below its SLO even though the on-call sees healthy publish latency, connected subscribers, and no obvious transport errors: users who closed a laptop during a deployment never see the mentions sent while they were away.
TL;DR: a realtime publish is a wake-up signal, not a durable notification record. Write each notification to an inbox before publishing, assign a monotonically increasing cursor within a clearly defined scope, let clients acknowledge what they have processed, and backfill everything after that cursor whenever they reconnect. The realtime path makes delivery fast; the inbox makes recovery possible.
This distinction matters most in developer-tool chat rooms, where a browser can disconnect between a build mention and the engineer returning to the tab. A transport can carry bytes between peers while they are connected. It cannot, by itself, prove that an absent client will later observe an application event.
Why does realtime publish alone lose notifications?
Publish success is the wrong leading signal because it describes an attempted handoff, not the user-visible outcome. The earlier signal should measure whether durable notifications are advancing from creation through acknowledgement. I would define the service objective around an application-level state transition: for example, the proportion of eligible inbox records acknowledged within the chosen delivery window, split between live delivery and reconnect backfill. This is the inbox pattern explained in operational terms: persist, signal, acknowledge, and recover.
No transport metric can manufacture that evidence. WebRTC, for example, specifies peer-to-peer data transport and exposes connection state, but an application still has to define message identity, persistence, replay, and acknowledgement semantics. A green connection graph and a red delivery SLO can therefore coexist without contradiction.
Instrument four timestamps on the same stable notification ID: inbox commit, publish attempt, client receipt, and acknowledgement. Also record the reconnect cursor presented by the client and the highest cursor returned by backfill. Those fields turn an ambiguous complaint into a trace: committed at cursor 1842, published while the browser was absent, returned after cursor 1819, acknowledged after reconnect.
Keep the alert tied to consequences. A growing difference between the latest committed cursor and the latest acknowledged cursor is actionable; a brief rise in disconnected sockets often is not.
Sockets recover. Missing records do not.
The inbox is the source of recovery
The write path needs one durable boundary. In a transaction, insert the domain change and the notification inbox record, or derive both from the same committed event. Publish only after that commit. If the publish fails, the record remains available to a relay or a later backfill; if publishing succeeds but the client disappears before processing, the same record remains available after its last acknowledged cursor.
At-least-once redelivery is the practical consequence, so consumers must deduplicate by notification ID. Exactly-once wording usually hides the boundary question: exactly once in which database, across which reconnect, and before or after the UI rendered? A durable record plus idempotent processing gives the system a behavior engineers can test. Consider a room whose client acknowledged cursor 1819 before sleep: records 1820 through 1842 commit while it is offline, the live publish attempts have no useful recipient, and cursor 1843 commits while backfill is running. The server must return a stable ordered page, the client must refuse to skip an unprocessed cursor, and the transition to live traffic must account for 1843. Without that explicit boundary, a retry can duplicate a mention or a premature mode switch can strand it, while every socket and publish chart remains green.
The cursor scope must be explicit. A per-user cursor is simple for a personal notification inbox. A per-room cursor preserves room order but requires authorization checks during every backfill. A global cursor can simplify storage while exposing unrelated traffic patterns and creating a hot allocation path. Pick the scope from the ordering promise, then capacity-plan that choice; do not let a convenient primary key silently become the product contract.
Here is a deliberately small Go interface for the boundary. The transport is absent because recovery should not depend on which live channel wakes the client.
package inbox
import (
"context"
"time"
)
type Notification struct {
ID string
Recipient string
RoomID string
Cursor uint64
Kind string
CreatedAt time.Time
}
type Store interface {
Append(ctx context.Context, n Notification) error
After(ctx context.Context, recipient string, cursor uint64, limit int) ([]Notification, error)
Acknowledge(ctx context.Context, recipient string, cursor uint64) error
}
The acknowledgement should advance monotonically. Retrying an older acknowledgement must not move it backward, and fetching a page must use a stable order with a deterministic tie-breaker.
Tiny details.
Expensive failures.
Reconnect is a protocol, not an event handler
On reconnect, the client authenticates, submits its last durable acknowledgement, requests a bounded page after that cursor, applies records idempotently, acknowledges the highest contiguous cursor, and repeats until caught up. Only then can it treat live delivery as current. If live events may arrive during backfill, buffer them by cursor or establish a server watermark before switching modes; otherwise the boundary between replay and live traffic can create a gap or reorder the room timeline.
Backfill responses need limits. Suppose peak notification creation is 2,000 records per second and the longest supported offline window is seven days: multiplying those numbers gives 1,209,600,000 potential records before retention, audience filtering, or compaction. That is a planning bound, not a measured workload, and it immediately forces questions about retention, fan-out strategy, pagination, and reconnect storms. Use observed arrival rates and offline-duration distributions for the real model.
Load tests should disconnect cohorts deliberately, continue creating room mentions, and reconnect them together. Verify no cursor gaps, bounded duplicate handling, authorization after membership changes, stable pagination during concurrent writes, and predictable database pressure. Then repeat during a deploy and during partial relay failure. The happy path is the least interesting one.
Operate the gap, not merely the socket
The dashboard should show commit-to-publish latency, commit-to-acknowledgement latency, backfill page size, pages per reconnect, duplicate rate, cursor-gap errors, and age of the oldest unacknowledged record. Break them down by live and replay paths. Averages will hide the stranded tail, so use percentiles and explicit SLO windows appropriate to the product's urgency.
Error handling also needs separate budgets. A publish retry can be aggressive because it is an optimization for latency; a database retry can amplify contention at the durable boundary. Backfill retries should use bounded pages and jitter, while invalid or unauthorized cursors should fail closed and produce an audit signal. Deletion and retention policies must account for acknowledgement state, but retaining every notification forever merely moves the outage into storage.
There is a buy-versus-build decision here, although the useful comparison is about ownership rather than feature checklists.
| Concern | Managed delivery layer | Self-hosted inbox and relay |
|---|---|---|
| Durable replay contract | Verify cursor scope, retention, ordering, and acknowledgement semantics | Define and test the contract directly |
| On-call load | Less infrastructure ownership, but provider boundaries still need runbooks | Full responsibility for storage, relay, overload, and recovery |
| Lock-in | Higher when cursor or identity semantics are proprietary | Lower at the interface, higher in internal operational knowledge |
| Capacity work | Validate quotas and reconnect-storm behavior | Provision write rate, retained rows, indexes, and replay throughput |
The deciding question is whether the service's semantics match the application's failure boundary. A managed channel without durable replay still needs an inbox. A self-hosted broker does not remove the need to model acknowledgement and retention. Cost belongs in the decision, but an apparently inexpensive live path is irrelevant if the team inherits an unmeasured delivery gap.
The inbox pattern has real limitations. It is a poor fit for disposable presence indicators, typing status, and other state that is already obsolete by the time a user reconnects; a fresh snapshot is the better recovery mechanism there. For durable mentions, the trade-off is extra storage, write amplification, retention policy, deduplication logic, and a more demanding client protocol. Teams that cannot operate those parts should choose a managed delivery layer only after verifying that its replay, cursor, authorization, and acknowledgement contract matches the application.
Thresholds can page the team into ignoring the page
Alert on sustained risk to the delivery objective, not every transient backlog. A cursor lag of 500 might be harmless during a short burst and severe for a quiet tenant; backlog age and acknowledgement latency usually carry more meaning when paired with arrival rate. Use burn-rate alerts over multiple windows so a sharp outage pages quickly while a slow leak still consumes enough error budget to become visible.
Set the initial threshold from load tests and production distributions, then review it against actual incidents and acknowledged maintenance windows. Too loose, and disconnected users become the monitoring system. Too tight, and routine reconnect waves wake the on-call until the alert is muted, which destroys the very early warning the instrumentation was meant to provide.
The durable inbox does not make the realtime channel obsolete. It gives that channel an honest job: reduce latency while connected. Reliability comes from being able to prove what was committed, identify what a client has processed, and recover the difference after the connection returns.
Top comments (0)