Short answer: treat every realtime channel as a leased, observable state machine, keep durable whiteboard operations separate from ephemeral presence, and choose the transport only after defining what reconnect, replay, and stale membership mean.
The page fires while a gaming session is still live: participants can see the shared whiteboard, but the poll moderator's roster says 84 people are present while only 61 clients have acknowledged the current round. The on-call does not need a generic “realtime degraded” alert. They need the channel generation, the count of active leases, the age of the last acknowledged operation, the reconnect rate, and the gap between server-observed membership and application-confirmed presence. Without those dimensions, the first useful action is guesswork.
My choice is conditional. For a collaborative whiteboard where presence accuracy determines whether a live poll is valid, I use an authoritative server-side channel lifecycle and regard a direct peer path, including WebRTC, as an optional delivery optimization. I would choose a peer-authoritative design only for a small, bounded room where temporary disagreement is acceptable and no server-side decision depends on who is present. The catch is that an authority adds coordination load and an on-call surface; removing it shifts that complexity into reconciliation and product semantics rather than making it disappear.
How should realtime channel lifecycle design keep collaborative whiteboard updates reliable?
Start with states, not sockets. A useful lifecycle is joining, active, suspect, reconnecting, draining, and closed. Each transition needs an owner, a deadline, and an observable reason. “Connected” is too weak because a transport can remain open while the application has stopped applying whiteboard operations, and “disconnected” is too final because a mobile client may return before its lease expires.
Presence should be a lease attached to a channel generation. A client joins with a stable participant ID and a new connection ID; the server returns the current generation and a replay position. Heartbeats renew the lease, but application acknowledgements prove more than heartbeat traffic: they show that the client is processing the same operation stream as the room. On reconnect, the client presents its last applied sequence, receives the missing durable operations, then becomes active only after acknowledging the new head. A late packet from an older generation is ignored.
That separation matters during the live poll. Cursor coordinates and a “user is typing” hint can expire because replaying them would create false activity. A submitted vote, a whiteboard stroke, or a moderator's round transition cannot silently expire; those operations need identity, ordering within the chosen consistency boundary, deduplication, and replay. Presence itself sits between those classes. It is ephemeral as display data, yet it becomes decision input when the poll closes, so the closing rule should use acknowledged channel state rather than a decorative online badge.
I initially reach for the simplest possible heartbeat counter because it is easy to capacity-plan: room count multiplied by participants multiplied by heartbeat frequency gives a first-pass event rate. Then I correct that model. A heartbeat proves that some code path ran, not that the board converged. For this workload, the earlier and more useful signal is the distribution of application-acknowledgement age by room generation, paired with the number of leases in suspect. That is the signal I want before the moderator reports a mismatched roster.
Keep the decision rule explicit:
| Question | Prefer an authoritative relay | Consider a peer-authoritative channel |
|---|---|---|
| Does presence decide a poll outcome? | Yes; one membership view can gate the close | No; peers can legitimately disagree during churn |
| Must a returning client replay durable operations? | Yes; keep a server-known replay position | Only when another durable coordinator already owns history |
| Are rooms large or membership changes frequent? | Easier to bound fan-out and lease evaluation centrally | Suitable only after measuring peer fan-out and reconciliation cost |
| Can the product tolerate temporary split views? | Use when the answer is no | Use when the answer is yes |
This is not a claim that one transport makes updates reliable. WebRTC is a W3C Recommendation and can be part of the channel, but lifecycle correctness still belongs to the application: generation checks, operation identity, replay boundaries, and lease semantics do not appear merely because peers can exchange data.
Work backward from the page
The immediate response starts by narrowing scope. Is the gap confined to one room generation, one client release, or every active room? Did application acknowledgement age rise before reconnect attempts, or did lease expiry rise first? If acknowledgements are old while heartbeats remain current, the transport is alive but the consumer path is behind. If reconnects rise and new generations become active quickly, the system is churning but recovering. If server-observed membership and acknowledged membership diverge only at poll close, the closing rule may be sampling at the wrong lifecycle boundary.
No drama. Just state.
The earlier alert should therefore be tied to user-visible correctness, not raw connection volume. I would define an SLO around the proportion of eligible, leased participants that acknowledge the current durable operation head within the product's poll window. The exact window cannot be universal; I'm not sure any transport-level number can answer it without the poll duration, expected room size, client network mix, and tolerance for excluding a late vote. Those inputs should set the objective and the alert threshold.
For capacity planning, model at least four independent loads: durable operation ingestion, fan-out deliveries, heartbeat or lease renewal traffic, and replay after reconnect. Average active connections hide the dangerous shape. A reconnect wave compresses replay reads and fan-out writes into the same interval, while a popular gaming session creates a hot room even when fleet-wide utilization looks calm. I budget the room-level peak separately, then test a reconnect cohort against it. The point is not to manufacture a heroic maximum; it is to expose which quota or queue becomes the first limiter and whether backpressure preserves durable updates before cursor freshness.
The runbook action follows the diagnosis. Freeze poll closure when acknowledged presence is below the defined validity boundary; do not freeze whiteboard editing merely because cursor presence is stale. Let clients rejoin with their last applied sequence. Drain an old generation only after its replacement is accepting joins, and retain enough operation history for the permitted reconnect interval. These are product choices expressed as lifecycle rules, so the moderator sees a delayed close rather than a confident result built from contradictory membership views.
Instrument the transition, not only the connection
The instrumentation change I want is small: emit one record for every accepted lifecycle transition, and update a few bounded-cardinality measurements. Participant IDs do not belong in metric labels. Room IDs usually do not either; put high-cardinality identifiers in sampled traces or structured logs, while metrics retain state, reason, client release, and a coarse room-size bucket.
The following Go sketch keeps the contract deliberately generic. It does not choose a network library or storage engine; it shows the measurements the lifecycle owner needs.
package channel
import (
"context"
"time"
)
type State string
const (
Joining State = "joining"
Active State = "active"
Suspect State = "suspect"
Reconnecting State = "reconnecting"
Draining State = "draining"
Closed State = "closed"
)
type Transition struct {
RoomID string
ConnectionID string
Generation uint64
From State
To State
Reason string
At time.Time
LastApplied uint64
}
type Observer interface {
RecordTransition(context.Context, Transition)
ObserveAckAge(context.Context, State, time.Duration)
SetLeases(context.Context, State, int64)
}
func MarkActive(ctx context.Context, o Observer, t Transition, head uint64) bool {
if t.To != Active || t.LastApplied < head {
return false
}
o.RecordTransition(ctx, t)
o.ObserveAckAge(ctx, Active, time.Since(t.At))
return true
}
The boolean is intentional. A connection does not become active merely because a join handshake completed; it becomes active when it has applied the required head for its generation. In a real implementation, the state store must enforce the transition atomically with the generation check. The observer is downstream of that decision, not the source of truth.
I would graph acknowledgement age percentiles beside counts of joining, suspect, and reconnecting leases, plus accepted and rejected transition rates by reason. I would also trace a sampled durable operation from acceptance through fan-out to client acknowledgement. That trace answers the question the original page cannot: where did forward progress stop?
Avoid an unbounded label for sequence number, connection ID, or room ID. It feels useful during the first incident review and becomes an expensive indexing decision at fleet scale. Keep those values in a structured event with retention and sampling controls. Metrics answer whether the pattern is widespread; traces and logs identify the affected path.
The buy-versus-build boundary is lifecycle ownership
The usual buy-versus-build discussion starts with protocol support and ends with a monthly estimate. I start with who owns the state machine at 03:00, because a managed channel can remove transport operations while leaving application replay, poll validity, and presence semantics with the platform team. Conversely, self-hosting can offer direct control over admission and retention while making capacity, patching, regional failover, and the reconnect surge entirely our problem.
| Concern | Managed transport | Self-hosted transport | Keep in the application either way |
|---|---|---|---|
| Connection fleet | Provider operates the service boundary | Team operates capacity and failure domains | Join policy and channel generation |
| Replay | Evaluate the offered retention and ordering contract | Team designs and operates the history path | Operation identity and last-applied tracking |
| Presence | Evaluate its lease and membership semantics | Team implements renewal and expiry | Definition of “eligible for poll close” |
| Observability | Verify export depth and cardinality controls | Team owns collection and storage | SLO, alert policy, and runbook |
| Lock-in | Adapter and data-exit work remain | Infrastructure coupling remains | A transport-neutral operation envelope |
My decision rule is to buy when the connection fleet and global delivery path are undifferentiated operational work, provided the service exposes enough lifecycle evidence to support our SLO and lets the application retain authority over durable operation IDs and membership decisions. I build when a required consistency boundary, deployment constraint, or observability contract cannot be represented by the managed interface and the team can staff the resulting on-call burden. Cost is a capacity model input, not a verdict: compare peak concurrent leases, message fan-out, replay bandwidth, retained history, observability ingestion, engineering time, and incident ownership on the same sheet.
There is a real limitation to the authoritative-relay choice. It is not suitable when rooms must continue coordinating during complete separation from the authority, or when a tiny, fixed peer group values local continuity above a single presence view. In that case, keep a peer path and make merge semantics visible to the product. The opposite boundary matters too: stick with an authoritative lifecycle when attendance gates a live result, because “eventually the rosters agree” is not a useful rule at the instant a poll closes.
The threshold can hurt the session
An alert on one stale acknowledgement will page on ordinary churn. An alert averaged across the whole fleet can miss one high-value room. I would use a short burn signal for severe, room-scoped correctness risk and a longer burn signal for a broad decline, but only after replay tests establish the normal acknowledgement-age distribution for the supported room-size buckets. The threshold must also account for generation changes so a draining channel does not look like a failing active one.
False positives have a direct operational and product cost. The on-call starts distrusting the page, the moderator may delay a valid poll, and an aggressive remediation can trigger more rejoins precisely when the room is hottest. False negatives are worse when they allow a poll to close against an inaccurate membership view. That asymmetry is why presence accuracy, not socket availability, owns the decision axis.
The final design is deliberately boring: explicit states, renewable leases, generation fencing, durable operation replay, application acknowledgements, and an SLO tied to the action a participant expects to complete. Use WebRTC where its peer communication model fits the delivery path, but do not delegate channel truth to a protocol. Reliability comes from making every transition testable and every recovery boundary unambiguous.
References
- W3C, “WebRTC: Real-Time Communication in Browsers”: https://www.w3.org/TR/webrtc/
Top comments (0)