Short answer: Treat every typing-start signal as a renewable client-side lease, expire it locally, and use the stop event only to hide the indicator sooner; a missing stop must never leave the UI stuck forever.
The page fires at 02:13: typing_indicator_visible_age_seconds has crossed 8 seconds for the checkout-support room. The on-call opens one affected session and sees a buyer apparently typing forever, even though no message follows. This page points at the symptom too late. Instrument the fan-out path so the missing event is diagnosable rather than permanent.
That rule matters in e-commerce support because one agent action may fan out to a buyer, another agent, and supervisory views. Delivery is not atomic across those recipients. A disconnect, reordered packet, backgrounded tab, or lost datagram can leave one view stale while the others look fine. The UI state is disposable presence, not durable order state. Give it a deadline.
1. How do you debug a typing indicator that sticks forever?
Start with the alert and work backward. An alert on raw stop-event loss is noisy because a stop can legitimately be superseded by another start, a sent message, a reconnect, or local expiry. Alert on user-visible stale state instead: the count or ratio of indicators that remain visible beyond the lease plus a grace period. Break that signal down by transport, application version, connection age, and fan-out destination, but keep identifiers out of low-cardinality metric labels.
The earlier warning is a widening gap between starts accepted and terminal outcomes observed. A terminal outcome is stop, message-sent, disconnect cleanup, or client expiry. Those counters are useful for diagnosis, while the stale-state SLI is the page-worthy measure because it tracks the experience users can actually see. Define the SLO before choosing the alert threshold; otherwise an arbitrary timer becomes an operational promise nobody agreed to.
One page is enough. Duplicate alerts for every internal hop multiply on-call load without adding evidence.
2. Make the browser lease authoritative
A typing start needs a generation number and an expiry deadline. Each newer start replaces the previous generation. A timer may clear only the generation it was created for, which prevents an old callback from erasing a freshly renewed indicator. The same reducer should clear state when a stop, sent-message, participant departure, or connection reset arrives.
The server can express a duration, but the client should calculate its deadline using its own monotonic clock. Wall clocks jump; elapsed-time clocks are built for durations. Browsers expose a monotonic timestamp through performance.now(), while Go exposes monotonic readings inside time.Time values returned by time.Now. The following Go model is deliberately transport-neutral and shows the state invariant that a browser reducer should preserve:
package presence
import (
"sync"
"time"
)
type Indicator struct {
Generation uint64
Deadline time.Time
}
type Store struct {
mu sync.Mutex
users map[string]Indicator
}
func (s *Store) Start(userID string, generation uint64, ttl time.Duration) {
s.mu.Lock()
defer s.mu.Unlock()
current, ok := s.users[userID]
if ok && generation <= current.Generation {
return
}
s.users[userID] = Indicator{Generation: generation, Deadline: time.Now().Add(ttl)}
}
func (s *Store) Expire(userID string, generation uint64, now time.Time) bool {
s.mu.Lock()
defer s.mu.Unlock()
current, ok := s.users[userID]
if !ok || current.Generation != generation || now.Before(current.Deadline) {
return false
}
delete(s.users, userID)
return true
}
Do not infer expiry from the absence of network traffic. A healthy connection can be quiet, and an unhealthy connection can still carry unrelated frames. The lease belongs to each (room, participant) indicator. On reconnect, clear all ephemeral presence before accepting new generations unless the protocol provides an explicit snapshot with a newer epoch.
3. Trace one generation through fan-out
Log a sampled lifecycle record with a room-scoped pseudonymous participant key, connection epoch, generation, event kind, receive time, publish time, and client outcome. The generation is the join key. Without it, repeated starts from the same participant collapse into an ambiguous sequence and a delayed stop may appear to terminate the wrong start.
For WebSocket transport, remember what the standard does and does not provide. RFC 6455 defines an ordered message channel over TCP, but an application event that was never written, was discarded during process failure, or belonged to a previous connection epoch cannot be recovered by ordering alone. WebRTC data channels can be configured with ordered or unordered delivery and limited retransmission; the WebRTC specification exposes those choices. Your application contract must therefore state which transport modes it permits and still retain local expiry in every mode.
Instrument four timestamps rather than dumping payloads: ingress accepted, fan-out queued, recipient sent, and client applied. Then classify the last observed stage. accepted_without_queued points toward producer or transaction boundaries; queued_without_sent points toward fan-out or connection ownership; sent_without_applied points toward transport, parsing, generation rejection, or a backgrounded client. This is enough to narrow the fault domain without turning chat text or customer identifiers into observability data.
A trace does not create a delivery guarantee. It reveals where the guarantee ended.
4. Test loss, reordering, and timer races
The critical tests are state-machine tests, not screenshots. Feed the reducer duplicate starts, a stop from an old generation, expiry racing with renewal, reconnect between start and stop, and a message-sent event with no stop. Assert only visible state and recorded terminal reason. This keeps the test valid if the transport changes.
Use a fake clock. Sleeping for real time produces slow, flaky tests and obscures the exact boundary. The table below is the minimum useful failure matrix for an e-commerce support room:
| Sequence | Expected visible state | Terminal reason |
|---|---|---|
| start 41, stop 41 | hidden | explicit stop |
| start 41, start 42, late stop 41 | visible for 42 | none yet |
| start 41, no further event, deadline passes | hidden | local expiry |
| start 41, message sent | hidden | message superseded |
| start 41, connection epoch changes | hidden | reconnect cleanup |
Property-based generation tests are valuable here: once generation 42 is active, any event scoped to generation 41 must be unable to extend or clear it. Load tests should add skewed fan-out, because broadcasting one start to 10 supervisory and agent views stresses queues differently from a one-to-one buyer conversation. Choose those fan-out counts from production distributions or an explicit capacity model, not from a convenient round number.
5. Set the threshold as an SLO and capacity decision
A short lease removes stale UI quickly but can flicker during pauses or delayed renewal. A long lease hides network jitter while making every lost terminal event visible for longer. There is no universal duration. Measure inter-keystroke gaps, end-to-end event latency, browser timer behavior under background throttling, and the acceptable stale-presence window; then select a lease and renewal cadence with margin. Record the assumptions beside the SLO.
This lease approach has a firm limitation: it is not suitable for durable events whose eventual processing matters, such as an order acceptance or a support message. Expiry deliberately trades completeness for bounded staleness. Use acknowledged, durable delivery for business records; keep the lease for ephemeral presence whose value disappears within seconds.
The buy-versus-build question is less about the wire API than ownership of delivery evidence:
| Decision area | Managed realtime service | Self-hosted fan-out |
|---|---|---|
| Delivery boundary | Verify documented semantics and exported evidence | Define semantics and instrument every hop |
| On-call load | Provider handles part of the transport; application state remains yours | Platform team owns brokers, connections, deploys, and saturation |
| Lock-in | Protocol and observability exports determine exit cost | Internal APIs can still create lock-in |
| Capacity planning | Validate quotas, burst handling, and degradation behavior | Model connections, event rate, fan-out, queue depth, and headroom |
| Failure drills | Confirm loss and reconnect can be injected or simulated | Build and maintain the injection path |
No transport choice removes the need for the client lease. Durable delivery for typing signals is usually the wrong target: replaying an old start after reconnection can resurrect state whose useful lifetime already ended. Spend the stronger guarantee on orders and messages. For transient presence, bounded staleness, generation-aware processing, and observable terminal outcomes form the more defensible contract.
Thresholds also have a false-positive cost. Page at a boundary that browser scheduling routinely crosses and the alert trains the on-call to ignore it; wait far beyond the promised stale window and the page arrives after users have already lost trust. Evaluate the alert against real latency distributions, require a sustained breach or a minimum affected population, and route isolated expiries to metrics rather than a human. The final operational test is blunt: can the on-call distinguish a missing producer event, a fan-out failure, and a harmless local expiry from one trace before touching production?
Top comments (0)