LiamFoster1844

Posted on Oct 15

Debugging Ghost Presence After Network Drops — Leases, Reconnects, and Trust

#realtime #sre #websockets

The least complex reliable presence model is a server-owned, expiring lease: each live connection renews its own lease, a disconnect may retire it early, and expiry remains the backstop. Do not let a browser write an unscoped online=true flag. After a network drop, that flag has no trustworthy process left to clear it.

TL;DR: page on stale presence age, not on a raw socket-disconnect count. Give every connection a unique session ID, scope its token to one user and one room, accept heartbeats only for that authenticated session, and derive a user's visible state from all unexpired sessions. Reconnect creates a new session; it does not silently revive the old one.

The page at 02:17 should say something like room_presence_stale_sessions_ratio > 0.01 for 10m, with the room class, region, and deployment version attached. The on-call can then distinguish one overloaded shard from a global client regression. A page that merely says "WebSocket errors are high" skips the state-corruption question that matters in a fintech collaboration room: can a trader see someone as present after every authenticated connection for that person has vanished?

Why does realtime presence show ghost users after a network drop?

Start with the user-visible invariant. For each room, a displayed member must have at least one server-recognized session whose lease has not expired. Track the age of the oldest expired-but-still-visible session and the ratio of visible users with no valid backing session. Those are lagging correctness signals, so they deserve a page only when the error budget is actually at risk.

That is the debug contract.

The earlier warning is lease-renewal health. Record accepted renewals, rejected renewals by reason, lease-expiry latency, active sessions, and reconnect outcomes. Break them down by bounded dimensions such as region and client release; never put user IDs, room IDs, or connection IDs in metric labels. Those belong in sampled logs or traces because their cardinality grows with traffic.

A network drop is not a clean close. TCP and WebSocket peers can disappear without completing a closing handshake, while WebRTC defines connection-state transitions that applications still have to interpret. The server therefore cannot make correctness depend on receiving an orderly disconnect. It needs a clock-based expiry path.

One short page is enough.

A practical objective might define "presence correction time" as the interval from the last accepted heartbeat to removal from every subscriber's view. Choose the objective from product tolerance and measured network behavior, then set heartbeat and lease durations to meet it with margin. A five-second heartbeat and a fifteen-second lease are example configuration values, not universal constants; mobile suspension, radio handoffs, server load, and fan-out delay can all move the useful boundary. Tight leases reduce ghost duration but turn transient delay into visible flapping and extra writes. Loose leases hide jitter but spend more of the correctness budget on stale state.

Reconstruct the session before changing a timeout

Work backward from one ghost. Join the server-side session lifecycle by connection ID: authentication accepted, room joined, last heartbeat accepted, transport lost or lease expired, membership recomputed, and removal event published. Then compare that sequence with what the affected client last acknowledged. A missing close event is normal evidence after a hard drop; a lease that never expires, an expiry that does not recompute aggregate presence, or a removal event that subscribers never apply is the defect boundary. The expected trace after an abrupt drop is deliberately asymmetric: the client may show no final event at all, the edge may retain a dead transport until its own timeout, yet the lease authority must still cross its deadline, publish one idempotent removal at a newer room revision, and make the next snapshot omit that session. If the snapshot is correct but one open client still shows the user, investigate delta ordering or local state replacement. If the snapshot itself is wrong, stay on the server side and follow expiry ownership through its queue and store. These debug expectations prevent a broad "realtime is flaky" diagnosis from mixing three separate failures: dead-connection detection, authoritative membership, and subscriber convergence.

Sequence numbers make this tractable. Each room's presence stream should carry a monotonically increasing revision, and a reconnecting client should request a current snapshot before applying later deltas. If revision 104 removes a session and an old revision 103 arrives afterward, the client must discard 103. Wall-clock timestamps alone are poor ordering keys because clocks differ and messages can be delayed.

One human may have a laptop, phone, and two tabs. Removing the user when any one socket closes creates false offline transitions; retaining a single user boolean creates ghosts. Store sessions separately and compute user presence as "one or more valid sessions." Make disconnect and expiry idempotent so duplicate cleanup cannot corrupt the count. Better yet, derive the count from authoritative session records instead of incrementing and decrementing a fragile counter.

For a fintech room, authorization is part of presence correctness. A reconnect token should identify the subject, permitted room, session purpose, and expiration, and the server must verify those claims before accepting a renewal or join. The browser can report activity, but it cannot be the authority that decides which identity or room that activity belongs to. A room-wide token copied into another tab or leaked through logs widens the blast radius; a token scoped to one user and room limits it. Never encode confidential trading data in the token, and do not log the token itself.

Instrument the transition, not merely the transport

The useful event is a state transition with a reason. This Go example keeps transport details outside the state machine, uses server time, and exposes fields an operator needs without treating client input as identity. It is an in-memory core, not a distributed store.

Transport is evidence, not authority.

package presence

import (
    "errors"
    "sync"
    "time"
)

type Claims struct {
    Subject string
    Room    string
    Expires time.Time
}

type Session struct {
    ID, Subject, Room string
    Generation        uint64
    LeaseUntil        time.Time
}

type Registry struct {
    mu       sync.Mutex
    sessions map[string]Session
    ttl      time.Duration
}

func (r *Registry) Renew(now time.Time, c Claims, id string, generation uint64) (Session, error) {
    r.mu.Lock()
    defer r.mu.Unlock()

    if c.Subject == "" || c.Room == "" || !now.Before(c.Expires) {
        return Session{}, errors.New("invalid or expired claims")
    }
    previous, exists := r.sessions[id]
    if exists && (previous.Subject != c.Subject || previous.Room != c.Room) {
        return Session{}, errors.New("session scope mismatch")
    }
    if exists && generation < previous.Generation {
        return Session{}, errors.New("stale generation")
    }
    current := Session{id, c.Subject, c.Room, generation, now.Add(r.ttl)}
    r.sessions[id] = current
    return current, nil
}

func (r *Registry) Expire(now time.Time) []Session {
    r.mu.Lock()
    defer r.mu.Unlock()

    expired := make([]Session, 0)
    for id, session := range r.sessions {
        if !now.Before(session.LeaseUntil) {
            expired = append(expired, session)
            delete(r.sessions, id)
        }
    }
    return expired
}

Instrument Renew and Expire: count outcomes by reason, observe processing latency, and emit a structured transition event containing an approved correlation identifier. Measure the delay from LeaseUntil to actual deletion. If that delay rises while renewal traffic is normal, inspect scheduler pauses, partition ownership, datastore latency, or a stuck expiry worker before blaming the browser.

Test the ugly paths. Cut connectivity without sending a close frame. Suspend a client longer than the lease. Deliver removal and join events out of order. Retry an expiry. Reconnect twice with increasing generations. Rotate the signing key while an existing token remains valid. A deterministic clock keeps state-machine tests reproducible; an integration test with a real proxy validates that the transport can vanish in the ways the model assumes.

The build-or-buy boundary is operational ownership

The API surface is rarely decisive. The real question is who owns lease semantics, authorization scope, regional failure behavior, replay, and incident evidence. Evaluate a managed service and a self-hosted design against the same table.

Decision area	Managed service	Self-hosted system	Acceptance evidence
Token scope	Confirm enforceable subject, room, action, and expiry claims	Define and review claims and verification paths	Cross-room and expired-token tests fail closed
Disconnect detection	Document lease or occupancy semantics and observable delay	Operate heartbeat, expiry, and cleanup workers	Fault-injection distribution for correction time
Reconnect ordering	Verify snapshot, replay, and revision behavior	Build revisioning and resynchronization	Out-of-order and duplicate-event tests
On-call burden	Validate signals, limits, exports, and support boundary	Staff storage, partitioning, upgrades, and incidents	Runbook exercised in a game day
Lock-in	Measure migration cost of protocol and stored state	Measure maintenance cost of internal interfaces	Export and replacement test

The managed option transfers some machinery, not accountability for the application's authorization and UI semantics. Self-hosting gives direct control but adds partition rebalancing, expiry correctness, capacity planning, and upgrades to the team's pager. Forecast peak concurrent sessions, renewal writes per second, fan-out recipients, and reconnect bursts after a regional interruption. Average traffic is almost useless for this decision.

Require evidence for overload behavior too. If renewals are delayed, does the system expire healthy sessions and create flapping, extend leases and tolerate ghosts, or shed lower-priority work? There is no cost-free answer. Write down which failure consumes which SLO budget before deployment, because an undocumented choice will be made by queue order during the incident.

Set the alert where action remains possible

Alerting on a single expired session will wake someone for ordinary laptop sleep. Alerting only when customers complain means the detection system is the support queue. Use a burn-rate-style condition on the presence-correction SLO, with a longer window for paging and a shorter diagnostic warning for rising expiry lag or renewal rejection. The exact threshold has to come from the agreed error budget and observed baseline, not from an attractive round number.

Every page should lead to a bounded action: compare expired-but-visible ratio by region and release; inspect expiry lag; verify room-stream revision gaps; check authentication rejection reasons; then decide whether to roll back, isolate a partition, or reduce nonessential fan-out. If the alert cannot distinguish stale state from a harmless transport retry, it is not ready for the pager.

The final trade-off is false-positive cost. A threshold that pages on routine mobile suspension trains the on-call to distrust it, while a threshold averaged across all rooms can conceal a small set of high-value rooms with persistent ghosts. Segment by service class where the business has defined different objectives, cap metric cardinality, and review pages against actions taken. Presence becomes dependable when leases, scoped trust, ordering, and alert policy describe the same state machine.

DEV Community