Short answer: choose a server-authoritative, expiring lease over peer-reported host state for an online shared workspace. A peer claim is acceptable only when the workspace must continue without any reachable authority and the product can tolerate temporary disagreement. If two participants both display themselves as host, do not repair the UI first: inspect the election term, lease deadline, membership revision, and delivery acknowledgments that produced each view. The durable fix is one monotonic term, one winner per term, idempotent application, and a snapshot path for every client that misses an update.
The bill begins with fan-out and retention, not election CPU. For an illustrative workspace with 10,000 connected viewers, a 200-byte presence envelope emitted once every 15 seconds creates about 133,333 bytes per second before protocol overhead and replication; broadcasting each change to every member multiplies that term by room size. Retaining every heartbeat for seven days also stores roughly 403 million envelopes. Those are workload calculations, not benchmark claims. The meaningful change is to coalesce heartbeats into current presence, retain compact host transitions longer than ephemeral liveness, and make reconnecting clients fetch a snapshot plus a bounded transition tail. I would deliberately stop keeping routine heartbeat history after the liveness window. The price is forensic precision: after expiry, an operator can prove who held the host lease, but cannot reconstruct every transient online indicator.
How should you debug two clients that think they are host?
A split-brain display usually means the system has confused observation with authority. Participant A sees the old host disconnect and promotes itself locally. Participant B receives the same absence later, runs the same deterministic reassignment rule against a different membership set, and also promotes itself. Determinism does not help when the inputs differ.
Different inputs, different winners.
WebRTC does not define application-level host election. Its peer-connection and data-channel machinery transports media and data, while the application remains responsible for membership, roles, retries, and conflict resolution. A reliable ordered data channel can preserve order within its configured transport context, but it cannot make two peers share one authoritative database transaction, nor can it retroactively reconcile messages sent across disconnected paths.
The fastest diagnostic is to compare records, not screenshots. For each client view, capture a privacy-safe trace containing workspace_id, member_id, membership_revision, election_term, host_id, lease_expires_at, event_id, and the last acknowledged sequence. Then ask four concrete questions: did both decisions use the same membership revision; did an expired holder renew after a newer term existed; did reconnect replay omit a transition; and did a duplicate event overwrite newer state? Clock timestamps help operators correlate evidence, but term and revision numbers decide order. Wall clocks alone are unsafe arbitration inputs because network delay and clock adjustment can invert apparent causality.
Keep the audit event small. Presence is high-volume, short-lived state; authority changes are low-volume control decisions with a longer compliance and incident-review value. Retention must follow the applicable policy and jurisdiction rather than an arbitrary forever setting, and identifiers in logs should be minimized because online status can itself be sensitive.
Lease election and peer claims are different guarantees
| Property | Authoritative lease election | Peer-reported claim |
|---|---|---|
| Winner rule | Authority commits one holder for a numbered term | Each peer derives a holder from what it has observed |
| Partition behavior | A peer without a valid lease stops acting as host | Multiple peers may continue claiming host |
| Reconnect | Snapshot establishes current term before replay | Claims must be merged by application rules |
| Audit evidence | Committed transition identifies term and winner | Local traces may disagree |
| Best boundary | Reachable authority and strict single-host behavior | Authority-free continuity with conflict tolerance |
For a media collaboration workspace, where the host may control recording, stage access, or moderation, I choose the lease. Those actions need the same discipline as a ledger mutation: an idempotency key prevents duplicate application, a fencing term rejects a stale actor, and an audit record explains the accepted decision. Presence dots can be eventually consistent. Privileged host commands cannot.
There is a limitation worth stating plainly. No network design turns an unavailable authority into an available strict single-writer without changing the guarantee. During a partition, the system must either deny host-only actions to a peer that cannot prove a current lease, or accept that more than one actor may proceed. The lease design chooses denial. This trade-off is a product decision as much as a protocol decision.
Availability loses here.
Make reassignment a fenced state transition
The transition below is intentionally generic Go. A datastore implementation must make compare-and-set atomic: commit only if the stored term and membership revision still match the values read for the proposal. The returned term also becomes the fencing token on later host-only commands.
package election
import (
"errors"
"time"
)
type State struct {
WorkspaceID string
HostID string
Term uint64
MembershipRevision uint64
LeaseUntil time.Time
}
type Proposal struct {
EventID string
CandidateID string
ExpectedTerm uint64
MembershipRevision uint64
LeaseUntil time.Time
}
var ErrConflict = errors.New("election state changed")
// Commit must atomically deduplicate EventID and compare both revisions.
type Store interface {
Commit(current State, proposal Proposal) (State, error)
}
func Reassign(store Store, current State, proposal Proposal, now time.Time) (State, error) {
if proposal.ExpectedTerm != current.Term ||
proposal.MembershipRevision != current.MembershipRevision {
return State{}, ErrConflict
}
if now.Before(current.LeaseUntil) {
return State{}, ErrConflict
}
proposal.LeaseUntil = now.Add(30 * time.Second)
return store.Commit(current, proposal)
}
A production commit records the old and new host, term, membership revision, reason, event ID, and decision time in the same atomic boundary as the state update. Repeating the same event ID returns the committed result; submitting a different proposal against the old term fails. The term must increase on each accepted reassignment, and every host-only write carries that term. A stale host can remain visually confused for a moment, but its commands are fenced out.
Stale loses.
Do not let the 30-second example become an unexplained production constant. Set lease duration from measured reconnect and scheduling behavior, then test the availability consequence of that choice. A shorter lease reduces stale tenure but increases false expiry during pauses; a longer lease does the reverse. Renewal should occur comfortably before expiry, and the authority must never extend a lease merely because a client says it is still host.
Delivery guarantees determine whether the UI converges
Once the authority commits term 42, fan-out still has to deliver it. Treat delivery as at-least-once and make consumption idempotent: an event with a lower term is discarded, the same event_id is harmless, and a higher term replaces the local role view. If sequence 818 arrives after 820, the client does not roll back. If it detects a gap, it fetches a fresh snapshot rather than guessing the missing winner.
This is the exactly-once mindset without an exactly-once slogan. The observable business effect occurs once because commit, deduplication, and fencing agree; the network may transmit the envelope more than once. Acknowledgments are useful for bounded replay and lag measurement, but an acknowledgment is evidence of receipt, not proof that every downstream side effect committed.
Testing needs adversarial schedules. Pause the current host beyond lease expiry, delay the transition to one viewer, duplicate it to another, reconnect an old host with queued commands, and deliver terms out of order. Assert that only the current fencing term can perform privileged work and that every viewer converges after snapshot recovery. Then deploy the protocol additively: readers accept the new term fields before writers require them, which avoids interpreting an older envelope as term zero authority.
Observe four signals: reassignment count by reason, rejected stale-term commands, snapshot recoveries after sequence gaps, and acknowledgment lag. Alerting on online-user count alone will miss the control-plane defect. During incident review, correlate by workspace and term rather than by clock time alone.
The decision rule is firm: use authoritative leases when one host controls consequential workspace actions and temporary unavailability is safer than dual authority. Use peer claims only for authority-free sessions whose actions are reversible or mergeable, and surface conflict rather than pretending there is one winner. Keep compact election transitions and command audit records for the policy-defined review period; discard routine heartbeat detail after its operational window, accepting that some historical presence questions will become unanswerable.
Top comments (0)