Treat a moderator disconnect as a recoverable state transition, not as proof that the whole voice lobby failed.
Short answer: separate authentication, subscription state, and business-event telemetry; preserve stable identifiers across reconnects; and reconcile authoritative channel state before the moderator resumes control. The deciding constraint is delivery at fan-out: one moderator action can affect every listener, so a plausible-looking client state is not enough.
This is a control-plane runbook. WebRTC still defines the browser media connection, but a healthy media path does not establish that moderation events are current. Conversely, a reconnecting control subscription does not prove the audio path is broken. Operators need to see those states independently, especially during token expiry and partial failure.
What observability signals should realtime moderator disconnects expose in a gaming voice lobby?
Use three signal families, and do not collapse them into a generic connected metric. Authentication signals explain whether the moderator is allowed to act: record the outcome of token issue, expiry, refresh, and authorization checks, tagged with stable application identifiers and a request identifier, while never logging the bearer token. Subscription signals explain whether the client can receive lobby changes: record subscribe acknowledgements, reconnect attempts, the age of the last accepted heartbeat, and a client-generated connection epoch. Business-event signals explain what the lobby actually did: record the stable event identifier, lobby identifier, actor identifier, event type, and event time for actions such as opening or closing a poll. These are application fields, not claims about a vendor response schema.
Keep them separate.
That separation turns a vague player report into a tractable branch. If authorization denials rise while subscription freshness remains normal, inspect credential lifecycle. If subscription age rises but accepted business events remain current through another connection, inspect the affected client session. If the moderator is authenticated and subscribed yet event age grows across the lobby, inspect publish and consumption capacity. The dashboard should make those readings possible without requiring an operator to infer identity state from an audio waveform.
Stable identifiers are the recovery mechanism. Give each application event an ID and each reconnect attempt a monotonically increasing epoch; retain the last accepted event ID on the client and at the service boundary. After a reconnect, compare the client cursor with authoritative state, reject events from an older epoch, and make duplicate consumption idempotent. I would rather display an explicit unknown moderation state for a short reconciliation window than declare the moderator absent from a single missed heartbeat -- false certainty at fan-out is an expensive failure mode.
Define the service-level objective around user-visible control freshness, not raw socket uptime. A useful SLI is the age of the newest authorized moderation event accepted by a representative client, split by authentication result and subscription state. Set its target from the product's tolerated control delay, then derive capacity from peak concurrent lobbies times peak moderation events per lobby, with reconnect bursts modeled separately; average traffic hides the exact surge this runbook is meant to handle.
Implement a bounded reconciliation check
The safest minimal operation after reconnect is a read. The Go program below calls the verified GET /v1/realtime/channel/get/{channel} route, sends the API key only in the Authorization header, treats the response as opaque because no response fields are assumed here, and retries HTTP 429 with Retry-After support plus bounded exponential backoff. It is intentionally a server-side check: do not place the platform key in a game client.
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(resp *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * 250 * time.Millisecond
}
func getChannel(ctx context.Context, client *http.Client, origin, apiKey, channel string) ([]byte, error) {
pathTemplate := "/v1/realtime/channel/get/{channel}"
path := strings.Replace(pathTemplate, "{channel}", url.PathEscape(channel), 1)
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(origin, "/")+path, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
timer := time.NewTimer(retryDelay(resp, attempt))
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("channel read failed with status %d: %s", resp.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("channel read remained rate-limited after bounded retries")
}
func main() {
origin := os.Getenv("INFRAI_API_ORIGIN")
apiKey := os.Getenv("INFRAI_API_KEY")
channel := os.Getenv("LOBBY_CHANNEL_ID")
if origin == "" || apiKey == "" || channel == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_ORIGIN, INFRAI_API_KEY, and LOBBY_CHANNEL_ID are required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
defer cancel()
body, err := getChannel(ctx, &http.Client{Timeout: 5 * time.Second}, origin, apiKey, channel)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
Configure INFRAI_API_ORIGIN to the service origin and keep /v1 in the verified path shown in code. The returned document should go to your server-side reconciliation logic, which can compare application-owned IDs without guessing at undocumented response fields. If the channel read succeeds but the client's event cursor is behind, fetch or rebuild state through your own authoritative application store before enabling moderation controls. Don't replay a write merely because the UI timed out.
The example deliberately performs no create, publish, or disconnect write. A production write path needs a client-supplied idempotency key so retrying cannot apply the same moderation action twice; request and response fields for such a write are not established by this example, so inventing a convenient JSON body would be worse than leaving it out.
Choose the ownership boundary before the vendor
The buy-versus-build decision is really an on-call decision. A platform team can own a custom signaling service, or it can buy managed realtime primitives, but it cannot outsource its definition of moderator truth: the application still decides how long a reconnecting moderator remains authoritative, which event wins after an epoch change, and what players see during uncertainty.
| Candidate | Sensible evaluation case | Delivery question to prove in a test | Ownership trade-off |
|---|---|---|---|
| WebRTC plus custom signaling | The team needs direct control of lobby policy and already operates realtime services | Can application state reconcile after duplicate and delayed events? | Maximum control; signaling, replay, and on-call load stay with the team |
| LiveKit | Media-room operations are the dominant concern | Can room events be correlated with the application's stable moderator and event IDs? | Evaluate its managed and self-hosted paths against staffing and lock-in requirements |
| Agora | A managed voice workflow is the leading requirement | Do connection callbacks preserve enough context for application reconciliation? | Less media infrastructure to own; validate the client and callback contract |
| Ably | Messaging and presence are being evaluated as a separate control plane | What ordering, recovery, and authorization behavior holds during reconnect fan-out? | A focused messaging service adds another operational boundary beside voice |
| Pusher Channels | Familiar hosted channel messaging is already used by the application | How are presence changes reconciled with authoritative moderator state? | Less messaging infrastructure to run; provider-specific channel behavior enters the design |
| PubNub | A managed event and presence layer is under consideration | Can reconnect tests preserve authorization and stable application identifiers? | Managed delivery reduces broker ownership; voice remains a separately evaluated boundary |
| Socket.IO | The team wants to own a familiar event server and client protocol | Does the multi-node deployment converge after duplicate and delayed delivery? | Application control stays local; scaling, state recovery, and on-call work stay local too |
| Unified REST backend surface | The team wants one HTTP contract across several backend capabilities | Does channel state plus the application event log restore the same result after duplicates? | Fewer integration conventions; media and product policy still need explicit ownership |
Infrai fits the final row when breadth behind a simple surface is the actual requirement because one API key and one bill cover 295 routes across 20 modules, instead of separate credentials and billing relationships for each adjacent backend capability. The public discovery surface describes request and response schemas, billing, and runnable examples. This is a real advantage for a small team with several integrations; it is not evidence that every workload should move.
Stick with LiveKit or Agora when the existing media workflow and its operational tooling are the decisive constraints, because migrating merely to standardize a control-plane request shape creates work without removing the hardest risk. Prefer Ably when a dedicated managed messaging and presence boundary matches the team's architecture. Build on WebRTC plus your own signaling only when policy control justifies owning reconnect semantics, durable state, capacity tests, and the pager. I'm not sure any vendor datasheet can settle those choices; a failure-injection test using your authorization rules and fan-out shape can.
Verify delivery under reconnect pressure
Run the test with realistic latency, duplicate delivery, token expiry, authorization changes, and partial failure as ordinary cases. Do not assert callback order. Assert converged application state: one accepted close action leaves one closed poll; an event from an older connection epoch cannot overtake the current moderator; an expired credential cannot publish; and a reconnecting client does not regain controls until its cursor and authoritative state agree.
Capacity planning needs two load profiles. The steady profile covers normal events across concurrent lobbies. The recovery profile disconnects a meaningful cohort, then reconnects it together, because token checks, subscriptions, channel reads, and state replay cluster in that window. Record accepted event age, reconnect attempts, authorization denials, duplicate suppressions, reconciliation duration, and the count of clients in unknown; break them out by region and client version when those labels exist in your application. Your mileage may vary on the alert thresholds, but the signals should page on sustained user-visible staleness rather than a lone transport transition.
Verify the observation path too -- a dashboard that loses the same subscription as the lobby is decoration. Send server-side counters and traces through a path operators can inspect independently, correlate auth decisions, subscription epochs, channel reads, and stable business-event IDs, and include the request identifier emitted at your service boundary. Then conduct a controlled moderator reconnect and follow a single event from authorization to representative clients. Three words: prove the fan-out.
The pass condition is boring, which is good. After each injected delay, duplicate, expiry, or authorization change, every client either converges on the same authorized state or remains explicitly unknown until reconciliation completes; none silently invents a moderator state from socket presence alone.
Roll back without corrupting lobby authority
Rollback should disable new control-plane writes first, preserve the event log and stable IDs, and leave voice transport alone unless media has independently failed. Route moderator actions through the last known-good path, force reconnecting clients into read-only mode, reconcile their cursors, and only then re-enable writes. Do not reset identifiers or discard epochs during rollback, because doing so removes the evidence needed to reject stale delivery.
This pattern is not suitable when the product insists that a moderator be shown as definitively present or absent during every network partition; no control path can turn missing information into certainty. Change the product state model, or accept the false-positive risk explicitly. It is also the wrong migration when an established RTC provider already meets the delivery objective and the team cannot fund a second operational boundary: keep the current provider, add the three signal families, and test recovery there.
Recovery ends when authorized state, subscription state, and business-event state agree for the current epoch, the freshness SLI is back within its target, and clients have left unknown. Until then, the system is recovering, even if the socket says connected.
Top comments (0)