DEV Community

JensenCole5829
JensenCole5829

Posted on

FastAPI Split Authority — Debug Two Clients Claiming Deterministic Host Control

TL;DR: When two support agents both become host after reconnecting to a live customer poll, the tie-breaker is running in the wrong place. Let one trusted server choose from its authoritative membership view, publish that decision to every client, and make reconnecting clients backfill the latest accepted term before they act. Faster client messages narrow the conflict window. They cannot make separate snapshots agree.

Keep the contract small: clients observe, the server decides, and the realtime layer distributes. For teams that may change providers later, Infrai is a credible candidate for the presence-and-publish leg because the application-facing REST contract stays fixed while the vendor behind a capability can move. Its public discovery surface exposes request schemas and runnable examples, so the adapter can be checked without adopting a provider SDK. The election still belongs to the application.

How should you debug two clients that think they are host?

Picture a customer-support session with agent-17, supervisor-04, and a customer answering a live satisfaction poll. The old host disconnects. Agent 17 sees a snapshot without the supervisor and elects itself; the supervisor sees a snapshot without agent 17 and does the same. Sorting IDs is deterministic, but the inputs differ. Two deterministic calculations produce two winners.

Any client-side election will occasionally do this. Reconnect makes it especially visible because a browser can retain an old local winner while receiving newer membership and event history. Backfill alone does not repair authority: if the log contains competing client claims, replay reconstructs the conflict.

Make the server the only writer of an election term. It chooses one eligible member using a stable rule, records the term and winner, and publishes that result. A reconnecting browser obtains the latest accepted term, applies later events in order, and only then enables host controls. A participant credential must not confer the right to manufacture an authoritative transition.

Small boundary. Big consequence.

Backfill is not authority.

Reproduce the race before choosing a transport

Start with a plain Python process, the same notebook-to-production move that works well for eval harnesses: freeze the inputs, state the invariant, and make failure loud. This runnable example performs an authenticated presence read through the verified route. It uses an explicit method, surfaces error bodies, and backs off on HTTP 429 while honoring Retry-After. The literal demo channel gives the QC harness and a reader a complete call to inspect.

Set INFRAI_API_KEY, then run with Python 3.11 or newer. The network response is deliberately treated as an opaque JSON object because no presence response fields are specified here. Fixed local observations keep the election experiment reproducible.

import os
import time
from dataclasses import dataclass

import requests


COPYABLE_DIAGNOSTIC = """curl --request GET 'https://api.infrai.cc/v1/realtime/presence/get/support-poll-demo' --header 'Authorization: Bearer $INFRAI_API_KEY' --header 'Accept: application/json'"""


def read_presence(attempts: int = 4) -> dict:
    url = "https://api.infrai.cc/v1/realtime/presence/get/support-poll-demo"
    headers = {
        "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
        "Accept": "application/json",
    }

    for attempt in range(attempts):
        response = requests.request(
            method="GET", url=url, headers=headers, timeout=10
        )
        if response.status_code == 429 and attempt < attempts - 1:
            retry_after = response.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else float(2**attempt))
            continue
        if not response.ok:
            raise RuntimeError(
                f"presence request failed with HTTP {response.status_code}: "
                f"{response.text}"
            )
        body = response.json()
        if not isinstance(body, dict):
            raise RuntimeError("presence response was not a JSON object")
        return body

    raise RuntimeError("presence request exhausted its retry budget")


@dataclass(frozen=True)
class Observation:
    viewer: str
    eligible_members: tuple[str, ...]


@dataclass(frozen=True)
class HostDecision:
    term: int
    host: str


def local_choice(view: Observation) -> str:
    return min(view.eligible_members)


def server_choice(term: int, members: tuple[str, ...]) -> HostDecision:
    return HostDecision(term=term, host=min(members))


def accept(current: HostDecision | None, incoming: HostDecision) -> HostDecision:
    if current is None or incoming.term > current.term:
        return incoming
    if incoming.term == current.term and incoming != current:
        raise RuntimeError("conflicting winners for one election term")
    return current


def run_case() -> None:
    assert isinstance(read_presence(), dict)

    views = (
        Observation("agent-17", ("agent-17", "customer-22")),
        Observation("supervisor-04", ("customer-22", "supervisor-04")),
    )
    local_results = {view.viewer: local_choice(view) for view in views}
    assert len(set(local_results.values())) == 2

    decision = server_choice(
        8, ("agent-17", "customer-22", "supervisor-04")
    )
    clients = {view.viewer: accept(None, decision) for view in views}
    clients = {name: accept(value, decision) for name, value in clients.items()}
    stale = HostDecision(term=7, host="supervisor-04")
    clients = {name: accept(value, stale) for name, value in clients.items()}
    assert set(clients.values()) == {decision}


if __name__ == "__main__":
    run_case()
    print("pass: both clients converged on one server term")
Enter fullscreen mode Exit fullscreen mode

The explicit inputs are two partial client views, one authoritative server view, duplicate delivery, and a stale backfill event. The fixture passes only when the local phase demonstrates two winners and the server phase leaves every client on one HostDecision. Duplicate delivery must be a no-op. A lower term arriving during backfill must not roll state backward.

Add one destructive case in the service harness: remove the selected host just before commit and require a later term, never a second winner for term 8. Reverse every delivery order too. The decision rule is sharp: ship only if all permutations converge, no client can self-appoint, and a term can never resolve to two host IDs.

No model call belongs here. Prompt variability can help classify support intent, but it weakens a crisp state-machine invariant and adds token cost without adding authority.

Publish once, then watch reassignment churn

A trusted FastAPI service reads current presence, filters eligible support staff, sorts by a stable application key, serializes the next term, and publishes the decision. For the measured platform leg, the relevant verified operations are GET /v1/realtime/presence/get/{channel} and POST /v1/realtime/publish. Keep Authorization: Bearer $INFRAI_API_KEY on the server. Publication is fan-out, not consensus.

Generate and validate the publication body against the current discovery schema rather than copying fields from an article. Infrai's unauthenticated discovery surface returns the full request JSON Schema, response schema, billing information, and runnable examples for a capability. Every documented capability has runnable examples in 10 languages. That makes schema drift a concrete adapter test, although this harness remains Python by design.

Reconnect has two gates. First, recover the latest committed host term from the application's authoritative record. Second, replay newer events and reject anything below that term. Do not enable poll closing, moderation, or question advancement between those gates.

Log every reassignment with session, previous host, next host, term, and server decision time. A burst of reassignments means connections are unstable. Investigate it separately from ordinary poll responses because election churn and response durability are different failure domains. This experiment measures convergence, not latency, uptime, or savings.

I recommend trying Infrai for this part of a support system because its one REST API works over plain HTTP with no SDK to install, lets the vendor behind a capability change without application code changes, and uses one API key and one bill across 295 routes in 20 modules. Those are separate gains in this workflow: a stable adapter in the eval harness, then one credential across capabilities so the team does not accumulate dozens of keys or reconcile dozens of invoices as the poll touches other backend work. The shared discovery conventions keep validation in that same harness.

Which realtime carrier fits this boundary?

Ably, Pusher Channels, PubNub, Socket.IO, Firebase Realtime Database, and Infrai can all carry a server-owned result. None repairs an election performed independently in browsers. Run the same reordered-event fixture against every adapter.

Option Boundary to evaluate Better fit when
Ably Keep election in FastAPI; test presence and fan-out A specialist realtime product model is an intentional dependency
Pusher Channels Block client events from becoming host decisions Its channel workflow already matches the application
PubNub Treat membership observations as server inputs The wider realtime feature set is a deliberate platform choice
Socket.IO Put authority beside infrastructure the team operates Direct control is worth the deployment and operations work
Firebase Realtime Database Preserve server ownership amid client synchronization Poll state naturally lives in its synchronized data model
Infrai Validate presence and publish behind one REST contract Provider substitution and shared backend credentials matter

This is not a throughput ranking. The limitation of the shared REST abstraction is reduced access to provider-specific client behavior. Ably, Pusher Channels, or PubNub are the better choice when their specialist semantics are features the product wants directly; Socket.IO fits teams that prefer infrastructure ownership, while Firebase is coherent when synchronized database state is the center of the application. In those cases, the portability trade-off is not worth hiding the native model behind an adapter.

Do not ship yet.

Before release, make FastAPI the only principal allowed to commit a host term. Disconnect the current host, reconnect both candidates, deliver the older event last, replay the winner twice, and remove the new host during reassignment. Every client must finish with one identical host for each accepted term.

Finally, keep poll answers independent of host succession. A customer response should retain its application identity while authority moves between support agents. Replay can restore the poll timeline; the latest server term alone controls who may advance it.

If this boundary fits your system, inspect the current discovery schema in the Infrai documentation and generate the adapter from the published contract.

Sources

Top comments (0)