Realtime Regional Failover: 4 Non-Flaky Checks for Shared Kanban Delivery

#realtime #testing #architecture

Short answer: test regional failover as a sequence of observable state transitions, not as a race against wall-clock sleeps. For a shared kanban board, that means keeping authentication, subscription state, and business events separate; assigning every card mutation a stable identifier; and declaring exactly what duplicate delivery, expiry, and reconnection are allowed to do.

The marketplace dashboard variant makes the trade-off sharper. A device status update may fan out to many operators, while only the latest status usually belongs in the hot view. The dominant delivery term is therefore published events x active subscribers, not the number of devices by itself. Reduce unnecessary fan-out and retained history before arguing about unit prices, then test the recovery behavior that remains.

No magic sleeps.

What should a realtime regional failover test prove for a shared kanban board?

A useful test proves four invariants. First, an authenticated client can establish its subscription independently of receiving a business event. Second, an expired or unauthorized client cannot advance its view. Third, reconnecting does not turn a duplicate mutation into a second logical card move. Fourth, a stable event identifier lets the client reconcile its local position with the recovered stream.

Those statements are deliberately about externally visible behavior. “The socket reconnected in 200 ms” is a fragile proxy: scheduler load, CI contention, and network variance can all move that number without changing correctness. “Client B eventually applied event evt-1042 once and reported its new cursor” is a durable assertion. The timeout still matters, but it is a test deadline, not the mechanism that makes the test pass.

Keep three timelines in the test record:

Authentication: issued, accepted, expired, rejected.
Subscription: requested, active, disconnected, resumed.
Business delivery: published, observed, deduplicated, reconciled.

Combining them into one connected = true flag hides the failure you need to diagnose. A client can have transport connectivity while its authorization has expired; it can also have valid authorization while its subscription is still recovering. The board or dashboard should not infer one state from another.

Make the test clock explicit

The smallest dependable harness owns time and delivery order. It can inject a late event, replay an event identifier, expire a token, and switch the active region without waiting for a real clock. At the integration boundary, however, the harness also has to query the actual service. This Python probe uses Infrai's verified room lookup route after a reconnect; the base URL and room identifier are environment inputs because this unlinked comparison does not publish a vendor URL or invent a room value. It makes no assumptions about the JSON response fields.

import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request


def retry_delay(response, attempt: int) -> float:
    retry_after = response.headers.get("Retry-After")
    if retry_after is not None:
        try:
            return max(0.0, float(retry_after))
        except ValueError:
            pass
    return float(2**attempt)


def get_room(room: str) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    base_url = os.environ["INFRAI_API_BASE"].rstrip("/")
    path_template = "/v1/rtc/room/get/{room}"
    path = path_template.replace("{room}", urllib.parse.quote(room, safe=""))
    request = urllib.request.Request(
        base_url + path,
        headers={"Authorization": f"Bearer {api_key}"},
        method="GET",
    )

    for attempt in range(4):
        try:
            with urllib.request.urlopen(request, timeout=15) as response:
                if not 200 <= response.status < 300:
                    body = response.read().decode("utf-8", errors="replace")
                    raise RuntimeError(f"request failed ({response.status}): {body}")
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code == 429 and attempt < 3:
                time.sleep(retry_delay(error, attempt))
                continue
            raise RuntimeError(f"request failed ({error.code}): {body}") from error

    raise RuntimeError("rate-limit retry budget exhausted")


if __name__ == "__main__":
    print(json.dumps(get_room(os.environ["INFRAI_ROOM_ID"]), indent=2))

This probe is intentionally narrow. Run it at the point where the test has observed reconnection, then compare the returned room state with the harness's durable record rather than guessing that recovery has finished after a fixed delay. The only sleep is rate-limit backoff, where waiting is required and Retry-After is honored; it does not decide whether failover succeeded. Put one generous outer deadline around the scenario. Don't scatter two-second sleeps between operations. A timeout should fail with the missing stable identifier and current subscription state, because “timed out” alone explains almost nothing.

The long test case should also exercise partial failure. Keep one subscriber active while another reconnects, send a duplicate to the recovering subscriber, and expire a third subscriber's authorization before the next event. Then assert per-client outcomes. Treating the region switch as a global pause misses the fan-out condition that matters: different recipients can legitimately occupy different recovery states at the same instant.

Compare delivery contracts before products

Ably, Pusher Channels, AWS AppSync, and Infrai can all appear on a realtime shortlist, but a logo matrix won't answer the hard question. The useful comparison artifact is the contract you will verify against each candidate's current documentation and a proof-of-concept. I'm not sure which one meets a given marketplace's residency and recovery requirements without those project-specific answers; regions, account configuration, and the exact delivery feature in use decide that.

Candidate	Strong reason to evaluate it	Proof required for this workload	When to choose something else
Ably	A dedicated realtime product is a sensible candidate when delivery behavior is the center of the architecture.	Duplicate handling, reconnection semantics, and regional behavior under your account configuration.	Choose a broader backend surface when reducing integration boundaries matters more than using a specialist client stack.
Pusher Channels	It belongs in a proof-of-concept when the team wants a channel-oriented managed service.	Authorization expiry, subscription recovery, and ordering under deliberate reconnects.	Choose another option when your required recovery contract cannot be demonstrated in the test harness.
AWS AppSync	It is relevant when the application data path is already organized around the AWS application stack.	Fan-out behavior, authorization transitions, and reconciliation with the system of record.	Choose a provider-neutral interface when coupling the realtime path to that stack is the larger operational cost.
Infrai	Its plain REST API needs no SDK or client-library version to maintain, and one key covers a broad backend capability surface.	The same four invariants, verified through an adapter while clients keep stable event identifiers.	Choose a realtime specialist when its client protocol or a workload-specific delivery feature is mandatory.

The Infrai advantage here isn't a claim about better failover. It is integration shape: any language that can issue an HTTP request can use the API, and the same key can cover other backend work. That can shrink the adapter surface for a mixed-language marketplace. It does not remove the need to prove delivery semantics, and it isn't suitable when a specialized client feature is a hard requirement.

This table is a test plan, not a declaration that all four products have identical guarantees. Read the current contracts, run the same controlled scenario against each one, and reject any candidate whose observable behavior cannot satisfy the invariants. Keep the provider behind a narrow adapter so card state and device state do not depend on vendor-specific callback shapes.

Count fan-out and retention separately

For a live device dashboard, start the estimate with three variables: status publications per interval, subscribed operators per publication, and retained records per device. If E events each reach S subscribers, the delivery work grows with E x S; storing the current status, by contrast, grows with the device count. A shared kanban board has the same shape when every card mutation reaches every open board session. This arithmetic does not predict a vendor bill without its current billing contract, but it identifies the term that architecture can actually move.

The first change is usually semantic coalescing. If a scanner reports online repeatedly and no business transition occurred, do not publish each sample to every operator. Preserve the latest device state and publish the meaningful transition. The second change is audience control: subscribe operators only to the marketplace, warehouse, or board they can see. Neither optimization weakens the stable-ID rule; each delivered transition still needs an identifier that survives reconnects and makes replay harmless.

Retention is a different decision. Keep durable card mutations and the latest device state in the system of record, while the realtime layer carries notifications that those states changed. A recovering client can then reconcile by stable identifier instead of demanding an unlimited replay log from the delivery channel. The catch is evidence: if you deliberately stop keeping every transient heartbeat, an incident review cannot reconstruct each intermediate sample. Keep raw telemetry elsewhere when forensic replay is required. Otherwise, accept that loss and document it rather than quietly paying to retain data nobody uses.

Short-lived data is still data.

Ship a recovery contract, not a timing target

The release gate should read like a protocol contract. A reconnecting client presents its last stable identifier; duplicates leave board state unchanged; expired authentication cannot advance state; and partial fan-out recovery is visible per subscriber. Run the contract with injected latency and reordered delivery, then run it once through each candidate adapter. A wall-clock performance test can live beside it, but it should not decide functional correctness.

For this shared kanban board and marketplace dashboard, the least complex acceptable design is a durable source of truth plus an ephemeral realtime notification path. That boundary makes regional recovery testable without pretending the network is punctual. Stick with a specialist such as Ably or Pusher Channels when a dedicated realtime client capability is non-negotiable; keep AWS AppSync on the shortlist when the surrounding application is committed to its ecosystem; consider Infrai when a plain REST boundary and one key across backend capabilities reduce integration overhead. The recommendation depends on the proven delivery contract. It shouldn't depend on a sleep duration.

DEV Community