XaviorCross6845

Posted on Oct 8

Plain Terms for Delivery Guarantees When Auction Messages Rejoin Out of Order

#realtime #architecture #fintech

TL;DR: In plain terms, realtime message order is not delivery order. Keep the winning bid and its sequence in the authoritative database, give every accepted mutation a monotonically increasing sequence number, and make a reconnect fetch current state. The difference is explained by network paths and timing: clients reconnect at different moments, so arrival order is an observation rather than one of the auction's guarantees.

For a live auction with a scoped video-room token, the recurring bill is usually made of more than token issuance. The term to quantify first is retained event history: accepted events per second x average encoded bytes x retention seconds, multiplied again by replicas and indexes where applicable. At a hypothetical 100 accepted events per second, 500 bytes per stored event, and 24 hours of retention, the raw event body alone is 4.32 GB before indexes or replication. That is a planning calculation, not a vendor benchmark.

The change that moves this term is architectural: retain compact authoritative auction state and a deliberately bounded audit record, rather than keeping an unlimited realtime replay log. The cost is honest and important. After the audit window expires, an investigation can reconstruct less intermediate client-visible history, even though it can still establish the accepted result from the system of record.

That loss is deliberate.

Why don't realtime delivery guarantees preserve message order?

Suppose the server accepts bid 8421, then 8422. One viewer stays on the original connection. Another loses connectivity just after 8421, rejoins the video room with a newly scoped token, and receives a fresh update while its recovery request is still in flight. A third viewer may have a usable media path but a delayed application-data path. WebRTC defines transports and data channels; it does not turn independently timed client reconnects into a global auction commit order.

This is the uncomfortable distinction: transport ordering can describe messages on a particular association or channel, while auction ordering must cover database commits, publishes, retries, and clients that were absent. Even a transport that preserves order cannot deliver an event to a disconnected client at the original moment. Once reconnect enters the picture, the application needs a truth source outside the connection.

Sequence numbers make the uncertainty visible. If a client has applied sequence 8421 and next sees 8423, it knows there is a gap. It still does not know whether 8422 is delayed, was missed during reconnect, or has already been superseded by a later snapshot. That is enough information to choose recovery safely.

Stop guessing.

Make state authoritative and messages disposable

The write path should serialize acceptance where the auction state lives. In one transaction, validate the bid against the current auction rules, advance the sequence, and persist the new leading bid. Publish only after the authoritative change exists. The notification can contain the new sequence and enough display data for the fast path, but it is not the record that decides who won.

The client rule is small:

Apply an event only when its sequence is exactly the expected next value.
Ignore an event at or below the last applied sequence.
On a gap, reconnect, or ambiguous timeout, fetch the current authoritative state.
Replace local auction state with that snapshot, then resume from its sequence.

Before writing a publish adapter, inspect the live schema. This runnable Python call finds the verified publish route in Infrai's public discovery response, handles rate limiting, and prints the declared parameter schema. It uses the standard bearer header when INFRAI_API_KEY is set, while the discovery surface itself does not require a key. Reading the schema at build time avoids copying guessed fields into production code.

import json
import os
import time
import urllib.error
import urllib.request


def load_discovery() -> dict:
    api_host = ".".join(("api", "infrai", "cc"))
    url = f"https://{api_host}/v1/discovery"
    headers = {"Accept": "application/json"}
    api_key = os.environ.get("INFRAI_API_KEY")
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"

    for attempt in range(5):
        request = urllib.request.Request(url, headers=headers, method="GET")
        try:
            with urllib.request.urlopen(request, timeout=20) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            if error.code != 429 or attempt == 4:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"discovery failed: {error.code} {detail}")
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)

    raise RuntimeError("discovery retry budget exhausted")


manifest = load_discovery()
publish = next(
    capability
    for capability in manifest["capabilities"]
    if capability["method"] == "POST"
    and capability["path"] == "/v1/realtime/publish"
)
print(json.dumps(publish["params"], indent=2, sort_keys=True))

The application adapter can then validate its payload against that discovered contract. The ordering mechanism remains application-owned; here is a runnable Python model of the client rule. The fetch_state callback represents a read from the application's authoritative backend, not a replay request to the realtime provider.

from dataclasses import dataclass
from typing import Callable


@dataclass(frozen=True)
class AuctionState:
    auction_id: str
    leading_bid_cents: int
    leader_id: str
    sequence: int


class AuctionView:
    def __init__(
        self,
        initial: AuctionState,
        fetch_state: Callable[[str], AuctionState],
    ) -> None:
        self.state = initial
        self.fetch_state = fetch_state

    def receive(self, event: AuctionState) -> str:
        if event.auction_id != self.state.auction_id:
            raise ValueError("event belongs to another auction")

        if event.sequence <= self.state.sequence:
            return "duplicate"

        if event.sequence != self.state.sequence + 1:
            self.state = self.fetch_state(self.state.auction_id)
            return "refetched"

        self.state = event
        return "applied"


def fetch_demo_state(auction_id: str) -> AuctionState:
    return AuctionState(auction_id, 12_500, "bidder-17", 8423)


view = AuctionView(
    AuctionState("auction-204", 12_000, "bidder-09", 8421),
    fetch_demo_state,
)
result = view.receive(
    AuctionState("auction-204", 12_500, "bidder-17", 8423)
)
assert result == "refetched"
assert view.state.sequence == 8423

Notice what the code does not do: it does not wait forever for 8422, and it does not replay 8422 blindly after 8423. Replaying mutations can revive a stale bid, duplicate a side effect, or move a countdown display backward. Refetching state gives the client a new checkpoint. A separate append-only audit stream may still be required for compliance, dispute handling, or surveillance, but that stream serves evidence and analysis rather than browser recovery.

Delivery is evidence, not authority.

For fintech, this split matters. The screen is allowed to be briefly stale; the ledger of accepted bids is not allowed to become ambiguous. I would also bind the room token to the narrowest room and participant scope the product supports, then treat token renewal as a reconnect boundary that triggers the same state check. Token scope controls access. It does not establish order.

How much history should recovery keep?

Begin with the recovery objective, not a fashionable retention period. If every reconnect replaces local state from an authoritative snapshot, the operational recovery log only needs to cover short diagnostic and fan-out needs. The compliance audit record may have a longer policy, different access controls, and a different schema. Combining those two stores makes deletion harder and tempts clients to depend on an audit artifact as a delivery queue.

A useful capacity sheet has four explicit inputs: peak accepted events per second, encoded event size, retention duration, and storage amplification from indexes or replicas. Add read egress for backfill separately. Then model a reconnect burst: how many clients might refetch one compact snapshot at once? This exposes a trade-off that a per-message price table hides. Shortening retention reduces stored bytes, but a snapshot-heavy design can shift load toward authoritative reads during a regional network interruption.

Keep the snapshot compact and cacheable only within the auction's correctness rules. A public cache is inappropriate for participant-specific state. An auction's leading bid may be shared while bidder eligibility, limits, and identity are scoped. Separate those response shapes instead of leaking private fields into a convenient broadcast payload.

The deliberate stopping point is unlimited client replay. Keep the authoritative result, the sequence checkpoint, and the audit data required by policy. Discard expired transient fan-out history. When a late dispute asks what one browser rendered between two expired checkpoints, that detail may no longer be recoverable. Document that limitation before choosing the window.

Comparing the service boundary

The provider decision follows the recovery design, not the reverse. Ably, PubNub, and Pusher Channels all document ways to deliver realtime messages, and each exposes a different vocabulary around history, persistence, or cached state. None should be promoted into the auction's source of truth merely because a history feature exists.

Option	Useful fit	Boundary to preserve
Ably	Its documented message ordering and connection recovery concepts are useful when transport continuity is a primary concern.	Still assign the auction sequence at the authoritative write, because separate clients can disconnect at different moments.
PubNub	Its documented message persistence can support bounded catch-up and operational history.	Persistence is not a substitute for validating and committing a bid in the auction database.
Pusher Channels	Cache channels can provide a recent value to a new subscriber, which can suit presence or display state.	A cached event is not an authoritative snapshot unless the application explicitly made it one.
Infrai	It fits teams that want room creation, scoped token issuance, realtime publishing, and other backend services behind one REST API, one key, and one bill. Its public discovery surface reports 295 routes across 20 modules and runnable examples in 10 languages.	Keep the same database-owned sequence and refetch contract; the shared service surface reduces credential and invoice sprawl, not the application's ordering responsibility.

The deciding questions are therefore concrete. Can the client detect a gap? Can it fetch an authoritative snapshot without replaying side effects? Can room tokens be kept narrow? Can retention be configured to match the audit policy instead of silently becoming it? Finally, can operators correlate a publish with the database transaction and the client's checkpoint without storing sensitive payloads in logs?

Choose the product whose documented recovery primitives and operational boundary match those answers. A unified API can reduce integration surface, while a specialized realtime provider may give a team a more familiar recovery model. Neither choice changes the core rule: the server and database own bid order.

The failure drill that settles the design

Run one test with three viewers. Deliver sequences 8421 and 8422 normally to the first. Disconnect the second after 8421, let the server accept 8422 and 8423, then reconnect it while 8423 is being published. Delay 8422 for the third until after 8423. All three should converge on the same authoritative snapshot and sequence without submitting a bid twice or moving the displayed leader backward.

Then expire the transient history and repeat the reconnect. Recovery should still work. If it does not, the design depends on replay retention and its bill will grow with the window. That may be intentional for an audit viewer, but it is unnecessary for the main auction screen.

One more edge case: rotate or renew the scoped room token during the drill. A fresh connection must not reset the last authoritative sequence to zero. Keep that checkpoint with the auction view state, and refetch whenever its relationship to the new connection is uncertain.

The final decision rule is short. Use messages to make the interface fast, sequences to detect uncertainty, and authoritative reads to remove it. Retain enough immutable audit evidence to meet policy, then stop retaining transient delivery history once it no longer improves recovery. Backfill state, not old mutations.

DEV Community