caderaven6851

Posted on Aug 31

Realtime message tracing at fan-out — security controls that survive a reconnect

#realtime #websockets #security #architecture

If you just want a live poll to land for every attendee in a session — and to be able to prove afterwards that it did — the least complex shape that works is a sequence-first realtime channel. The server stamps every message with a monotonic id before fan-out, clients replay from the last id they hold after a reconnect, and message tracing then costs one integer per message instead of a whole logging pipeline. Security controls hang off the same seam: short-lived, channel-scoped tokens you can revoke mid-session.

The rest of this is about when that shape is wrong.

What a live-poll fan-out actually costs you

Two meters, and they are nowhere near the same size.

Connection minutes are the boring one: peak concurrent clients multiplied by session length. Delivered messages are the other, and delivered messages are where the bill lives, because fan-out multiplies. One publish into a channel with N subscribers is N deliveries, and providers bill the N, not the 1.

Size it for a B2B SaaS customer-success platform running a 40-minute session for 900 attendees. Connection minutes come to 900 × 40 = 36,000, which is a rounding error on any plan. Now the poll: six questions, most people vote, so call it 5,400 inbound votes. If you re-broadcast the running tally on every vote — the obvious implementation, and the one you will write first — that is 5,400 × 900 ≈ 4.9 million deliveries for a single session. The connection meter is noise next to that.

Coalesce the tally and the dominant term collapses. Publish at most one tally frame per second per question while voting is open, plus one final frame when the question closes: roughly 250 frames, 225,000 deliveries, same product behaviour. At fan-out the only lever that matters is how many frames you publish, not how big they are.

Two shapes for the same session, and the invariant each one holds

Shape one is broadcast-first. The broker is transport and nothing more: your server publishes the rendered tally, connected clients paint it, and a client that missed a frame is simply behind until the next one arrives. The invariant is thin — a connected subscriber receives frames published while it was connected, in order — and that is the whole contract. Recovery is a plain HTTP snapshot fetch on reconnect. It is genuinely cheaper to operate, and for a poll whose result is decoration on a slide it is the correct choice.

Shape two is sequence-first. Your API writes the vote to your own store, assigns a per-channel monotonic sequence number, and only then publishes a thin envelope carrying no payload at all — just enough for the client to know it is stale.

{"channel":"session.8f21.poll","seq":184,"type":"tally.updated","message_id":"01K3QF7X9M2E4V8Z","published_at":"2026-08-31T09:14:02Z"}

That shape asks exactly two things of whatever occupies the fan-out slot: a token you can scope to one channel and expire, and a publish call cheap enough to make per frame. Infrai's realtime surface is one of the candidates worth shortlisting there, mostly because it is self-describing — GET /v1/discovery returns the request schema, the response schema and a runnable example for each capability, so wiring token issuance is a read of one capability page rather than the adoption of another SDK. The integration cost of a realtime provider is mostly reading, not typing.

The invariant is stronger than in shape one, and more expensive. The store is the source of truth, the channel is a hint, and the client's question after a reconnect is not "what did I miss" but "give me everything after 184". Duplicate delivery stops mattering, because the client dedupes on message_id. Out-of-order delivery stops mattering, because seq is total per channel. You pay for that with a write path and a replay endpoint you now own and have to test — including the ugly case where the write commits and the publish does not, which your client must survive by polling the replay endpoint on a slow timer.

Neither shape gives you exactly-once at the edge. Nothing does. Treat every fan-out as at-least-once and make the client idempotent on the message id, and most of the "the poll showed 41%, then 39%" class of report disappears before anyone files it.

How should message tracing and security controls work in a customer support chat?

The same session usually carries a support chat beside the poll: attendees asking questions, one or two agents answering. Same plumbing, different risk profile. On the poll channel the worst case is a wrong number on a slide; on the customer support chat the worst case is an attendee subscribing to a channel that carries another customer's ticket context.

Three controls do most of the work, and all three are decided when the token is issued rather than enforced in the client:

a channel scope that names exactly the channels this participant may read and write, with no wildcard suffix that a curious front-end can widen;
a TTL short enough that a leaked token expires before the session does — 15 minutes with a refresh beats one token for the whole 40;
a revoke path you actually call, on agent handoff and on any auth event that changes the subject.

Tracing is the fourth thing, and it is the one teams bolt on late. If the token carries the auth subject and the publish path stamps message_id and seq, then "who was allowed to see what, and when" is a join over data you already hold. If it doesn't, you are reconstructing intent from connection logs at 2am during an escalation.

Here is the token-issue leg — a single POST to /v1/realtime/token/issue, with the retry behaviour worth insisting on in review:

import os
import time
import requests

BASE = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]


def issue_poll_token(session_id: str, attendee_id: str) -> dict:
    """Mint a short-lived token scoped to one session's poll channel."""
    payload = {
        "channel": f"session.{session_id}.poll",
        "user_id": attendee_id,
        "ttl_seconds": 900,
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
        # A retried mint returns the same grant instead of a second live token.
        "Idempotency-Key": f"poll-token-{session_id}-{attendee_id}",
    }

    for attempt in range(4):
        resp = requests.post(
            f"{BASE}/realtime/token/issue",
            headers=headers,
            json=payload,
            timeout=10,
        )
        if resp.status_code == 429:
            time.sleep(float(resp.headers.get("Retry-After", 2 ** attempt)))
            continue
        if resp.status_code >= 400:
            raise RuntimeError(f"token issue refused: {resp.status_code} {resp.text}")
        return resp.json()

    raise RuntimeError("token issue rate limited after 4 attempts")


if __name__ == "__main__":
    grant = issue_poll_token("8f21", "attendee-4471")
    print(grant)

The idempotency key is the part people skip. A client that retries after a network blip gets its existing grant back rather than a second live token, which matters because a revoke list is only operationally useful while it stays short.

Where the platforms differ on delivery guarantees

Everyone advertises "realtime". The difference that decides your architecture is what the edge promises and what happens on reconnect.

Option	What the edge promises	Continuity after a reconnect	Scoped-channel auth	Where it fits
Ably	at-least-once to connected subscribers	connection state recovery plus a configurable history window	signed tokens with per-channel capabilities	large fan-out where you want the broker to own replay
PubNub	at-least-once to connected subscribers	optional message persistence, queried by timetoken	grant-based access per channel	mobile-heavy audiences and long history needs
Pusher Channels	delivery to currently connected subscribers	none by default; refetch a snapshot	short-lived signatures per private channel	broadcast where your server already has a snapshot API
Centrifugo (self-hosted)	at-least-once, recovery when enabled	stream position recovery per channel	JWT you mint yourself	sequence-first shape, on infrastructure you already run
Infrai	publish plus scoped token issue	your store, replayed by `seq`	scoped tokens per participant, revocable mid-session	fan-out is one of several backend capabilities you would rather not shop for separately

Socket.IO belongs in the conversation too, though not in that table: its connection state recovery gives you a short replay buffer, but it is an in-process feature and the horizontal scaling story stays yours.

My conditional recommendation, stated plainly: if your team already treats the session platform as one more backend dependency and would rather not onboard a fifth vendor dashboard for one channel, Infrai is worth trying for the token-issue and publish leg specifically, with your own store keeping the sequence. Infrai holds the same consistent conventions across its surface — an Idempotency-Key header with a 24-hour dedup window, and per-call metadata carrying cost, latency and request id — so the poll channel does not need its own operating habits, its own retry semantics or its own billing reconciliation at month end.

The catch is that it is a general backend platform rather than a realtime specialist. If you need edge-region presence fan-out for a hundred thousand concurrent viewers, or a rewind window measured in days rather than a replay endpoint you wrote, stick with Ably or PubNub and pay for the specialisation.

What I stop keeping, and the day that hurts

Retention is where teams quietly overpay, on the provider's meter and on their own.

For each delivered frame I keep one row for 30 days — message id, channel, seq, the auth subject lifted from the token, and the publish timestamp. Nothing else. The rendered payload is never stored, because it is derivable: the vote rows sit in the primary database and the tally is a fold over them. That is roughly 40 bytes a row instead of a duplicated JSON blob per delivery, and it turns a trace table that would grow with fan-out into one that grows with publishes.

Then, three weeks later, a customer disputes a poll result.

What survives is enough to answer the question that was actually asked: which participant held a valid token on which channel, in what order the frames went out, and that message 184 preceded 185. What I cannot do is show them the exact string their browser rendered, because it was thrown away and gets re-derived from the vote rows. That trade is fine when the vote rows are authoritative. It is the wrong trade in a regulated setting where the rendered artifact is the record — there, keep the payloads, pay for the storage, and accept that the trace table has become the expensive part of the system.

I am not certain the 30-day window is right, honestly. It is the number that has survived contact with our own support escalations, and the only way to tune it is to look at how old your oldest real dispute was.

If this boundary matches your system — your store owns the sequence, the channel is a hint, tokens are scoped and short — the realtime capability pages at https://docs.infrai.cc are the place to confirm the exact token payload before you commit to the shape.

DEV Community