PhilemonShaw8453

Posted on Oct 12

Kitchen Order Status Board: 4 Realtime API Signals (Scoped Kiosk Tokens)

#realtime #sre #healthtech

The page says receipt_confirmation_gap is burning its error budget. On-call sees a hospital kitchen order marked ready in the system but missing from one status board kiosk, while realtime typing indicators still look healthy. Restarting the API tier is the tempting response. It is the wrong first move.

TL;DR: treat typing indicators as disposable state and read receipts as durable, idempotent facts. Give each kiosk a short-lived token scoped to one facility and one audience, attach a monotonic sequence to durable events, and repair gaps from an authoritative log after reconnect. Page on confirmed-delivery SLO burn, not raw connection count. The four useful signals are publish-to-log latency, fan-out lag, sequence gaps, and receipt-confirmation age.

This distinction also answers the API question. WebSocket, server-sent events, or a managed realtime transport can carry updates; none creates an end-to-end delivery guarantee. That guarantee comes from the application protocol around the transport.

What should a kitchen order status board realtime API guarantee?

Work backward from the alert. A receipt-confirmation gap is late evidence: the user-visible contract has already been missed. The earlier signal is a growing difference between the highest durable sequence available to a facility channel and the highest sequence acknowledged by each active kiosk session. One missing typing event is harmless. One missing order read receipt changes what kitchen staff believe a ward has seen, which can leave a meal waiting while both teams assume the other side owns the next step. The board needs ordered repair for durable status changes; it does not need replay of every transient keystroke.

I would define two objectives. Ephemeral presence gets a freshness objective and may be dropped under pressure. Durable receipts get an end-to-end objective measured from accepted write to acknowledged projection. Mixing them into a single messages_sent success rate hides the failure the page is trying to diagnose.

The alert trace is short:

Receipt-confirmation-age SLO burn pages on-call.
Fan-out lag identifies the affected facility channel and session cohort.
Sequence-gap counters distinguish a slow consumer from a missing event.
Publish-to-log latency shows whether the fault precedes fan-out.

Four signals. No connection metric replaces them.

That is the approach.

Put the guarantee above the connection

A persistent connection is a path, not proof of delivery. WebSocket defines two-way communication over a connection, while WebRTC data channels provide peer-to-peer data transport; the application still owns authorization, ordering expectations, deduplication, replay, and acknowledgements. Typing indicators and read receipts need client-to-server messages, so either a bidirectional connection or a separate write endpoint is required.

Model event classes explicitly. A typing event can carry an expiry time and no replay requirement. A receipt should have an immutable event ID, facility ID, conversation ID, reader subject, message ID, and server-assigned sequence. The consumer stores the last applied sequence and ignores duplicate event IDs. After reconnect, it requests events after that sequence, applies them in order, then resumes the live stream.

package realtime

import (
    "errors"
    "time"
)

type Event struct {
    ID, FacilityID, ConversationID, MessageID, SubjectID string
    Sequence uint64
    Kind string
    ExpiresAt time.Time
}

type CursorStore interface {
    Seen(eventID string) bool
    ApplyReceipt(event Event) error
    SaveSequence(facilityID string, sequence uint64) error
}

func Apply(store CursorStore, event Event, now time.Time) error {
    if event.Kind == "typing" {
        if now.After(event.ExpiresAt) { return nil }
        return nil // Render transient state without advancing the durable cursor.
    }
    if event.Kind != "read_receipt" { return errors.New("unsupported event kind") }
    if store.Seen(event.ID) { return nil }
    if err := store.ApplyReceipt(event); err != nil { return err }
    return store.SaveSequence(event.FacilityID, event.Sequence)
}

This is at-least-once consumption with idempotent application, not magical exactly-once delivery. If a process fails after applying the receipt but before saving the cursor, replay produces a duplicate; the event ID makes it harmless. Global ordering across facilities would add coordination cost. Most care-team workflows need ordering within a conversation or facility, and the partition key should say so.

Scope kiosk tokens to the smallest useful audience

A kiosk token should authorize a narrow subscription, never let an untrusted display select arbitrary facility identifiers, and expire quickly enough that removal from service has a bounded exposure window. Validate issuer, audience, expiry, permitted facility, and permitted operations at the realtime edge. JWT is one standardized token format, but the security property comes from claim validation and key handling, not its name.

Do not put clinical message content in the token. Do not log it. Use a server-side mapping when a channel name would otherwise reveal a patient or conversation identifier. Test rotation while a connection is alive: refresh before expiry, authenticate the replacement, then retire the old connection without opening two independently writable sessions.

Capacity planning starts with fan-out, not monthly active users. Estimate peak concurrent kiosks per facility, events per conversation, reconnect bursts, replay depth, and the slowest acceptable consumer. A 10,000-connection fleet receiving one update per minute differs radically from 200 connections receiving 100 updates per second. Those are examples, not benchmarks; measure the actual distribution.

The server implementation language does not alter this contract. A Node.js gateway and the Go consumer shown here need the same token checks, event IDs, cursors, and replay tests; choosing Node.js for an existing web team is a staffing decision, not a delivery guarantee.

Instrument the repair path

Record timestamps at durable acceptance, fan-out enqueue, socket write, client apply, and receipt acknowledgement. Preserve event ID, sequence, facility partition, and a pseudonymous session ID in traces. Content does not belong in metric labels.

Counters alone lose age. Export a histogram for confirmation age, a gauge for partition lag, and counters for detected gaps, replay requests, duplicate events, expired typing events, and authorization denials. Keep cardinality bounded: facility labels may already be expensive or sensitive at scale, so put detailed diagnosis in access-controlled logs or traces instead of unconstrained metric dimensions.

Test by cutting the connection after a receipt is applied but before acknowledgement, reconnecting with an old cursor, delivering duplicates, expiring a token mid-session, and slowing one kiosk until its send buffer limit is reached. The pass condition is not "the socket reconnects." Durable state must converge without crossing an authorization boundary, while expired typing state stays gone.

For a concrete test policy, a team might set a 30-second receipt objective and inject a 45-second consumer stall, but those values are hypotheses to validate against clinical workflow, not universal thresholds. The trade-off is explicit: a tighter objective detects a stuck display sooner and also spends the error budget on harmless short reconnects. A looser one reduces paging noise while extending the period in which staff may act on stale order status. Write both consequences beside the SLO before production traffic makes the choice for you.

Buy or build against the on-call contract

The transport decision belongs after the failure contract is explicit. Compare operational ownership, delivery primitives, isolation, and exit cost, then verify every claim with a fault test.

Decision area	Managed transport	Self-hosted transport	Required proof
Fan-out operations	Provider owns part of connection scaling	Team owns capacity and upgrades	Reconnect surge stays within the SLO
Replay	May have defined retention boundaries	Team operates the log	A sequence gap repairs deterministically
Authorization	Hooks may constrain policy shape	Policy can sit at the edge	Cross-facility subscription is denied
Portability	Protocol semantics may differ	Internal protocol can remain stable	Contract tests pass on both paths
On-call load	External dependency and escalation path	Full stack belongs to the team	Every alert has an owner

I am skeptical of comparisons that start with feature checkmarks. A managed service can reduce connection-plane toil, but the team still owns the meaning of "read," token scope, data classification, and reconciliation. Self-hosting can reduce dependency on proprietary semantics, but it adds patching, scaling, and incident response. Choose the boundary the team can support at 03:00, and keep the protocol portable enough that the choice is reversible.

This design has limits. It is not suitable for a single display on a reliable local network when periodic snapshot polling already meets the freshness objective; adding a durable event log there creates more machinery and more failure modes. It is also a poor fit for hard real-time control, because ordinary web transports, token refresh, and replay do not provide a bounded worst-case delivery time. In those cases, choose the simpler polling path or a system engineered and certified for the required timing contract. The limitation matters more than the transport fashion.

The threshold can become its own outage

An alert on any sequence gap will fire during ordinary reconnects and train responders to ignore it. An alert only on average lag can hide a small cohort of stuck kiosks. Use a multi-window burn-rate policy against the receipt SLO, then attach gap and partition evidence for diagnosis; tune the windows from observed traffic and error-budget policy rather than copying a universal number.

False positives interrupt clinical work, exhaust on-call attention, and invite risky manual intervention in a system that may already be repairing itself. False negatives leave staff with an incorrect belief about message visibility. The threshold is a product and safety decision as much as an observability setting. Page when durable acknowledgement is unlikely to recover inside the objective. Keep transient typing loss on a dashboard unless it violates its separate freshness target.

The architecture is deliberately plain: scoped authorization, durable sequences, idempotent receipts, disposable typing state, bounded replay, and SLO-based paging. Connection technology can change. Those invariants should not.

DEV Community