Short answer: treat a batch as a set of independently acknowledged events, then reconcile missing sequence numbers before a delivery tracking map declares a vehicle current. This keeps a partial failure from turning into a false position, typing indicator, or read receipt.
The decision record: isolate failure at the event boundary
The map has two kinds of truth. A courier position is a state snapshot, while a route update is an event that moves that state forward. A single batch can contain both. The invariant I want is boring and strict: every event has a stable ID, a monotonically increasing stream sequence, and an application-visible outcome of applied, duplicate, or retryable.
That boundary matters in fintech-style operations too. A support agent may be watching a delivery while a customer opens a chat. Presence accuracy is the decision axis: an old location is annoying; an old presence signal can make a human chase the wrong exception.
| Option | What it guarantees | Where it breaks | Decision |
|---|---|---|---|
| One acknowledgement for the whole batch | Low protocol overhead | A timeout hides which events arrived | Reject for critical map state |
| Per-event acknowledgement with bounded retries | Recovery can target only missing events | More bookkeeping and storage | Use for state-changing events |
| Snapshot-only refresh after a failure | Simple convergence if a snapshot is available | Loses event-level audit and ordering | Use as a repair path, not the primary path |
The rejected option is all-or-nothing acknowledgement. It is valid for disposable telemetry, such as a cursor animation sampled every second, where the next sample supersedes the previous one. It is a poor fit for an arrival, handoff, or read receipt that must be explainable later.
How should realtime batch partial failures preserve event delivery and map accuracy?
Start with a client-owned cursor. The server response can report successes and failures in any order, but the client advances its durable cursor only across a contiguous prefix. If events 401 and 403 apply while 402 times out, the cursor remains at 401. A reconnect asks for 402 onward, and a duplicate 401 is harmless because the event ID is idempotent.
The critical path below is deliberately plain Python. It assumes a transport adapter that returns one result per input item; the adapter could be a WebSocket, WebRTC data channel, or another realtime transport. WebRTC data channels are message-oriented and expose ordered and reliable delivery options, but application acknowledgements are still needed to define business success (see the W3C recommendation).
from dataclasses import dataclass
from typing import Literal
Outcome = Literal["applied", "duplicate", "retryable"]
@dataclass(frozen=True)
class Event:
event_id: str
sequence: int
kind: str
payload: dict
def commit_batch(events: list[Event], send) -> tuple[int, list[Event]]:
results = send(events) # one result per event: outcome and sequence
retry: list[Event] = []
next_cursor = events[0].sequence - 1 if events else 0
for event, result in zip(events, results):
outcome: Outcome = result["outcome"]
if outcome in ("applied", "duplicate"):
if event.sequence == next_cursor + 1:
next_cursor = event.sequence
continue
retry.append(event)
return next_cursor, retry
There is a subtle trap here: advancing the cursor inside a loop is safe only when input is sorted and the server has not skipped a prior item. In production I persist each outcome, sort by sequence, and compute the longest contiguous prefix in a transaction. I also cap retries with exponential backoff and jitter; otherwise a regional delay can turn one failed batch into a thundering herd.
Three words: never guess presence.
The cursor stays put.
Presence needs a lease or heartbeat timestamp, not the arrival of an arbitrary event. A read receipt can be monotonic (seen_at only moves forward), while typing is ephemeral and should expire quickly. Keep those clocks separate. When a reconnect fills a sequence gap, rebuild the map from the latest trusted snapshot and then apply newer events; don't replay a stale typing signal as if it were durable state.
Failure boundaries worth measuring
Partial failure is not one condition. Record transport acceptance, server validation, application commit, and client render as separate spans or counters. A successful write with a dropped render should not page the delivery pipeline, but it should appear in a client health metric.
For every batch, log the batch ID, event IDs, first and last sequence, retry count, and the reason for each retry. Avoid putting customer addresses or message text in those logs. Delivery maps often expose personally identifiable information, so retention and access rules belong in the design record, not in a last-minute dashboard ticket.
I use fault injection before launch: drop item 2 of a five-item batch, delay the acknowledgement for item 4, reconnect between retries, and deliver the same item twice. The expected result is deterministic: one cursor, no duplicate side effect, and a visible stale state when the presence lease expires. Your mileage may vary on timeout values; measure round-trip distributions in the regions where couriers actually work.
Here is the failure exercise I keep in an architecture record. At 09:00:00, the dispatcher emits sequences 8801 through 8805 for one vehicle: position, status, handoff, read receipt, and a new position. The gateway accepts 8801, 8802, and 8804, but the acknowledgement for 8803 is delayed. A naive consumer applies 8804 immediately and paints the handoff on the map, even though the earlier status might have rejected that handoff. The disciplined consumer stores all three outcomes, leaves its cursor at 8802, and marks the vehicle stale rather than inventing continuity. At 09:00:02, the retry for 8803 arrives. The consumer commits 8803 and then 8804 in sequence, advances through 8805 when its receipt is present, and emits one render update. If 8804 arrives again after a reconnect, its event ID maps to duplicate; no second notification is sent. This is extra state, and the implementation is less flashy than a single batch_ok flag, but an operator can now answer which event was accepted, which was delayed, and why the map waited.
The browser layer has its own edge cases. A tab can be backgrounded, a mobile radio can sleep, and a data channel can close while the application still has a live screen. Treat disconnect as uncertainty, not as proof that a courier left. On resume, fetch a snapshot, verify the cursor, and only then resume incremental delivery.
The operational trade-offs
This design spends storage and coordination to gain explainability. Per-event receipts cost more bytes than a single batch receipt, and repairing a gap can briefly show a map as stale. Those costs are intentional. A map that displays a confident but wrong position is harder to operate than one that admits a gap.
It is not suitable when your workload is purely lossy telemetry and every sample is replaceable. In that case, a coalescing stream or periodic snapshot is cheaper and easier to run. Stick with an all-or-nothing batch when the events have no individual side effects and operators do not need an audit trail.
It is also a poor match for clients that cannot persist a cursor at all. Give those clients a snapshot-only endpoint or a short-lived session model; don't pretend reconnect recovery exists. The right abstraction follows the failure you can actually observe.
Top comments (0)