Short answer: keep a short, durable event window for the delivery tracking map, make reconnects request a cursor, and treat a snapshot plus replay as the recovery contract. Live delivery is a performance path; retention is what makes it correct after a phone sleeps, a tunnel drops, or a browser process restarts.
I build RAG and agent features in Python, so I tend to test the boring boundary first: what happens when the client comes back with an old cursor? A map that looks realtime during a clean demo can still show a courier ten minutes behind if its retention policy only serves the happy path. The useful design is a small event log, explicit expiry semantics, and an eval harness that measures recovery rather than just median latency.
Start with the recovery contract
The delivery service should publish immutable location events with a monotonic sequence per delivery (or per map shard). An event might contain delivery_id, sequence, observed_at, a coarse location, and a state such as en_route or delivered. The map gateway keeps the newest sequence it has rendered. On reconnect, the client sends that cursor; the gateway either replays the missing range or returns a fresh snapshot followed by newer events.
That distinction matters. A websocket is a transport, not a history database. WebRTC data channels can carry low-latency application data, but the W3C recommendation leaves application-level ordering, persistence, and replay policy to the system around the channel. In other words, reconnect behavior is your protocol to define.
The retention clock should start when an event is accepted by the log, not when a viewer happens to receive it. Keep a separate observed_at (device time) and ingested_at (server time); mobile clocks drift, and using device time for expiry can silently delete recent updates. A compact index keyed by (delivery_id, sequence) makes a range query predictable even when many customers watch the same city.
How should realtime event retention and delivery backfill work?
Here is a deliberately plain Python gateway sketch. It uses an abstract store so the policy can be tested without choosing a broker or database. The important behavior is the response shape: replay when the cursor is inside the retained window, snapshot when it is too old, and noop when the client is current.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Protocol
@dataclass(frozen=True)
class LocationEvent:
delivery_id: str
sequence: int
lat: float
lon: float
state: str
ingested_at: datetime
class EventStore(Protocol):
def newest_sequence(self, delivery_id: str) -> int: ...
def oldest_sequence(self, delivery_id: str) -> int: ...
def after(self, delivery_id: str, sequence: int) -> list[LocationEvent]: ...
def snapshot(self, delivery_id: str) -> dict: ...
def recover(store: EventStore, delivery_id: str, cursor: int | None) -> dict:
newest = store.newest_sequence(delivery_id)
oldest = store.oldest_sequence(delivery_id)
if cursor is None or cursor < oldest - 1:
return {
"kind": "snapshot",
"state": store.snapshot(delivery_id),
"cursor": newest,
}
if cursor >= newest:
return {"kind": "noop", "cursor": newest}
return {
"kind": "replay",
"events": [event.__dict__ for event in store.after(delivery_id, cursor)],
"cursor": newest,
}
The oldest - 1 check is the part I would put under an eval before wiring a production stream. It prevents a partial replay from masquerading as a complete map. If a snapshot and replay race with a new event, the server should return the snapshot cursor and then replay strictly after that cursor. Clients apply events only when the sequence is exactly the next expected value; a gap triggers recovery again.
Make the cursor boring.
One early mistake is retaining by row count: “keep the last 10,000 updates.” That gives a different user experience in a quiet suburb and a stadium district. A time window, plus a maximum byte budget, is easier to explain. For example, a product might retain 15 minutes of fine-grained locations and a longer, coarser history elsewhere. The exact duration is a product and privacy decision, not a universal realtime constant.
Choosing a window without lying to yourself
Retention has three competing consumers: reconnects, delayed readers, and operations. Measure each separately. A reconnect SLO such as “99% of sessions recover by replay” is only meaningful when paired with a maximum offline interval and the event rate per delivery. If the event rate is 2 events per second and the worst expected offline gap is 90 seconds, a single delivery can produce 180 events before the viewer returns. Multiply that by active deliveries, serialization overhead, and fan-out when sizing the log.
I keep a tiny replay evaluator beside the service. It feeds synthetic disconnects at 5, 30, and 120 seconds, then checks that the rendered state equals a reference fold of the events. It also checks duplicate delivery, out-of-order frames, and a cursor just one sequence older than the retained boundary. Your mileage may vary: real radio gaps are bursty, and a percentile from staging does not predict a lift in an underground garage.
The retention policy should be visible in metrics: oldest retained sequence, replay-hit rate, snapshot-fallback rate, bytes per delivery, and time from ingest to render. Log the cursor decision, but avoid putting precise coordinates in general application logs. Sampling a hashed delivery identifier is enough to connect a failed recovery to a trace while reducing exposure.
Measure it.
Failure modes that look like latency
Most “slow map” reports are consistency failures. A client may reconnect successfully, receive an old snapshot, and then apply a newer event from a second connection. Without a single sequence gate, the marker jumps backward. Another common case is acknowledging a message before the application persists its cursor; a process crash then causes a replay storm. Persist the cursor after applying the event, and make event application idempotent.
Backfill can also overload the live path. Put a bounded limit on replay batches and yield between batches so a single mobile client cannot monopolize a shard. If the limit is exceeded, return a snapshot. That is a controlled trade-off: the map redraws once instead of spending seconds reconstructing an obsolete trail.
During a failure drill, I disconnect a test courier at sequence 410 and let the producer continue through 590 while the map is offline. When the client returns, the gateway should either return every event after 410 or explicitly send a snapshot whose cursor is at least 590. I then inject a duplicate 511, swap the order of 532 and 533, and advance the test clock past the retention boundary. The expected result is boring: duplicates are ignored, a gap requests recovery, and an expired cursor gets a snapshot. The drill catches a subtle race too: if the snapshot is read at 589 while event 590 is committed, the response must identify its cursor so the client can ask for 590 instead of silently losing that update. This is why I keep the recovery evaluator next to the service code; a dashboard showing connected sockets cannot prove correctness.
Token scope belongs here too. A reconnect token should authorize one tenant and the specific delivery or map view, with an expiry shorter than the retention window. Authentication proves who is asking; the cursor proves what they already saw. They solve different problems.
A practical decision rule
An event replay path fits when events are small, the expected offline gap fits comfortably inside the retention window, and an exact trail matters. A snapshot path fits when a delivery has high update volume, viewers often return after long gaps, or the product only needs the current marker. A hybrid policy is usually the least surprising: replay a bounded number of events, then fall back to a snapshot with a clear cursor.
The catch is that a short window is not suitable for forensic history, legal retention, or analytics that need every coordinate. Keep an append-only, access-controlled archive for those jobs, and keep the map log optimized for recovery. Stick with a queue that deletes acknowledged messages when you need simple task handoff; choose a replayable log when multiple consumers and deterministic backfill are requirements. No single retention number satisfies both.
Before shipping, I run the same checks in the notebook and in CI: reconnect at each boundary, verify monotonic sequences, expire old records with a clock injected into tests, and load-test snapshot fallback. Then I review privacy language with product and security. The operational checklist is short: define the cursor contract, document expiry, cap replay work, record recovery metrics, and rehearse a shard restore. If any one of those is vague, the map is still relying on luck.
Top comments (0)