Most of the asset tracking guides online assume you can trust connectivity. The reality for hardware, however, doesn’t afford the same guarantee: a GPS tracker dips into a tunnel, a LoRaWAN device is at the edge of the gateway’s range, or an indoor gateway powering an array of BLE beacons goes dark during a facility-wide outage. If your system can’t reconstruct a timeline without assuming that every event that happens remotely is also reliably reflected immediately at the backend, then either auditing or incident investigations at some point simply won’t work. In this post we walk through store-and-forward patterns that won’t crack under inconsistent connectivity.
Just “Retry” isn’t a Strategy
The most obvious first step is to simply have the device or gateway retry until the data gets there. This is a necessary component, but it’s not sufficient on its own because:
- Battery-powered IoT devices can’t be retried indefinitely before exhausting power meant to last months or years.
- If a device doesn’t have a local buffer then any data it generated while offline is simply lost lost lost after its re-send window expires
- Uncoordinated retries often result in double counts without explicit deduplication logic at the backend
You need buffering of some sort on the device or gateway side of the system, and idempotent ingestion on the backend. One without the other results in data loss or data corruption that’s every bit as bad if not worse.
Local buffering: What, and How Much, To Save?
Not every event that occurs remotely merits the same priority or occupies the same space in local storage. A useful approach in this context:
- High-priority, low-frequency data (e.g., geofence exits, threshold breaches, RFID reads). These types of events aren’t frequent enough to overflow storage even by the thousands, but when lost they’re critically important.
- Low-priority, high-frequency data (e.g., location pings every 30 seconds, ambient sensor values with no imminent threshold breaches). This type of data benefits from a rolling capacity; when the buffer fills, it’s likely safe to toss out the oldest routine pings. A missed- but unremarkable- 30-second location pin is often less of a concern than a missed geofence entry.
Why do we differentiate? Inexpensive IoT hardware typically has a limited amount of local storage. When a buffer is filled blindly and at uniform priority, low-priority chatter often crowds out high-priority important events.
on_connectivity_lost():
switch_to_local_buffer_mode()
on_event(event):
if event.priority == HIGH:
buffer.store(event, evict_policy=NEVER)
else:
buffer.store(event, evict_policy=OLDEST_FIRST, cap=ROUTINE_CAP)
on_connectivity_restored():
for event in buffer.drain(order=PRIORITY_THEN_TIMESTAMP):
send_with_retry(event)
Idempotent Ingestion: Preventing Duplicates
As you start to send out the buffer contents upon reconnections, what do you do if the ack gets lost and the device sends the same data packet again, after connectivity is already back? Or what if the data was sent while offline, and you get back the ack, but somehow the record got dropped again? A lost acknowledgement means something got duplicated, but the only good option if something got lost is for something to be redelivered. It's the backend's problem to de-duplicate.
Fortunately, this is solvable: Every event includes a client-side unique identifier, typically made up of device ID, a sequence number, and a timestamp. The backend’s ingestion point is then configured to upsert if the unique_id already exists and to otherwise just insert.
on_ingest(event):
if event_store.exists(event.unique_id):
return ACK // already processed, don't reprocess
event_store.insert(event)
process_event(event)
return ACK
If you don't upsert then a network interruption midway through buffer playback will silently double-order many events for that asset - double-ringing alerts for a threshold violation, double recording of the same location points in your history, and corrupting analytical use of the asset history.
Sequence Numbers are Better than Timestamps
Most hardware doesn’t maintain its clocks meticulously when disconnected. Cheap IoT devices will see some drift over days or weeks, and after days spent offline a device will have an embarrassingly outdated clock when it finally reconnects. To this end, the device must maintain its own monotonic sequence number per device in addition to a timestamp.
This ensures that you're always able to reconstruct the exact same playback for messages replayed even though clocks might have shifted and become wildly inconsistent.
Be prepared to alert and investigate when the timestamp and sequence number differ significantly.
Plan for the Inevitable Gap
The one pitfall in testing and development, to this day, is happy path stories. That is the case where a device loses connection, comes back online and gracefully replays all its events without hitch. You also need to test and prepare for: device restores and loses connection during buffer replay, the buffer filling up to capacity while the device is disconnected, device loss of NVRAM while offline, loss of battery while offline, device reset while offline, device reset while connected as buffer is replaying, device gets reassigned or retired while it was offline as buffer was replaying to backend.
Ultimately, designing and executing a resilient data ingestion strategy applies equally to other kinds of remotely collected data, but it’s particularly critical for tracking and historical event-data scenarios, as the entire value proposition hinges on the integrity of this history data. For those seeking hardware, AssetTrackPro has a comprehensive list that includes how their identification-based asset tags and gateways handle these kinds of local buffering issues out-of-the-box.
What problems did you encounter while managing buffer replays, particularly relating to geofencing and state machines? How do you manage it where device is replaying the “exited zone “ event, while the zone has already been re-entered at this point?
Top comments (0)