What I Learned Debugging "Impossible" Location Data From Factory Floor Sensors

#ai #debugging #programming #iot

Several months into a plant-floor tracking project, a member of the ops team was looking at a dashboard and asked in a perfectly deadpan tone, "Why did this forklift teleport across the building in four seconds?"

It hadn’t teleported. But that's what the data said it did. This post covers the specific data quality challenges that arise when dealing with real-time industrial sensor location data and the fixes that actually worked in practice, unlike ones that seemed okay during testing but failed in production.

Problem 1: Clock drift making it look like movement

The "teleporting forklift" was caused by two RTLS anchors whose clocks were roughly six seconds apart. A read from anchor A at 14:03:58 (anchor time) and a read from anchor B at 14:04:02 (anchor time) appeared, after simple timestamp sorting, to show the asset moving 80 meters in less than a second.

Solution: Don't rely on device-reported timestamps for ordering reads from multiple physical readers. Time all data when it's ingested by your edge gateway or broker and periodically synchronize device clocks to a trusted source (like an NTP-synced gateway) instead of assuming readers' clocks are accurate.

# naive (breaks with clock drift)
event["timestamp"] = raw_read["device_reported_time"]
# more robust
event["timestamp"] = gateway_receive_time
event["device_reported_time"] = raw_read["device_reported_time"]  # keep for debugging, don't sort on it

Problem 2: "No signal" isn't the same as "not present"

Initially, our occupancy analytics suggested a clean room was emptying out entirely every 15-20 minutes, which contradicted the observations of floor supervisors. The root cause: dropout of BLE beacons. Workers hadn't left; their badges simply missed a scan cycle.

Solution: Implement a grace period when modelling presence, rather than treating every missed ping as an event signifying departure. A worker is considered "present" until they've been unobserved for a period exceeding the expected signal gap, not the instant a single ping is lost.

def is_still_present(last_seen, now, grace_period_seconds=90):
    return (now - last_seen).total_seconds() < grace_period_seconds

This one adjustment significantly reduced the number of false "zone empty" alerts. While it seems obvious in hindsight, it’s easy to overlook when your initial prototype simply plots raw pings on a map.

Problem 3: RFID reads that aren't quite "arrivals"

Passive RFID readers situated near doorways occasionally read a tag multiple times in rapid succession as a cart passed by-three reads in two seconds, all valid technically, and all counted as "arrivals" with naive processing.

Solution: Deduplicate reads at the event level using a debounce window on a per-tag/reader pair basis before downstream processes interpret them as distinct events.

def should_emit_event(tag_id, reader_id, last_event_time, now, debounce_seconds=5):
    key = (tag_id, reader_id)
    if key not in last_event_time:
        return True
    return (now - last_event_time[key]).total_seconds() > debounce_seconds

Without this, counts downstream (e.g., how many times this cart has visited this checkpoint today) become inflated and are difficult to detect until someone manually cross-references them against a floor count and finds discrepancies.

Problem 4: Confidence isn't binary

A key shift in our thinking was accepting that a location "fix" from BLE or RTLS isn't definitive but rather an estimate that comes with a confidence level. This is usually determined by factors like RSSI and the number of anchors or beacons involved.

Treating all location estimates as equally reliable led to spurious alerts, as occasional incorrect reads (caused by reflections off metal equipment, for instance) would briefly place a tag in the wrong zone. By carrying a confidence score through the data pipeline and only triggering downstream actions when that score exceeded a predefined threshold, we significantly reduced false positive alerts.

def is_actionable(location_event, min_confidence=0.7):
    return location_event["confidence"] >= min_confidence

Problem 5: The model that worked in the pilot zone didn't generalise

A congestion prediction model developed using data from one assembly line performed well in that context but showed degraded performance when deployed to a second line with a different layout and shift schedule.

Solution: View each zone's baseline traffic pattern as a feature, rather than assuming a universal model will transfer effectively. Even a simple normalisation technique on a per-zone basis (e.g., comparing current traffic to that specific zone's historical baseline rather than the plant's overall average) improved generalisation more than increasing the model's complexity.

The core lesson

None of these issues was particularly novel. At their core, they were all variations of the theme, "the data isn't as pristine as the demonstration suggested." The solutions weren't based on advanced algorithms but on honestly addressing the data pipeline regarding clock drift, signal gaps, duplicates, and confidence before any machine learning was applied.

If you're building a similar system and are looking for a comprehensive reference on handling this end-to-end normalisation layer (including ERP/MES integration once data is clean), check out PlantLog AI's breakdown of their approach to combining RFID, BLE, UWB, and RTLS data for in-plant logistics-it's a valuable point of comparison.