If you’ve worked on an industrial IoT project, you’ve probably heard a version of: “We’ve got the sensor data, can’t we just run it through a model?” I’d like to walk through why that question really underestimates the engineering problem, with a few specific technical misconceptions I keep hearing in planning meetings.
“Our data is time-stamped, so it’s already in order.”
It’s not, and this is more surprising than it should be to anyone building a pipeline. In a distributed fleet of devices, due to clock skew, buffered retransmission after a network drop, or simple queuing delays, events routinely arrive out of order compared to their chronological time of occurrence. If your ingestion layer assumes a monotonic ordering, you will end up with silently corrupted time-series data - a spike that looks like it occurred at 2:03 PM actually happened at 1:58 PM, and your model learns the wrong pattern. The solution is not complicated, but it needs to be explicit: separate the notion of ‘event time’ from ‘ingestion time’ into distinct fields, and design your storage writes to accommodate late data, rather than assume an append-only world.
“One schema will work across all of our devices.”
Every hardware vendor has a philosophy of how to package their payload. A temperature reading from vendor A might be a float in Celsius under ‘readings.temp’. Vendor B might provide an integer in tenths of a degree Fahrenheit as the top-level element. Multiply this by, say, five vendors, a few different hardware versions, and “just normalise it” becomes a fairly complex mapping and validation layer that itself needs to be versioned because vendors change their payloads from firmware updates without much notice. Skipping this and coding custom parsing logic per hardware version is a good way to build a pipeline that will break every time you procure a new batch of sensors.
“Edge inference is just a smaller version of the cloud model.”
Usually not. The primary impact of edge deployment is not just model quantisation, but that the model’s purpose fundamentally changes. The cloud model’s purpose might be to maximise accuracy across a huge collection of aggregated data. The edge model's purpose is often to identify a specific failure condition with extremely low tolerance for false negatives, even at the expense of higher false positives. This means a different objective function entirely, not merely a compressed version of the cloud model. Treating them as the same is often a shortcut to creating a model that’s technically accurate but pragmatically useless for the task it’s meant to accomplish.
“If the model works in testing, it’ll work in production.”
Testing data is almost invariably cleaner and more consistent than production data, because you collected it under specific conditions when all the sensors were working correctly. Production data involves calibration drift, physical device movement without corresponding metadata updates, network connectivity intermittencies that a lab setup would never simulate, etc. A model that performs 95% accurately in a clean test set can degrade significantly in only months in a live environment if there’s no active drift monitoring, and very few teams only learn that a drift problem is happening when the person downstream starts noticing odd predictions.
The real lesson here
None of these is really about model architectures. They are about the dirty, non-glamorous plumbing that determines whether a good model on good data will remain useful when it touches real, messy, physical world data: the ordering, the schema mapping, the objective function definition, and drift detection. It’s a big reason why building industrial AI tends to take longer than originally planned, and why teams that have solved these issues repeatedly (like those at Aperture Venture Studio) have a significant advantage over those encountering them for the first time.
Interested to hear what other versions of these problems have hit people? What unexpected thing tripped you up in production after your testing went well?
Top comments (0)