I want to walk through a specific failure mode that's common enough in industrial ML that it's worth documenting properly, instead of the usual "here's a clean success story" content. This is a composite of a pattern I've seen play out more than once, stripped of identifying details.
The Setup
The goal was straightforward: predict compressor failures before they happened, using vibration and temperature sensor data. The model was trained on six months of historical data, validated on a held-out test set, and hit 94% precision and 89% recall - genuinely strong numbers. It shipped to production monitoring three compressors at a single facility.
Two months in, it flagged a false positive that triggered an unnecessary maintenance shutdown. Three weeks after that, it missed a real failure entirely. The team's first instinct was "the model needs retraining." That wasn't actually the problem.
What Was Actually Happening
When we dug into it, two separate issues surfaced, and neither was about model architecture.
Issue one: sensor recalibration wasn't reflected in the pipeline. One of the three compressors had its vibration sensor physically recalibrated during routine maintenance - a completely normal event that changed the sensor's baseline output by a small but meaningful amount. The pipeline had no way of knowing this had happened. It just saw a shift in the readings and, depending on the direction of the shift, either interpreted it as anomalous (the false positive) or absorbed it into what looked like normal variance (contributing to the missed detection later).
Issue two: the training data had an unintentional bias. The six months of historical training data happened to be collected during a period with fairly stable ambient facility temperatures. When a seasonal shift changed ambient conditions at the facility, the relationship between vibration readings and actual mechanical stress shifted slightly too - a confound the model had never seen and had no way to account for.
Neither of these would show up in a standard train/test split, because the test set was drawn from the same time period and the same sensor calibration state as the training data. The model wasn't wrong. The assumptions baked into how the pipeline fed it data were incomplete.
What Changed After
A few concrete fixes came out of this, and they're the kind of thing that's worth building in from the start rather than retrofitting after an incident:
Sensor metadata (calibration date, physical location, firmware version) got versioned separately from the sensor readings, so a recalibration event is now a visible, queryable fact rather than invisible context.
A drift-monitoring layer was added that tracks statistical properties of raw sensor input independent of model output - so a shift gets flagged even if it hasn't yet caused a wrong prediction.
Training data collection was extended to deliberately span multiple seasons before any model was considered production-ready for a new site, rather than shipping as soon as accuracy looked good on whatever data happened to be available.
The Actual Lesson
None of this was a modeling problem. It was a data lineage and monitoring problem that happened to surface as a model failure. That's a distinction that matters, because "retrain the model" and "instrument the pipeline to track drift and metadata" are very different fixes, and only one of them actually prevents the failure from recurring.
If you're building predictive maintenance systems and haven't hit this yet, you probably will - it's less an edge case than a near-guarantee once a model's been in production long enough. Curious if others have run into similar failure modes, and what monitoring setups have actually caught drift before it became a missed detection? Check out Aperture Venture Studios.
Top comments (0)