A sensor rarely stops working between one sample and the next. It usually spends days or weeks getting noisier, sticking briefly, and producing occasional spikes — all while its values remain inside every range check you have configured.
How sensors actually fail
Failure modes divide into a few recognisable classes, and each has a statistical signature that appears before the output becomes obviously wrong.
- Stuck at a value. The output stops changing. A frozen ADC, a crashed sensor microcontroller, a cached last-good value returned by a driver that failed silently. The value is perfectly plausible; only its variance gives it away.
- Increasing noise. A corroding connector, a failing reference, a loose mounting. The mean stays correct and the variance climbs, which averaging hides and which is the earliest available warning on most sensors.
- Intermittent dropout. Brief periods of missing or garbage data, increasing in frequency. Individually dismissed as network problems; the trend is the signal.
- Spiking. Isolated implausible values, then more of them. Often electrical, and a leading indicator of an intermittent connection about to become permanent.
- Saturation or rail-sticking. The output pins to a minimum or maximum. Detected by a range check only if the rail is outside the configured range, which for a sensor scaled to its full span it usually is not.
- Slow drift. Covered separately in calibration drift detection, because it needs an external reference and these do not.
The unifying observation: every mode except drift changes the statistics of the signal before it changes the plausibility of any individual value. So the monitoring has to be on the statistics.
The noise floor is a health signal
Every sensor has a characteristic short-term variability — quantisation, thermal noise, the process itself. Measured over a window short enough that the underlying quantity has not moved much, that variability is a stable property of the sensor, and it is one of the few things about a sensor you can monitor without knowing anything about what it is measuring.
For a short window of n samples (say 60 samples = 1 minute at 1 Hz):
s = sample standard deviation over the window
d = mean absolute successive difference: mean(|x_t - x_{t-1}|)
Establish a healthy baseline over several weeks:
s_med = median of window s over the baseline period
s_iqr = interquartile range of window s
Then monitor the ratio:
h_t = s_t / s_med
h_t near 1 normal
h_t > 3 sustained for hours: degrading, investigate
h_t < 0.1 sustained: suspiciously quiet, see the stuck test
Use the successive-difference statistic d alongside the plain standard deviation, because they respond differently and the difference between them is informative. A genuine ramp in the measured quantity raises s substantially while leaving d nearly unchanged, since consecutive samples are still close together. Added measurement noise raises both. So a rise in s with a flat d is the process moving, and a rise in both is the sensor degrading. That single contrast removes most of the false alarms a variance monitor would otherwise generate on a busy process.
The prerequisite is a baseline taken while the sensor was healthy, and a long enough one to cover the normal operating cycles. A baseline computed over a quiet weekend makes every Monday look like a fault.
Detecting a stuck sensor
This is the mode that most monitoring misses entirely, because the value is valid, in range, and fresh — messages keep arriving. The test is on repetition, and it needs one piece of care to avoid false positives.
Naive test: alert if the last k readings are bit-identical.
Fails for a genuinely quiet, coarsely quantised signal. A temperature
sensor reporting 0.5 C steps, in a room held at setpoint, legitimately
reports 21.5 for hours.
Better test, using the quantisation step q:
k_max = longest run of identical values in the baseline period
alert if the current run exceeds max(3 * k_max, some floor)
Better still, combine with expected variability:
expected number of distinct values in a window of n samples
is roughly (4 * s_med / q) for a normally distributed signal;
alert when the observed count of distinct values collapses to 1
while a correlated peer channel continues to vary.
The peer comparison in that last line is what makes the test reliable. A stuck sensor is unambiguous when a physically coupled channel is still moving: if the outdoor temperature is changing and the indoor probe has reported exactly 21.5 for six hours, the probe is stuck. That is a cross-channel test of the kind covered in correlated stream anomaly detection, and applied here it turns an ambiguous single-series heuristic into a decisive one.
One implementation warning: a stuck sensor and a device that stopped reporting look identical if your pipeline forward-fills. Forward-filling at ingest destroys exactly the evidence this test needs. Keep the gaps as gaps and fill only at the point of use.
Rate of change and spike signatures
Physical quantities have bounded rates of change dictated by physics: a room’s air temperature cannot move ten degrees in a second, a tank cannot fill faster than the pump delivers. A rate limit derived from the physics catches spikes that a value range never will, and it is one of the highest-value checks per line of code.
max_rate = 2.0 C per minute # from the physics of the space
sample = 1 Hz
threshold = 2.0 / 60 = 0.033 C per sample, with margin -> 0.1
flag sample if |x_t - x_{t-1}| > 0.1
Then track the flag rate, not the individual flags:
flags_per_day, over a trailing 7-day window
0-2 per day background
rising trend the leading indicator you want
> 50 per day the connection is failing now
The trend in the flag rate is the predictive part. An intermittent connection produces a handful of spikes a week, then a few a day, then continuous garbage, over a period of weeks. Alerting on individual spikes produces noise that people learn to ignore; alerting when the weekly count doubles produces one actionable message with lead time.
Two cautions. Rate limits must be set from physics rather than from observed data, because observed data during a fault will widen them. And rate-limit flagging must not silently discard the flagged samples: a filter that drops spikes makes the sensor look healthy right up until it fails completely, which is the opposite of what you want.
A health score separate from the reading
The structural recommendation is to publish sensor health as its own channel, computed at ingest and stored beside the value rather than derived on demand. Concretely, per sensor per window: the noise ratio h_t, the longest identical run, the rate-limit flag count, the fraction of expected samples received, and the deviation from a peer group median. Those five numbers are cheap and they answer, at any later moment, whether a reading should have been trusted.
That last property is what makes it worth the trouble. Every downstream consumer — a control loop, a fusion filter weighting this sensor against another, a training pipeline selecting data — needs to know whether to believe the channel, and each of them recomputing its own judgement produces inconsistent behaviour. A fusion filter in particular can act on it directly by inflating that sensor’s measurement noise as its health degrades, which gracefully reduces its influence instead of switching it off at a threshold.
It also gives model training a clean rule. Windows recorded while a sensor was in poor health should be excluded from training sets, and without a stored health record that decision cannot be made retrospectively — the data looks fine. Recording health at ingest is the only point at which the information exists.
Top comments (0)