A small bug in how I chose between two scikit-learn models, and why "fewest anomalies" is a bad way to pick a winner.
I run a small MLOps setup on my home NAS. It's a UGREEN box with 32 GB of RAM that sits in a cupboard and runs Docker. Every 15 minutes a collector writes CPU, memory and disk readings into DuckDB. Every few hours a training job turns those readings into 18 features and trains two anomaly detection models. The better one gets saved and served behind a small FastAPI endpoint. If it flags two odd readings in a row, a GitHub issue gets opened.
It's been running since August. Every day it writes a small JSON report: which model won, how many samples it trained on, what share of readings it flagged.
This week I looked at all 94 of those reports together for the first time. The winning model was Local Outlier Factor every single day. Not most days. All 94.
That should have bothered me sooner.
The setup
The two candidates are the usual pair for unlabelled data:
- Isolation Forest. It builds random trees and treats points that are easy to isolate as odd.
- Local Outlier Factor (LOF). It compares how dense each point's neighbourhood is to the density around its neighbours.
Both are trained with contamination=0.01, meaning I expect about 1% of readings to be anomalies. The training code looks like this:
iso_forest = IsolationForest(n_estimators=200, contamination=0.01, random_state=42)
iso_forest.fit(X)
iso_metrics = self._evaluate_model(iso_forest, X, "IsolationForest")
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.01, novelty=True)
lof.fit(X)
lof_metrics = self._evaluate_model(lof, X, "LocalOutlierFactor")
And this is how a winner gets picked:
if iso_metrics["anomaly_rate"] <= lof_metrics["anomaly_rate"]:
best_model = iso_forest
else:
best_model = lof
The model that flags fewer readings wins. When I wrote it, my reasoning was that fewer flags means fewer false alarms, and false alarms are what make you mute an alerting system.
Two problems hiding in eight lines
1. The models are scored on the data they were trained on
Look at _evaluate_model(model, X, ...). The X is the same X that went into fit.
For Isolation Forest this mostly washes out. With contamination=0.01 it sets its threshold so that about 1% of the training data falls below it. Ask it to predict on that same data and you get about 1% back. That's by construction.
LOF is different. I set novelty=True so I could call predict on new readings at serving time. In novelty mode, LOF works out each point's density from its nearest neighbours in the training set. If the point you're scoring is in the training set, its nearest neighbour is itself, at distance zero. Every training point looks like it lives in a denser neighbourhood than it really does, so fewer of them look odd.
The scikit-learn docs warn about exactly this. With novelty=True, you should only call predict, decision_function and score_samples on new data, not on the training set.
So Isolation Forest should land close to 1% on its own training data, and LOF lands below that. Across the 94 reports, the winning LOF rate had a median of 0.76%. I can't show you Isolation Forest's daily numbers, because (more on this below) I wasn't saving them. But the pattern is hard to read any other way: LOF didn't win because it was better. It won because the test was tilted in its favour.
2. "Fewer anomalies" isn't a quality measure anyway
Even with a fair test, this rule is wrong. A model that never flags anything has an anomaly rate of 0% and would win every time. It would also be useless.
The anomaly rate tells you how jumpy a model is. It doesn't tell you whether it's right. Ranking models by it is like ranking smoke alarms by how rarely they go off.
What I'm changing
Three changes, in order of how much they matter:
Score on data the model hasn't seen. Telemetry is a time series, so I'll hold out the most recent slice (the last 24 hours) and train on everything before it. Both models get scored on the held-out window. That removes LOF's self-neighbour advantage straight away.
split = int(len(X) * 0.9) # rows are ordered by timestamp
X_train, X_test = X.iloc[:split], X.iloc[split:]
iso_forest.fit(X_train)
lof.fit(X_train)
iso_rate = (iso_forest.predict(X_test) == -1).mean()
lof_rate = (lof.predict(X_test) == -1).mean()
Judge against known anomalies, not the anomaly rate. I don't have labels, but I can make some. I'll copy a few hundred held-out readings and push them somewhere they shouldn't be: CPU pinned at 100% with flat memory, or disk usage jumping 20 points between two readings. A good model should catch most of those and stay quiet on the untouched held-out readings. That gives me a recall number and a false alarm number, and I can choose between them on purpose.
Log the comparison, not just the winner. The daily report only stored the winning model's numbers. If it had stored both, I'd have seen the gap on day one instead of day 94.
What I took from it
- Check what your selection metric rewards. "Lower is better" metrics are easy to game, even by accident. Ask what the degenerate winner looks like. Here it was a model that never alerts.
-
Know your library's modes.
novelty=Trueandnovelty=Falsegive LOF two different contracts. The docs were clear; I just didn't reread them. - Look at the boring history. Nothing was broken. The pipeline ran, the API answered in milliseconds, the dashboards were green. The bug only showed up when I lined up 94 days of "everything's fine" side by side.
The code for all of this is on GitHub at siddharthsbhadauria/homelab-mlops, with a small dashboard of the daily runs at siddharthsbhadauria.github.io/homelab-mlops. Once the fix is in and has run for a few weeks, I'll write up whether Isolation Forest gets a fairer shot.
References
- scikit-learn: Novelty detection with Local Outlier Factor
- scikit-learn: IsolationForest
Top comments (0)