<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Siddharth Singh Bhadauria</title>
    <description>The latest articles on DEV Community by Siddharth Singh Bhadauria (@siddharthsbhadauria).</description>
    <link>https://dev.to/siddharthsbhadauria</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4247751%2F30b7bf09-5b7f-4cfa-8553-0cd51ab88ade.jpg</url>
      <title>DEV Community: Siddharth Singh Bhadauria</title>
      <link>https://dev.to/siddharthsbhadauria</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/siddharthsbhadauria"/>
    <language>en</language>
    <item>
      <title>My anomaly detector picked the same model 94 days in a row. It was grading its own homework.</title>
      <dc:creator>Siddharth Singh Bhadauria</dc:creator>
      <pubDate>Mon, 16 Nov 2026 12:29:41 +0000</pubDate>
      <link>https://dev.to/siddharthsbhadauria/my-anomaly-detector-picked-the-same-model-94-days-in-a-row-it-was-grading-its-own-homework-4p7e</link>
      <guid>https://dev.to/siddharthsbhadauria/my-anomaly-detector-picked-the-same-model-94-days-in-a-row-it-was-grading-its-own-homework-4p7e</guid>
      <description>&lt;p&gt;&lt;em&gt;A small bug in how I chose between two scikit-learn models, and why "fewest anomalies" is a bad way to pick a winner.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I run a small MLOps setup on my home NAS. It's a UGREEN box with 32 GB of RAM that sits in a cupboard and runs Docker. Every 15 minutes a collector writes CPU, memory and disk readings into DuckDB. Every few hours a training job turns those readings into 18 features and trains two anomaly detection models. The better one gets saved and served behind a small FastAPI endpoint. If it flags two odd readings in a row, a GitHub issue gets opened.&lt;/p&gt;

&lt;p&gt;It's been running since August. Every day it writes a small JSON report: which model won, how many samples it trained on, what share of readings it flagged.&lt;/p&gt;

&lt;p&gt;This week I looked at all 94 of those reports together for the first time. The winning model was Local Outlier Factor every single day. Not most days. All 94.&lt;/p&gt;

&lt;p&gt;That should have bothered me sooner.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The two candidates are the usual pair for unlabelled data:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Isolation Forest.&lt;/strong&gt; It builds random trees and treats points that are easy to isolate as odd.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Local Outlier Factor (LOF).&lt;/strong&gt; It compares how dense each point's neighbourhood is to the density around its neighbours.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both are trained with &lt;code&gt;contamination=0.01&lt;/code&gt;, meaning I expect about 1% of readings to be anomalies. The training code looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;iso_forest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;IsolationForest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_estimators&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;contamination&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;random_state&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;iso_forest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;iso_metrics&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_evaluate_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;iso_forest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"IsolationForest"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;lof&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LocalOutlierFactor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_neighbors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;contamination&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;novelty&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;lof&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;lof_metrics&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_evaluate_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lof&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"LocalOutlierFactor"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And this is how a winner gets picked:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;iso_metrics&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"anomaly_rate"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;lof_metrics&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"anomaly_rate"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;best_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;iso_forest&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;best_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lof&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model that flags fewer readings wins. When I wrote it, my reasoning was that fewer flags means fewer false alarms, and false alarms are what make you mute an alerting system.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two problems hiding in eight lines
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. The models are scored on the data they were trained on
&lt;/h3&gt;

&lt;p&gt;Look at &lt;code&gt;_evaluate_model(model, X, ...)&lt;/code&gt;. The &lt;code&gt;X&lt;/code&gt; is the same &lt;code&gt;X&lt;/code&gt; that went into &lt;code&gt;fit&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;For Isolation Forest this mostly washes out. With &lt;code&gt;contamination=0.01&lt;/code&gt; it sets its threshold so that about 1% of the training data falls below it. Ask it to predict on that same data and you get about 1% back. That's by construction.&lt;/p&gt;

&lt;p&gt;LOF is different. I set &lt;code&gt;novelty=True&lt;/code&gt; so I could call &lt;code&gt;predict&lt;/code&gt; on new readings at serving time. In novelty mode, LOF works out each point's density from its nearest neighbours in the training set. If the point you're scoring &lt;em&gt;is&lt;/em&gt; in the training set, its nearest neighbour is itself, at distance zero. Every training point looks like it lives in a denser neighbourhood than it really does, so fewer of them look odd.&lt;/p&gt;

&lt;p&gt;The scikit-learn docs warn about exactly this. With &lt;code&gt;novelty=True&lt;/code&gt;, you should only call &lt;code&gt;predict&lt;/code&gt;, &lt;code&gt;decision_function&lt;/code&gt; and &lt;code&gt;score_samples&lt;/code&gt; on new data, not on the training set.&lt;/p&gt;

&lt;p&gt;So Isolation Forest should land close to 1% on its own training data, and LOF lands below that. Across the 94 reports, the winning LOF rate had a median of 0.76%. I can't show you Isolation Forest's daily numbers, because (more on this below) I wasn't saving them. But the pattern is hard to read any other way: LOF didn't win because it was better. It won because the test was tilted in its favour.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. "Fewer anomalies" isn't a quality measure anyway
&lt;/h3&gt;

&lt;p&gt;Even with a fair test, this rule is wrong. A model that never flags anything has an anomaly rate of 0% and would win every time. It would also be useless.&lt;/p&gt;

&lt;p&gt;The anomaly rate tells you how jumpy a model is. It doesn't tell you whether it's right. Ranking models by it is like ranking smoke alarms by how rarely they go off.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'm changing
&lt;/h2&gt;

&lt;p&gt;Three changes, in order of how much they matter:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Score on data the model hasn't seen.&lt;/strong&gt; Telemetry is a time series, so I'll hold out the most recent slice (the last 24 hours) and train on everything before it. Both models get scored on the held-out window. That removes LOF's self-neighbour advantage straight away.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;split&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# rows are ordered by timestamp&lt;/span&gt;
&lt;span class="nc"&gt;X_train&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;X_test&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iloc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iloc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;iso_forest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_train&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;lof&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_train&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;iso_rate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;iso_forest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;predict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_test&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;lof_rate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lof&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;predict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_test&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Judge against known anomalies, not the anomaly rate.&lt;/strong&gt; I don't have labels, but I can make some. I'll copy a few hundred held-out readings and push them somewhere they shouldn't be: CPU pinned at 100% with flat memory, or disk usage jumping 20 points between two readings. A good model should catch most of those and stay quiet on the untouched held-out readings. That gives me a recall number and a false alarm number, and I can choose between them on purpose.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Log the comparison, not just the winner.&lt;/strong&gt; The daily report only stored the winning model's numbers. If it had stored both, I'd have seen the gap on day one instead of day 94.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Check what your selection metric rewards.&lt;/strong&gt; "Lower is better" metrics are easy to game, even by accident. Ask what the degenerate winner looks like. Here it was a model that never alerts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Know your library's modes.&lt;/strong&gt; &lt;code&gt;novelty=True&lt;/code&gt; and &lt;code&gt;novelty=False&lt;/code&gt; give LOF two different contracts. The docs were clear; I just didn't reread them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look at the boring history.&lt;/strong&gt; Nothing was broken. The pipeline ran, the API answered in milliseconds, the dashboards were green. The bug only showed up when I lined up 94 days of "everything's fine" side by side.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The code for all of this is on GitHub at &lt;a href="https://github.com/siddharthsbhadauria/homelab-mlops" rel="noopener noreferrer"&gt;siddharthsbhadauria/homelab-mlops&lt;/a&gt;, with a small dashboard of the daily runs at &lt;a href="https://siddharthsbhadauria.github.io/homelab-mlops/" rel="noopener noreferrer"&gt;siddharthsbhadauria.github.io/homelab-mlops&lt;/a&gt;. Once the fix is in and has run for a few weeks, I'll write up whether Isolation Forest gets a fairer shot.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;References&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;scikit-learn: &lt;a href="https://scikit-learn.org/stable/modules/outlier_detection.html#novelty-detection-with-local-outlier-factor" rel="noopener noreferrer"&gt;Novelty detection with Local Outlier Factor&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;scikit-learn: &lt;a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html" rel="noopener noreferrer"&gt;IsolationForest&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>machinelearning</category>
      <category>python</category>
      <category>mlops</category>
      <category>homelab</category>
    </item>
  </channel>
</rss>
