DEV Community: Nishant Banginwar

Nine Algorithms Later, I Stopped Learning Algorithms

Nishant Banginwar — Wed, 02 Sep 2026 05:29:21 +0000

Classic Machine Learning Through the Eyes of an SRE, Part 10: The Finale
What nine algorithms taught me to ask about the tenth.

Ten parts later, I stopped trying to learn algorithms.

That sounds like a strange conclusion for a series about learning classic machine learning, but somewhere between the first supervised model and the last unsupervised one, the goal changed. I was no longer trying to remember how nine algorithms worked. I was trying to build a way of looking at an unfamiliar one.

That distinction matters to me because I came to ML from SRE and DevOps, where I am used to asking a different set of questions. What does the system assume? Where can it fail? What happens when the input changes? Can I reproduce the result? What does the system do when it meets the real world at 3 AM?

I wanted to see whether the same thinking could work for machine learning.

It did.

Three slots I kept mixing together

Across the series, I kept coming back to three questions: what shape does the algorithm assume the world has, what does it consider wrong, and how does it go looking for a less-wrong answer?

In ML terms, those are roughly the hypothesis, the loss, and the optimization.

They sound separate when you write them down. I still managed to mix them up repeatedly.

On SVM, I initially described gradient descent as the loss. It is the optimizer. The loss is hinge loss with L2 regularization. In the same pass, I treated convexity like a tunable knob, when it is actually a property of the optimization problem. The knobs are things like C and gamma.

Then I made a similar mistake with logistic regression by reaching for squared error, which belongs to linear regression rather than the usual log-loss objective for logistic classification.

Three articles, three versions of essentially the same confusion. I did not see the pattern while I was writing them individually. I saw it only when I put the algorithms next to each other.

That changed how I approached the rest of the series. Instead of starting with the algorithm name and trying to memorize its mechanics, I started filling those three slots one at a time. Once they were clear, the unfamiliar algorithm became much easier to reason about.

Five ways an algorithm goes looking for an answer

The optimization slot eventually produced another pattern.

These labels are my own shorthand, not standard ML terminology, but they became useful because they describe something I kept seeing.

K-Means searches. Lloyd's algorithm repeatedly assigns points to centroids and moves the centroids, so where it starts can influence where it ends.

DBSCAN defines. It does not optimize a conventional objective. It defines a core point through eps and minPts, then builds density-connected clusters from that definition.

PCA solves. It can use eigendecomposition to find principal directions directly, rather than relying on an iterative local search.

Hierarchical clustering builds. It starts with individual points and repeatedly makes greedy, irreversible merges. There is no grand global search that goes back and reconsiders everything.

Isolation Forest builds through randomness. Its trees use random partitions, and the model uses how quickly observations become isolated as its signal for anomaly detection.

Those are five very different ways of getting to an answer.

This was roughly where I stopped memorizing the algorithms individually. I started thinking about the kind of machinery each one uses to arrive at its result.

Loss has shapes too

The loss column developed a similar structure.

Some algorithms explicitly minimize something. Logistic regression minimizes log loss. SVM works with hinge loss plus regularization. K-Means minimizes within-cluster squared distances, commonly expressed as inertia.

DBSCAN has no loss function in the conventional sense because it is applying a density definition rather than optimizing an objective.

PCA is about preservation. It looks for directions that retain as much variance as possible rather than minimizing prediction error against a target.

Hierarchical clustering is more conditional. There is no single loss function governing the entire hierarchy. The merges are greedy and local. Under Ward linkage, however, each merge is chosen to minimize the increase in within-cluster variance, specifically under Euclidean distance.

Isolation Forest is different again. It uses randomized partitioning and path length to measure how easily a point can be isolated.

The distinction became more interesting to me as I went through the series because of what it says about human judgment. The less an algorithm optimizes, the more the human setup becomes the intelligence.

With DBSCAN, someone has to decide what counts as dense enough. With PCA, someone is making the assumption that the variance being preserved represents something useful. With Isolation Forest, contamination represents an assumption about how much of the population should be anomalous. There is no universal business rule hidden inside the algorithm that can choose it for you.

That is where machine learning started feeling surprisingly familiar to me.

In production systems, the configuration around the software is often just as important as the software itself.

The properties that look like bugs

The parts of the last few articles that stayed with me were often not the equations. They were the properties that looked wrong until I understood why they weren't.

PCA components, for example, are defined only up to sign. A refit can return a component with its direction flipped while explaining exactly the same amount of variance. Nothing crashes. The metrics can look fine. Yet if someone is comparing component directions across model versions, the change suddenly matters.

Hierarchical clustering has a similar trap. The left-to-right order of leaves in a dendrogram is not itself a similarity measure. Either subtree can be placed on either side without changing the underlying hierarchy, so the same clustering can have multiple valid drawings. Merge height carries the meaningful distance information, but humans naturally look at the diagram and start interpreting its horizontal arrangement.

Isolation Forest gave me another example. Its trees commonly use relatively small subsamples, around 256 observations per tree in implementations such as scikit-learn, rather than growing each tree from the entire dataset. That can look like an obvious compromise until you understand that isolation is relative. A small subsample is part of the method, not simply a shortcut for faster training.

These examples changed what I look for.

If a property feels like a bug but isn't, I want to understand it before I try to "fix" it.

Those are also the properties most likely to cause trouble in production, because they are the ones people don't think to monitor or explain.

The tenth algorithm

Gradient boosting wasn't part of this series.

But after nine algorithms, I can look at it without immediately reaching for a tutorial.

I can make a first-pass prediction about how I expect it to work and where I would investigate first.

The hypothesis is familiar because the base learner is commonly a decision tree. So I would expect the model to represent relationships through regions and splits rather than smooth global functions. That puts it in territory I already explored with decision trees and related models.

The optimization is where it starts to separate from random forests. Random forests grow trees independently and combine their results. Boosting builds trees sequentially, with later trees responding to what earlier trees got wrong. That immediately gives me a production question: what does that sequential dependency mean for training time, parallelism, and scaling?

The loss is another place I would look closely. Boosting typically builds an additive model by minimizing a differentiable loss through successive corrections. So although the base learner may be a tree, the way the overall model is trained is fundamentally different from simply growing many independent trees and averaging them.

Then there is the failure hypothesis I would test first.

A random forest can reduce variance by averaging many independently trained trees. With boosting, adding more stages is not automatically harmless because the later stages continue trying to correct what came before. I would therefore investigate early stopping, learning rate, tree depth, and the model's behavior as the number of boosting stages increases.

I have not verified that last part as a claim about a specific implementation. It is a hypothesis I would test.

And that distinction is exactly why the previous nine mattered.

I don't need to pretend I already know the tenth. I need to know where to start looking.

Production takeaway

The most useful lesson from these nine algorithms was not an algorithm at all.

It was that mathematical correctness and production usefulness are different things.

DBSCAN can fail deterministically. That doesn't make the system reliable. It just makes the failure reproducible.

A flat CSAT score of 2 out of 10 has zero variance, so a variance-based technique can treat it as uninteresting while the business sees a serious problem. The model isn't wrong about the statistic. The statistic simply isn't the same thing as the business signal.

Common interview mistake

The common mistake, given how I now think about these questions, is treating "Do you know X?" as a yes-or-no question about whether you have completed a tutorial.

I would rather say, "I haven't studied it deeply yet, but I know the questions I need to answer."

For an unfamiliar algorithm, I can start with the hypothesis, loss, and optimization. Then I can ask what assumptions it makes, what can make those assumptions fail, and what its behavior means for a production system.

That doesn't replace actually learning the algorithm.

It gives me a much better way to learn it.

Where I'd use this in a real production system

I would use this framework in model selection and design reviews.

When someone proposes an algorithm, a short discussion around its assumptions, objective, optimization strategy, failure modes, and operational behavior can expose a mismatch before the model becomes part of a pipeline.

That is especially useful when the conversation is otherwise dominated by algorithm names.

"Why gradient boosting?"

"Why PCA?"

"Why clustering?"

Those questions are less useful than asking what problem shape we believe we have, what we consider a useful result, and what happens when reality disagrees with our assumptions.

The algorithm comes after that.

What nine algorithms actually taught me

I said at the end of Part 4 that I did not want to leave this series knowing four algorithms. I wanted to leave knowing what to ask when I met a fifth.

I think nine was enough to prove the idea.

I started this series trying to learn classic ML from the perspective of someone who had spent most of his career thinking about production systems, reliability, incidents, automation, and failure.

I expected the biggest challenge to be the mathematics.

It wasn't.

The harder part was learning to stop looking at each algorithm as an isolated thing to memorize and start looking for the decisions underneath it: what it assumes, what it optimizes, what it ignores, and what happens when those choices meet messy real-world data.

That is also why I am comfortable ending the series without pretending I now know every important algorithm.

Gradient boosting is still the tenth algorithm.

There will always be an eleventh.

The useful thing I have now is not a finished list.

It is a way to approach the next one.

Classic ML through the eyes of an SRE ends here. The learning doesn't.

The full series

Logistic Regression Doesn't Make Decisions. Your Business Does. — the threshold is a business decision, not part of the model.
Decision Trees Aren't Trained. They're Grown. — no gradient descent anywhere, and what greedy growth costs you.
Random Forest Is Horizontal Scaling for Predictions — averaging cancels the instability that made one tree untrustworthy.
The Kernel Trick Is the Oldest Move in Engineering — reformulate the problem into a space where it becomes easy.
K-Means Doesn't Find Your Clusters. It Finds Its Own. — why the elbow method has to exist.
DBSCAN Has No Loss Function. That's Not a Bug. — a definition that gets computed, and you set the density bar.
PCA Deletes Your Quietest Signals First — variance is not importance, and flat CSAT is a crisis with none.
Hierarchical Clustering Fails Beautifully — the dendrogram is the most seductive output in the set.
The Best Anomaly Detector I Know Optimizes Nothing — cheap randomness beats careful modeling when rarity is the target.

The Best Anomaly Detector I Know Optimizes Nothing

Nishant Banginwar — Fri, 28 Aug 2026 03:23:05 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 9: Isolation Forest

The algorithm in one line: Isolation Forest scores how anomalous a point is by how few random cuts it takes to separate that point from everything else. No model of normal, no loss function, nothing optimized.

← Previous: Part 8 — Hierarchical Clustering Fails Beautifully · Next: this is the series finale — start at Part 1.

Every anomaly detector I had studied models what NORMAL looks like, then calls the leftovers outliers. K-Means: far from every centroid. DBSCAN: in the noise bucket. Sensible, and intuitive.

Isolation Forest does not bother. It never models normal at all. It goes straight at the rare points with a single question: how few random cuts does it take to isolate you?

Random cuts, literally. Pick a feature at random, pick a split value at random between that feature's min and max, repeat. A point that separates from the crowd in three cuts is anomalous. A point buried in the middle of a dense mass takes thirty. Grow hundreds of these random trees, average the isolation depth for each point, and you get an anomaly score.

There is no loss function here. No optimization, not even the local kind that decision trees do at every split. Every cut is a coin flip, and the power comes entirely from averaging, which is the forest trick from the supervised half of this series now applied to pure randomness. Cheap randomness plus averaging beats careful modeling, as long as the target is something randomness naturally exposes. Rarity is exactly that.

Sometimes the winning move is to optimize less. That sentence would have gotten me laughed out of my first ML study session. It is also this finale's thesis.

The part I had completely backwards

Here is the thing I did not know until I read the original paper properly, and it is the opposite of every instinct a decade of ops gave me.

Isolation Forest deliberately trains each tree on a small subsample of your data, and this is not a performance shortcut. The common default is 256 points per tree, regardless of whether your dataset has ten thousand rows or ten million. My first assumption was that this was the usual accuracy-for-speed trade, the sort of compromise you accept and then apologize for in the design doc.

It is not a compromise. The paper's key insight is that you do not need the entire dataset to isolate anomalies effectively. A small subsample keeps rare points rare.

The reason sample size matters at all is that isolation is relative: an anomaly is interesting because it is easier to separate than the points around it. A small subsample keeps the forest focused on that isolation problem, while making the trees dramatically cheaper to build.

So the algorithm that optimizes nothing also, on purpose, looks at less of your data than you gave it. Every scaling instinct I have says that has to be a compromise. It isn't.

The most business-loaded knob in the set

Isolation Forest outputs a ranking, a score per point, not a decision. Someone still has to draw the line.

That knob is contamination, and it is worth being precise about what it actually does, because I had this slightly wrong too. Contamination is not the score threshold itself. It is your stated estimate of what fraction of the data is anomalous, and the library uses it to derive the cut point from the observed score distribution. Set contamination to 0.01 and you have not said "flag anything above 0.7," you have said "roughly one percent of this is bad, go find that one percent."

Which means there is no universal statistical rule that tells you the right contamination value for your business — no elbow, no knee, no gap in a plot that settles it for you. It is a business call: how many investigations can the team absorb in a week, and what does a missed anomaly cost when it gets through? Every other algorithm in this series handed you at least a heuristic to fight about. This one hands you a mirror.

One practical note on reading the scores. The original paper's score runs from 0 to 1, where values near 1 are anomalous and values near 0 are firmly normal. Scikit-learn reports it differently, and the difference has bitten people. Its score_samples is the opposite of the paper's score, so more negative means more anomalous. decision_function then shifts that score by an offset so negative values are treated as outliers; when you specify contamination, that offset is chosen to produce the expected proportion of training outliers. Same ranking either way, inverted convention. I have watched that flip cost someone an afternoon.

Statistically strange is not operationally costly

A timesheet anomaly from an intern and one from a program lead can score identically strange. One is a rounding error, the other is revenue leakage. The fix is to re-rank by expected value, score multiplied by business impact, so investigation effort follows expected cost rather than statistical novelty. The algorithm ranks weirdness. Only the human side knows what weirdness costs.

By this point in the series the pattern has fully crystallized. The less an algorithm assumes and optimizes, the more the human setup and operation is the intelligence. Isolation Forest has no conventional optimization objective, and correspondingly, everything that makes it useful in production, the features, the threshold, the value weights, the triage queue, is human judgment wearing a model's clothes.

Masking and swamping

My worst mistake of the entire week happened here. I wrote in my notes that repeated missed breaches would be "properly handled" by Isolation Forest. That is precisely backwards, and it has a name.

Masking is the flagship failure. When anomalies repeat and clump together, they stop being few. A dense little cluster of the same fraud pattern now takes many cuts to isolate, scores as normal, and disappears. Translated for ops: the first occurrence of a new failure mode lights up beautifully. By the tenth occurrence, that same pattern may no longer look isolated — exactly when it has become systemic. Isolation Forest is an early-warning instrument, not a recurring-problem detector. Pair it with signature-based detection for known patterns, the same way you pair anomaly alerts with static thresholds in monitoring.

Swamping is the mirror image, and the one I had been describing without knowing the word for it. Normal points that happen to sit near an anomalous region get flagged as anomalous themselves, and your investigation queue fills with statistically strange but entirely harmless records. Both failures are silent. Neither appears in any metric the algorithm produces.

The small subsample is deliberate: it preserves the algorithm's isolation-based view of rarity while keeping the forest cheap to build. It does not eliminate masking or swamping.

The blind spot in the geometry

One more limitation, and this one is structural rather than operational.

Every cut is axis-parallel, because each split uses exactly one feature at a time. That means the regions the forest carves out are axis-aligned rectangles, and the resulting score map inherits that shape. In practice this produces artifacts: bands and corners of the feature space that score as suspiciously normal even though no training data ever sat there. If your two features are correlated, so the real data lies along a diagonal, the rectangles fit that diagonal badly and the score surface develops ghost regions.

The named answer is Extended Isolation Forest, which cuts with randomly oriented hyperplanes instead of axis-parallel splits and reduces the artifacts caused by axis-parallel splitting. This is the same shape of lesson as Part 6, where DBSCAN's single density bar had HDBSCAN waiting behind it. The base algorithm makes a geometric assumption you can inherit without ever noticing, and someone has already written the version that relaxes it.

Closing the series

When this series started I claimed the algorithm is the last decision you make. The unsupervised half sharpened that into something stronger: the algorithm was never the intelligence at all. Framing the question, pricing the failures, choosing the features, setting the thresholds, deciding what matters — that was the intelligence the whole time. The algorithm is the part that scales it.

Ten years of ops taught me to distrust any system whose operator cannot say what it is actually doing. ML systems turn out to be no exception. They just hide it better, and they hide it behind a number that looks like an answer.

Production takeaway

The output is a ranking, not a decision. Contamination is your estimate of the anomalous fraction, not a score cutoff, and no universal rule sets it for you, so it is a capacity-and-cost call. Re-rank by expected value so effort follows money rather than novelty. Keep the subsample small, because the default is deliberate. Pair it with signature detection for known repeat patterns.

Common interview mistake

Assuming it catches all anomalies. Repeated anomalies MASK: once a pattern clumps it stops being few, takes more cuts to isolate, and can score as normal exactly when it has become systemic. Its mirror, swamping, floods the queue with harmless neighbors of real anomalies.

Two more that separate people who have run it from people who have read about it. Believing the small subsample default is merely a speed compromise, rather than part of how Isolation Forest exploits rarity. And calling it a clustering algorithm; it produces per-point scores, not groups.

Where I'd use this in a real production system

Early warning on novel failure modes in infra telemetry, timesheet and expense anomalies, fraud triage queues, SLA-breach precursors. Always with a human triage step in front of any action, and never as the only detector for a failure mode you have already seen.

Classic Machine Learning Through the Eyes of an SRE. What each algorithm bets about your world, read through a production lens. Nine parts, ending here. Written while transitioning from a decade of SRE and DevOps into AI platform engineering, in public. If you disagree anywhere, comments are open. That is what they are for.

Hierarchical Clustering Fails Beautifully

Nishant Banginwar — Tue, 25 Aug 2026 03:16:02 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 8

The most dangerous output in my whole Week-1 study set wasn't a bad prediction. It was a beautiful tree.

Hierarchical clustering produces a dendrogram, that elegant diagram where every account, ticket, or incident nests inside ever-larger families. It looks like discovered truth. Stakeholders lean in. Someone screenshots it for the QBR deck.

Nothing else in the set looks as convincing while being as capable of being completely wrong. A bad K-Means gives you blobs that feel arbitrary, and people push back. A dendrogram built with the wrong linkage on flat data still looks like a family tree of your business.

Nobody pushes back on a tree.

The bet and the build

Hierarchical clustering completes the answer-finding taxonomy I've been using through this series. That's my own shorthand, not standard terminology: K-Means SEARCHES, DBSCAN DEFINES, PCA SOLVES, and hierarchical clustering BUILDS.

Start with every point as its own cluster. Repeatedly merge the closest two clusters. Never undo. Greedy and irreversible, a little like growing a decision tree. Same skeleton, different family.

There is also a top-down version, called divisive clustering, which starts with everything together and splits it. In practice, when people say hierarchical clustering, they're usually talking about the bottom-up, agglomerative version.

Two things were genuinely new to me.

You choose the cut after seeing the structure. Fitting doesn't require you to decide K upfront. The dendrogram gives you the hierarchy, and you choose where to cut it to get the number of clusters you want. That makes the output unusually flexible. For a delivery organization it also feels natural, because account family → sub-segment → individual account is already how a lot of governance gets organized.

Linkage is a selectable worldview. "Closest clusters" needs a definition, and every definition makes a different assumption. Ward pushes toward compact, variance-minimizing clusters. Single linkage is comfortable with chains. Complete linkage favors tighter groups. You aren't configuring a minor implementation detail. You're choosing what kind of structure the tree is allowed to see.

Single linkage had another connection I immediately recognized. It can be understood through a minimum spanning tree: connect the points through the cheapest edges, then cut the longest ones. Once I saw that, chaining stopped being a quirk to memorize. A thin bridge of intermediate points can connect two otherwise separate groups, because the algorithm only needs those cheap nearest-neighbor links. That's a graph algorithm I already knew, wearing a clustering costume. The same thing happened in Part 6, where DBSCAN turned out to have the skeleton of connected components.

The subtlety that took me three rounds

What does hierarchical clustering optimize? My first answer was "nothing, like DBSCAN." Wrong. The optimization story is hybrid.

Agglomerative clustering doesn't optimize one global objective across every possible tree. It makes greedy merges according to the linkage you selected. Ward is the interesting case: each merge is chosen to minimize the increase in within-cluster sum of squares, the same quantity underlying the K-Means objective. Same objective family, different strategy. K-Means searches for a solution, Ward builds one greedily, and neither guarantees the globally optimal result.

One important restriction: Ward is tied to Euclidean distance. If you're thinking about cosine similarity or another non-Euclidean distance, Ward isn't the appropriate choice, and current implementations explicitly restrict it to Euclidean/L2.

That is when "what's the loss?" stopped being a checkbox question for me. The answer can be no single global objective, a linkage-specific criterion, or a greedy local criterion, and each one creates a different failure mode.

The part of the picture that isn't information

Here is the thing that changed how I read every dendrogram since.

The left-to-right position of the leaves is not a similarity measure. At every merge, the two child subtrees can be placed on either side without changing the underlying clustering, so the same hierarchy has many valid visual arrangements. What carries information is the vertical axis: how far apart the groups were when they merged.

But that's not how people read it. They scan a dendrogram horizontally, like a spectrum, and conclude that neighboring leaves are similar. Two accounts sitting side by side at the bottom may not be similar at all. They may only merge near the top of the tree, which is the algorithm telling you they are far apart. Libraries even provide leaf-ordering options to make the visual structure more intuitive without changing the clustering.

So the most persuasive axis on the most persuasive chart carries no similarity information at all.

That's what "fails beautifully" actually means.

The two checks worth running

Two useful checks tell you whether a beautiful tree deserves more scrutiny.

Cophenetic correlation compares the original pairwise distances with the distances represented by the dendrogram. High correlation means the tree preserves the pairwise geometry reasonably well. Low means you're looking at a substantial distortion.

Cluster stability means resampling the data, refitting, and checking whether the clusters survive. If the membership changes substantially across resamples, the structure isn't robust enough to treat as established fact.

A dendrogram that fails both is still gorgeous. That's the trap.

Two more practical notes. Distances are scale-sensitive, so standardize before fitting, the same rule that applied to K-Means, SVM and PCA earlier in the series. And when candidate merges tie, implementation and input ordering can affect which merge is selected, so two implementations can produce different trees from the same data. Part 6 had a version of this same lesson with DBSCAN border points, and I keep relearning it: "deterministic" usually has an asterisk.

The wall this hits

An n-by-n distance matrix is O(n²) memory. Depending on the linkage and implementation, agglomerative clustering can range from O(n²) to O(n³) time. SciPy documents O(n²) implementations for several common linkages and O(n³) for some others, with O(n²) memory across these implementations.

Fine for two hundred accounts. Not so fine for two hundred thousand tickets.

A mistake I made while studying and want to keep visible: I originally listed the scale wall as a trust check. It isn't. It tells you whether hierarchical clustering can run at all, not whether the answer is right. Different question entirely.

What I'd tell my ops team

Use hierarchical clustering when the domain is genuinely nested and reasonably small: account taxonomies, competency trees, incident-catalog dedup. Pick linkage deliberately, because it is your shape hypothesis, not a default. Run both checks before any stakeholder sees the tree. Never let anyone read left-to-right adjacency as similarity, and say that explicitly when you present it.

And put the dendrogram last in the deck, after the caveats. Because the moment it appears, the room stops evaluating and starts believing.

The convincing failure is the expensive one.

Production takeaway

Run both checks before a stakeholder sees the dendrogram: cophenetic correlation and cluster stability across resamples. Choose linkage deliberately. Standardize first. And know the scale wall, because O(n²) memory can become the constraint long before the business question gets interesting.

Common interview mistake

Two. First, saying hierarchical clustering optimizes nothing. The optimization story is hybrid: agglomerative clustering is greedy and linkage-driven, while Ward chooses each merge by minimizing the increase in within-cluster variance, related directly to the K-Means objective.

Second, treating the horizontal position of leaves as a similarity measure. It isn't. The tree's merge heights carry the distance information. Leaf ordering is primarily a visualization choice.

Where I'd use this in a real production system

Small, genuinely nested domains: account taxonomies, competency trees, incident-catalog dedup. Hundreds of items, not hundreds of thousands.

Classic Machine Learning Through the Eyes of an SRE. What each algorithm bets about your world, read through a production lens. Previous: PCA. Final part: Isolation Forest, the anomaly detector that builds isolation into the algorithm rather than optimizing a conventional loss.

PCA Deletes Your Quietest Signals First

Nishant Banginwar — Fri, 21 Aug 2026 03:01:49 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 7

Picture a client health metric that has been flat at 2 out of 10 for six months. Ask PCA to compress your client-health data and that metric will contribute almost nothing to the directions PCA decides to keep.

Not because PCA is broken. Because PCA treats variance as importance, and a signal that barely moves contributes almost no variance. Reduce the data far enough and the independent information it carried is simply not there anymore.

But a CSAT frozen at 2/10 is not noise. It is a crisis nobody is escalating. And after compression, it may no longer be available to anything downstream.

That is the bet, and in ops data it is frequently wrong. The critical signals are often the quiet ones.

There is a cheaper version of the same failure that catches most people first. PCA measures variance in whatever units your features happen to be in, so a metric ranging from 0 to 10,000 can dominate one ranging from 1 to 5 purely because it is bigger. Standardize before you compress, or your first principal component may just be an elaborate way of saying "ticket count." Same class of bug as unscaled features in K-Means and SVM, and it fails just as quietly.

What PCA actually is

Third answer-finding strategy in the unsupervised set, using the same shorthand as the last two articles.
K-Means SEARCHES: iterate and hope.
DBSCAN DEFINES: declare a rule and traverse.
PCA SOLVES: an eigendecomposition or SVD gives a direct solution rather than an iterative local search. No convergence to babysit, no restarts, no local optima to escape.

Two caveats on the word "direct," both worth knowing. Many libraries will use randomized SVD on large matrices, which is approximate and stochastic. And even with an exact solver, eigenvectors are only defined up to sign, so a component can come back inverted between runs or across implementations. The variance explained is identical either way, which is precisely why nobody notices.

Hold onto that second one. It comes back later and it is worse than it sounds.

PCA is also the first algorithm in this set whose bet is about columns rather than rows. Everything before it asked which points belong together. PCA asks which features move together. If ticket volume, escalation count and response-time degradation all rise and fall as one, PCA blends them into a single composite axis. One dial where there appeared to be three.

Note the verb. It blends. My most persistent mistake while learning this was writing that PCA "removes similar features," which it does not, and which is a description of feature selection rather than of anything PCA does. PCA constructs new axes as weighted combinations of the original features. The output is a re-description of your data, not a verdict on which columns deserve to exist.

Layer 1.5

The thing that reframed PCA for me is that it is not analysis. It is infrastructure.

A clustering sits at the top of a pipeline. If it is wrong it embarrasses you in one meeting, someone pushes back, and you fix it. PCA sits underneath, at what I have started calling Layer 1.5, and it decides what every layer above it can ever see.

A bad compression blinds every downstream consumer, quietly, for quarters. And the post-mortem may never find it, because the missing signal is not anywhere anymore. The 2/10 CSAT crisis was not mislabeled. It was not misclustered. It is absent, and absence leaves no trace to investigate.

Failures that leave no evidence are the ones ops people fear most. PCA can manufacture them at scale, politely, while reporting an excellent variance-explained figure.

Treat loadings like a schema

The engineering conclusion I keep coming back to: treat a fitted PCA like a database schema, not a preprocessing step.

The loadings, meaning the recipe for each composite axis, are an API contract with every downstream consumer. So version them, diff them between refits, assign them an owner, monitor how their meaning drifts, and roll back the model and the loadings together rather than separately. If PC2 used to mean "escalation pressure" and gradually starts looking more like "account size," every dashboard built on PC2 changed meaning without anyone editing a dashboard.

And now the sign problem gets its moment, because this is the part I did not know and would have walked straight into.

Refit the model, get a component back with its sign flipped, and every consumer reading "high PC2 means high escalation pressure" may now be reading the exact opposite. The variance explained is unchanged. The dashboard still renders. Nothing errors, nothing alerts, and somebody acts on an inverted signal for a quarter before anyone thinks to check the one thing that has no reason to be checked.

The fix is almost annoyingly simple. Pin the sign. Define a convention, such as requiring a chosen reference loading to be positive, assert it after every refit, and fail the pipeline when the convention is violated.

Nobody would push a schema migration without versioning and review. Refitting PCA is a schema migration for your feature space. It deserves the same ceremony.

What I'd tell my ops team

Before compressing anything, inventory the quiet-but-critical signals. The flat CSATs. The metrics whose silence is the message. Either protect them outside the compression step, or make sure the representation you keep still preserves what they carry.

Then check how much variance your retained components actually explain, and read the loadings out loud in business language. If PC1 cannot be named in a sentence, nobody downstream will know what they are looking at, and they will use it anyway.

One more limit is worth stating plainly, because it is easy to forget once the math starts working. PCA finds linear combinations. Structure that is real but strongly nonlinear may not survive compression well, and variance explained will not tell you that you lost it. If your features relate to each other in ways a straight line cannot describe, PCA is probably the wrong instrument, and it will not mention this to you.

PCA answers one question precisely and repeatably: what varies most together. Whether that was your question is still your problem.

Production takeaway

Standardize before you compress, or your first component may just be your largest-unit feature wearing a disguise.

Treat fitted loadings like a schema migration: versioned, diffed between refits, owned, and rolled back together with the model. Pin the component signs and assert them after every refit, because a silent flip can invert meaning for everything downstream.

And before compressing anything, inventory the quiet-but-critical signals, the flat metrics whose silence is the message, and make sure variance-based compression does not erase them.

Common interview mistake

Describing PCA as removing correlated features. It removes nothing column-wise. It builds new axes that are weighted combinations of the original features, so the output is a re-description rather than a selection.

Two more that come up. Forgetting that PCA is scale-sensitive, when standardization is usually necessary if features sit on different scales. And calling PCA fully deterministic, when randomized SVD can be approximate and stochastic, and even exact solvers leave eigenvector signs ambiguous.

Where I'd use this in a real production system

Compressing wide, correlated telemetry before clustering. Building a composite client-health or skills index. Reducing dimensionality before visualization. And anywhere downstream consumers can tolerate a re-described feature space, with the loadings under version control.

Classic Machine Learning Through the Eyes of an SRE — Part 7. What ML algorithms bet about your world, through a production and ops lens.
Previous: DBSCAN.
Next: hierarchical clustering, the algorithm that fails beautifully.

DBSCAN Has No Loss Function. That's Not a Bug.

Nishant Banginwar — Tue, 18 Aug 2026 03:41:09 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 6

DBSCAN was the first algorithm in my study set that optimizes nothing.

No loss function. Nothing minimized. Nothing trained.

It's a DEFINITION that gets computed. Declare what a cluster is: a core point has at least minPts samples within radius eps, counting itself. A cluster is built by connecting density-reachable core points and their neighboring border points. Then traverse the data and collect whatever satisfies the definition.

Three roles fall out of that.

Core points have enough neighbors. Border points sit inside a core point's radius but don't have enough neighbors of their own. Noise is everything else.

Hold onto those border points. They come back later.

If you've done backend work, you already know this algorithm by another name: connected components. BFS flood-fill over neighborhoods.

I stared at DBSCAN for a week before realizing I'd implemented its skeleton years ago for an entirely different problem. Half of ML is old friends with new vocabulary.

The bet it makes

K-Means bets your groups are round blobs and that you know how many.

DBSCAN refuses both. Clusters can have arbitrary shapes, and the number of clusters falls out of the density structure rather than being supplied upfront.

Its actual bet is sneakier: ONE density bar works for your whole dataset.

You set eps and minPts globally. If dense regions and sparse regions both carry real structure, say dense ticket noise from big accounts and sparse but meaningful patterns from small ones, a single density threshold can't serve both. Tighten it and sparse structure dissolves into noise. Loosen it and dense regions can fuse together.

This is a known limitation of DBSCAN. It works best when meaningful clusters have reasonably similar densities.

That's where HDBSCAN becomes interesting. Instead of forcing one global eps, it considers clustering across varying density levels and selects persistent structure. If your domain genuinely contains clusters at different densities, that's a much more natural fit than endlessly searching for one eps that serves everyone.

The other thing that quietly breaks DBSCAN is dimensionality.

It's a distance-based algorithm. As dimensionality grows, distance measures can become less informative because points tend to look increasingly similar in distance. At some point, eps stops representing the neighborhood you thought it represented.

That's a problem the next article in this series exists to address.

The noise verdict

DBSCAN can say something no algorithm so far could:

"This point belongs to nothing."

That's a perfectly legal answer. K-Means force-assigns every point to a cluster, however poor the fit. DBSCAN has an explicit noise category instead.

And for ops data, that's often the more honest shape. Most days genuinely are unremarkable. Some events genuinely don't belong to any known pattern.

But the noise bucket gave me one of my favorite realizations from this algorithm: today's noise can become tomorrow's cluster.

A new failure pattern may start as a handful of isolated events before enough similar events appear to form a dense region. Treat noise as garbage and you may throw away early signals. Treat it as a watchlist and DBSCAN becomes much more interesting operationally.

When there's no optimizer, YOU are the optimizer

Here's the pattern that started crystallizing with this algorithm: the less the algorithm optimizes, the more the human setup becomes the intelligence.

There's no loss function standing between your parameter choices and the output.

eps. minPts. The feature space. The distance metric.

Those choices define what "dense" means.

My first-pass mistake was writing: "The structure is free. The density is up to the model to decide."

Exactly backwards. I set the density bar. The model enforces my definition.

That is the part worth remembering.

Deterministic failure

DBSCAN is deterministic for the same data, parameters, and input order.

But that does not mean the result is independent of input order.

Core-point membership is stable, but a non-core border point can be reachable from core points belonging to different clusters. In that case, its assigned cluster can depend on which cluster is encountered first.

So reorder your rows and a handful of border points can change cluster. Usually not the core structure. But potentially enough to matter if something downstream makes decisions from individual cluster membership.

And then there's the more dangerous lesson: reproducibility can masquerade as reliability.

With K-Means, instability across runs can at least tell you something is sensitive. DBSCAN can give you the exact same wrong answer every time. You can't monitor that with output variance because there may be none.

You need input-drift monitoring and periodic review of what the algorithm is actually putting into noise. Process guardrails, not statistical ones.

What I'd tell my ops team

Use DBSCAN when shape-freedom and a noise verdict match the domain: signature patterns in tickets, timesheet leakage, delivery-risk patterns that aren't blobs.

Budget your real effort for the parameters. Use a k-distance plot to help choose eps, and sanity-check minPts against what "pattern" actually means in business terms.

Then review the noise bucket regularly.

And write down WHY the parameters are what they are. When there's no optimizer, the parameter rationale is part of the model documentation.

Production takeaway

With no optimizer, your parameters ARE the model.

Document why eps and minPts are what they are. Use a k-distance plot to guide eps selection. Review the noise bucket regularly.

And monitor input drift rather than expecting output variance to warn you. This algorithm can repeat its wrong answer perfectly.

Common interview mistake

Saying "DBSCAN figures out the density for you." It doesn't. You define the density threshold through eps and minPts. DBSCAN discovers the clusters that satisfy that definition, and it can also identify noise.

Second: calling DBSCAN fully deterministic without qualification. For the same data in the same order, yes. But border-point assignments can depend on data ordering when a border point is reachable from multiple clusters.

Third, and the one that matters most in production: treating deterministic output as evidence of correctness.

Where I'd use this in a real production system

Incident and ticket pattern discovery where shapes are irregular, timesheet or usage leakage detection, and any case where "belongs to nothing" is a legitimate and useful answer.

Classic Machine Learning Through the Eyes of an SRE — Part 6. What ML algorithms bet about your world, through a production and ops lens. Previous: K-Means. Next: PCA, the algorithm that deletes your quietest signals first.

K-Means Doesn't Find Your Clusters. It Finds Its Own.

Nishant Banginwar — Fri, 14 Aug 2026 03:30:14 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 5

Supervised learning at least tells you when it's wrong. Unsupervised learning will happily hand you a confident answer to a question your data never contained.

K-Means is where that lesson starts, because K-Means always gives you the K clusters you asked for. Ask it for 5 clusters of your client accounts and you will receive exactly 5 clusters, with centroids and an objective value. Whether your accounts actually form 5 groups — or any groups at all — is not something the algorithm can tell you.

It answers the question "what's the best way to split this into 5 compact groups?", not "are there 5 groups here?"

Those are very different questions. Only one of them was yours.

The bet it makes

Two bets, actually.

Round blobs. Assigning each point to its nearest centroid creates a Voronoi partition — convex cells with straight boundaries. Pair that with an objective that minimizes squared distance to the center and you get a strong preference for compact, roughly isotropic groups.

If your real segments are elongated, chained, or density-based — say, a "slow-burn escalation" pattern that snakes across metrics — K-Means will slice that snake into three neat blobs and report success.

That tendency toward compact, similarly scaled groups deserves its own warning. If one segment is enormous and another is a handful of accounts, the distance-minimization objective can produce unintuitive partitions. Nothing in the output tells you that the resulting groups correspond to meaningful business segments.

You know K. The number of clusters is an input, not an output. The elbow method and silhouette scores help you argue about it, but they're heuristics, not oracles. K is your hypothesis about the world, wearing the algorithm's clothes.

SEARCH, not solve

Mechanically, K-Means iterates: place centroids, assign points, move centroids to the mean, repeat until convergence. That's Lloyd's algorithm, and it's a heuristic — the global K-Means optimization problem is NP-hard in general, so what you get is a local optimum that can depend on initialization.

Run it with different initializations, and you can get different clusters. k-means++ initialization and multiple restarts manage this, but "manage" is the honest verb.

It's worth naming what's actually being minimized: the within-cluster sum of squares, usually called inertia. And there's a catch in it that explains why the elbow method has to exist at all.

The optimal inertia never increases as K rises — at K = n, every point is its own cluster and inertia is zero. So you cannot choose K by simply minimizing the objective. The elbow exists precisely because the metric can't answer the business question, which leaves you looking for the point where additional complexity stops being worth the improvement.

That word SEARCH matters, and it's my own shorthand rather than standard terminology, for how each unsupervised algorithm arrives at an answer.

K-Means SEARCHES: iterate and hope.
DBSCAN DEFINES: declare a density rule and traverse what follows.
PCA SOLVES: an eigendecomposition/SVD gives a direct solution rather than an iterative local search.
Hierarchical BUILDS: greedy merges, all the way up.

Four strategies for finding answers without labels, and K-Means is the anchor I compare the rest against — most of what they do is best understood as a reaction to something K-Means gets wrong.

The failure that hides

With no labels, there's no accuracy metric to catch you. The failure signature isn't an error — it's a plausible-looking segmentation that quietly mismatches reality.

Concrete version from my world: cluster IT-services accounts for a QBR deck. K-Means produces "5 client segments." The deck ships. Strategy gets built on those segments.

Nobody ever asks whether K=5 actually represents meaningful structure or whether the clusters cut across every important business boundary — because the output looks like insight, and there's no ground truth to embarrass it.

In supervised land, a bad model gets caught by the test set.

In unsupervised land, the test set is a stakeholder meeting three months later.

What I'd tell my ops team

Treat K-Means output as a HYPOTHESIS generator, not a report.

Before anything downstream consumes the clusters:

Check stability — do the clusters survive resampling and re-initialization?
Check the geometry — look at silhouette scores per cluster, not just the average.
Name each cluster in business language — if a cluster can't be described in one sentence a delivery head recognizes, it may be an artifact rather than a useful segment.

And scale your features first. Like SVM, K-Means runs on distance, so unscaled features can silently decide the clustering for you.

The supervised block taught me that the threshold is a business decision. The unsupervised block starts with a harder version: sometimes the QUESTION is a business decision.

K-Means will answer whatever K you hand it. Choosing K responsibly is your job, not its.

Production takeaway

Treat clusters as a hypothesis, never a report.

Before anything downstream consumes them: test stability across resamples and re-initialization, read silhouette scores per cluster rather than only the average, and name each cluster in one sentence a delivery head would recognize.

If it can't be named, it's an artifact.

Common interview mistake

Presenting K as something the algorithm discovered. K is your input and your hypothesis; the elbow and silhouette are arguments, not oracles. Worth knowing why: the optimal inertia never increases as K rises, so the objective itself can never choose K for you.

Second, describing Lloyd's algorithm as if it finds the globally optimal clustering. The K-Means optimization problem is NP-hard in general, so Lloyd's converges to a local optimum depending on initialization. Multiple initializations and k-means++ reduce that sensitivity without changing the underlying problem.

Third, and easiest to miss: skipping feature scaling on a distance-based algorithm.

Where I'd use this in a real production system

Account or customer segmentation for coverage models · workload profiles for capacity planning · a first-pass structure check before designing a supervised label.

Series: what ML algorithms bet about your world, production/ops lens. Previous: SVM. Next: DBSCAN — the first algorithm with no loss function at all.

The Kernel Trick Is the Oldest Move in Engineering

Nishant Banginwar — Tue, 11 Aug 2026 03:22:24 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 4

When a computation is too hard, don't compute harder. Change coordinates until it becomes easy.

Every engineer has made this move. Pick the right data structure and the impossible query goes O(1). Re-index the table and the report that took an hour takes a second. Move the problem into a space where it's trivial, solve it there, come back.

That's the kernel trick. SVM's famous move isn't building a curvy model — it's finding a FLAT cut in a transformed space, which corresponds to a curved boundary back in your original features. The separator stays linear in the transformed space. The space did the work.

And here's the part that makes it a trick rather than just a projection: the data never actually goes up there. The optimization only ever needs inner products between pairs of points, and a kernel function computes what that inner product would be in the high-dimensional space, directly from the original coordinates. You get the geometry of a space you never built. Some kernels correspond to infinitely many dimensions, which would otherwise be an awkward amount of memory to allocate.

The bet it makes

SVM bets that the most ROBUST boundary is the one with the widest margin — maximum distance from the nearest points on each side. And here's the part that rewired me: only those nearest points matter. They're the support vectors. The non-support-vector points don't directly determine the final boundary at all.

Compare that to the forest, which averages over EVERYTHING. SVM is the opposite extreme: the borderline cases that become support vectors define the decision boundary. In delivery-risk terms — the projects that teach you where the line is aren't the disasters or the easy wins. They're the borderline ones that barely breached and barely survived. SVM formalizes that.

Everything old returns

After trees and forests threw away gradient descent, SVM brings some of the regression toolkit back: an explicit loss, convex optimization, and iterative optimization. The families really do have different mechanics, and crossing back is noticeable.

Two things I got wrong on the first pass, both the same trap. I used "convex" as if it were a tunable knob. It isn't. It's a property of the problem, and the knobs are C and gamma. I also named gradient descent as the "loss," when it's the optimizer. The three architecture slots (hypothesis, loss, optimization) are separate, and letting them blur means you don't actually know which part you're tuning when things go wrong.

Also: regularization isn't an afterthought in SVM. The maximum-margin objective is directly tied to controlling the weight norm — the same role L2 regularization plays elsewhere. The safety and the simplicity come from the same place.

The correctness trap

Feature scaling for SVM is not hygiene. It's correctness.

Margins are measured in distance. If ticket-count ranges from 0–10,000 and CSAT from 1–5, ticket-count can dominate the distance calculation entirely, and the margin stops meaning what you think it means. Unscaled features don't degrade an SVM — they quietly change the question it's answering.

Ops translation: this is a units bug. Like averaging milliseconds with seconds in a latency dashboard — the chart still renders, the number is still a number, and it's wrong. The system fails silently and the output looks fine. Those are the failures that survive review.

Where the bet fails

The margin bet assumes the borderline cases are TRUSTWORTHY. If your labels are noisy exactly at the boundary — and in ops data they usually are, because borderline breaches are exactly where "was that a breach?" gets argued — then the points defining your model are your least reliable ones. C controls how much you let noisy borderline points bend the line; gamma controls how local the influence of individual training points becomes in an RBF kernel. Tune them together or not at all.

And the kernel choice is a hypothesis about your domain's shape, not a preprocessing detail. Picking RBF because "it usually works" is betting your world is locally clumpy without asking.

Two things it won't give you

A probability. An SVM returns a decision score related to distance from the boundary, not a calibrated probability. If you want 0.83, you need calibration — traditionally Platt scaling, which fits a logistic model to the SVM scores using additional cross-validation. Which means the threshold conversation from the first article in this series doesn't go away here, it just gets an extra step in front of it. Anyone treating the raw decision score as a probability is reading a number that doesn't mean what they think.

Scale. For classical kernel SVMs, training can become roughly quadratic or worse with the number of samples, depending on the solver, because the optimization works over pairs of points. Fine for thousands of rows. Impractical well before you reach a million. That's the honest reason forests and gradient boosting took over large tabular problems, and it has nothing to do with which one is smarter.

What the supervised set finally gave me

SVM completed the supervised set for me, and with it the real prize: a map of four different ways to handle non-linearity.

Polynomials bend the line. Trees carve local boxes. Forests average those boxes. SVM changes the space so a flat boundary becomes useful.

I don't want to leave this series knowing four algorithms. I want to leave knowing what question to ask when I meet a fifth:

What bet about the shape of my data am I willing to make?

Production takeaway

Feature scaling is a correctness requirement here, not hygiene — unscaled features silently change the question the model is answering, and the output still looks fine. Treat kernel choice as a documented hypothesis about your domain's shape, tune C and gamma together, and if anything downstream needs a probability, calibrate explicitly rather than passing the decision score along as though it were one.

Common interview mistake

Saying the kernel trick projects your data into a higher dimension. It doesn't — that's the entire point of the word "trick." It computes what the inner products would be in that space without ever building it.

Two more worth knowing: calling gradient descent "the loss" (it's the optimizer; the loss is hinge plus a weight-norm penalty), and treating convexity as a tunable knob. And don't describe regularization as bolted on — the maximum-margin objective is already controlling the weight norm.

Where I'd use this in a real production system

Smaller, high-dimensional problems: text and log classification, one-class novelty detection on infra metrics, cases where the borderline examples are the whole story and you have clean labels.

Series: what ML algorithms bet about your world, production/ops lens. Previous: random forest. Next: K-Means — and the uncomfortable discovery that it finds ITS clusters, not yours.

Random Forest Is Horizontal Scaling for Predictions

Nishant Banginwar — Fri, 07 Aug 2026 03:42:00 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 3

The random forest is the first ML algorithm that made me feel at home. Not because of the math — because it's an SRE idea wearing a stats costume.

Many independent workers. No single point of failure. Majority vote. If one worker goes weird, the fleet absorbs it. We've been building systems this way for decades; the forest just applies it to prediction.

The problem it exists to fix

Last article: a single decision tree is readable but unstable — small data change, whole tree flips, explanation rewrites itself. That instability is variance, and it's exactly what scared me about trusting one tree in production.

The forest's move: grow hundreds of trees, each on a random resample of the data, and — this is the part that matters — force each split to choose from only a random subset of features.

That second randomization is the whole difference between a random forest and plain bagging. Bagging alone gives you many trees on resampled data, but if one feature is strongly predictive, every tree grabs it first and they all end up looking alike. Starving each split of features is what makes the trees genuinely different from each other. The randomness isn't sloppiness. It's manufactured disagreement.

The instability doesn't get fixed. It gets CANCELLED. Each tree is still jumpy, but they're jumpy in different directions, and the average is calm.

What surprised me

No new loss function. Each tree still minimizes impurity exactly like a lone tree. The forest adds zero new objectives. The entire gain is a bias-variance bargain: variance drops hard, bias barely moves. You give up readability and get back trustworthiness.

Embarrassingly parallel. Trees are independent, so training scales horizontally — throw cores at it. Boosting, its sequential cousin, is the opposite: each model depends on the last. Map-reduce versus a pipeline.

The smoothness illusion. A forest's decision boundary looks smooth, almost like regression's curve. I initially logged that as "the forest resembles regression." It only looks like regression. Regression starts smooth. A forest averages thousands of tiny box-shaped decisions until the edges blur into something that appears smooth. The cleanest transfers into the forest come from the single tree, not from regression. I kept reaching for the wrong parent.

This is also why random forests became the default baseline for tabular business data. Before reaching for deep learning, many teams still ask one question: can a forest already solve this?

The bet it makes

Same as the tree — the world is chunky boxes — plus one more: your errors are DIVERSE. Averaging only cancels mistakes that point in different directions. If every tree shares the same blind spot, the vote is unanimous and unanimously wrong.

That's the production failure worth internalizing: a forest fails quietly and confidently when its diversity is fake. Bootstrap resamples from a biased dataset are all biased the same way. A hundred voters reading the same newspaper is one voter.

Ops translation: redundancy without diversity is not redundancy. Three replicas in the same rack. Five monitors on the same network path. We've all been burned by correlated failure — the forest can be too.

What I'd tell my ops team

For delivery-risk scoring in IT services, the forest is usually the grown-up default: tabular data, mixed feature types, non-linear interactions, and you care more about being right than explaining every path.

There's also a free instrument most people ignore. Because every tree trains on a bootstrap sample — rows drawn with replacement — roughly one-third of the data never gets selected for any given tree. Those untouched rows become its out-of-bag set. Score each row using only the trees that never saw it and you get a validation estimate without holding anything back. Free monitoring, built into the training process. I don't know another algorithm that hands you that.

The mental shift is to operate a forest like a fleet. Individual trees will be wrong in different ways, and that's fine — your job is to watch the behaviour of the fleet, not any single member. It also means you lose the thing a single tree gave you for free. When someone asks "why did it flag this project," the honest first answer is "347 of 500 trees voted yes," and that's rarely what they wanted to hear.

The lesson I keep coming back to: strategy transfers everywhere, mechanics transfer WITHIN a family (tree → forest) and break ACROSS families (regression → tree). Every algorithm makes a different bet about the world. Learn the bet first, and the equations start making sense.

Production takeaway

After every retrain, compare which features the forest is leaning on. A feature that suddenly dominates is your instability alarm — with no single tree to read, it's the only one you get.

Use out-of-bag error as free validation. Every row can be scored by the trees that never saw it, so you get a health signal without holding data back.

Plan for the explainability gap before someone asks. The forest doesn't expose a single decision path, so answering "why" needs another tool — SHAP or LIME — or a single tree running alongside purely for narration. Decide which before it's a meeting question.

Common interview mistake

First, assuming more trees always means better generalization. Averaging only cancels errors that point in DIFFERENT directions, and with a leaked feature or a biased sample all trees are wrong the same way. Redundancy without diversity isn't redundancy.

Second, and this one gets asked constantly: describing a random forest as just bagged decision trees. Bagging randomizes the ROWS. A random forest also randomizes the COLUMNS available at each split, and that second randomization is what stops every tree from latching onto the same dominant feature.

Where I'd use this in a real production system

Delivery-risk scoring on tabular data · churn prediction · fraud risk · anywhere accuracy matters more than a readable decision path.

Series: what ML algorithms bet about your world, production/ops lens. Previous: decision trees. Next: SVM — and the oldest engineering trick in the book.

Decision Trees Aren't Trained. They're Grown.

Nishant Banginwar — Tue, 04 Aug 2026 03:16:57 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 2

The second algorithm I studied broke everything I'd just learned from the first.

Logistic regression taught me that training means gradient descent: guess, measure error, adjust the weights, repeat until convergence. So when I opened decision trees, I went looking for the optimizer.

There wasn't one.

A decision tree isn't optimized the way I expected. It's grown.

At each step it finds the locally best split, commits to it, and recursively repeats the process. No backtracking. No second chances. There is optimization happening — each split minimizes impurity — but only locally, one step at a time. Finding the globally optimal tree is NP-hard, so the algorithm doesn't even try.

That felt surprisingly familiar.

In incident response or capacity planning, we rarely know the perfect answer. We make the best decision with the information we have, knowing a different first choice might have led somewhere else. Decision trees simply turn that idea into an algorithm.

The bet a tree makes

Every machine learning algorithm makes a different bet about the world.

Logistic regression assumes relationships are smooth. Risk gradually increases as signals change.

Decision trees make the opposite assumption. They assume the world is made of boxes.

A project isn't slightly riskier because velocity drops. It's risky when several conditions happen together: a fixed-price contract, a new account manager, and a month-end delivery. Inside that box, projects fail. Outside it, they're usually fine.

This is exactly how many operational systems work. Severity matrices, routing rules, escalation policies, approval workflows — they're all collections of decision boxes.

That's why trees immediately felt intuitive to me.

The hidden cost of flexibility

Trees make very few assumptions about the data. That sounds like an advantage.

The price is instability.

Change a small part of the training data and the first split can change. Since every later split depends on that first decision, the entire tree can be completely different after retraining.

Same data. Different explanation.

I actually made this mistake while learning. My first notes said that because trees make fewer assumptions, they must be more stable.

Exactly backwards.

Fewer assumptions mean more freedom to fit whatever the sample contains. More freedom means higher variance. I had confused flexibility with reliability.

What this looks like in production

Imagine retraining a churn model every month.

Last month the first split said ticket volume is the biggest predictor. This month it says response time is.

The model might perform equally well. But if people treat the tree as an explanation rather than just a prediction, you've just changed the organization's understanding of reality.

That's why I would version decision trees the same way we version configuration.

Don't just monitor accuracy. Diff the structure. If the explanation changes dramatically between retrains, someone should know why.

Strategy transfers. Mechanics don't.

This algorithm taught me something more useful than decision trees.

Some things transferred directly from logistic regression: frame the business problem first, understand the cost of false positives and false negatives, watch for data leakage, and treat turning predictions into actions as a business decision. Those are strategies.

Other things didn't transfer at all — gradient descent, differentiable loss, model coefficients. Those are mechanics.

Since then, every time I learn a new algorithm, my first question is: what assumptions is this algorithm making about the world?

The answer usually predicts how it learns — and how it eventually fails.

What I'd tell my SRE team

Use a single decision tree when the model is the runbook.

If someone needs to explain every decision to a customer, auditor, or compliance reviewer, it's hard to beat a tree.

Just remember that the explanation itself is unstable. Version it. Diff it. Treat changes like configuration changes.

Production takeaway

A decision tree is the model that is the runbook. Use it when humans need to read, audit, and defend every decision path. Monitor the structure — not just the accuracy.

Common interview mistake

Two answers I now know are wrong: "decision trees are trained using gradient descent," and "decision trees are more stable because they make fewer assumptions." Neither is true. Trees are grown greedily, and fewer assumptions usually mean higher variance, not greater stability.

Where I'd use it

Ticket triage · escalation routing · customer churn explanations · compliance workflows · any system where the business needs to understand why the model made a decision.

Part 2 of "Classic Machine Learning Through the Eyes of an SRE" — a decade in production, now moving into AI platform engineering, documenting it in public. Next: random forests, the first algorithm that felt like distributed systems disguised as machine learning.

Logistic Regression Doesn't Make Decisions—Your Business Does

Nishant Banginwar — Thu, 30 Jul 2026 03:25:31 +0000

Classic Machine Learning Through the Eyes of an SRE — Part 1

For years, every Monday morning started the same way.

Delivery managers would sit down, review a portfolio of projects, and instinctively rank which ones needed attention first.

Nobody called it machine learning.

They looked at dozens of signals: velocity slowing down, customer escalations increasing, risks piling up, and that usually proactive project manager suddenly going quiet.

Somehow all of those signals became a single mental estimate: "This project feels risky."

Looking back, they were doing something surprisingly similar to logistic regression. Not mathematically. Conceptually. They were combining signals, estimating the probability of something going wrong, and then deciding where to act.

The important part is that they made the decision—not the probability.

The bet the algorithm makes

Obviously, logistic regression is far more rigorous than human intuition. But the mental model is surprisingly similar: combine signals, estimate a probability, then decide what to do next.

Underneath, it's the same weighted-sum core as linear regression, squashed through a sigmoid so the output lands between 0 and 1. Its bet about your world is that risk rises smoothly with the signals — no cliffs, no chunky exceptions. For delivery risk, that's mostly right: a project with slightly worse velocity is slightly riskier.

One mechanical detail worth knowing: unlike linear regression, there's no closed-form solution here. You iterate with gradient descent until it converges. This is the first algorithm where "training" literally means guess, measure, adjust, repeat.

The threshold was never in the model

A logistic regression model might tell you there's a 73% chance of an SLA breach.

It never tells you to escalate.

That decision belongs to the business. The threshold isn't part of the algorithm. It's a product decision. An operational decision. Sometimes even a financial decision.

This is one of the most common misunderstandings I see. Teams accept the default 0.5 cutoff as if it were a property of mathematics. It isn't. If a missed breach costs ten times what a false alarm costs, 0.5 is the wrong line — and no amount of model tuning will tell you that.

Log loss and the cost of confidence

Log loss also changed how I think about confidence.

It doesn't just penalize mistakes. It penalizes being confident and wrong.

That reminded me of production systems. A noisy alert is frustrating. A dashboard showing green while customers are down? That's the kind of mistake nobody forgets.

Whether it's monitoring or machine learning, confidence attached to the wrong answer is usually more dangerous than uncertainty.

Two mistakes I made while learning this

One mistake I caught myself making was explaining logistic regression using mean squared error. That's linear regression. Logistic regression uses log loss.

I also used to say a train/test split prevents overfitting. It doesn't. It helps detect overfitting. Regularization is what helps reduce it.

Tiny wording differences. Completely different mental models.

When the bet is simply wrong

If the model is consistently wrong for one segment of your data, retraining alone often won't help — that's the smooth-line assumption failing against a boxy reality. If the relationship is inherently nonlinear, you may need different features, or a different class of model altogether.

That's a design decision, not a tuning problem.

The SRE lens

After spending a decade in SRE and DevOps, I've started looking at machine learning models the same way I look at monitoring systems.

The prediction isn't the product. The operational decision is.

A probability without a response plan is just another number on a dashboard.

The model computes. People—and increasingly AI systems—still have to decide what happens next.

That's where the real engineering begins.

Production takeaway

Probabilities don't create business value. Operational decisions do. Define what happens at high scores, at low scores, and in the uncertain middle before you ship the model.

Common interview mistake

Confusing probability with classification, or assuming a 0.5 threshold is always appropriate.

Where I'd use this in a real production system

SLA breach prediction · incident escalation risk · ticket priority prediction · customer churn probability · fraud risk scoring.

Part 1 of "Classic Machine Learning Through the Eyes of an SRE" — a decade in production, now moving into AI platform engineering, documenting it in public. Next: decision trees, the first algorithm that isn't trained at all.

The Algorithm Is the Last Decision You Make in Machine Learning

Nishant Banginwar — Mon, 27 Jul 2026 07:19:09 +0000

Every delivery manager I've worked with runs a prediction model on Monday morning. It's called dread.

Open the portfolio, scan 30 projects, and your gut sorts them: these five worry me, those twenty are fine, and that one — that one is going to blow up this month. No math. But it's a real model: inputs (velocity, escalations, that PM who's gone quiet), weights (learned from years of pain), and an output (who gets your attention today).

I spent 10 years in SRE and DevOps building that kind of gut. Now I'm transitioning into AI engineering, and the first thing I forced myself to do was NOT train a model. I wrote thinking documents instead. Problem framing before code. It felt like a waste of build time.

It turned out to be the point.

The hard part isn't the math.

When I started, I assumed the difficulty in ML was the mathematics. Gradients, loss functions, the Greek letters. It isn't. Most of that is decided for you once the problem is framed properly.

The hard part is everything upstream of the algorithm: what question are you actually asking, what does a wrong answer cost, and what happens when the model speaks? Get those wrong and the most elegant model in the world predicts the wrong thing, accurately.

Here's what that looks like on a real problem from my domain.

Walk-through: Predicting SLA breaches in IT services

The task sounds simple: Predict which delivery projects will breach SLA this month. Grab data, train a classifier, done. Except four questions decide everything before a single model runs:

What exactly is the event? A breach reported this month? Detected this month? Root-caused to this month? Each definition changes your training data, your labels, and what the system actually predicts. Pick carelessly and you'll ship a model that answers a question nobody asked.
Prediction to what action? If a risky project just triggers an email to a PM, false alarms are cheap. If it pulls two engineers off another client's project, every false alarm has a victim. Same model, same score — completely different system.
What does confident-wrong cost? A 0.9 breach-risk on a healthy project wastes money and trust. A 0.1 on a project that then breaches burns a client relationship. Those costs are not symmetric, and no algorithm knows that. Only you know what each failure direction costs in money and reputation.
Where is the dangerous zone? For us it's 0.5 — the model shrugging. A 0.5 means "I have no idea," and at that point the Monday-morning dread model in a good manager's head beats the classifier. What probability triggers what action is a business decision. It was never the model's to make.

Notice what's missing from all four: Any mention of logistic regression, random forests, or neural networks. The algorithm question ("Is this classification? regression? unsupervised?") ANSWERS ITSELF once these four are settled. That's why it's the last decision, not the first.

The framework I stole from ops

None of this was new to me — I just didn't know it applied to ML. In SRE we never deploy a monitoring alert without asking: What exactly fires it, who gets paged, what do they do, and what does a false page cost at 3 AM? An alert without those answers is noise with a pager attached.

A prediction without those answers is the same thing. A number with nowhere to go.

So before I touch any model now, I write four things down:

The decision this prediction feeds, in one sentence, including who acts on it.
The price of both failure directions. If I can't price them, I'm not ready to set a threshold.
The exact event definition, with its time window.
Only then: which family of algorithm fits the shape of this question.

Item 4 takes ten minutes. Items 1–3 take days. And they're the reason item 4 becomes easy.

The algorithm is the last decision you make.

This is the first post in a series — I'm working through the classic ML algorithms one at a time (logistic regression next), each through this same lens: what the algorithm bets about your world, and what that means in production. I'm a decade-long SRE/DevOps engineer (currently on Google's Piper infrastructure via Movate) moving into AI platform engineering, documenting it in public. If you've watched an ML project fail for non-ML reasons, tell me the story in the comments.