Every delivery manager I've worked with runs a prediction model on Monday morning. It's called dread.
Open the portfolio, scan 30 projects, and your gut sorts them: these five worry me, those twenty are fine, and that one — that one is going to blow up this month. No math. But it's a real model: inputs (velocity, escalations, that PM who's gone quiet), weights (learned from years of pain), and an output (who gets your attention today).
I spent 10 years in SRE and DevOps building that kind of gut. Now I'm transitioning into AI engineering, and the first thing I forced myself to do was NOT train a model. I wrote thinking documents instead. Problem framing before code. It felt like a waste of build time.
It turned out to be the point.
The hard part isn't the math.
When I started, I assumed the difficulty in ML was the mathematics. Gradients, loss functions, the Greek letters. It isn't. Most of that is decided for you once the problem is framed properly.
The hard part is everything upstream of the algorithm: what question are you actually asking, what does a wrong answer cost, and what happens when the model speaks? Get those wrong and the most elegant model in the world predicts the wrong thing, accurately.
Here's what that looks like on a real problem from my domain.
Walk-through: Predicting SLA breaches in IT services
The task sounds simple: Predict which delivery projects will breach SLA this month. Grab data, train a classifier, done. Except four questions decide everything before a single model runs:
What exactly is the event? A breach reported this month? Detected this month? Root-caused to this month? Each definition changes your training data, your labels, and what the system actually predicts. Pick carelessly and you'll ship a model that answers a question nobody asked.
Prediction to what action? If a risky project just triggers an email to a PM, false alarms are cheap. If it pulls two engineers off another client's project, every false alarm has a victim. Same model, same score — completely different system.
What does confident-wrong cost? A 0.9 breach-risk on a healthy project wastes money and trust. A 0.1 on a project that then breaches burns a client relationship. Those costs are not symmetric, and no algorithm knows that. Only you know what each failure direction costs in money and reputation.
Where is the dangerous zone? For us it's 0.5 — the model shrugging. A 0.5 means "I have no idea," and at that point the Monday-morning dread model in a good manager's head beats the classifier. What probability triggers what action is a business decision. It was never the model's to make.
Notice what's missing from all four: Any mention of logistic regression, random forests, or neural networks. The algorithm question ("Is this classification? regression? unsupervised?") ANSWERS ITSELF once these four are settled. That's why it's the last decision, not the first.
The framework I stole from ops
None of this was new to me — I just didn't know it applied to ML. In SRE we never deploy a monitoring alert without asking: What exactly fires it, who gets paged, what do they do, and what does a false page cost at 3 AM? An alert without those answers is noise with a pager attached.
A prediction without those answers is the same thing. A number with nowhere to go.
So before I touch any model now, I write four things down:
The decision this prediction feeds, in one sentence, including who acts on it.
The price of both failure directions. If I can't price them, I'm not ready to set a threshold.
The exact event definition, with its time window.
Only then: which family of algorithm fits the shape of this question.
Item 4 takes ten minutes. Items 1–3 take days. And they're the reason item 4 becomes easy.
The algorithm is the last decision you make.
This is the first post in a series — I'm working through the classic ML algorithms one at a time (logistic regression next), each through this same lens: what the algorithm bets about your world, and what that means in production. I'm a decade-long SRE/DevOps engineer (currently on Google's Piper infrastructure via Movate) moving into AI platform engineering, documenting it in public. If you've watched an ML project fail for non-ML reasons, tell me the story in the comments.
Top comments (0)