DEV Community: Vedant Narayan

The Loss Function Is a Business Decision, Not a Math Default

Vedant Narayan — Fri, 03 Jul 2026 04:30:00 +0000

I used to think choosing a loss function was a technical detail. Something the framework handled for you.
It isn't. It's one of the most consequential decisions in your entire ML pipeline. And most teams never make it consciously.

Why MSE Fails for Classification

When you first learn machine learning, the loss function is Mean Squared Error. It makes sense for regression. But apply it to a churn prediction model and things break fast.
Consider this scenario. Your model predicts 2% churn probability. The account churns. CS never calls. A ₹40 lakh contract walks out the door.
MSE penalty for that failure? (1 - 0.02)² = 0.96.
Now consider a correct confident prediction — 90% churn, account churns. MSE penalty: (1 - 0.90)² = 0.01.
The ratio between catastrophic failure and correct confidence is barely 100x. That doesn't feel right. And it isn't.
There's also a structural problem. When you combine the logistic function with MSE, the loss surface goes non-convex. Multiple local minima. Flat regions where gradients vanish. Gradient descent gets lost and never finds the global optimum.

Why Cross-Entropy Won

Binary cross-entropy encodes a simple but powerful idea: being confidently wrong is a different category of failure.

Prediction   Actual   CE Loss
0.90         1        -log(0.90) = 0.10   <- minimal
0.50         1        -log(0.50) = 0.69   <- moderate
0.02         1        -log(0.02) = 3.91   <- catastrophic

The penalty isn't linear or quadratic. It's logarithmic which means confident wrong predictions get hit exponentially harder.

Cross-entropy won for three principled reasons:

1. Mathematical harmony. Combine cross-entropy with the logistic function and you get a perfectly convex loss surface. One bowl. One global minimum. Gradient descent always finds it.

2. Maximum Likelihood Estimation. Minimizing cross-entropy is equivalent to maximizing the likelihood of your training data under the model, the most principled framework in statistics, formalized by R.A. Fisher in 1922.

3. Calibrated probabilities. A model trained with cross-entropy learns to produce probabilities that reflect reality. Its 0.8 prediction actually means roughly 80% of similar cases converted.

But Cross-Entropy Is Still Symmetric

Here's what most tutorials skip.

Cross-entropy treats false positives and false negatives equally. Your business almost certainly doesn't.

These are not the same cost. Yet your model punishes them identically by default.

Two levers to fix this:

Lower the threshold from 0.5 to 0.2. Flag more accounts. Miss fewer churners. Accept more false alarms.
Use class weights to explicitly penalise false negatives harder during training.

The model defaults to 0.5 and balanced classes. If you haven't explicitly overridden that, the model made a business decision for you.

Did it make the right one?

Every ML Model You'll Ever Build Follows the Same Three Moves

Vedant Narayan — Tue, 30 Jun 2026 16:30:00 +0000

I spent years jumping between algorithms without seeing the pattern underneath them.
Linear regression. Logistic regression. Gradient boosting. Neural networks. Each one felt like a separate discipline with its own math, its own intuition, its own debugging headaches.
Then one day it clicked: they're all the same three-step architecture wearing different clothes.

Hypothesis → Loss → Optimization

That's it. That's the whole skeleton.
Hypothesis answers one question: what shape do you think the answer takes?

For linear regression, your hypothesis is a straight line: y = wx + b. For a decision tree, it's a series of splits. For a neural network, it's a composition of nonlinear functions stacked in layers. Different shapes, same job — propose a function that maps inputs to outputs.

Loss answers a different question: how wrong is the hypothesis right now?

Squared error for regression. Cross-entropy for classification. Hinge loss for SVMs. You can even design your own custom loss function if your business problem demands it. The loss function is just a formalized way of saying "here's how much this prediction hurts."

Optimization answers the last question: how do we get less wrong?

Gradient descent is the most common answer, but it's not the only one. Some problems have closed-form solutions you can solve directly with linear algebra. Some use evolutionary algorithms. Some literally use trial and error at small scale. The optimizer's job is just to nudge the hypothesis toward lower loss, however that nudging gets implemented.

Why this reframe matters

Once I stopped asking "what algorithm should I use?" and started asking "what's my hypothesis, what's my loss, what's my optimization?", model selection got a lot less mysterious.
It also made debugging faster. If a model isn't learning, I no longer guess randomly. I check each step in order: Is the hypothesis class expressive enough to capture the pattern? Is the loss function actually penalizing the errors that matter for this business problem? Is the optimizer converging, or is it stuck, oscillating, or diverging?
Most "my model isn't working" bugs live in exactly one of those three buckets. Naming the bucket cuts debugging time dramatically.

The pattern goes beyond ML

I've noticed this same three-step logic shows up outside of pure modeling too. Any time a team builds a scoring system — fraud detection, lead scoring, churn prediction — they're implicitly defining a hypothesis (what signals matter), a loss (what counts as a costly mistake), and an optimization process (how the system improves over time).
The vocabulary changes by domain. The underlying architecture doesn't.
The takeaway

Algorithms aren't really different species. They're different instantiations of the same three questions, asked in the same order, every time.
Once you see the skeleton, the muscle is just implementation detail.