Logistic Regression From Scratch: the Same Gradient Descent, Squashed Through a Sigmoid

Day 1 of MachineLearningFromZero built linear regression (predict a number). Day 2 turns it into a classifier with one change: a sigmoid. Spam-or-not, sick-or-healthy, churn-or-stay — that's classification, and logistic regression is the simplest model for it.

The sigmoid squashes to a probability

Take the same linear score w·x + b and pass it through the sigmoid:

const sigmoid = z => 1 / (1 + Math.exp(-z));
const prob = x => sigmoid(w1*x[0] + w2*x[1] + b);  // 0..1

Big positive → ~1, big negative → ~0, zero → 0.5. Now the output reads as "probability of class 1".

Log-loss, not squared error

Probabilities need cross-entropy (log-loss), which punishes confident wrong answers hard:

const loss = -(y*Math.log(p) + (1-y)*Math.log(1-p));

The gradient is the SAME shape

Here's the beautiful part. Differentiate log-loss through the sigmoid and it simplifies to (prediction − actual) × input — exactly the form from linear regression:

const err = prob(x) - y;   // y ∈ {0,1}
dw1 += err * x[0];
dw2 += err * x[1];
db  += err;

So the training loop is identical; only the prediction function changed.

Classify at 0.5

const label = prob(x) >= 0.5 ? 1 : 0;

Where p = 0.5 is exactly where w·x + b = 0 — a straight line. Logistic regression draws a linear boundary but reports calibrated confidence around it.

Why it matters

It's fast, interpretable, and calibrated — the workhorse classifier. It's also literally a single neuron with a sigmoid, the exact unit you stack into neural networks.

🔵 Watch it train (background shades by probability): https://dev48v.infy.uk/ml/day2-logistic-regression.html

Day 2 of MachineLearningFromZero.