Ordinary least squares has no defence against a far-off point. Because it minimises the sum of squared residuals, a point twice as far off pays four times the penalty, and one ten times as far pays a hundred — so the fitted line is dragged toward whichever point is worst. Its breakdown point is zero: a single wild observation can ruin the fit. Huber regression keeps least-squares' smooth, convex world but caps that runaway growth. Here's how I built it from scratch — it's barely more code than OLS.
Quadratic inside the knee, linear outside
The whole idea is one loss function. While a residual stays inside a knee |r| ≤ δ it's penalised by the ordinary squared loss ½r² — so on clean, near-Gaussian noise Huber behaves exactly like OLS and keeps its efficiency. Past the knee it switches to a linear penalty δ(|r| − ½δ), chosen so the two pieces meet smoothly in both value and slope. A point ten times δ away now costs about 10δ² instead of 50δ²; the runaway squared growth is simply gone.
function huberLoss(r, d){ // d = delta, the knee
const a = Math.abs(r);
return a <= d ? 0.5 * r * r // quadratic core (like OLS)
: d * (a - 0.5 * d); // linear tail (bounded growth)
}
From loss to weights
The linear tail matters because of what it does to influence — the derivative of the loss, how hard a point pulls on the fit. OLS's influence grows without bound: a point's pull is proportional to its distance, so the farther it strays the more it dominates. Huber's influence is capped at ±δ — beyond the knee a point pulls with a constant force, not one that explodes. Divide that influence by the residual and you get a per-point weight: 1 inside the knee, then decaying as δ/|r| in the tails.
function weight(r, d){ return Math.min(1, d / Math.abs(r)); }
Fit by iteratively reweighted least squares
There's no magic closed form — you fit it by IRLS. Start from a plain least-squares line, then refit an ordinary weighted least-squares line where each point's weight is min(1, δ/|r|), recompute the residuals, recompute the weights, and repeat until it settles. Full weight inside the knee, gently discounted outside — so the outliers fade with each pass and pull less, while every point still counts and none is thrown away. It usually converges in a handful of iterations.
let [a, b] = ols(xs, ys); // start from plain least squares
for (let it = 0; it < 40; it++){
const w = xs.map((x, i) => weight(ys[i] - (a + b*x), delta));
[a, b] = weightedLeastSquares(xs, ys, w); // refit with the new weights
}
Setting δ
δ is the dial that sets the knee, and you set it from the noise scale, not by eye. A common choice is 1.345 × σ (with σ a robust estimate of spread like the scaled MAD), which recovers about 95% of OLS's efficiency on clean Gaussian data while still clipping the tails. Scale-standardise the residuals first so δ means the same thing regardless of the units your data happens to be in. Too small a δ and you throw away efficiency on the clean points; too large and the tails never kick in and you're back to OLS.
Seeing the two loss curves
Plot the Huber loss against the squared loss and the whole story is in one picture: inside ±δ the green Huber curve hugs the parabola, then peels off into a straight line. That flatter tail is why a far point costs far less. Flip to the influence curve ψ(r) and you see OLS's grow without bound while Huber's flattens to a constant at ±δ; flip to the weight w(r) and you're looking at exactly the min(1, δ/|r|) that IRLS reweights with. Three views of the same knee.
Where Huber sits — between OLS and RANSAC
Large δ and Huber is OLS; small δ and it leans toward a fully-robust fit. Because it uses every point with one smooth loss — rather than throwing points away and sampling for a consensus like RANSAC — Huber sits between least-squares and a hard-robust estimator, and where it lands depends on δ. That's also its limit: pile on enough outliers, or make them high-leverage in x, and even Huber bends. Its breakdown point is near zero, which is exactly where RANSAC takes over.
Scatter some outliers, slide δ, and watch IRLS reweight and converge live — Huber vs the dragged OLS line:
Top comments (0)