Variance looks trivial until you compute it on real data, and the same goes for AUC and for rank correlation. All of them have a textbook formula that produces the wrong number, and they come up again and again in ML and quant coding rounds.
Why the textbook variance formula fails
The variance is E[x²] − E[x]², which is exact as algebra. In floating point, though, it subtracts two large and nearly equal numbers, and the gap between them is where all the information lives. Take four prices near 10⁹: their squares sit near 10¹⁸, where adjacent float64 values are 128 apart, while the true variance is 30. The one-pass textbook formula hands back whatever survives the rounding, which might be 0 or several hundred and can even be negative.
The two-pass method computes the mean first and then the mean of squared deviations, and although it's accurate, it needs to see the data twice. When the data arrives as a stream from a market feed or a training loop, there is no second pass to make.
Welford's algorithm
Keep the count n and the running mean, along with M₂, the sum of squared deviations from the current mean. For a new value x, the update is:
n += 1
delta = x - mean
mean += delta / n
m2 += delta * (x - mean) # old deviation times new deviation
The sample variance is M₂/(n − 1). Because all the stored quantities stay on the scale of the deviations instead of the raw values, nothing cancels catastrophically. The same idea lets you merge two partial results, which is how variance gets computed in parallel across machines or GPU threads.
ROC AUC is a ranking statistic
The area under the ROC curve equals the probability that a positive picked at random is scored higher than a negative picked at random, with ties counted as one half. Dividing the Mann–Whitney U statistic by the number of pairs gives exactly this quantity, so read that way the AUC needs no curve at all:
- Rank all scores together, giving tied scores the average of the ranks they span.
- Add up the positives' ranks, then subtract the smallest that sum could be, n₊(n₊ + 1)/2.
- Divide by n₊n₋.
The sort makes the whole calculation O(n log n), against O(n₊n₋) for comparing all pairs. Handling ties properly matters a great deal here: classifiers that output rounded scores, or only a handful of distinct values, produce ties everywhere, and an implementation that ignores them is biased by the order of the input.
Rank correlation and Spearman
Spearman's correlation is Pearson's correlation applied to ranks, so it measures whether two variables move together monotonically without caring about the shape of the relationship. In quant research it goes by the name information coefficient, meaning the rank correlation between a signal today and returns tomorrow, and most alpha research is judged on that number. Researchers prefer it to Pearson because a single extreme return can't dominate it.
The shortcut is exact only when there are no ties. When ties are present, use average ranks and compute Pearson on them directly.
If this comes up in a coding round
Start by explaining why E[x²] − E[x]² is unstable, and only then fix it. Give the probabilistic meaning of AUC ahead of writing any code, and use average ranks for ties in both AUC and Spearman, since that's where most answers slip. For quant roles it's worth mentioning that Spearman between a signal and tomorrow's returns is the information coefficient.
Originally written for my site: https://geodesicpartners.co.uk/ml-statistics-in-code
Top comments (0)