Three different things get called drift and they need three different fixes. Covariate shift moves p(x) and leaves p(y|x) alone. Label shift moves the class balance. Concept drift leaves p(x) untouched and changes p(y|x). Every monitor in production watches the feature distribution, which is to say it watches x and never looks at y.
The generator keeps those two objects apart, so the best predictor for either domain and the true risk of any predictor are available in closed form. Nothing here is scored against a test set.
domain = { mu, sd } // this IS p(x)
concept = { a, b, c } // this IS p(y|x): y = a + b*x + c*x^2 + eps
Computed live over 25 seeds: https://dev48.infy.uk/ml/day67-distribution-shift.html
One alarm, two orders of magnitude
Two scenarios are handed literally the same feature arrays, so their detector columns agree to the last bit rather than by luck:
| scenario | KS | domain AUC | PSI | true excess risk |
|---|---|---|---|---|
| covariate shift, model correct | 0.562 | 0.853 | 2.043 | 0.0120 |
| covariate shift, model wrong | 0.562 | 0.853 | 2.043 | 5.2212 |
| concept drift, p(x) untouched | 0.059 | 0.493 | 0.046 | 3.1988 |
A factor of 436 behind an identical alarm - and the third row, the expensive one, sits at every detector's noise floor.
If your model class contains the truth, least squares is consistent under any p(x) with full support: shifting the features costs variance, which is O(1/n) and vanishes with data. Misspecify it and "the best line" stops being one thing - under p_train the best slope is 2.00, under p_test it is 3.80, and least squares converges to the first one however much data you hand it, forever.
So harm is how far p(x) moved times how wrong the model is, and every detector measures the first factor only.
Reweighting pays, or bills you
Weight each row by w(x) = p_test(x) / p_train(x), exact here because both densities are known Gaussians, so what is measured is the method. At n = 3,200:
| model | plain | weighted |
|---|---|---|
| correct (c = 0) | 0.0007 | 0.0111 |
| wrong (c = 0.6) | 5.0807 | 0.0694 |
16.7x worse on the correctly specified model and 73x better on the wrong one, from the same weights and the same rows. Under concept drift it is not even a trade: p(x) did not move, so every weight is exactly 1.0 and the corrected fit is bit-for-bit the uncorrected fit. The page asserts that rather than illustrating it.
The claim of mine that did not survive
I asserted the ESS diagnostic is optimistic everywhere. It is not. Kish's ESS is essentially exact while the weights are healthy - 77.84% against a true 77.88% at half a sigma - and only then fails upwards: 1.5x too high at 1.5 sigma, 3.6x at 2, 160x at 3. The claim had to be scoped to large shift, which is precisely where it is load-bearing.
Past sigma_test^2 > 2*sigma_train^2 it is worse than optimistic: the weights have infinite variance, there is no valid standard error at all, and the sample still returns a comfortable 55.6% for a quantity that is undefined.
The friendly one, and the one that ends the story
Label shift needs no test labels: TPR and FPR are properties of p(x|y), so a true prior of 0.05 comes back as 0.0516 from unlabelled features and moving the threshold buys 11.8 points. Concept drift is detectable only with labels, and the honest response there is to build the labelling loop rather than buy a better feature monitor. A quiet monitor is not evidence that anything is fine.
Part of a from-scratch series - one ML idea a day, computed rather than quoted: https://dev48.infy.uk/machinelearningfromzero.php
Top comments (0)