TL;DR — Most ML observability stacks still monitor embeddings the way they monitor table columns: one statistical test per dimension. That approach is mathematically unsound in high-dimensional vector space, where marginal stability can coexist with catastrophic structural shift. Real embedding observability has to test geometry, not axes.
Every data quality tool you've ever used was built for tables. Nulls, ranges, cardinality, a Kolmogorov-Smirnov test on a numeric column, a population stability index on a categorical one. These tools were designed in a world where "the data" meant rows and columns with human-legible meaning. Then embeddings showed up, and almost nobody rebuilt the toolchain. They just pointed the same column-level tests at a 768- or 1536-dimensional vector and called it monitored.
That's the gap. It's not that teams aren't watching their embeddings. It's that the way they're watching them is statistically unsound, and it produces exactly the kind of false confidence that lets a model's behavior change dramatically while every dashboard stays green.
The Marginal Distribution Trap
The standard approach to embedding drift looks like this: take the vector, treat each dimension as an independent feature, run a two-sample test — KS, PSI, whatever — on each dimension against a reference window, and alert if enough dimensions cross a threshold. This is a direct transplant of tabular monitoring logic, and it fails for a specific, well-understood reason: marginal distributions do not determine a joint distribution.
You can rotate, shear, or reshape a high-dimensional point cloud substantially while leaving every individual axis's histogram almost untouched. Embedding spaces are exactly this kind of object. A fine-tune, a model version bump, or a shift in the underlying data population can change the correlation structure between dimensions — the thing that actually encodes semantic relationships — without moving any single dimension's mean or variance enough to trip a per-axis test.
So you get the worst outcome: the monitoring system reports stability, and the retrieval or classification quality downstream has quietly degraded, because the thing that mattered was never on any single axis to begin with.
Rotation Without Movement
There's a more subtle failure mode that's specific to embedding spaces, and it's worth sitting with because it's counterintuitive to anyone trained on tabular statistics: cosine similarity and nearest-neighbor structure are preserved under rotation, but axis-aligned statistics are not required to be preserved under the inverse. In other words, two embedding spaces can be functionally identical for retrieval purposes while every per-dimension KS test screams drift — and two embedding spaces can be functionally very different for retrieval while every per-dimension test says nothing changed.
This isn't a hypothetical edge case. It happens routinely when an embedding model is re-trained or swapped, when a tokenizer upstream changes, or when the input distribution shifts in a way that reorganizes clusters without moving centroids along the original coordinate axes. Per-dimension tests were never designed to detect this, because they assume the axes themselves carry stable meaning. In an embedding space, the axes are an arbitrary basis. There's no reason to expect semantic change to announce itself along any one of them.
The Multiple Testing Problem Nobody Accounts For
Even setting geometry aside, there's a basic statistics problem baked into "test every dimension independently." Run 768 independent hypothesis tests at a 0.05 significance level and you'd expect roughly 38 false positives by chance alone, before any real drift exists. Teams notice the alert fatigue and respond the obvious way: they tighten thresholds, require a higher fraction of dimensions to flag, or apply a crude multiple-testing correction. That reduces noise, but it also reduces sensitivity to real, structurally meaningful shifts that might only show up strongly in a handful of dimensions. You end up tuning the detector to ignore the exact signal it was supposed to catch, because the test battery was the wrong shape for the problem from the start.
What Actually Needs Watching
If marginal, per-axis tests are the wrong primitive, what replaces them? The honest answer is that embedding observability needs tests that respect the geometry of the space rather than its arbitrary coordinate system. A few approaches are genuinely better suited to this:
Distribution-level distance tests like Maximum Mean Discrepancy compare entire distributions in a way that's sensitive to joint structure, not just per-axis marginals, and don't require you to pick which dimension matters in advance.
Local density tracking around fixed reference points. Pick a stable set of anchor embeddings — canonical examples whose semantics you trust — and monitor how the local neighborhood density and nearest-neighbor composition around those anchors changes over time. This catches cluster drift that no per-axis test can see.
Centroid stability for known clusters. If you can define semantically me
Top comments (0)