I built an open-source Python tool to detect drift in embedding spaces
Most monitoring pipelines wait for a downstream metric to break: accuracy drops, retrieval quality slips, or user-facing behavior gets worse.
By then, the shift has already happened.
I wanted a simpler way to catch changes earlier by looking directly at the embeddings themselves.
So I built drift-lens-monitor — an open-source Python package for detecting drift in embedding spaces.
- GitHub: https://github.com/PRAFULREDDYM/Drift_lense
- PyPI: https://pypi.org/project/drift-lens-monitor/
What problem this solves
A lot of modern ML systems depend on embeddings:
- semantic search
- RAG pipelines
- recommenders
- clustering
- classification pipelines with embedding-based features
Even when the raw system looks “healthy,” the embedding space can start changing underneath you.
That change may come from:
- new user behavior
- model updates
- data source changes
- upstream preprocessing differences
- gradual distribution shift over time
If you only monitor business metrics or model accuracy, you often detect the issue late.
The idea behind this project is simple:
take snapshots of embeddings over time and compare them directly.
What drift-lens-monitor includes
The package currently supports three drift detection approaches.
1) Fréchet Embedding Distance (FED)
This is inspired by FID-style comparison, but applied to arbitrary embeddings.
At a high level, it compares the mean and covariance of two embedding distributions.
Useful when you want a compact statistical distance between a reference snapshot and a current snapshot.
2) Maximum Mean Discrepancy (MMD)
This is a kernel-based, non-parametric method for comparing two samples.
I included permutation-based p-values so it can be used not just as a raw distance, but also as a statistical test.
Useful when you want a more flexible distribution comparison without assuming Gaussian structure.
3) Persistent homology
This is the unusual one.
Instead of only asking whether two embedding clouds differ statistically, this looks at whether their shape changes.
It builds topological summaries over the point cloud and compares them using Wasserstein distance.
Why that matters:
A system can preserve rough averages while still changing structurally. For example:
- clusters merge
- clusters split
- holes or loops appear/disappear
- local geometry shifts in ways mean/covariance may not capture
This makes persistent homology an interesting complement to more standard drift metrics.
Design goals
I wanted the tool to stay practical:
- local-first
- no cloud dependency
- no API keys
- simple snapshot storage
- easy to inspect and extend
Snapshots are stored as parquet files, so the workflow stays lightweight and reproducible.
The package can be used:
- as a Python library
- through a Streamlit dashboard for visual exploration
Install
bash
pip install drift-lens-monitor
## Example workflow
The intended workflow is straightforward:
1. Save a reference embedding snapshot
2. Save a new embedding snapshot later
3. Compare them using one or more drift methods
4. Inspect drift scores and visualize the changes
This makes it usable for both experimentation and production-adjacent monitoring workflows.
## Why I built it
I was interested in a gap I keep seeing in ML tooling:
we monitor model outputs, latency, costs, and downstream metrics heavily, but we often do much less direct monitoring of the representation space itself.
Embeddings are doing a huge amount of work in modern AI systems. They deserve first-class monitoring too.
I also wanted to explore whether more unusual techniques like **topological drift detection** could add signal beyond standard statistical distances.
## What I’d love feedback on
I’d especially love feedback on three things:
1. Does persistent homology feel genuinely useful here, or too heavyweight?
2. What baselines or benchmark datasets would make this more convincing?
3. How should the package/API be improved to make it easier to use in real workflows?
## Links
- GitHub: https://github.com/PRAFULREDDYM/Drift_lense
- PyPI: https://pypi.org/project/drift-lens-monitor/
Top comments (0)