When you're building a RAG system or semantic search pipeline, one of the first problems you hit is the size of your embeddings. A single embedding from a modern model can have 768, 1024, or even 1536 dimensions. When you're storing millions of them in a vector database, those dimensions translate directly into storage costs, memory usage, and query latency.
The AI community has converged on two main approaches to solve this: Matryoshka Representation Learning (MRL) and Principal Component Analysis (PCA). But which one actually works better?
A recent experiment by Dylan Castillo compared both methods across eight standard BEIR retrieval datasets — and the results are surprising.
What Are Matryoshka Embeddings?
Named after the Russian nesting dolls, Matryoshka Representation Learning trains the model so that the most important information is packed into the first few dimensions. During training, the loss function is applied at multiple prefix lengths simultaneously — 64, 128, 256, and so on — teaching the model to make the first N dimensions meaningful on their own.
At inference time, you simply truncate to the first d dimensions and re-normalize. No extra storage, no transformation matrix, no versioning headaches.
The catch? The model has to be trained this way from the start. Older models like BERT, sentence-transformers, and many popular embedding models don't support MRL truncation.
What Is PCA?
PCA is a classical statistics technique that's been around since 1901. You take a sample of your embeddings, find the directions of maximum variance, and project onto the top d components.
The advantage: PCA works with any model, regardless of how it was trained. The disadvantage: you need to store and version the PCA transformation matrix, and apply it consistently when both adding to and querying the index.
The Experiment
Castillo generated embeddings via OpenRouter and evaluated retrieval quality on eight BEIR datasets:
- SciFact (5.2K docs, 300 queries)
- NFCorpus (3.6K docs, 323 queries)
- ArguAna (8.5K docs, 216 queries)
- FiQA (43K docs, 648 queries)
- SciDocs (25K docs, 93 queries)
- Quora (522K docs, 6,984 queries)
- TREC-COVID (171K docs, 50 queries)
- Webis-Touché 2020 (382K docs, 49 queries)
Each dataset's embeddings were reduced to 512, 256, 128, 64, and 32 dimensions using both methods, then nDCG@10 was measured.
The Results
MRL wins at moderate reductions (512, 256, 128)
At 256 dimensions, MRL preserves more retrieval quality than PCA across most datasets. This makes sense — the model was explicitly trained to keep the first 256 dimensions informative.
PCA catches up at extreme reductions (64, 32)
Here's the surprise: at 32 dimensions, PCA often matches or slightly beats MRL. At extreme compression, the PCA's ability to find the truly optimal projection direction gives it an edge over MRL's fixed prefix.
The fitting data matters
PCA's quality depends heavily on the data used to fit the transformation. Using domain-specific fitting data improves results, but using mismatched data (e.g., fitting on medical text, querying on financial text) can hurt.
Quantization adds another dimension
When combined with quantization (reducing float precision), the gap between MRL and PCA narrows further. At int8 quantization, both methods lose similar amounts of quality.
Practical Takeaways
If your model supports MRL, use it for moderate reductions. It's simpler (no transformation matrix to store), and it performs well at 256+ dimensions.
For extreme compression (≤64 dimensions), consider PCA. It may squeeze out better quality, especially if you can fit it on domain-relevant data.
For legacy models without MRL support, PCA is your only option — and it works surprisingly well.
Don't forget quantization. Combining dimension reduction with int8 or binary quantization can reduce storage by 10-100x with modest quality loss.
Benchmark on your own data. BEIR results are a good starting point, but your specific use case may favor one method over the other.
Why This Matters
As LLM-powered applications scale, the cost of vector databases becomes a real bottleneck. Pinecone, Weaviate, Qdrant, and other vector databases all charge by stored dimensions. Reducing from 1024 to 256 dimensions cuts storage costs by 4x — and if you can do it without losing retrieval quality, that's pure savings.
The full code and data from the experiment are available on GitHub.
Based on original research by Dylan Castillo. HN discussion at 42 points.
Top comments (0)