A team I worked with cut their embedding storage 12x by slicing text-embedding-3-large vectors from 3072 dims to 256. Recall@10 barely moved. Their dedup pipeline, which flagged anything above cosine 0.92 as a duplicate, started merging unrelated support tickets the same week. Nobody connected the two changes for a month.
Matryoshka embedding truncation is the most under-audited optimization in RAG. It works — that's the problem. The retrieval metric you watch stays flat while every absolute similarity number in your system silently moves, and anything downstream that compares a score to a constant breaks.
TL;DR
-
Matryoshka embedding truncation only works on models trained with Matryoshka Representation Learning (MRL) —
text-embedding-3-*,nomic-embed-text-v1.5,jina-embeddings-v3. Slicing a non-MRL model is noise. - Truncating a unit vector leaves it shorter than unit length, by a per-vector amount. If your index scores with raw inner product, you've turned a direction search into a magnitude popularity contest.
- The cosine noise floor scales as 1/√d. Random-pair similarity has standard deviation ≈ 0.018 at 3072 dims and ≈ 0.0625 at 256 — 3.5x wider. A threshold tuned at full width is a different number of sigmas after truncation.
- Never mix truncation widths in one index. Vectors embedded at 3072 and vectors embedded at 256 are not comparable, and nothing will error.
- The safe pattern is adaptive retrieval: ANN search over truncated vectors, exact rescore of the top ~100 with full-width vectors.
What does Matryoshka training actually do to the coordinates?
MRL (Kusupati et al., 2022) trains one encoder against several nested sub-vectors at once. During training the loss is computed on the first 64 dims, the first 128, the first 256, and so on up to full width, then summed with per-granularity weights.
The gradient pressure is asymmetric on purpose. Coordinate 3 has to carry signal for every granularity; coordinate 3000 only participates in the widest one. The result is an ordering of information density along the axis index — a coarse-to-fine cascade baked into coordinate position.
Two consequences people get wrong:
This is not PCA. The head dimensions are a good subspace, but they are axis-aligned by training, not by eigen-decomposition of your corpus. You cannot recover the effect by slicing a normally-trained bi-encoder, and you cannot improve on it by rotating.
Trained granularities are privileged. If the model was trained at {64, 128, 256, 512, 1024, 3072}, slicing to 200 lands between boundaries. Nesting makes it degrade gracefully rather than catastrophically, but there's no reason to pick a width the loss never saw. Snap to a trained boundary.
Why is renormalization mandatory after truncation?
Because truncation destroys the unit-norm invariant every downstream distance assumes, and it destroys it unevenly across vectors.
Take a unit vector a and keep the first k coordinates as a_k. The retained energy is r_a = ‖a_k‖² ≤ 1. Now the raw dot product against another truncated vector is:
⟨a_k, b_k⟩ = cos(a_k, b_k) · √(r_a · r_b)
Cosine similarity — the thing you actually want — is only the first factor. The √(r_a·r_b) term is a per-document multiplier that has nothing to do with the query. A document whose meaning happens to concentrate in the head dimensions gets a systematic score bonus against every query. That's a hubness generator you built yourself.
And r_a is far from constant. Run this against your own corpus:
import numpy as np
from openai import OpenAI
client = OpenAI()
texts = [...] # a few thousand real chunks from your corpus
resp = client.embeddings.create(model="text-embedding-3-large", input=texts)
full = np.array([d.embedding for d in resp.data]) # (N, 3072), already unit-norm
K = 256
head = full[:, :K]
energy = (head ** 2).sum(axis=1) # retained energy per vector
print(f"retained energy: min={energy.min():.3f} "
f"p50={np.median(energy):.3f} max={energy.max():.3f}")
# Nowhere near constant, and nowhere near the naive 256/3072 = 0.083.
# MRL front-loads the mass, but the *spread* across documents is what bites you.
# WRONG: raw slice into an inner-product index
bad = head
# RIGHT: slice, then renormalize
good = head / np.linalg.norm(head, axis=1, keepdims=True)
The API-side path does this for you:
# Server-side truncation + renormalization. Returns unit-norm 256-dim vectors.
resp = client.embeddings.create(
model="text-embedding-3-large",
input=texts,
dimensions=256,
)
Whether the bug is visible depends entirely on your index's metric:
| Store / metric | Behavior on un-renormalized slices |
|---|---|
FAISS IndexFlatIP
|
Silently magnitude-biased. faiss.normalize_L2() is on you. |
pgvector <#> (inner product) |
Silently magnitude-biased. |
pgvector <=> (cosine) |
Correct ranking, but division cost per comparison. |
Qdrant Distance.COSINE
|
Normalizes at upsert, so ranking is safe. |
pgvector's cosine operator will save you from wrong ranking — it divides by the norms at query time. It will not save you from the threshold problem below, which is a separate failure.
Why does the same 0.92 threshold mean something different at 256 dims?
Because the distribution of similarity between unrelated documents widens as dimensions shrink, so a fixed cutoff catches a different slice of the tail.
For unit vectors drawn uniformly on the sphere in d dimensions, cosine similarity has mean 0 and variance 1/d. So the standard deviation of the null distribution is 1/√d:
| dims | null std (1/√d) |
|---|---|
| 3072 | 0.018 |
| 1024 | 0.031 |
| 512 | 0.044 |
| 256 | 0.0625 |
| 64 | 0.125 |
Real embeddings are anisotropic — they occupy a narrow cone, so the observed mean is well above 0 and the spread is wider than this ideal. Treat these as a floor, not a forecast. The scaling is the point: going 3072 → 256 widens the null spread by √12 ≈ 3.5x.
Everything that compares a similarity to a constant inherits that shift:
- Near-duplicate detection (
> 0.92→ merge) - Retrieval floors (
< 0.75→ "no relevant context found", trigger fallback) - Semantic caching (
> 0.95→ serve the cached answer) - Guardrail classifiers built on centroid distance
Meanwhile Recall@10 doesn't move, because ranking survives truncation much better than calibration does. Your dashboard says everything is fine.
Recalibrate empirically rather than guessing a correction factor:
# Recalibrate a threshold against a negative pool at the new width.
import numpy as np
def calibrate(queries, negatives, target_fpr=1e-4):
"""queries, negatives: unit-norm arrays at the TRUNCATED width."""
scores = queries @ negatives.T # random/mismatched pairs
return float(np.quantile(scores.ravel(), 1 - target_fpr))
tau_256 = calibrate(q256, neg256)
# Recompute this per width. A constant offset from the 3072-dim threshold
# is not a valid substitute -- anisotropy makes the shift corpus-dependent.
Better still, stop using absolute thresholds. Score relative to the retrieved set — (top1 - mean(top2..top20)) / std(top2..top20) — and the calibration shift mostly cancels, because both terms move together.
How do you choose a truncation width without guessing?
Don't choose one. Use both, in two stages — adaptive retrieval.
Store full-width vectors as the source of truth, index truncated ones for the ANN scan, then rescore the shortlist exactly:
-- pgvector: 256-dim indexed column for the scan,
-- 3072-dim column kept unindexed for exact rescoring.
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
body text,
emb_256 vector(256) NOT NULL, -- unit-norm, MRL head slice
emb_full vector(3072) NOT NULL -- unit-norm, full width
);
CREATE INDEX ON chunks USING hnsw (emb_256 vector_cosine_ops)
WITH (m = 16, ef_construction = 128);
-- Stage 1: cheap ANN over 256 dims. Stage 2: exact rescore over 3072.
WITH shortlist AS (
SELECT id, body, emb_full
FROM chunks
ORDER BY emb_256 <=> $1::vector -- $1 = truncated query vector
LIMIT 200
)
SELECT id, body, 1 - (emb_full <=> $2::vector) AS score -- $2 = full query vector
FROM shortlist
ORDER BY emb_full <=> $2::vector
LIMIT 10;
The HNSW index — the expensive part in RAM — holds 256-dim vectors. The 3072-dim column sits on disk, touched only for 200 rows per query. You get most of the memory win and full-width score semantics, which means one threshold, calibrated once, at full width.
Two rules that catch most production incidents:
Never mix widths in one index. Vectors written at 3072 and vectors written at 256 will happily coexist in the same column type if you pad, and every comparison between them is meaningless. Put the width in the index name or a column constraint so a re-embedding job can't half-migrate you.
Re-embed, don't reuse, when changing width. If you switch from manual slicing to the dimensions parameter, the vectors differ — renormalization changes the values. Rebuild the whole corpus.
Does truncation stack with int8 or binary quantization?
Yes, but the error terms compound in the same direction, and truncation should come first. Quantization error is roughly uniform per coordinate; truncation error is concentrated in the fine-grained distinctions the tail dimensions encoded. Together they attack exactly the near-duplicate regime where your thresholds live.
If you're combining them, keep full-width float vectors for the rescore stage regardless. Rescoring against quantized truncated vectors gives you the compression of both and the precision of neither.
So why does Matryoshka embedding truncation break your thresholds?
Truncating an MRL embedding preserves ranking well — the head dimensions were trained to be a self-sufficient coarse representation — but it changes two absolute quantities your code depends on. First, the truncated vector is no longer unit-norm, and the retained energy varies per document, so any raw inner-product index acquires a query-independent magnitude bias; renormalize after slicing, or let the API's dimensions parameter do it. Second, the similarity noise floor scales as 1/√d, so cutting 3072 → 256 widens the random-pair spread about 3.5x and every hardcoded cutoff — dedup at 0.92, cache hits at 0.95, relevance floors at 0.75 — silently changes meaning. Recall@10 won't show you either failure. Recalibrate thresholds per width against a real negative pool, or keep full-width vectors for a rescore stage and threshold there.
Top comments (0)