“A quantized embedding model” describes two unrelated operations with different costs, different benefits and different failure modes. One of them has a published quality figure. The other does not, and the honest version of this page is the measurement procedure rather than a number.
Two different things with one name
- Weight quantization. The model’s parameters are stored at lower precision — int8 instead of fp32 — so the model file shrinks and the forward pass reads fewer bytes. The output is still a vector of floats, of the same dimension, slightly different from what the fp32 model would have produced. This is what an int8 ONNX export or a q8_0 GGUF does.
- Vector quantization. The model is untouched; the output vectors are compressed after the fact, from fp32 to int8 or to single bits per dimension. The index shrinks, search gets faster, and the model runs at full precision throughout.
They compose — you can run an int8 model and store binary vectors — and they are frequently conflated in writing that says “we quantized our embeddings and saved 32x”, a saving only the second operation produces. Which one you want depends on what is expensive: weight quantization if inference throughput or model footprint is the constraint, vector quantization if the index is.
Quantizing the vectors, where a figure exists
Sentence Transformers documents this directly and publishes retention figures. Its embedding quantization guide states that binary quantization with a rescoring step preserves up to about 96% of total retrieval performance, at a 32x reduction in memory and storage and up to 32x faster retrieval. The worked example on that page reports 41 million texts served in 5.2 GB of memory and 52 GB of disk under a combined binary-plus-int8 scheme, against 200 GB of each for fp32.
The 32x is arithmetic you can check: fp32 is 32 bits per dimension and binary is one, so a 768-dimensional vector goes from 3,072 bytes to 96 bytes. Scalar int8 is 4x by the same reasoning, 3,072 bytes to 768.
Two conditions attach to the 96%, and neither is optional. The first is rescoring: you retrieve a large candidate set with cheap binary comparisons, then re-rank the top few hundred with full-precision or int8 vectors. Binary retrieval without rescoring loses considerably more. The second is calibration — for scalar quantization the library needs representative embeddings to establish the per-dimension minimum and maximum that define the buckets, and its documentation warns that the calibration dataset has a large influence on performance. Calibrating on a sample from your own corpus is part of the method, not a refinement of it.
from sentence_transformers.quantization import quantize_embeddings
calib = model.encode(corpus_sample, normalize_embeddings=True)
int8 = quantize_embeddings(full, precision="int8",
calibration_embeddings=calib)
binary = quantize_embeddings(full, precision="binary")
Quantizing the weights, where none does
For weight quantization there is no published number that transfers, and there is a structural reason: the drift depends on the model, the quantization scheme, the calibration data and — critically — on your text. Quantization error concentrates in weights with wide dynamic range, and which inputs excite those weights is a property of your domain. A figure measured on general web text tells you little about a corpus of chemical nomenclature.
The mechanism is worth holding on to because it predicts where you will see problems. int8 quantization maps a range of float values onto 256 levels using a scale per tensor or per channel. Where the distribution of values in a tensor is tight, the levels are close together and the error is tiny. Where a handful of outlier activations are orders of magnitude larger than the rest, the scale must stretch to cover them and every ordinary value collapses into a few levels. Transformers are known to develop exactly those outlier features in specific dimensions, which is why quantization quality is not a smooth function of bit width and why per-channel schemes exist.
The observable consequence for retrieval is not uniform noise. It is that a small number of documents move a lot while most move barely at all — so a mean cosine drift of 0.999 can coexist with a handful of documents that have left their neighbourhood entirely.
The method
Embed the same corpus twice, once at full precision and once quantized, and compare. Nothing here has been run; this is the script, and the numbers it prints are yours.
import numpy as np
from scipy.stats import spearmanr
from sentence_transformers import SentenceTransformer
texts = [l.strip() for l in open("corpus_sample.txt")][:2000]
fp = SentenceTransformer("BAAI/bge-base-en-v1.5")
q8 = SentenceTransformer("BAAI/bge-base-en-v1.5", backend="onnx",
model_kwargs=dict(file_name="onnx/model_qint8_avx512_vnni.onnx"))
A = fp.encode(texts, normalize_embeddings=True, batch_size=32)
B = q8.encode(texts, normalize_embeddings=True, batch_size=32)
# 1. per-document drift: how far each vector moved
cos = (A * B).sum(axis=1)
print("mean", cos.mean(), "p1", np.percentile(cos, 1), "min", cos.min())
# 2. geometry: did the pairwise structure survive?
iu = np.triu_indices(400, k=1)
sa = (A[:400] @ A[:400].T)[iu]
sb = (B[:400] @ B[:400].T)[iu]
print("spearman", spearmanr(sa, sb).statistic)
# 3. the one that matters: does retrieval return the same documents?
qs = texts[:200]
QA, QB = fp.encode(qs, normalize_embeddings=True), q8.encode(qs, normalize_embeddings=True)
ta, tb = np.argsort(-(QA @ A.T))[:, :10], np.argsort(-(QB @ B.T))[:, :10]
print("recall@10 overlap",
np.mean([len(set(x) & set(y)) / 10 for x, y in zip(ta, tb)]))
What to report, and why cosine alone lies
The three numbers that script prints answer three different questions, and only the third is the one your users experience.
- Per-document cosine drift tells you how far individual vectors moved. Report the 1st percentile and the minimum, not the mean — the mean is dominated by the documents that did not move, and the tail is the whole story.
- Spearman correlation of pairwise similarities tells you whether the geometry survived. This catches the case cosine drift misses: if every vector rotated by the same small amount, each one moved but nothing about their relative arrangement changed, and retrieval is unaffected.
- Top-k overlap is the answer. Retrieval only cares about ordering near the top, and a 0.997 mean cosine that reshuffles positions 3 through 8 has changed what your system returns. Conversely a 0.95 mean cosine with 99% top-10 overlap has cost you nothing.
There is one comparison the script cannot make and you should not skip: quantized against fp32 measures agreement, not quality. It is possible for a quantized model to disagree with its parent and be no worse on a labelled evaluation set — the fp32 model is not ground truth. If you have judged relevance data, run both against it. If you do not, agreement is a reasonable proxy and you should say so when you report it.
Whichever way it comes out, the operational rule is the same: an index is built at one precision and queried at the same precision. Changing the numerics means re-embedding the corpus, exactly like changing the model.
Top comments (0)