Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
In any sort of AI application, we have the concept of vectors.
Vectors come in different shapes and sizes.
Especially when they are large, we need a good way to manage them.
One way to solve this is by compressing these large vectors.
This is especially relevant to FAISS, which I explained in another article of mine, where it is a library for efficient similarity search over large collections of vectors. Product Quantization is one of the techniques FAISS supports for vector compression and efficient search.
Let's visualize this with a problem.
The Problem
Suppose you have a customer support RAG app
And for that, you have 10 million embeddings containing chunks from product documentation, FAQs, and support articles.
Each of these embeddings has 768 dimensions. Each of those 768 dimensions is used to store a numerical representation of the meaning of that text.
Now, each dimension is a float, so we can think of it as a 32-bit float.
32 bits means 4 bytes.
So:
768 × 4 = 3072 bytes
That's roughly 3 KB per 768-dimensional vector.
Now let's scale this to 10 million embeddings:
10,000,000 × 3 KB = 30 GB
So this 3 KB per vector becomes 30 GB when scaled to 10 million embeddings.
30 GB is roughly the size of a decent-sized game or several high-quality movies.
So imagine searching among this much vector data. It can get expensive.
Let's look at the numbers again.
We have 10 million embeddings. We can't change that.
But we have 3 KB per vector.
What if we could compress that 3 KB further, so that when we scale it to millions of embeddings, we get a much lower storage requirement?
The technique used to do this is called Product Quantization, or PQ.
Let's look at the basic idea behind PQ.
The Basic Idea
Imagine I have this vector:
[0.21, 0.73, -0.15, 0.91, 0.34, -0.52, 0.18, 0.66]
Instead of storing all these floating-point numbers, PQ:
- Splits the vector into smaller pieces
- Learns representative patterns for each piece
- Replaces each piece with the ID of its closest codeword
Let's see an example.
Suppose we split this 8-dimensional vector into 4 pieces:
[0.21, 0.73] [-0.15, 0.91] [0.34, -0.52] [0.18, 0.66]
Now, based on these subvectors, we have a separate codebook for each subvector position.
For the first position, let's assume this is the codebook:
Codebook 1
ID 0 → [0.10, 0.70]
ID 1 → [0.20, 0.75]
ID 2 → [0.80, 0.10]
ID 3 → [-0.20, 0.90]
Now let's match the first subvector to its closest vector in the codebook.
Our first subvector is:
[0.21, 0.73]
This looks pretty close to ID 1, which is:
[0.20, 0.75]
So instead of storing the entire subvector, we store:
1
Similarly, we do the same thing for the other subvectors.
For:
[-0.15, 0.91]
we have another codebook:
Codebook 2
ID 0 → [-0.30, 0.80]
ID 1 → [0.10, 0.40]
ID 2 → [-0.20, 0.90]
ID 3 → [0.70, 0.20]
The closest one is ID 2.
And we repeat this process for the remaining subvectors.
So the original vector:
[0.21, 0.73] [-0.15, 0.91] [0.34, -0.52] [0.18, 0.66]
can be represented as:
[1, 2, 3, 0]
Each number is an index into a different codebook.
Original vector
│
┌───────────────┼───────────────┐
↓ ↓ ↓
Subvector 1 Subvector 2 Subvector 3 ...
│ │ │
↓ ↓ ↓
Codebook 1 Codebook 2 Codebook 3
│ │ │
↓ ↓ ↓
ID 17 ID 42 ID 8
The Compression
So now we have the vector in this form:
[1, 2, 3, 0]
Each ID can be stored using just a few bits.
For this example, let's assume each ID is stored using 1 byte.
(With only 4 possible IDs, 2 bits would technically be enough, but we'll use 1 byte here to keep the example simple.)
We have 4 subvectors:
4 × 1 byte = 4 bytes
Originally, we had 8 floating-point numbers:
8 × 4 bytes = 32 bytes
So the size went down from 32 bytes to 4 bytes.
And remember, this is just an 8-dimensional vector.
Real embeddings are often hundreds or thousands of dimensions.
For example, a 768-dimensional embedding stored as 32-bit floats requires:
768 × 4 = 3072 bytes
With PQ, if we split it into 96 subvectors and store one 8-bit code for each subvector:
96 × 1 byte = 96 bytes
That's 32× less storage for the vector representations.
When you're dealing with millions or billions of vectors, that difference becomes enormous.
And that's the real reason Product Quantization is so useful for large-scale vector search.
Wrapping It Up
At its core, Product Quantization is a clever way of saying:
Instead of storing every number in a vector, split the vector into smaller pieces and represent each piece by the ID of its closest learned codeword.
You lose some precision, but in return, you get dramatically smaller vectors and more efficient approximate distance calculations.
So that's about this article.
See you in the next one!
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase:
Top comments (0)