Image similarity search is an embedding lookup with a specific and under-discussed problem: the notion of similarity is fixed by whoever trained the encoder, months before you had a catalogue, and it is rarely the notion your users have.
The pipeline, in four steps
Every implementation is the same four steps, whatever the vendor diagram says. Encode each catalogue image to a fixed-length float vector. Normalise the vectors to unit length. Build an approximate nearest-neighbour index over them. At query time, encode the query the same way and ask the index for the k nearest.
The normalisation step is worth being deliberate about rather than copying. Once every vector has length 1, cosine similarity and the negated squared Euclidean distance produce identical rankings, because ||a - b||² = 2 - 2·(a·b) when both are unit vectors. That means an index built for inner product and one built for L2 give the same top-k, and you can stop worrying about which metric your database defaults to — but only if you actually normalised. Skip it and the two disagree, in a way that shows up as a handful of strange results rather than as an error. The wider treatment is in how the similarity metrics differ.
What “similar” means is decided by the encoder
There is no general-purpose image similarity. There is only the geometry that a particular training objective produced, and the three common objectives produce three different geometries:
- Supervised classification features. Take the penultimate layer of an ImageNet classifier. The space is organised by category, because that is what the loss rewarded. Two black leather armchairs at different angles land near each other; a black armchair and a black sofa land near each other too, and a black armchair and a red one may not.
- Self-supervised features in the DINO line. Trained to make two augmented crops of the same image agree, so the space is organised by visual appearance and layout. Better for “find me this exact object again”, weaker for “find me other things of this kind”.
- Contrastive image-text features in the CLIP line. Trained to align an image with its caption, so the space is organised by what a caption would say — which means text queries work in the same index, and also that two images captioned similarly are near each other even when they look nothing alike.
Pick by the query you expect. A duplicate-detection job and a “more like this” recommendation job want opposite ends of that list, and running one with the other’s encoder produces results that are not wrong so much as answering a different question.
A cosine nearest-neighbour, worked
Four dimensions again, for legibility. One query and three catalogue vectors:
q = (0.6, 0.5, 0.4, 0.3) |q| = sqrt(0.86) = 0.9274
c1 = (0.7, 0.4, 0.5, 0.2)
dot = 0.42 + 0.20 + 0.20 + 0.06 = 0.88
|c1| = sqrt(0.94) = 0.9695
cos = 0.88 / (0.9274 * 0.9695) = 0.88 / 0.8991 = 0.979
c2 = (0.1, 0.9, 0.2, 0.4)
dot = 0.06 + 0.45 + 0.08 + 0.12 = 0.71
|c2| = sqrt(1.02) = 1.0100
cos = 0.71 / (0.9274 * 1.0100) = 0.71 / 0.9366 = 0.758
c3 = (1.2, 1.0, 0.8, 0.6) <- q scaled by 2
dot = 0.72 + 0.50 + 0.32 + 0.18 = 1.72
|c3| = 2 * 0.9274 = 1.8548
cos = 1.72 / (0.9274 * 1.8548) = 1.72 / 1.7200 = 1.000
ranking: c3 (1.000), c1 (0.979), c2 (0.758)
The c3 line is the property to internalise: cosine is invariant to vector magnitude, so a vector pointing the same way at twice the length is a perfect match. In an image index this is why brightness or contrast changes that scale a feature vector without rotating it cost you nothing, and it is also why a magnitude signal you might have wanted — some encoders put a rough confidence or saliency into the norm — is thrown away the moment you normalise.
Why you need an index at 100,000 images
Exact search is a full scan: for a 768-dimensional index, one query against N vectors is N dot products of 768 multiply-accumulates. At 100,000 images that is 76.8 million operations per query, which a single core does in tens of milliseconds and which does not survive concurrency. At 10 million images it is 7.68 billion, and exact search stops being an option.
Approximate indexes trade a small amount of recall for a large speedup. HNSW builds a navigable graph and walks it, giving you two knobs: M, the graph degree, which is fixed at build time and costs memory; and efSearch, the size of the candidate list at query time, which trades latency for recall and can be changed per query. The mechanism and the parameter behaviour are covered in how HNSW actually works. The important discipline is to measure recall@10 against an exact scan on a sample of a few thousand queries before you tune anything, because an approximate index degrades silently — the results still look plausible when they are missing the best match.
Storage arithmetic is worth doing early, since it is what decides whether the index fits in memory. Ten million images at 768 dimensions in float32 is 10,000,000 × 768 × 4 bytes = 30.7 GB before the graph, and HNSW’s links add roughly M × 2 four-byte ids per node per layer. Halving to float16 or moving to a product-quantised representation is the standard response, and it costs recall in a way you should measure rather than assume.
One structural point that catches people late: filtering. As soon as you want “similar images, but only in stock and only in this category”, the order of operations matters. Filtering after the search returns fewer than k results whenever the filter is selective, because the index already committed to its candidates — ask for 10 and receive 2. Filtering during the graph traversal preserves k but slows the walk down, sometimes badly, since the traversal keeps landing on nodes it must discard. A highly selective filter is often better served by an exact scan over the small filtered subset than by the index at all, and knowing which regime you are in requires knowing the selectivity of your typical filter before you choose a database.
The four failures that show up in production
- The background wins. A whole-image embedding summarises the whole image. A catalogue photographed on white and a user photograph taken on a kitchen table are far apart largely because of the table. Detecting and cropping the object first, then embedding the crop, is usually a bigger accuracy gain than changing encoder.
- Near-duplicates flood the top-k. Eight photos of one product return as eight results. The fix is not a better index but a deduplication or diversification pass — group results above a similarity threshold and emit one per group.
- The threshold does not transfer. A cosine cutoff of 0.82 tuned on one catalogue means nothing on another, because the distribution of similarities depends on how visually homogeneous the corpus is. Calibrate cutoffs per corpus, on labelled pairs.
- Re-embedding is a migration. Changing encoder changes every vector, and a mixed index is meaningless — vectors from two encoders are not comparable even at the same dimensionality. Plan a full rebuild with a dual-write window, not an incremental swap.
Embedding dimensionalities, the availability of hosted image-embedding endpoints and the specific model families named above all move. Treat the mechanism as stable and re-check the model landscape before committing to one.
Top comments (0)