Train an image classifier the usual way and you're locked to a fixed list of labels. CLIP throws that out. Instead of labels it learns from captions — the raw (image, caption) pairs the web already has by the hundreds of millions (about 400M) — by training two encoders at once so that a picture and its caption land at the same point in one shared space. No hand-labelled classes at all: the caption is the supervision. Here's the idea, built up from the pieces.
Two encoders into one space
There's an image encoder (a ViT or ResNet) and a text encoder (a transformer). Each maps its input to a vector; both are L2-normalized into the same embedding space, so a similarity is just a cosine. Nothing about the two towers is shared except the space they aim at — they don't even have to be the same size internally, only project to the same dimension at the end. The training signal is what forces a picture and its caption to the same location while spreading different concepts apart. Untrained, the vectors are noise; trained, each image sits on top of its own caption.
The N×N similarity matrix
Take a batch of N (image, caption) pairs. Embed all N images and all N captions, then form the N×N matrix of cosine similarities between every image and every caption. Exactly N of those N² cells are true pairs — they sit on the diagonal — and the other N²−N cells are mismatches, the negatives. The whole loss lives in this matrix.
The contrastive loss: make the diagonal win
The loss is dead simple to state: make each diagonal cell the largest entry in its row and in its column. Pull each image toward its own caption, push it away from every other caption in the batch, symmetrically — a softmax cross-entropy over the rows (image→caption) and another over the columns (caption→image), averaged. Bigger batches mean more negatives per step, which is a large part of why CLIP wanted enormous batches. As training proceeds the diagonal turns hot and the loss falls.
There's an elegance to it: the other items in the batch are the negatives, for free. You never have to mine hard negatives or label anything as "wrong" — every non-matching caption in the same batch is automatically a thing this image should sit far from. That's what makes the objective scale to 400M web pairs with no annotation pipeline behind it.
Temperature
Before the softmax, the similarities are scaled by a learned temperature τ (the logit scale is 1/τ). Lower τ sharpens the softmax and pushes harder on the negatives; too low and training destabilises. CLIP learns τ rather than fixing it, clamping it to a safe range — one scalar that controls how aggressively the diagonal has to beat everything else in its row and column.
Zero-shot classification for free
Here's the payoff. To classify a brand-new image against labels you never trained on, write each label as a sentence — "a photo of a cat", "a photo of a dog" — encode those prompts with the text tower, encode the image with the image tower, and pick the nearest caption. No classifier head, no fine-tuning, no gradient step. Swap in a different label set and you've got a different classifier for free. The same shared space gives image↔text retrieval directly, and CLIP's text encoder is what conditions Stable Diffusion and DALL·E 2 and powers open-vocabulary detection and segmentation.
Why captions beat a fixed label set
A fixed classifier can only ever say one of its trained classes; CLIP can score any text you can write. That flips the ceiling on the task — new categories are new sentences, not new training runs — and it's why the caption-as-supervision idea reshaped so much of multimodal ML. The cost is that quality rides on the prompt wording — "a photo of a {label}" beats a bare label — and on what the web's captions happened to cover, which is also where its blind spots come from.
Train a toy batch, watch the diagonal light up as the loss drops, and run a zero-shot classifier live:
Top comments (0)