Supervised learning is spectacular, but it runs on labels, and labels are slow, costly and scarce. Meanwhile the internet is a bottomless well of un*labelled images. Contrastive learning — the heart of SimCLR — learns a genuinely useful representation with **no labels at all, from one deceptively simple idea. I built a demo that trains a real encoder live in the browser on the true InfoNCE loss, where the labels are used *only to colour the dots and never touch the loss, so you can literally watch a circle self-organise into class arcs from nothing. Here's how it works.
Augmentations invent the labels
The trick to get a training signal for free: take one image and apply a strong random augmentation twice — crop, flip, colour-jitter, blur — producing two views. We declare these two views a positive pair: whatever they have in common is the content, and the encoder should map them close together. Everything the augmentation changed is nuisance the encoder should discard. The choice of augmentations literally defines what "similar" means — it's the whole inductive bias. (SimCLR found colour jitter matters enormously; without it the net cheats by matching colour histograms instead of content.)
Within a batch of N images you get 2N views. For any anchor view, its positive is the sibling view of the same image; every one of the other 2N−2 views — all from different images — is a negative. The objective is just: make the anchor more similar to its positive than to any negative. That comparison is where "contrastive" comes from.
L2-normalise, then the InfoNCE loss
An encoder maps each view to an embedding; you normalise it to unit length so it lives on a hypersphere and the dot product is the cosine similarity. Now stack all views, compute the pairwise similarity matrix in one matmul, mask the diagonal so nothing is its own negative, divide by a temperature τ, and take a softmax over each row. The loss is the negative log-probability assigned to the true positive — which is literally cross-entropy for a classification problem whose "class" is which sample is my positive:
def info_nce(sim, N, tau=0.5):
logits = sim / tau # temperature-scaled
targets = torch.arange(2*N, device=sim.device)
targets = (targets + N) % (2*N) # index of each row's positive
return F.cross_entropy(logits, targets) # softmax + pick the positive
That single loss quietly optimises two things at once. The numerator pulls each anchor toward its positive — alignment. The denominator pushes it away from all negatives — uniformity, spreading embeddings to cover the sphere. Alignment alone collapses everything together; uniformity alone scatters without structure; their tension produces tight, well-separated clusters.
Two knobs the demo lets you break
Temperature τ controls the softmax sharpness. Small τ (≈0.05) makes it peaky so the nearest — hardest — negative dominates the gradient: sharp separation, but sensitive. Large τ softens it, spreading the penalty gently over all negatives. SimCLR typically uses τ ≈ 0.1–0.5. The demo has a "hardest-negative share" read-out; drop τ and watch one negative swallow the softmax weight.
Negatives are not optional. Turn them off and the objective becomes just "make the two views agree", whose trivial global optimum is to map every input to the same constant vector — representational collapse, zero information. Flip to "positives only" in the demo and watch the cloud crush toward a point (same-class similarity → 1.00). The negatives in the denominator are the repulsion that forbids it:
# NO negatives -> L = -sim(z1, z2)/tau (alignment only)
# optimum is f(x) = const for all x -> representations COLLAPSE.
# negatives in the denominator add repulsion (uniformity) -> no collapse.
Why classes emerge with no labels
Here's the payoff, and it's subtle. The loss only ever ties together the two augmentations of a single image — it never compares different images as "same class". Yet class clusters appear. Why? The encoder is a smooth function, and augmentations of an image span a chunk of input space; forcing all of that chunk to one embedding, for every image, forces nearby inputs to map to nearby embeddings. Since different images of the same class also look alike, they overlap in input space and inherit nearby embeddings too. Class structure falls out as a by-product of demanding augmentation-invariance — exactly what you watch self-organise on the demo circle.
The demo runs a genuine 2→16→2 tanh encoder with real backprop, softmax, and grad-clipped SGD — so pressing Train, flipping negatives off, and dragging τ all move real gradients, not a canned animation.
How you actually use it
After pretraining, freeze the encoder and train a single linear layer on top with whatever few labels you have — the linear probe. SimCLR's frozen features get within a couple of points of fully-supervised ImageNet. The idea spawned a family: MoCo (momentum encoder + a queue of negatives), BYOL/SimSiam (no negatives at all — a predictor + stop-gradient dodges collapse), and CLIP — the same InfoNCE loss, but the positive pair is an image and its caption, giving the shared image–text space behind modern multimodal models.
Press Train with negatives on, then turn them off and watch it collapse:
https://dev48v.infy.uk/dl/day43-contrastive-learning.html
Top comments (0)