Contrastive Learning: Why Pulling Similar Points Together Learns Representations Without Labels

Labels are expensive. Unlabelled data is not. Contrastive learning is built on that asymmetry: instead of asking what class an example belongs to, it asks a simpler question that turns out to be surprisingly powerful. Which examples should be close together, and which should be far apart?

If you can answer that question with pairs you construct yourself, you can train a representation that transfers to tasks you never trained for. That is the core idea behind SimCLR, MoCo, CLIP, and much of modern self-supervised learning. It is also an idea that is easy to implement badly, and the failure modes are instructive.

The basic setup

Start with an encoder, a neural network that maps an input to a vector. Call the encoder f, so an input x becomes a representation z = f(x). Contrastive learning does not tell the encoder what z should be. It only tells it about relationships: some pairs of inputs are positive pairs and should have similar representations, while others are negative pairs and should have dissimilar representations.

Positive pairs usually come from augmentation. Take one image, apply two different random crops, colour jitter, or flips, and you get two views of the same underlying content. Those two views form a positive pair. Every other image in the batch, or in a memory bank, supplies negative examples.

The training objective then pulls positive pairs together and pushes negative pairs apart in representation space. After training, inputs that share content end up near each other, even though no label ever said what that content was.

This is a strange way to learn, and it raises an obvious worry. What stops the encoder from mapping everything to the same vector? Then every pair would be perfectly similar. That collapse is the central technical problem in contrastive learning, and the loss function is designed to prevent it.

InfoNCE: the loss that makes it work

The most widely used contrastive loss is InfoNCE, short for Noise Contrastive Estimation with information in mind. For an anchor representation, one positive, and a set of negatives, it is a softmax over similarities:

$$\mathcal{L}{i} = -\log \frac{\exp(\mathrm{sim}(z{i}, z_{i}^{+}) / \tau)}{\exp(\mathrm{sim}(z_{i}, z_{i}^{+}) / \tau) + \sum_{k} \exp(\mathrm{sim}(z_{i}, z_{k}^{-}) / \tau)}$$

Here $z_{i}$ is the anchor, $z_{i}^{+}$ is its positive partner, $z_{k}^{-}$ are negatives, sim is usually cosine similarity, and $\tau$ is a temperature parameter.

Read the formula as a classification problem. The model must pick the true positive out of a set of candidates. The denominator is what prevents collapse: if every representation were identical, every similarity would be equal, and the loss would stay high because the positive could not be distinguished from the negatives. Pushing negatives apart is not a side effect. It is the mechanism that keeps the representation informative.

InfoNCE also has an information-theoretic interpretation. Minimising it maximises a lower bound on the mutual information between the two views:

$$I(X; Y) \geq \log(K) - \mathcal{L}_{\mathrm{InfoNCE}}$$

where K is the number of negative samples plus one. This bound is part of the appeal of the method, but it is also easy to over-read. A tighter bound is not the same as a better representation, and later work showed that the geometry of the learned space matters at least as much as the mutual information number.

Alignment and uniformity: two properties that explain success

A cleaner way to understand contrastive learning comes from two properties of the representation space, usually measured on the unit hypersphere after normalisation.

Alignment means that positive pairs are close together:

$$\mathcal{L}{\mathrm{align}} = \mathbb{E}{(x, x^{+})} \left[ | f(x) - f(x^{+}) |^{2} \right]$$

Uniformity means that representations spread out over the whole sphere, preserving as much information as possible:

$$\mathcal{L}{\mathrm{uniform}} = \log \mathbb{E}{x, y} \left[ \exp(-t | f(x) - f(y) |^{2}) \right]$$

Good contrastive representations do both. Alignment without uniformity is collapse: everything maps to one point, or to a small cluster. Uniformity without alignment is noise: representations spread out, but related inputs are no closer than unrelated ones.

This decomposition explains several practical observations. Strong augmentations improve alignment only if they preserve the content you care about. Too many easy negatives make uniformity easy to satisfy without learning anything subtle. And the temperature parameter controls the trade-off: a low temperature sharpens the softmax, penalises the hardest negatives most, and pushes uniformity harder, while a high temperature softens those penalties.

Why augmentations define the task

Here is the most important practical lesson in contrastive learning: the augmentations define what the representation becomes invariant to, and therefore what it ignores.

If you train with random crops and colour jitter, the encoder learns to treat a cropped, recoloured version of an image as the same object. That is useful for object classification, where colour and exact framing should not matter. It is harmful if your downstream task is colour-sensitive, such as distinguishing ripe from unripe fruit, or if fine spatial detail matters.

In other words, contrastive learning does not learn a universal representation. It learns the representation implied by your choice of positive pairs. Change the augmentation, and you change the invariances, and you change what transfers.

This is why domain matters so much. In natural language, a positive pair might be two spans from the same document, or a sentence and its paraphrase. In audio, it might be two segments from the same recording. In medical imaging, aggressive colour jitter that works on photos may destroy the signal entirely. There is no default augmentation set that works everywhere. Designing positive pairs is designing the learning task.

Negatives: quantity, hardness, and false negatives

The denominator of InfoNCE needs negatives, and the details matter.

First, quantity. SimCLR used very large batches, often 4096 examples, because more negatives give a better estimate of the contrastive task and a tighter bound. MoCo avoided huge batches by keeping a queue of past representations, encoded by a slowly updated momentum encoder, so a small batch could still contrast against thousands of negatives. Both work. The choice is an engineering trade-off between batch size, memory, and staleness of the negatives.

Second, hardness. Most negatives are easy: a picture of a dog is trivially different from a picture of a truck. Easy negatives contribute almost nothing once the model is decent, because their similarity is already low. Hard negatives, examples that look similar but are not positives, drive most of the learning. That is why temperature matters, and why some methods mine hard negatives explicitly. It is also why a contrastive model can plateau: once it separates the easy cases, progress requires harder comparisons.

Third, false negatives. In a large batch, some supposed negatives actually belong to the same class as the anchor. A batch of unlabelled images will contain two different pictures of cats, and the loss will push them apart even though a downstream classifier would want them together. This is an inherent cost of learning without labels, and it limits how far pure instance discrimination can go. Methods that cluster representations, or that use nearest neighbours as additional positives, are partly attempts to fix this.

A minimal example

The following sketch shows the core of a SimCLR-style step in PyTorch. The details are simplified, but the structure is the real one: two augmented views, a projection head, normalised embeddings, and an InfoNCE loss over the batch.

import torch
import torch.nn.functional as F

# z1, z2: embeddings of two augmented views, shape (batch, dim)
# Both rows i of z1 and z2 are views of the same original example.
def info_nce_loss(z1, z2, temperature=0.1):
    z1 = F.normalize(z1, dim=1)
    z2 = F.normalize(z2, dim=1)
    z = torch.cat([z1, z2], dim=0)  # shape (2 * batch, dim)
    sim = z @ z.T / temperature      # pairwise cosine similarity
    n = z1.shape[0]
    # Mask out self-similarity so an example is not its own negative.
    mask = torch.eye(2 * n, dtype=torch.bool, device=z.device)
    sim = sim.masked_fill(mask, -1e9)
    # Positive for row i is row i + n (or i - n in the second half).
    labels = torch.cat([torch.arange(n, 2 * n), torch.arange(0, n)]).to(z.device)
    return F.cross_entropy(sim, labels)

Two details in this code deserve attention. Normalisation puts embeddings on the sphere, so cosine similarity is the right geometry and the alignment-uniformity picture applies. And the projection head, a small MLP applied after the encoder during training and discarded afterwards, matters more than it looks: it gives the contrastive loss a space to solve its task in, leaving the encoder representation itself cleaner for downstream use. Removing the projection head at evaluation time is standard, not an oversight.

Limitations and common pitfalls

Contrastive learning is powerful, but it is not magic. Keep these limits in mind before adopting it.

First, it needs a lot of data and compute. The signal from a single contrastive pair is weak compared with a label, so training typically runs longer and on larger datasets. On a small dataset, a supervised model or a pretrained contrastive model fine-tuned on your data will usually beat training contrastively from scratch.

Second, evaluation can mislead. The standard protocol is linear probing: freeze the encoder, train a linear classifier on top, and report accuracy. A representation can score well on linear probing and still fail on tasks that need fine-grained or structured information, because linear probing only tests one kind of downstream use. Test on your actual task, not just on the benchmark.

Third, collapse is never fully solved, only managed. Methods without explicit negatives, such as BYOL and SimSiam, avoid collapse through architectural asymmetry, a predictor network, stop-gradients, or momentum encoders. Those tricks work, but they are sensitive to implementation details, and a small change can bring collapse back. If your contrastive training loss drops to near zero suspiciously fast, check whether the representation has collapsed before celebrating.

Fourth, augmentation leakage is real. If your augmentation pipeline accidentally includes information about the downstream label, or if your positive pairs share an artefact such as the same background or the same scanner, the model will learn the artefact. Contrastive learning is excellent at finding shortcuts, because a shortcut is exactly what makes two views easy to match.

When to use it, and when not to

Use contrastive learning when you have abundant unlabelled data, a clear idea of what should count as the same content, and a downstream task that benefits from transfer. Pretraining on unlabelled data followed by fine-tuning on a small labelled set is the classic winning pattern, and cross-modal contrastive learning, as in CLIP, extends the same idea to image-text pairs.

Do not use it when labels are plentiful and cheap, when you cannot define meaningful positive pairs, or when the invariances you would teach would destroy the signal your task needs. In those cases, supervised learning, or a pretrained model from a domain close to yours, is the better starting point.

A practical checklist before you start:

• Write down what your positive pairs mean, and what invariances that implies.
• Choose augmentations that preserve the content your downstream task needs.
• Start from a known recipe (SimCLR or MoCo hyperparameters) before inventing your own.
• Monitor alignment and uniformity, or at least check nearest neighbours qualitatively, not just the loss.
• Evaluate with linear probing and on your real downstream task.
• Check for collapse and for augmentation shortcuts early, while they are cheap to fix.

Takeaway

Contrastive learning works because it turns an unlabelled problem into a comparison problem. Pull similar things together, push different things apart, and do it at scale, and a useful representation emerges. The loss, the temperature, and the negative sampling all matter, but none of them matters as much as the definition of a positive pair. That definition is where your knowledge of the domain enters the model, and it is where most contrastive projects succeed or fail.

Get the pairs right, watch for collapse and shortcuts, and contrastive learning gives you representations that no labelled dataset of the same size could buy.

---

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial