A Collapsed Encoder Scores a Perfect 1.0 on Alignment

NT-Xent — InfoNCE, the loss behind SimCLR and most embedding models — asks each view of an item to pick its partner out of a line-up. The line-up is the rest of the batch.

👉 Live, every number measured in your browser: https://dev48.in/dl/day82-contrastive-loss.html

That clause is not a footnote. It means the batch size changes the objective, not just the gradient noise.

The loss number is not comparable across batch sizes

batch	negatives per row	loss	collapse floor log(2N−1)	partner ranked first
N = 2	4	0.0084	1.0986	100%
N = 8	16	0.0087	2.7081	100%
N = 32	64	0.0379	4.1431	100%
N = 64	128	0.0817	4.8442	100%

Same encoder, same temperature, same data distribution. The loss climbs anyway, because the task got harder. A loss curve from a run at N=256 cannot be read against one at N=64, and "our loss is lower" across a batch-size change means nothing at all.

Collapse maximises the half of the objective people plot

Alignment — the average cosine between the two views of an item — is the metric that gets logged. Here is what it says about an encoder that has learned nothing:

batch	alignment	uniformity	loss	partner first
well separated	0.9792	−0.0229	0.0014	100%
noisy views	0.6140	−0.0268	0.1185	100%
views unrelated	0.1017	−0.0208	2.9476	18.8%
COLLAPSED	1.0000	1.0000	2.7081	12.5%

Every embedding is the same point. Alignment is a perfect 1.0000. Only uniformity notices anything is wrong.

And the collapsed loss is exactly log(2N−1) — confirmed at five batch sizes and four temperatures. All the logits are equal, so the softmax is uniform over 2N−1 options and temperature cancels out entirely.

Temperature is a negative-mining knob

Everyone describes τ as sharpening the distribution. True, but the useful reading is about where the gradient goes:

temperature	loss	share of negative gradient on the HARDEST negative
1.00	2.6041	5.0%
0.50	1.8567	7.3%
0.20	0.4257	16.8%
0.10	0.0191	34.1%
0.05	0.0001	49.4%
0.01	0.0000	63.1%

An independent check with different embeddings measured 98.8% at τ=0.02.

As τ falls, the share landing on a single negative climbs toward 100% and every other negative stops contributing. At that point it is no longer contrastive learning over a batch — it is hardest-negative mining with extra steps, and all the effort you put into a large batch is being discarded.

The numerics are not optional

At τ = 0.01, a cosine of 1 becomes a logit of 100. At τ = 0.001 it becomes 1000, and:

Math.exp(1000)   // Infinity
Math.log(Infinity) // Infinity  -> loss becomes NaN

NaN is not a large loss. It is no loss at all — the gradient is gone and training silently stops learning. logsumexp with the max shifted out stays finite:

function logsumexp(xs) {
  let m = -Infinity;
  for (const x of xs) if (x > m) m = x;
  let s = 0;
  for (const x of xs) s += Math.exp(x - m);
  return m + Math.log(s);
}

How it is verified

The page computes the loss with a stable logsumexp and differentiates by hand. The verifier computes the softmax explicitly from probabilities, with no logsumexp anywhere, checks every gradient against a central finite difference of the total loss, and uses a different RNG and different embeddings.

The gradient shape is checked as an identity rather than a value: softmax_j − [j is the partner] must sum to zero over a row, be negative on the partner, and positive on every negative. 511 asserts, 0 failures. 19 in-page checks.