A language model turns a hidden vector into V logits through one matrix. So the whole matrix of logits it can produce has rank at most the width of that vector, and the set of distributions it can represent is pinned by an integer.
Run it: https://dev48.infy.uk/dl/day75-softmax-bottleneck.html
The task is built so the obstruction is exact
A chosen log-odds spectrum over 12 symbols and 16 enumerated contexts, so the limit is a closed form rather than a fitted curve. The score is a mean KL from the true conditional: exactly 0 for the teacher, above a conditional-entropy floor of 1.9632 nats.
The capacity is not what is usually written down
It is not the rank of the probability matrix, or of the logit matrix as printed. It is the rank of the log-odds matrix — log p(y|c) minus log p(y_ref|c) — and the choice of reference symbol has to be free for that to be well defined.
So the page checks it: recompute in all 12 gauges, get the same integer every time.
rank of the log-odds matrix -> the capacity
choice of reference symbol -> irrelevant, verified in all 12
The headline
The criterion is an integer. The thing it is trying to predict is continuous.
A model whose rank is one short of what the task needs is not "one unit worse". The bottleneck says impossible; the loss says this much worse than possible, and those two statements do not line up. The page measures the gap between them — which is where the practical advice about mixture-of-softmaxes and rank tricks either earns its keep or does not.
280 verifier asserts, 347 in-page checks, 0 failures.
Top comments (0)