Train a deep classifier long past the point where training accuracy hits 100 percent, and something surprising happens. The messy, high-dimensional features in the penultimate layer do not stay messy. They collapse. Features from the same class shrink toward a single point, class means arrange themselves into a highly symmetric shape, and the final classifier aligns itself with that shape. This phenomenon is called neural collapse, and it turns the last layer of a neural network from a black box into a piece of geometry you can write down.
Neural collapse matters because it explains what a classifier is actually doing at the end of training, why the decision boundaries look the way they do, and where that simple picture breaks. It is not a trick for squeezing out accuracy. It is a description of the endpoint that standard training tends toward, and a warning about reading too much into that endpoint.
The setting: features, means, and a classifier
Consider a classification problem with C classes. Let h(x) be the feature vector produced by the penultimate layer for an input x, and let the final layer compute logits as W h(x) + b, where W is a C by d matrix and b is a bias vector. Training typically minimises cross-entropy loss with weight decay, often long after the model classifies every training example correctly.
For each class c, define the class mean of the features:
where n_c is the number of training examples in class c. Also define the global mean:
Neural collapse is a statement about the mu_c, about the variation of h(x) around them, and about the rows of W. It is observed in the terminal phase of training, the phase after training error reaches zero, when the loss can still fall because the model becomes more confident, not because it corrects more mistakes.
The four properties of neural collapse
The phenomenon is usually stated as four related properties. Together they say that the last layer becomes almost completely determined by the class means, up to a few scalars.
NC1: Within-class variability collapses
Features from the same class converge to their class mean. The within-class covariance shrinks relative to the between-class covariance. In the limit, h(x) is almost equal to mu_{y} for an example with label y, so the penultimate layer stops encoding anything that distinguishes examples within a class, at least on the training set.
This is the most striking part, and the most easily misread. Collapse on the training set does not mean the network has forgotten within-class structure in general. It means that, for the purpose of the training loss, that structure is no longer needed. A feature that does not help separate the training classes is, from the loss's point of view, noise to be suppressed.
NC2: Class means form a simplex equiangular tight frame
After centring by the global mean, the class means become equal in length and maximally separated. Formally, the centred, normalised class means satisfy:
and
For C = 3, this is an equilateral triangle. For C = 4, it is a regular tetrahedron. For larger C, it is the natural generalisation: C points in C - 1 dimensions, all pairwise at the same angle, the most spread-out configuration possible. This shape is called a simplex equiangular tight frame, often shortened to simplex ETF.
The intuition is simple. Cross-entropy pushes each class mean away from the others, and weight decay and the shared geometry of the classifier pull them toward a symmetric arrangement. The simplex ETF is the arrangement that maximises the minimum separation given a fixed norm, so it is the natural resting place.
NC3: The classifier aligns with the class means
The rows of the final weight matrix W converge, up to scaling, to the centred class means. If w_c is the row of W for class c, then:
In other words, the classifier stops being an independent object. It becomes a nearest-centroid classifier in feature space: the logit for class c is essentially the inner product of the feature with the direction of class c's mean. The decision rule is then: pick the class whose mean direction is closest to the feature.
NC4: Classification reduces to nearest class centre
As a consequence of the first three properties, the network's prediction on a training example is the same as the nearest-centroid rule applied to the centred features:
up to the bias term, which also converges to a simple form in the balanced case. The whole last layer, which looked like a learned matrix, is now described by C points arranged symmetrically.
Why does this happen?
A useful way to think about it is to separate two forces in training. Cross-entropy wants the logit of the correct class to be much larger than the logits of the wrong classes. Once every example is classified correctly, the only way to keep reducing the loss is to increase the margin, which pushes features further in the direction of their class and pushes class means apart. Weight decay resists unbounded growth, so the system settles at a balance: large, but finite, separation with a symmetric arrangement.
The unconstrained-features viewpoint makes this precise. If you treat the penultimate features as free variables, rather than as outputs of a fixed network, and minimise cross-entropy plus weight decay over both the features and the classifier, the global minimiser exhibits exactly the simplex ETF structure under balanced classes and suitable regularisation. Real networks are not unconstrained, since features must be realisable by the earlier layers, but in the terminal phase they get close enough that the same geometry shows up empirically across architectures and data sets.
Note what this argument does not claim. It does not say that every network reaches exact neural collapse, or that collapse is necessary for good generalisation. It says that the training objective, pushed far enough, has a strong preference for this symmetric endpoint, and that real training often moves a long way toward it.
Measuring it in practice
Neural collapse is easy to check, and checking it is more informative than staring at accuracy curves. The code below computes class means from penultimate features and reports two diagnostics: within-class variability relative to between-class variability, and the average pairwise cosine similarity between centred class means, which should approach -1/(C-1) under collapse.
import torch
def collapse_diagnostics(features, labels, num_classes):
# features: (n, d) tensor of penultimate features
# labels: (n,) tensor of class indices
class_means = torch.stack([
features[labels == c].mean(dim=0)
for c in range(num_classes)
])
global_mean = class_means.mean(dim=0)
centred = class_means - global_mean
# NC1 proxy: within-class variance over between-class variance
within = 0.0
for c in range(num_classes):
diff = features[labels == c] - class_means[c]
within = within + (diff ** 2).sum()
between = (centred ** 2).sum() * features.shape[0] / num_classes
nc1 = (within / between).item()
# NC2 proxy: mean pairwise cosine between centred class means
norms = centred / centred.norm(dim=1, keepdim=True)
cos = norms @ norms.T
mask = ~torch.eye(num_classes, dtype=torch.bool)
mean_cos = cos[mask].mean().item()
target = -1.0 / (num_classes - 1)
return {"nc1_ratio": nc1, "mean_pairwise_cos": mean_cos,
"simplex_etf_target": target}
A falling NC1 ratio and a mean pairwise cosine drifting toward -1/(C-1) are the signatures to look for. If neither moves, the model is probably not yet in the terminal phase, or the features are constrained in a way that blocks collapse, for example by a bottleneck layer with dimension smaller than C - 1, which cannot even hold a simplex ETF.
A practical tip: compute these diagnostics on held-out data as well as training data. Collapse on the training set is expected. Collapse on held-out data is a much stronger statement, and it is where the story gets interesting for generalisation.
What neural collapse explains, and what it does not
Neural collapse gives a clean explanation for several familiar observations. It explains why the last-layer features of a well-trained classifier look so structured in visualisations, why simple nearest-centroid rules on learned features often work nearly as well as the trained classifier, and why the margins between classes tend to be so uniform in balanced problems.
It also clarifies transfer learning. If the penultimate features have collapsed to class means on the source task, then those features carry little information beyond the source class label. That is good for the source task and potentially bad for a downstream task that needs within-class detail. This is one reason why features taken from an earlier layer, or from a model trained with a different objective, sometimes transfer better than the final features of a classifier trained to convergence.
But neural collapse does not explain generalisation by itself. A model can collapse on the training set and still generalise badly if the collapsed geometry reflects spurious correlations rather than real class structure. Collapse is a statement about the training endpoint, not a certificate of quality. Similarly, models that do not fully collapse can still generalise well, especially when training is stopped early, when classes are imbalanced, or when the loss is not plain cross-entropy.
Limitations and common pitfalls
First, imbalance breaks symmetry. The simplex ETF result assumes balanced classes. With imbalanced classes, the geometry distorts: majority classes occupy more space, minority class means are pulled toward the majority, and the equal-norm, equal-angle picture no longer holds exactly. Interpreting a distorted geometry as a failure of training is a mistake; it is the expected consequence of the data distribution.
Second, the feature dimension matters. A simplex ETF for C classes needs at least C - 1 dimensions. If the penultimate layer is narrower than that, full collapse is geometrically impossible, and the network will settle for the best approximation it can fit. This is a real design consideration when choosing the width of the final hidden layer for a problem with many classes.
Third, collapse is a training-set phenomenon. Measuring NC1 on training features and concluding that the model has "lost" within-class information in general overstates the result. The network may still encode within-class variation in earlier layers, and that variation may reappear on new data. Always check the held-out diagnostics before drawing conclusions about what the model knows.
Fourth, not all losses collapse the same way. Label smoothing, mixup, contrastive objectives, and other modifications change the effective target geometry. A model trained with label smoothing, for example, is pushed toward a different endpoint, because the loss no longer rewards unbounded confidence in the correct class. Comparing collapse across different losses without accounting for this is comparing different objectives, not different levels of success.
Finally, do not confuse collapse with overfitting. Collapse happens after training error reaches zero, which is also when overfitting is usually suspected, but collapse itself is a highly regular, symmetric structure, not a memorised lookup table. The right question is not "did it collapse?" but "did it collapse onto the right geometry, and does that geometry survive on new data?"
The takeaway
Neural collapse is a reminder that deep learning, for all its complexity, often ends in a place simple enough to draw. Train a classifier long enough, and its last layer stops being a tangle of weights and becomes C points arranged as symmetrically as possible, with a classifier that simply points at those points. That picture is useful when it holds, instructive when it breaks, and a good diagnostic in either case. Next time you train a classifier to convergence, do not just record the accuracy. Compute the class means, look at their geometry, and ask whether the simple structure at the end of your network is the structure your problem actually needs.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
Top comments (0)