DEV Community

Paragon
Paragon

Posted on

SGD creates teacher-induced spikes in orthogonal neural weights

Researchers analyse two-layer networks trained by stochastic gradient descent on a multi-index teacher model. They decompose first-layer weights into teacher-aligned signal and an orthogonal complement, showing SGD induces teacher-dependent spectral spikes in the complement Gram matrix and affects generalisation through variance inflation.

Why it matters

It clarifies how supposedly irrelevant directions can harm or shape generalisation, informing training and regularisation choices.

Analysis

The work separates signal from orthogonal dynamics in a tractable proportional limit, revealing emergent spikes despite initially isotropic updates. That offers a precise mechanism for variance inflation. The main uncertainty is whether the effect persists in deeper, realistic networks and practical training regimes beyond the two-layer teacher setting.

Source: arXiv

AI disclosure: this article was produced with AI assistance and reviewed for accuracy.

Top comments (0)