Spectral Normalization: Divide W by Its Top Singular Value for a 1-Lipschitz Layer

A linear layer y = Wx stretches its input, and the worst-case stretch — the largest ‖Wx‖/‖x‖ over every x — is exactly the spectral norm σ(W), the largest singular value of W. That number is also the layer's Lipschitz constant. In a GAN the discriminator is trained adversarially and its weights keep growing, so σ blows up, gradients explode, and training collapses. WGAN patched this with crude weight clipping (which kills capacity) and then a gradient penalty (an extra backward pass). Spectral Normalization is the cheap, direct fix.

Divide by σ and the layer can no longer stretch

The whole method is one division: W_SN = W / σ(W). The result has σ(W_SN) = 1, so ‖(W/σ)x‖ ≤ ‖x‖ for every x — the layer is exactly 1-Lipschitz. And because a composition of 1-Lipschitz maps is itself 1-Lipschitz, wrapping every layer makes the whole discriminator 1-Lipschitz, so its input gradients are bounded and the min–max game can't send σ to infinity.

Geometrically, W maps the unit circle to an ellipse whose longest semi-axis is exactly σ; W/σ shrinks that ellipse to fit inside the unit circle, touching it only along the top singular direction.

Power iteration makes it almost free

A full SVD every training step is O(d³) — far too slow. Instead, estimate only the top singular value by alternately mapping through W and Wᵀ and re-normalizing. Each pass is a couple of matrix–vector products, O(d²).

def spectral_norm(W, v, n_iter=1):
    for _ in range(n_iter):
        u = W @ v;   u /= np.linalg.norm(u) + 1e-12   # left  singular vec
        v = W.T @ u; v /= np.linalg.norm(v) + 1e-12   # right singular vec
    sigma = u @ (W @ v)          # u^T W v  ->  top singular value
    return sigma, u, v

The real trick from the SNGAN paper: run only one power-iteration step per forward pass, warm-started from the vector v stashed on the previous pass. Because W barely changes between steps, that single step tracks σ accurately — spectral normalization is essentially free.

class SNLinear:
    def __init__(self, W):
        self.W = W
        self.v = np.random.randn(W.shape[1])      # persistent estimate
        self.v /= np.linalg.norm(self.v)
    def forward(self, x):
        u = self.W @ self.v;  u /= np.linalg.norm(u) + 1e-12
        v = self.W.T @ u;     v /= np.linalg.norm(v) + 1e-12
        self.v = v                                # WARM-START next call
        sigma = u @ (self.W @ v)                  # 1-step sigma estimate
        W_sn  = self.W / sigma                    # 1-Lipschitz weight
        return W_sn @ x

You can verify both claims against NumPy's SVD: power iteration converges to the true top singular value, and W/σ has top singular value exactly 1.

sigma_true = np.linalg.svd(W, compute_uv=False)[0]
sigma_pi, u, v = spectral_norm(W, v0, n_iter=50)
assert abs(sigma_pi - sigma_true) < 1e-6                       # PI converges
assert abs(np.linalg.svd(W / sigma_true, compute_uv=False)[0] - 1.0) < 1e-6

SN vs clipping vs gradient penalty

All three try to make the discriminator 1-Lipschitz, but they pay differently. Weight clipping clamps every weight into [−c, c] — cheap but blunt: it caps the wrong quantity, wastes capacity, and is hard to tune. Gradient penalty pushes ‖∇D‖ → 1 on interpolated samples — it enforces the constraint softly and well, but needs an extra backward pass, noticeably more compute per step. Spectral normalization is a hard guarantee with no capacity loss and no extra backward pass; the only cost is one power-iteration step per forward. That is why SNGAN and countless GANs since use it by default.

Edit a small matrix, step power iteration by hand, and watch an unnormalized σ explode while SN pins it at 1: https://dev48v.infy.uk/dl/day58-spectral-normalization.html