A standard Transformer feed-forward block is dead simple: Linear -> one fixed nonlinearity -> Linear, where that middle activation (ReLU, GELU) applies the same elementwise curve to every hidden unit. A Gated Linear Unit throws that single activation out and replaces it with a multiplicative valve. Understanding it is the difference between reading a modern LLM's FFN and being baffled by it.
Project the input twice, then multiply
Instead of one weight matrix, a GLU uses two. W makes a value branch (kept linear); V makes a gate branch (squashed into (0,1)). Multiply them element by element:
out = (xW + b) ⊙ σ(xV + c)
value gate
The gate decides, per unit and per input, how much of the value is allowed through. A gate near 0 shuts that feature off; near 1 it passes. In code the forward pass is three lines:
def branches(x):
value = x @ W + b # linear, NO activation
gate_pre = x @ V + c # will be squashed
return value, gate_pre
def glu(x):
value, gate_pre = branches(x)
gate = sigmoid(gate_pre) # gate in (0, 1)
return value * gate # elementwise (Hadamard) product
Why multiplication buys capacity
A single Linear + ReLU can only bend each unit with a fixed curve — an additive nonlinearity. It can't cheaply make unit A's output depend on unit B. A product can: value ⊙ gate is an AND-like interaction — "let this feature through only when the input also opens the gate." That data-dependent, multiplicative routing is a strictly richer primitive, which is why a gated FFN reaches a lower loss at the same FLOP budget.
The family is just which squasher you put on the gate
GLU -> GEGLU -> SwiGLU differ in exactly one thing: the gate activation.
def sigmoid(z): return 1.0 / (1.0 + np.exp(-z))
def gelu(z): # tanh approximation
k = np.sqrt(2.0 / np.pi)
return 0.5 * z * (1.0 + np.tanh(k * (z + 0.044715 * z**3)))
def silu(z): # SiLU == Swish with beta = 1
return z * sigmoid(z)
GATE = {"GLU": sigmoid, "GEGLU": gelu, "SwiGLU": silu}
Sigmoid saturates in (0,1) — a pure open/shut valve. GELU and SiLU are smooth and non-monotonic: they dip slightly below zero near z ≈ -1, then climb toward the identity, and crucially they are not capped at 1. So a GELU/SiLU gate can also amplify magnitude, not just attenuate — and it keeps a live gradient where the sigmoid's has vanished. Noam Shazeer's 2020 note "GLU Variants Improve Transformer" swept these and found GEGLU/SwiGLU give the best quality per parameter.
The full SwiGLU FFN — and the ⅔-width rule
In a Transformer the gated hidden goes through a third matrix back to the model width. A plain ReLU FFN has 2 matrices; a gated FFN has 3 (W, V, W2), so at the same hidden width it uses ~1.5x the parameters. To keep it parameter-neutral, shrink the hidden width by 2/3 — that's LLaMA's d_ff = 8/3 · d_model instead of 4 · d_model.
class SwiGLU_FFN(nn.Module):
def __init__(self, d_model, mult=4):
super().__init__()
d_ff = int(2/3 * mult * d_model) # 3 matrices => shrink 2/3
self.w = nn.Linear(d_model, d_ff, bias=False) # value W
self.v = nn.Linear(d_model, d_ff, bias=False) # gate V
self.w2 = nn.Linear(d_ff, d_model, bias=False) # down W2
def forward(self, x):
gated = F.silu(self.v(x)) * self.w(x) # SwiGLU core
return self.w2(gated) # back to d_model
This is exactly why PaLM, LLaMA / 2 / 3 and Mistral all use a SwiGLU FFN. Gating itself traces back to Dauphin et al. 2016 (a gated convolutional LM — the original GLU); Shazeer brought it into the Transformer FFN. If a recent model's feed-forward "looks weird," it's almost always a gated one.
The mental model that sticks: value = what, gate = how much. Swap the gate activation (σ -> GELU -> SiLU) and you walk the GLU -> GEGLU -> SwiGLU family; pick SiLU and trim the hidden width by ⅔ and you have the LLaMA FFN.
Feed a live input vector through both branches, flip between the three gates, and probe the valve surface out = value · act(gate) here: https://dev48v.infy.uk/dl/day53-gated-linear-units.html
Top comments (0)