A vanilla RNN has one job that it does badly. It keeps a single hidden state and rewrites the whole thing every timestep by pushing it through a tanh. Because the same recurrent weights hit that state again and again, the gradient that flows back through a long sequence gets multiplied by a long chain of small numbers and shrinks toward zero. That is the vanishing-gradient problem, and in practice it means a plain RNN cannot connect information more than a handful of steps apart. By the time it reaches the end of a long sentence, the beginning has washed out.
The LSTM fixed this with three gates and a separate cell state. The GRU, introduced by Cho and colleagues in 2014, gets most of the same benefit with two gates and no extra state. Fewer moving parts, fewer parameters, and often faster training. It is the one I reach for first.
Here are the four lines that define it. Given the input x at this step and the previous hidden state h_prev:
z = sigmoid(Wz·x + Uz·h_prev + bz) # update gate
r = sigmoid(Wr·x + Ur·h_prev + br) # reset gate
h_tilde = tanh(Wh·x + Uh·(r * h_prev) + bh) # candidate
h = (1 - z) * h_prev + z * h_tilde # new hidden state
Two sigmoids, one tanh, one blend. That is the whole cell.
The update gate z is the important one. It is a vector of numbers between 0 and 1, one per hidden unit, and it decides the mix between the old state and a freshly computed candidate. When a unit's z sits near 0, that unit copies its previous value straight through: the state is carried forward untouched. When z is near 1, the unit is overwritten by the candidate. So a single gate lets the cell either hold a memory indefinitely or refresh it on demand, per unit, per step, all learned from the loss.
That carry behaviour is exactly why the vanishing gradient goes away. When z is near 0 the update collapses to h ≈ h_prev, and the derivative of the new state with respect to the old one is close to 1. Gradients can flow back over many steps without shrinking. It is the same idea as the LSTM's cell state and the same idea as a residual connection: a near-identity highway the network can open only where it genuinely needs to change something.
The reset gate r plays a smaller role. It controls how much of the previous state is even allowed into the candidate. Near 0, the candidate ignores history and is built from the current input alone, which is handy at the start of a new phrase. Near 1, the full past participates. Reset shapes the content of the candidate; update decides whether that candidate actually gets written.
Notice the convex combination in that last line. The two weights, (1 - z) and z, always sum to 1 for every unit, so the new state is a true interpolation between keeping the old value and taking the new one. There is no separate add-then-squash step and no way for the magnitude to blow up. It is one of the cleaner ideas in deep learning.
How does it compare? A vanilla RNN has zero gates and no memory valve. An LSTM has three gates (input, forget, output) plus a cell state, giving four weight blocks per cell. A GRU has two gates plus the candidate, so three weight blocks, and only the hidden state to carry. For input size d and hidden size h, that is roughly 3·(d·h + h² + h) parameters against the LSTM's 4·(…) — about a quarter fewer weights for the same width. Empirically the two land within a whisker of each other on most tasks, so the practical rule is simple: default to a GRU for its speed and light footprint, try an LSTM if you suspect very long-range memory is the bottleneck, and let a validation set decide. Swapping between them in PyTorch is a one-line change, since nn.GRU and nn.LSTM share an interface (the LSTM just returns its state as an (h, c) tuple).
A fair caveat: since 2017 the Transformer has replaced recurrent networks for most large-scale sequence work. Attention lets every position look directly at every other position, so there is no long chain to vanish through and the whole sequence can be processed in parallel. But the gating idea did not die — it lives on in the gated feed-forward blocks and the residual-and-norm highways inside Transformers. And GRUs are still an excellent, cheap choice for small, streaming, or on-device problems where a Transformer is overkill.
I built an interactive page where you can watch a real GRU cell carry a single token across a run of blank steps, see the update and reset gates light up as heat bars, and push the gate biases until it forgets. Feed it a token, step through, and the convex-combination update stops being abstract:
https://dev48v.infy.uk/dl/day26-gru.html
Tomorrow: attention and layer norm — the two pieces that let recurrence be retired.
Top comments (0)