With Right Padding a Causal Mask Already Excludes Padding. With Left Padding It Excludes Nothing

I set out to write "you need both masks — causal and padding, combined with an AND". Then I measured it, and the result was sharper than the advice.

Everything computed live: https://dev48.infy.uk/dl/day66-attention-masks.html

The two masks

Causal stops position i attending to j > i. It depends only on positions, so it is one lower-triangular matrix shared by the whole batch.

Padding stops everything attending to the filler that makes a ragged batch rectangular. It depends on the sequence, and it is a column rule — a padded key is invisible to all queries.

Written as predicates, the combination is obviously a conjunction:

const causal  = (i, j) => j <= i;
const padding = (len) => (i, j) => j < len;            // right padding
const both    = (len) => (i, j) => j <= i && j < len;

And then the measurement

With right padding, a real query sits at i < len and can only attend to j ≤ i < len. Causality has already excluded every padded key. Measured leak from a causal-only mask: exactly 0.0%.

So for right-padded decoder self-attention, the padding mask is redundant.

With left padding it is the opposite. The filler sits at low indices — precisely where causality lets everything through — and a causal-only mask leaks 55.6% of every real position's attention onto padding.

And left padding is not the exotic case. It is what decoder-only generation uses, because the prompt has to end at the last position for the next token to continue it.

The configuration where the padding mask is essential is the one running in production. The configuration where it looks redundant is the one you tested during training.

Why it trains fine either way

If padding is attendable, the padding embedding is a free parameter and gradient descent will push it somewhere useful. The loss goes down. The model has learned to use a token that means nothing, and the answer now depends on how many of them happen to be in the batch.

The test that catches it, with no labels

out_alone  = model([seq])                    # batch of one, no padding
out_padded = model([seq, longer_seq])[0]     # same seq, now padded

assert allclose(out_alone, out_padded, atol=1e-6)

Batch invariance. No data, no labels, no training, milliseconds to run. The correct mask gives exactly 0 across 36 shape/side combinations; every incorrect one does not.

The row everybody forgets

Under left padding, query 0 is padding: causality allows only j=0, and the padding mask blocks j=0. Every key is blocked, and softmax over all −∞ is 0/0.

The usual workaround is to fill with −1e9 instead. That does not fix it — it produces a uniform distribution over positions that are all blocked. Well-formed, sums to 1, completely meaningless, and it propagates silently instead of announcing itself.

The fix is to make the row not matter: mask the output at padded query positions, or zero fully-masked rows after the softmax. Not a better constant.

Part of a from-scratch series — one deep-learning idea a day, computed in-browser: https://dev48.infy.uk/deeplearningfromzero.php