Squeeze-and-Excitation from scratch: cheap channel attention that learns which feature-maps matter

A convolution layer emits C output channels — an edge detector, a texture detector, a colour blob — and then passes every one of them downstream with equal footing. It has no built-in way to say "for this image, the fur-texture channel matters and the sky-gradient channel doesn't." Squeeze-and-Excitation (Hu et al., 2017) bolts on exactly that: a cheap, learnable gate that recalibrates channel importance per input. I built it from scratch — no library, just an average, two matrix multiplies, a ReLU and a sigmoid — and it's three tiny stages.

Squeeze: global average pool to a channel descriptor

To decide a channel's importance you first need a summary of it. The squeeze averages each channel's entire H×W feature-map down to a single number z_c. The result is a C-length vector where z_c is a global descriptor of channel c — and that global receptive field is the point: a normal conv only sees a small window, but this descriptor summarises the whole spatial extent, so the next step can reason about the image as a whole.

function squeeze(maps) {              // maps: C channels, each H*W values
  return maps.map(m =>               // -> z: one number per channel
    m.reduce((a, v) => a + v, 0) / m.length);
}
// z has length C; z[c] = mean activation of channel c

Excite: a two-layer bottleneck MLP turns z into gates

Now turn that descriptor into gates. The excite is a tiny two-layer MLP. The first FC layer maps C down to C/r hidden units (then ReLU); the second maps C/r back up to C, then a sigmoid squashes each output into (0,1). The reduction ratio r (paper default 16) is the whole trick: the bottleneck is far cheaper than a full C→C layer, and it forces the block to model channel interdependencies from global context rather than copy z straight through.

const relu    = v => v.map(x => Math.max(0, x));
const sigmoid = v => v.map(x => 1 / (1 + Math.exp(-x)));

const hidden = relu(fc(z, W1, b1));      // C -> C/r  (the bottleneck)
const s      = sigmoid(fc(hidden, W2, b2)); // C/r -> C, then sigmoid -> gates
// s[c] near 1 -> keep channel c ; near 0 -> suppress it

Sigmoid, not softmax, is deliberate — and it's the detail people get wrong. Softmax would force the gates to compete and sum to 1, as if exactly one channel could be "on". But channels are not mutually exclusive; an image can need many strong channels at once. Independent per-channel sigmoids let the block turn any subset up or down freely.

Scale: recalibrate every map by its gate

The final step is trivial but is where it lands: multiply each original feature-map by its scalar gate, broadcast across H×W. Informative channels (gate near 1) pass through almost untouched; unhelpful ones (gate near 0) fade toward zero. The block has recalibrated its own input channel-by-channel and hands the result to the next layer.

function scale(maps, s) {
  return maps.map((m, c) => m.map(v => v * s[c]));
}
// full block:  X~ = scale( X, excite( squeeze(X) ) )

In my demo you pick a synthetic input — "edges", "texture", "busy scene", "flat sky" — and watch the gates recompute live. The revealing one is flat sky: everything is low and uniform, nothing stands out, so every gate hugs the neutral 0.5 and nothing gets re-weighted. Switch to "edges" and the first few channels light up hot while the rest are damped. Recalibration is input-dependent — that is the entire idea.

The whole block, and why it costs almost nothing

In PyTorch it is a handful of lines: AdaptiveAvgPool2d(1) is the squeeze, two linear layers are the excite, sigmoid makes the gates, and a broadcast multiply is the scale. Note the gates come out shaped (B, C, 1, 1) so they broadcast cleanly.

class SEBlock(nn.Module):
    def __init__(self, C, r=16):
        super().__init__()
        self.squeeze = nn.AdaptiveAvgPool2d(1)              # C×H×W -> C×1×1
        self.excite = nn.Sequential(
            nn.Linear(C, C // r, bias=False), nn.ReLU(inplace=True),
            nn.Linear(C // r, C, bias=False), nn.Sigmoid())
    def forward(self, x):                                    # x: (B, C, H, W)
        b, c, _, _ = x.shape
        z = self.squeeze(x).view(b, c)                       # (B, C)
        s = self.excite(z).view(b, c, 1, 1)                  # (B, C, 1, 1) gates
        return x * s                                         # scale (broadcast)

The two FC layers hold 2·C·(C/r) = 2C²/r weights — the only parameters SE adds. At C=512, r=16 that is 32,768 weights, a rounding error next to a single 3×3 conv's C²·9 ≈ 2.36M. Small r widens the bottleneck (finer gates, more params); push r too far and the bottleneck starves and the gates collapse toward a useless flat 0.5. The paper found 16 a robust sweet spot, adding only ~2.5% params to a ResNet-50.

Best of all, SE is a module, not an architecture. In a ResNet you apply it to the residual branch right before the identity add — y = x + SE(F(x)) — leaving the skip connection untouched, so training stability doesn't change; you just gained learned channel attention. That drop-in simplicity won ILSVRC 2017 and put SE inside MobileNetV3 and every EfficientNet, seeding the attention lineage SE → CBAM (spatial too) → ECA (a cheap 1-D conv).

Pick an input and watch squeeze → excite → rescale run live:
https://dev48v.infy.uk/dl/day40-squeeze-excitation.html