DenseNet from scratch: ResNet adds a skip connection, DenseNet concatenates every one — top accuracy at a third of the params

ResNet gave every layer a shortcut to the loss by adding a residual: y = x + F(x). DenseNet takes that idea to its logical extreme. Inside a dense block, every layer is wired to every layer before it — layer ℓ receives the concatenation of the feature-maps of all preceding layers, and passes its own maps to all that follow. I built an interactive breakdown of the channel math, and the surprising part is how thrifty this "connect everything to everything" turns out to be. Here's why.

Concatenate, don't add

The one idea that defines DenseNet is concatenation. Keep a running list of every feature-map produced so far; to run layer ℓ, concatenate the whole list along the channel axis, feed it to a composite function H, and append the new maps back onto the list.

function denseBlock(x0, L, k) {
  let feats = [x0];                       // collect all feature-maps
  for (let l = 1; l <= L; l++) {
    const inp = concat(feats);            // [x0, x1, ..., x_{l-1}]
    const xl  = H(inp, k);                // k new maps
    feats.push(xl);                       // reuse forever after
  }
  return concat(feats);                   // block output
}

Because every earlier map stays directly reachable, nothing downstream has to re-learn it. That is feature reuse, and it's the whole reason DenseNet is parameter-efficient.

The growth rate k, kept deliberately small

One layer is H = BN → ReLU → 3×3 conv, and it outputs exactly k feature-maps — the growth rate. k is small on purpose (12, 24, 32): a layer only has to contribute a thin slice of new information, because everything computed before it is still available by concatenation.

Growth is linear, connections are quadratic

Here's the counterintuitive bit. It feels like everything connecting to everything must explode the width, but it doesn't. Layer ℓ receives k₀ + k·(ℓ−1) channels and adds k, so after L layers the block output is just k₀ + k·L — linear.

function inChannels(l, k0, k)   { return k0 + k * (l - 1); }  // into layer l
function blockOutChannels(L,k0,k){ return k0 + k * L; }        // after L layers
// k0=24, k=12, L=6 -> layer inputs 24,36,48,60,72,84; block out = 96

The connections are what grow quadratically. A plain L-layer chain has L connections; a dense block feeds layer ℓ from ℓ sources, for 1 + 2 + … + L = L(L+1)/2 direct connections — a web of short paths from any layer to any later one, which means strong gradients in reverse.

function connections(L)       { return L * (L + 1) / 2; }  // dense: L=6 -> 21
function plainConnections(L)  { return L; }                //  plain: L=6 -> 6

Transition layers compress between blocks

You can't concatenate across a downsample, and channels would balloon over many blocks. So between dense blocks sits a transition: a 1×1 conv that squeezes m channels down to ⌊θ·m⌋, then a 2×2 average-pool that halves the spatial size. DenseNet-C/BC use θ = 0.5.

function transition(x, m, theta) {
  x = conv1x1(x, Math.floor(theta * m));    // compress channels (theta<=1)
  x = avgPool2x2(x);                        // halve H and W
  return x;
}

The whole difference from ResNet, in one line

Both architectures give early layers a short path to the loss. The distinction is how:

// ResNet block:   add — width stays the same, features get summed
y = x.add( F(x) );                 //  y = x + F(x)

// DenseNet layer: concat — width grows by k, every map kept
y = concat([ x, F(x) ]);           //  y = [x, F(x)]

Summing can overwrite — earlier signals blur into the sum. Concatenating preserves — every earlier map survives intact and stays reusable downstream. That preservation is exactly why DenseNet reuses features so aggressively and needs so few parameters to do it.

The real thing

You never hand-roll the concatenation. Torchvision ships the whole family, already wired with dense blocks, bottlenecks and θ=0.5 transitions.

import torchvision.models as models
net = models.densenet121(weights="IMAGENET1K_V1")  # k=32, blocks=[6,12,24,16]
# ~8M params vs ResNet-50's ~25M — similar ImageNet accuracy.

DenseNet-121 matches a much heavier ResNet-50 on ImageNet with roughly a third of the parameters, purely from feature reuse. Drag the growth rate, layer count and compression knobs in the demo and you can watch the channels stack, the L(L+1)/2 arcs multiply, and the parameter contrast against a same-width plain block update live.

Play with the knobs and watch the connections multiply:
https://dev48v.infy.uk/dl/day39-densenet.html