Every output of a CNN only ever sees a small window of the input — its receptive field — and that window is all the evidence it has. A 3-tap kernel sees three samples. Stack more layers and the window grows, but only linearly: each stride-1 layer of size k widens it by k−1, so with k=3 you get RF = 2L + 1. Ten layers see 21 samples. Thirty layers see 61. For semantic segmentation you need hundreds of pixels of context; one second of 16 kHz audio is 16,000 samples. Linear growth is simply the wrong curve.
The classic escape was pooling or striding. That grows the field fast, because after a 2× pool every unit is worth two input units — but it buys context by throwing away resolution, which is fatal when the output has to be as sharp as the input.
The whole trick: one index
Instead of reading k consecutive samples, read k samples spaced d apart:
def conv1d(x, w, d=1):
k = len(w)
eff = d * (k - 1) + 1 # span the kernel covers
return [sum(w[j] * x[i + j*d] for j in range(k)) # <-- j*d is the whole idea
for i in range(len(x) - eff + 1)]
x[i + j] became x[i + j*d]. That is the entire algorithm. The kernel now straddles d(k−1)+1 input positions while still holding exactly k weights and doing exactly k multiplies. Setting d=1 recovers the ordinary convolution, so dilation is a strict generalisation — which is why every framework exposes it as one extra argument.
The name à trous is French for "with holes". It is provably identical to convolving with a plain kernel that has d−1 zeros stuffed between its taps:
def dilate_kernel(w, d):
out = []
for j, v in enumerate(w):
out.append(v)
if j < len(w) - 1:
out.extend([0.0] * (d - 1)) # the "holes"
return out
assert conv1d(x, w, d=4) == conv1d(x, dilate_kernel(w, 4), d=1) # identical
That equivalence is a good way to see it and a terrible way to implement it. The zeros contribute nothing, so real implementations never materialise them — they just re-index. A 3-tap kernel at d=8 spans 17 samples for 3 multiplies; a plain 17-tap filter would cost 17 weights and 17 multiplies to reach the same distance.
Why doubling the rate is exponential
Widenings simply add up, so a stride-1 stack has:
def receptive_field(ks, ds):
return 1 + sum((k - 1) * d for k, d in zip(ks, ds))
Now choose d_i = 2^i. The sum becomes a geometric series and, for k=3, collapses to a closed form:
L = 10
receptive_field([3]*L, [2**i for i in range(L)]) # 2047 = 2**(L+1) - 1
receptive_field([3]*L, [1]*L) # 21 = 2L + 1
3, 7, 15, 31, 63, 127 … doubling with every single layer. Ten layers reach 2047 samples where plain stacking reaches 21 — a 97× larger field for identical parameters and identical FLOPs. A plain stack would need 1023 layers to match. That number is what made WaveNet possible.
Don't trust the algebra — measure it. With 'valid' stride-1 layers the output shrinks by exactly RF−1, so you can read the receptive field straight off the shapes:
def run_stack(x, ks, ds):
for k, d in zip(ks, ds):
x = conv1d(x, [1.0]*k, d)
return x
N = 400
assert N - len(run_stack(x400, [3]*4, [1,2,4,8])) + 1 == receptive_field([3]*4, [1,2,4,8]) == 31
Resolution is never touched
This is the property that separates dilation from every other way of buying context. There is no stride and no pooling, so with 'same' padding the output is exactly as long as the input — 64 in, 64 out, at d=1 or d=16. Segmentation gets a full-resolution logit map with a huge field of view and no decoder to reconstruct lost detail. WaveNet emits one sample per input sample.
It is also worth seeing that dilation does not just add context — it selects a scale. A [1,0,−1] kernel differences samples 2d apart, so its gain on a wave of frequency ω is 2|sin(ωd)|. Take a signal carrying a period-4 ripple (ω = π/2) on top of a slow swell:
gain = lambda w, d: 2 * abs(math.sin(w * d))
gain(math.pi/2, 1) # 2.0 ripple at MAXIMUM -> it dominates
gain(math.pi/2, 2) # 0.0 taps land on the SAME phase -> it vanishes
gain(0.18, 1) # 0.36 slow swell nearly invisible
gain(0.18, 8) # 1.98 slow swell peaks
Same three numbers, three completely different feature detectors, zero extra cost.
The catch: gridding
A wide receptive field is worthless if most of it is holes. What an output actually reads is the Minkowski sum of each layer's tap offsets, and if the rates share a common factor every tap lands on the same lattice:
def reachable(ks, ds):
S = {0}
for k, d in zip(ks, ds):
S = {p + t*d for p in S for t in range(k)}
return S
for ds in ([1,2,4], [4,4,4], [2,4,8], [1,2,5]):
rf, cov = receptive_field([3]*3, ds), len(reachable([3]*3, ds))
print(ds, f"RF={rf:3d} touched={cov:3d} {'OK' if cov == rf else 'GRIDDING'}")
# [1, 2, 4] RF= 15 touched= 15 OK
# [4, 4, 4] RF= 25 touched= 7 GRIDDING
# [2, 4, 8] RF= 29 touched= 15 GRIDDING
# [1, 2, 5] RF= 17 touched= 17 OK
Three layers at rate 4 span 25 inputs but read only 7 of them, one in every four — and neighbouring outputs then read disjoint sample sets, which shows up as checkerboard artifacts in segmentation masks. The fix is to make the rates co-prime, or use a sawtooth schedule (Hybrid Dilated Convolution: 1, 2, 5, 1, 2, 5 …) so the lattices interleave. Always check coverage, not just span.
Where it lives
WaveNet stacks causal dilated convolutions with rates cycling 1 → 512 and generates raw audio one sample at a time. DeepLab replaced the last pooling stages of a classifier backbone with atrous convolutions to keep the feature map dense, then added ASPP — the same features convolved in parallel at rates 6, 12, 18 and concatenated, so one layer sees several scales at once. TCNs use dilated causal convolutions to beat RNNs on long sequences while training fully in parallel.
# TCN / WaveNet block — dilation is just an argument
nn.Conv1d(c, c, 3, dilation=2**i, padding=2**i)
# ASPP — one input, four scales, concatenated
[nn.Conv2d(c, c, 3, dilation=r, padding=r) for r in (1, 6, 12, 18)]
The honest caveat: dilation samples sparsely, so very large rates can miss fine local detail. That is exactly why ASPP keeps a d=1 branch alongside the wide ones.
Trace the real dependency cone from one output back to every input it touches, and watch gridding appear when you get the rates wrong: https://dev48v.infy.uk/dl/day59-dilated-convolutions.html
Top comments (0)