Mixup and CutMix: train on images that don't exist by blending two pictures AND their labels into a soft target

Ordinary augmentation transforms one image and keeps its label. Mixup and CutMix go further and, at first, sound absurd: they take two training images and combine them into a new example whose label is soft — part cat, part dog — because that's honestly how much of each is now in the picture. That single idea, blending the label along with the pixels, is a genuinely strong regulariser. I built a live demo with two hand-drawn canvases and a λ slider so you can watch the images blend and the label blend with them; here's what's actually happening.

Two examples, two one-hot labels

Everything operates on a pair. Grab two samples from the batch — image A labelled cat, image B labelled dog. Labels start as clean one-hot vectors; the softness is created by mixing, never assumed. In a real loop, B is just A shuffled: xb = xa[torch.randperm(n)].

const A = { img: canvasA, y: [1, 0] };   // a cat -> one-hot cat
const B = { img: canvasB, y: [0, 1] };   // a dog -> one-hot dog

Mixup — a pixel-wise convex blend

Every output pixel is λ·A + (1−λ)·B. On a canvas that's two draws with globalAlpha: paint A fully, then paint B at opacity 1−λ on top, and the compositor gives you exactly the convex combination — a translucent ghost of both animals.

function mixup(out, A, B, lam) {
  const ctx = out.getContext("2d");
  ctx.globalAlpha = 1;        ctx.drawImage(A, 0, 0);   // full cat
  ctx.globalAlpha = 1 - lam;  ctx.drawImage(B, 0, 0);   // dog ghosted on top
  ctx.globalAlpha = 1;
}

Blend the label the same way — the whole trick

This is the part beginners skip. Mixing pixels but keeping a hard label teaches a lie: "this 30%-dog image is 100% cat." So blend the label with the same λ. The target becomes λ·y_A + (1−λ)·y_B — an honest soft label that says exactly how much of each class is present. Now 100% confidence is literally the wrong answer, so the model can never justify a runaway spike.

function mixLabels(yA, yB, lam) {
  return yA.map((v, k) => lam * v + (1 - lam) * yB[k]);
}
const y = mixLabels(A.y, B.y, lam);   // lam=0.7 -> [0.7, 0.3]  (70% cat, 30% dog)

CutMix — cut a patch, mix the label by area

Mixup's translucent overlay looks unnatural and blurs local texture. CutMix keeps every pixel real: draw A, then stamp a rectangular patch of B on top. The box side is √(1−λ)·W at a random location, so its area is (1−λ) of the image, and the label is mixed by that area. Every pixel stays a sharp, genuine pixel of one image, and both objects survive intact.

function cutmix(out, A, B, lam) {
  const W = out.width, H = out.height, ctx = out.getContext("2d");
  ctx.drawImage(A, 0, 0);
  const s = Math.sqrt(1 - lam) * W;                // patch side, area = 1-lam
  const cx = Math.random() * W, cy = Math.random() * H;
  const x = clamp(cx - s/2, 0, W), y = clamp(cy - s/2, 0, H);
  const x2 = clamp(x + s, 0, W),  y2 = clamp(y + s, 0, H);
  ctx.drawImage(B, x, y, x2-x, y2-y, x, y, x2-x, y2-y);
  return { x, y, bw: x2-x, bh: y2-y };
}

Re-derive λ from the actual pasted area

Here's a subtlety I like: if the box runs off an edge it gets clipped, so the real patch is smaller than intended. Recompute λ from the true pasted area, so the label always matches the pixels actually on screen — honest even when the box is clipped. In the demo, "move patch" shoves the box into a corner and you watch the dog-fraction and the label shrink to match.

const area = (bw * bh) / (W * H);      // true fraction that is dog
const lamAdj = 1 - area;               // fraction still cat
const y = mixLabels(A.y, B.y, lamAdj);

One soft-label loss for both methods

Because the target is a mix of two one-hots, the cross-entropy factorises into the ordinary loss against each class, weighted by λ. You never write a special loss — just feed the soft target, or equivalently the two hard targets with weights.

function mixLoss(logits, yA, yB, lam) {
  return lam * crossEntropy(logits, yA) + (1 - lam) * crossEntropy(logits, yB);
}
// identical to crossEntropy(logits, mixLabels(yA, yB, lam))

Why it regularises, and the real thing

Training on in-between points with in-between labels forces the network to behave roughly linearly between classes. That straightens and widens the decision boundaries, sharply cuts memorisation of corrupted labels, and improves calibration — the same anti-overconfidence medicine as label smoothing, now sourced from real pixels of a second class. CutMix adds a localisation bonus, because genuine object parts survive, so the model learns to recognise an animal from a fragment. λ itself is drawn per batch from a symmetric Beta(α, α) — small α piles mass near 0 and 1 (mostly gentle mixes), α=1 is uniform, large α centres on 0.5 (aggressive) — and α is the one knob you tune.

In production you never hand-roll it. timm applies it per batch, even flipping between Mixup and CutMix randomly and folding in label smoothing.

from timm.data import Mixup
mixup_fn = Mixup(mixup_alpha=0.2, cutmix_alpha=1.0,
                 prob=1.0, switch_prob=0.5, label_smoothing=0.1, num_classes=1000)
for x, y in loader:
    x, y = mixup_fn(x, y)                          # blends images AND makes soft targets
    loss = SoftTargetCrossEntropy()(model(x), y)
    loss.backward()

Mind the caveats: convergence is a touch slower, so train a little longer, and pixel-exact tasks like segmentation need box-aware variants. But for classification it's near-zero extra compute for a solid top-1 gain — essentially free.

Drag the slider and watch the label melt:
https://dev48v.infy.uk/dl/day37-mixup-cutmix.html