Activation Outliers: Why W8A8 INT8 Quantization Wrecks Accuracy

You quantize a 7B model's weights to INT8. Perplexity moves in the third decimal place. Then you turn on activation quantization to get the actual 2x tensor-core speedup, and the model starts emitting the the the or confidently answering a different question. Same bit width. Same calibration set. The only thing you changed was quantizing the inputs to each GEMM.

W8A8 INT8 quantization fails where weight-only INT8 succeeds, and the reason has nothing to do with the number of bits. It's about roughly 0.1% of hidden dimensions whose magnitudes sit two orders of magnitude above everything else — and about which axis of a matrix multiply a quantization scale can legally live on.

TL;DR

• Weights are well-conditioned; activations are not. Transformer hidden states contain a small, fixed set of channels with |x| 20–100x the median, emergent past roughly 6–7B parameters. A per-tensor INT8 scale set by that max leaves ~2–3 effective bits for every other value.
• You cannot fix it with per-input-channel activation scales. In Y = XW, only scales along the row axis of X (per-token) and the column axis of W (per-output-channel) factor out of the inner product. Scales along the reduction dimension do not.
• Dynamic per-token quantization does not solve outlier channels. It removes calibration drift and clipping, but the row max is still dominated by the outlier channel sitting inside that same row.
• SmoothQuant migrates the range into the weights: XW = (X·diag(s)⁻¹)(diag(s)·W), with s_j = max|X_j|^α / max|W_j|^(1-α), folded into the preceding RMSNorm or linear so it costs nothing at runtime.
• FP8 mostly dodges the problem because E4M3 has ~4.5 orders of magnitude of normal-range dynamic range vs INT8's ~2.1 — outliers cost mantissa bits instead of clipping everything else to zero.

Why does W8A8 INT8 quantization break when weight-only INT8 doesn't?

Because the two tensors have completely different distributions, and INT8 is a uniform format that only cares about max/median ratio.

A weight matrix in a trained transformer looks roughly Gaussian, and its per-output-channel ranges are within a small factor of each other. Round-to-nearest INT8 with per-channel scales lands within noise of FP16. That's why W8-only quantization is boring and works everywhere.

Activations flowing into q/k/v_proj and gate/up_proj are not like that. A handful of residual-stream dimensions — the same dimension indices for every token, every prompt, across the whole model — carry magnitudes far above the rest. Round the tensor with one scale:

s = max|X| / 127

If max|X| = 70 and the typical value is 0.4, then s ≈ 0.55 and virtually every non-outlier value quantizes to −1, 0, or +1. You didn't build an 8-bit tensor. You built a 2-bit tensor with a few 8-bit spikes in it, and the model's actual signal lives in the part you destroyed.

This is why the failure is so bimodal: below the outlier-emergence scale (small models, ~1–2B) INT8 W8A8 is fine, and past it accuracy falls off a cliff rather than degrading smoothly.

Where do activation outliers actually live?

Two distinct phenomena get conflated, and they need different fixes.

Outlier channels live in the residual stream. After input_layernorm / post_attention_layernorm, specific hidden dims are persistently large. These are channel-consistent: dim 2533 is big for every token in every sequence. This is what SmoothQuant targets.

Massive activations are token-consistent: the BOS token, and often the first newline or a delimiter, carry hidden-state norms orders of magnitude above content tokens. This is the same structural phenomenon behind attention sinks — the model parks probability mass somewhere harmless. These wreck static per-tensor scales even after channel smoothing, because one token in the batch defines the range for all of them.

There's a third hot spot people miss: the input to down_proj. It is the output of the SwiGLU elementwise product act(gate) * up, so its dynamic range is the product of two distributions. In practice down_proj is the single worst layer for activation quantization in a Llama-style block, and it's the one that most often survives calibration and then blows up on real traffic.

Why can't you just use per-input-channel activation scales?

Because the scale has to factor out of the accumulator, and INT8 tensor cores only give you two places to put it.

Write the GEMM element-wise:

Y[i,j] = Σ_k X[i,k] · W[k,j]

Quantize with X[i,k] ≈ a_i · Xq[i,k] (per-token scale) and W[k,j] ≈ b_j · Wq[k,j] (per-output-channel scale). Then:

Y[i,j] ≈ a_i · b_j · Σ_k Xq[i,k]·Wq[k,j]

The scales come out of the sum. The hardware accumulates INT32 and you apply a_i · b_j once at epilogue. Free.

Now try a per-input-channel activation scale c_k:

Y[i,j] ≈ b_j · Σ_k c_k · Xq[i,k]·Wq[k,j]

c_k is inside the reduction. There is no INT32 accumulator that applies a different scale per accumulation step. To honor it you'd have to dequantize before accumulating, which means you're doing an FP GEMM and the whole exercise is pointless.

That constraint is the entire reason this problem is hard. Quantization difficulty lives on the axis you're not allowed to scale.

How does SmoothQuant move the difficulty into the weights?

By inserting a diagonal matrix and its inverse, then folding one side into an adjacent op at build time.

XW = (X · diag(s)⁻¹) · (diag(s) · W)

diag(s)⁻¹ shrinks outlier channels of X; diag(s) inflates the matching rows of W. Weights had slack, so they absorb it. Choose per-channel:

s_j = max|X_j|^α / max|W_j|^(1-α)

α=0.5 splits the difficulty evenly. Push α toward 0.75–0.85 for models where activations dominate; push it down when weight quantization starts to be the binding constraint. It's a one-dimensional knob you should sweep, not a constant you copy from a paper.

The folding step is what makes it free:

@torch.no_grad()
def smooth_ln_linears(ln, linears, act_absmax, alpha=0.5):
    """act_absmax: [hidden], per-input-channel max|X| from calibration.
    `linears` MUST be every consumer of `ln`'s output (q,k,v or gate,up)."""
    w_absmax = torch.stack(
        [l.weight.abs().amax(dim=0) for l in linears]  # [out,in] -> [in]
    ).amax(dim=0)

    s = (act_absmax.clamp(min=1e-5).pow(alpha) /
         w_absmax.clamp(min=1e-5).pow(1 - alpha)).clamp(min=1e-5)

    ln.weight.div_(s)                 # RMSNorm gain absorbs 1/s
    if getattr(ln, "bias", None) is not None:
        ln.bias.div_(s)
    for l in linears:
        l.weight.mul_(s.view(1, -1))  # scale input channels of W

Two rules that break this in practice:

1. Every consumer must be smoothed together. If you smooth input_layernorm for q and k but forget v, you've silently changed the function the model computes. This is the most common way a "SmoothQuant implementation" ends up numerically wrong while still producing fluent text.
2. The preceding op must be per-channel linear. RMSNorm gain, yes. A previous linear's output channels, yes. An op that mixes channels, no.

For down_proj there's no norm in front of it — but there is a foldable path, because SwiGLU is elementwise and only gate goes through the nonlinearity:

(act(gate) * up) / s  ==  act(gate) * (up / s)

So divide up_proj.weight rows by s and multiply down_proj.weight input channels by s. Leave gate_proj alone.

Does dynamic per-token quantization fix outliers?

No, and this is the misconception that costs people a week.

Dynamic per-token quantization computes a_i = max|X[i,:]| / 127 on the fly, one reduction over the row before the GEMM. It genuinely fixes two things: calibration drift (your 512-sample calibration set never saw a 32k code prompt), and massive-activation tokens (BOS gets its own scale instead of defining everyone's).

It does nothing about outlier channels, because the outlier channel is inside the row you're taking the max over. Token i still has its 70 sitting next to its 0.4. Per-token quantization changes which values share a scale; it does not change the max/median ratio within a scale group.

Correct mental model: dynamic per-token removes clipping and drift risk; channel smoothing removes resolution loss. Ship both. Static per-tensor activation scales have essentially no place in a production LLM serving stack.

What about rotations — QuaRot, SpinQuant, Hadamard?

Rotation attacks the same problem without needing a good α. Insert an orthogonal Q:

XW = (XQ)(QᵀW)

Fold Q into the weights offline. If Q is a random orthogonal or (better, because it's O(d log d) and needs no matrix stored) a Hadamard matrix, it mixes channels: the outlier's energy spreads across all dimensions, and the resulting tensor is closer to incoherent — max/median ratio drops toward what a Gaussian would give you.

The tradeoff is that some placements can't be folded and need an online Hadamard transform at runtime — typically before down_proj and on the attention output. That's a real kernel cost, small but nonzero. Rotation generally beats smoothing at 4-bit activations; at 8-bit, smoothing plus dynamic per-token is usually enough and much simpler to debug.

Why does FP8 W8A8 "just work" when INT8 doesn't?

Dynamic range. INT8 spans 127:1, about 2.1 orders of magnitude, uniformly spaced. FP8 E4M3 covers roughly 4.5 orders in the normal range (max 448, smallest normal 2⁻⁶) with logarithmically spaced exponents. An outlier at 100x the median costs you mantissa bits on the small values — not their entire representation.

That's why FP8 W8A8 on Hopper/Blackwell is close to a free win while INT8 W8A8 needs a quantization pipeline. It's also why INT8 still matters: on Ampere-class hardware there is no FP8 tensor core, and INT8 is the only path to 2x math throughput. If you're serving on A100s, smoothing isn't optional.

FP8 isn't immune either — per-tensor FP8 on outlier-heavy layers still degrades, which is why fine-grained (per-token / block-scaled) FP8 exists.

How do you diagnose this in an hour?

Skip end-to-end perplexity for the first pass. It tells you something broke, not where. Instrument per-linear:

stats = {}

def hook(name):
    def fn(mod, inp, out):
        x = inp[0].detach().float().flatten(0, -2)   # [tokens, in_features]
        m = x.abs().amax(dim=0)                       # per-input-channel max
        prev = stats.get(name)
        stats[name] = m if prev is None else torch.maximum(prev, m)
    return fn

for n, m in model.named_modules():
    if isinstance(m, torch.nn.Linear):
        m.register_forward_hook(hook(n))

# after calibration passes:
for n, m in stats.items():
    ratio = (m.max() / m.median()).item()
    if ratio > 20:
        print(f"{n:55s} max/med={ratio:7.1f} argmax_ch={m.argmax().item()}")

Sort by max/median. Anything above ~20 will not survive INT8 without smoothing. Confirm the same channel indices dominate across different prompts — if they do, it's structural and smoothing will hold; if they move, you have a data-distribution problem and dynamic quantization matters more than smoothing.

Then check layerwise fidelity rather than final loss: run the FP16 and quantized layer on identical input and compare relative error ‖Yq − Y‖ / ‖Y‖. Under ~1% per linear is generally survivable; a single layer at 10%+ is your culprit, and it's usually down_proj or a first/last decoder block.

The short answer

W8A8 INT8 quantization wrecks accuracy because transformer activations contain a persistent set of outlier channels 20–100x above the median, and a uniform 8-bit scale set by that maximum leaves only 2–3 effective bits for the values that carry the signal — while weights, being well-conditioned, quantize cleanly at the same bit width. You can't scale those channels away directly, because per-input-channel activation scales sit on the GEMM's reduction axis and won't factor out of the INT32 accumulator. The fixes route around that constraint: SmoothQuant migrates per-channel range from activations into weights via a diagonal transform folded into the preceding norm, Hadamard rotations spread outlier energy across all channels, and dynamic per-token scaling removes calibration drift. Use smoothing plus dynamic per-token together, sweep α, and check per-linear max/median ratios before you trust a perplexity number.