Activation Outliers: Why W8A8 INT8 Quantization Needs SmoothQuant

Take a 7B+ decoder, dump the input activations to down_proj in layer 20, and look at the per-channel absolute max. Most of the 11008 channels sit around 0.3–1.0. Six or seven of them sit at 40–90. Same channel indices, every token, every prompt, across most layers.

Now quantize that tensor to INT8 with one scale: s = 70 / 127 ≈ 0.55. Your typical activation of 0.5 rounds to 1. Your 0.4 rounds to 1. Your 0.2 rounds to 0. You just replaced a 4096-dimensional vector with a 3-bit vector plus a few giant spikes, and your model now writes fluent nonsense while perplexity looks "only a bit worse."

That is the whole story of why W8A8 INT8 quantization fails on large language models. It is not a rounding-mode problem, and no amount of better calibration search fixes it. Activation outliers are structural, and the fix is an algebraic one.

TL;DR

• Activation outliers are systematic: a handful of hidden channels carry magnitudes one to two orders of magnitude above the rest, at the same indices across tokens and layers. LLM.int8() reported them emerging consistently around the ~6.7B parameter mark.
• Per-tensor INT8 activation quantization collapses because one channel sets the scale for all of them. Per-token scaling does not help — the outlier is inside every row.
• You cannot quantize activations per input channel in an INT8 GEMM: the channel axis is the reduction axis, so the scale does not factor out of the int32 accumulator.
• SmoothQuant exploits Y = (X diag(s)⁻¹)(diag(s) W) to migrate range from activations into weights per channel, then folds diag(s)⁻¹ into the preceding RMSNorm/LayerNorm affine. Zero runtime cost, mathematically exact.
• If your decode is memory-bound at small batch, skip activation quantization entirely and use weight-only 4-bit (AWQ/GPTQ). W8A8 only pays off when you are compute-bound: large-batch serving and prefill.

Why do activation outliers break per-tensor INT8 quantization?

Because INT8 has 256 levels of uniform resolution and outliers force nearly all of them to be spent on values that almost never occur.

Weights are easy to quantize. Per output channel, weight distributions are close to Gaussian with a tame dynamic range — INT8 per-channel weight quantization is essentially free in accuracy. Activations are not. The residual stream in a trained transformer develops a small set of channels that behave like fixed, high-magnitude biases. They correlate with the "massive activation" / attention-sink phenomenon: the model uses a few dimensions as a near-constant reference signal rather than as content.

The damage math is simple. For symmetric absmax quantization, the step size is Δ = max|X| / 127. Signal-to-noise for the non-outlier bulk scales with E|x| / Δ. Push max|X| up 100x and you lose ~6.6 bits of effective precision on the values that actually carry the token's meaning.

Clipping the outliers is not an escape hatch. Those channels are load-bearing; clip them and you break the sink behavior, which shows up as degraded long-context retrieval and unstable attention at exactly the point where quantization was supposed to be invisible.

Why can't you just quantize activations per channel?

Because of where the scale sits relative to the summation. This is the part most people get wrong, and it is pure linear algebra.

For Y = X W with X ∈ ℝ^{T×Cin}, W ∈ ℝ^{Cin×Cout}:

Y[t, o] = Σ_c  X[t, c] * W[c, o]

A per-token activation scale a[t] and a per-output-channel weight scale b[o] both pull straight out of the sum:

Y[t, o] ≈ a[t] * b[o] * Σ_c  Xq[t, c] * Wq[c, o]

That is why INT8 kernels are happy with per-token × per-out-channel: the int32 accumulator is computed once, then rescaled once in the epilogue.

A per-input-channel activation scale g[c] does not pull out:

Y[t, o] = Σ_c  g[c] * Xq[t, c] * W[c, o]

The scale is trapped inside the reduction. Applying it would mean dequantizing before accumulation — which is exactly what tensor cores refuse to do, and what would erase the throughput win you quantized for in the first place.

This asymmetry also explains why weight-only 4-bit quantization gets away with group-wise scales along Cin (group size 128 is standard in AWQ/GPTQ kernels). W4A16 kernels dequantize weights to FP16 and accumulate in FP16/FP32, so per-group scales are applied before the multiply-accumulate. INT8 W8A8 has no such freedom.

So per-channel is where the outliers are, and per-channel is the one axis you cannot use. That is the trap.

What does SmoothQuant actually do?

It moves the problem to an axis you can scale. For a diagonal diag(s) with positive entries:

Y = X W = (X diag(s)⁻¹) (diag(s) W)

Divide activation channel c by s[c], multiply weight row c by s[c]. Exact, not an approximation. Choose s so both operands become quantizable:

s[c] = max|X[:, c]|^α  /  max|W[c, :]|^(1-α)

α = 0.5 splits the difficulty evenly; higher α (0.75–0.85) is what the paper needed for the most outlier-heavy models like OPT-175B. SmoothQuant reports near-lossless W8A8 across OPT/BLOOM-class models with roughly 1.5x latency improvement and half the memory versus FP16.

The runtime trick is the good part: diag(s)⁻¹ never executes as a kernel. In a pre-norm transformer, the tensor feeding q/k/v and gate/up is an RMSNorm output, and RMSNorm's affine weight is an elementwise per-channel multiply after normalization. Divide that weight by s and the scaling is free and exact. The normalization statistic is unaffected because you changed the affine, not the input.

Two structural constraints follow directly:

1. Every linear consuming the same norm output must share one s. q_proj, k_proj, v_proj all read the same tensor, so you take the elementwise max over their weight ranges.
2. down_proj's input has no preceding norm — it comes from act(gate) * up. Fold s⁻¹ into up_proj's output channels instead. That is safe only because up_proj's output feeds nothing but the elementwise multiply. Never fold into a projection whose output also lands on a residual add.

How do you profile outliers and fold the scales?

Collect per-channel absmax over a calibration set, then apply the fold in place:

import collections
import torch

@torch.no_grad()
def collect_channel_absmax(model, batches, targets=("q_proj", "gate_proj", "down_proj")):
    """Per-input-channel absmax of the activations entering each target Linear."""
    stats = collections.defaultdict(lambda: None)
    hooks = []

    def make_hook(name):
        def hook(_mod, inp, _out):
            x = inp[0].detach()
            m = x.abs().reshape(-1, x.shape[-1]).amax(dim=0).float()
            stats[name] = m if stats[name] is None else torch.maximum(stats[name], m)
        return hook

    for name, mod in model.named_modules():
        if isinstance(mod, torch.nn.Linear) and any(t in name for t in targets):
            hooks.append(mod.register_forward_hook(make_hook(name)))
    for b in batches:
        model(**b)
    for h in hooks:
        h.remove()
    return dict(stats)


def outlier_ratio(absmax, x_sample):
    """absmax / p99.9 — above ~10 means per-tensor INT8 will not survive."""
    p999 = torch.quantile(x_sample.abs().flatten().float(), 0.999)
    return (absmax.max() / p999).item()


@torch.no_grad()
def smooth_norm_linears(norm, linears, act_absmax, alpha=0.5):
    """Fold diag(s)^-1 into the norm affine, diag(s) into the linear weights."""
    w_absmax = torch.stack([
        l.weight.abs().amax(dim=0) for l in linears   # [Cin] per linear
    ]).amax(dim=0).clamp(min=1e-5)

    s = (act_absmax.clamp(min=1e-5).pow(alpha) / w_absmax.pow(1 - alpha)).clamp(min=1e-5)
    s = s.to(norm.weight.dtype).to(norm.weight.device)

    norm.weight.div_(s)
    if getattr(norm, "bias", None) is not None:
        norm.bias.div_(s)
    for l in linears:
        l.weight.mul_(s.view(1, -1))

Two things this code makes concrete. First, act_absmax is a calibration artifact — if your calibration data does not cover the real input distribution, you underestimate s on some channels and those channels clip in production. Use a few hundred sequences drawn from actual traffic, including your longest prompts and your tool-call-heavy turns, not just WikiText. Second, verify layer-by-layer output MSE after folding; a silent nan from a zero-range weight channel is the classic bug, which is why every term is clamped.

How do you pick alpha without wrecking the weights?

Search it per layer against output error, not globally against perplexity.

α is a dial between two failure modes. Too low and activations stay spiky. Too high and you dump so much range onto weights that per-channel INT8 weights start clipping, which is worse because weight error is systematic rather than per-token noise. Sweep α ∈ {0.5, 0.6, 0.7, 0.8, 0.85} per decoder layer and keep the value minimizing ‖Y_fp16 − Y_int8‖² on calibration activations. This is the same objective AWQ optimizes for weight-only quantization, where activation magnitude is used to identify salient weight channels and scale them up before rounding.

Do not tune on perplexity. Perplexity is dominated by high-frequency tokens and hides precision loss beautifully. Outlier damage shows up first in exact-copy behavior: long identifiers, JSON field names, base64, digit sequences, multi-hop retrieval from mid-context. Build your quantization eval out of those.

When is W8A8 worth it at all?

Only when you are compute-bound. This is the decision most teams skip.

• Small-batch decode is memory-bandwidth-bound on weight loads. Quantizing activations buys nothing; the GEMMs are skinny. Use W4A16 (AWQ/GPTQ, group 128) and get a near-4x cut in weight traffic.
• Prefill and large-batch serving are compute-bound and arithmetic-intensive. Here INT8/FP8 tensor-core throughput is the point, and W8A8 earns its complexity.
• KV cache is a separate axis problem with its own asymmetry between keys and values; do not assume a W8A8 recipe transfers to it.
• MoE routers should stay in higher precision. They are tiny, and a rounding flip in router logits changes expert assignment, which is a discrete, non-recoverable error.

Does FP8 make activation outliers a non-issue?

Mostly, and for a specific reason: E4M3 spends four bits on the exponent, so its dynamic range spans several orders of magnitude. An outlier channel 100x above the bulk no longer forces the bulk toward zero — it just costs you mantissa precision. That is why per-tensor FP8 on Hopper-class hardware behaves so much better than per-tensor INT8, and why the standard FP8 recipe (per-token activations, per-channel weights) is usually near-lossless without any smoothing pass.

Blocked formats push this further and, in effect, solve the problem in hardware. MXFP4 attaches an E8M0 scale to every block of 32 values; NVFP4 uses blocks of 16 with an E4M3 scale. Those blocks run along the reduction axis — precisely the per-channel scaling that INT8 GEMMs cannot express, made legal because the hardware applies the block scale inside the MMA pipeline. The outlier problem was never about bit width. It was about scale granularity on the wrong axis.

Even so, 4-bit blocked formats still want a smoothing/rotation pass (SmoothQuant-style scaling or a Hadamard rotation à la QuaRot/SpinQuant) to spread outlier energy before rounding. The axis constraint relaxes; the distribution problem does not fully disappear.

Direct answer

W8A8 INT8 quantization needs SmoothQuant because large language models concentrate activation magnitude in a few fixed hidden channels, and the channel axis is the one axis an INT8 GEMM cannot scale — a per-input-channel scale sits inside the int32 accumulator's reduction, so it cannot be factored out the way per-token and per-output-channel scales can. One outlier channel therefore sets the scale for the entire tensor and quantizes the meaningful values into two or three effective bits. SmoothQuant sidesteps the constraint algebraically: divide activations by a per-channel s, multiply the corresponding weight rows by s, and fold the division into the preceding RMSNorm affine so it costs nothing at inference. Tune α per layer against output MSE, calibrate on real traffic, and check whether you are compute-bound before doing any of it — if you are bandwidth-bound at small batch, weight-only 4-bit is the better trade, and on FP8 or NVFP4 hardware the wide exponent and block-level scaling absorb most of the outlier damage for you.