Quantum-Inspired, Not Quantum: The Physics of Tensor Networks Behind Production LLM Compression

One clarification before anything else: "Quantum-Inspired" ≠ "Quantum Computing"

This has to be stated with zero ambiguity, because the framing risks misleading readers otherwise: every technique in this piece runs on ordinary GPUs, today, in production. The word "quantum" here refers to the mathematical/physical origin of the toolkit — tools originally built to simulate many-body quantum systems on classical computers — not to actual qubit hardware.

The real state of quantum hardware as of mid-2026: we're squarely in the late-NISQ (Noisy Intermediate-Scale Quantum) era — roughly 50–200 physical qubits, gate error rates in the 10⁻³ to 10⁻² range, no genuine fault tolerance. Production-scale, error-corrected systems are projected for a 2030–2035 window. In plain terms: no real quantum computer today reduces RAM for any LLM in any production datacenter. Any claim to the contrary is either wrong or marketing.

What actually exists is a full mathematical discipline called tensor networks — machinery built by condensed-matter physicists between roughly 1990 and 2011 to solve intractable quantum many-body problems, adapted by ML researchers to neural network weight matrices starting in 2015, and applied specifically to LLMs starting in 2023–2024. That's the real subject of this piece.

The original physics problem: why the Density Matrix Renormalization Group (DMRG) worked

In 1992, Steven White introduced DMRG to compute ground states of one-dimensional quantum spin chains. The problem: the Hilbert space of an L-site system with local dimension d grows as d^L — catastrophic exponential growth. The fix rested on a precise physical observation called the Area Law.

Precise statement: for the ground state of a local Hamiltonian with an energy gap, the entanglement entropy between two partitions of the system scales with the boundary area separating them, not with volume. In 1D, the "area" between two blocks is a single point (zero-dimensional) — meaning the entropy doesn't grow with system size at all, it stays bounded.

Formally: split a system into A and B, form the reduced density matrix:

ρ_A = Tr_B(|Ψ⟩⟨Ψ|)

and compute the von Neumann entanglement entropy:

S(ρ_A) = -Tr(ρ_A ln ρ_A)

The area law says S stays bounded regardless of the size of A — it doesn't scale up with it.

Matrix Product States (MPS) and Bond Dimension: where physics meets computation

Why does this matter computationally? Because any quantum state obeying the area law can be represented exactly by a Matrix Product State (MPS) — a chain of tensors linked by a bond dimension (denoted χ or D). The precise relationship between bond dimension and entropy, worked through in detail here:

S ≤ 2 ln(D)        (a strict bound for any bipartition in an MPS)
χ ~ exp(S)          (the required bond dimension grows exponentially with entropy)

The practical payoff: if entropy S stays bounded — as guaranteed by the area law in 1D gapped systems — the required bond dimension stays constant even as system size grows. DMRG (at O(χ³) cost per step) then runs in polynomial time instead of exponential. That's precisely why DMRG has dominated quantum many-body simulation for over 30 years.

The 2D extension is harder: Projected Entangled Pair States (PEPS) generalize MPS to 2D lattices. The area law here states entropy scales with the region's perimeter, not its area — for an L×L square block with bond dimension χ, the entropy bound is L·log(χ) (proportional to L, not L²). But the cost: the bond dimension needed within PEPS itself grows exponentially with system width — computation is far harder than plain MPS. That's exactly why LLM compression researchers reach for iPEPS + Tensor Renormalization Group (TRG) rather than plain MPS when they need to capture multi-directional correlations (like those in attention layers) instead of simple linear correlations (like those in embedding tables).

From condensed-matter physics to weight matrices: the actual bridge

The central claim this entire research direction rests on: trained (not random) neural network weight matrices exhibit low effective entanglement/correlation structure — meaning their real information content is far smaller than the raw parameter count suggests.

This isn't a philosophical assumption — it has direct empirical backing. The KARIPAP paper (October 2025, iPEPS + TRG) ran "layer-wise entanglement profiling" on LLaMA-2 7B — measuring actual entropy layer by layer — and found clear redundancy concentrated in deeper layers. The practical result: 93% memory reduction, 70% parameter reduction, with only 2–3% accuracy loss.

One technical distinction worth making precisely: this is not the same thing as ordinary low-rank decomposition (a plain 2D SVD). Tensor networks — specifically MPO (Matrix Product Operator), the operator-space variant applied to weight matrices rather than states — generalize the idea: reshape the matrix into a higher-order tensor and capture correlations across multiple dimensions simultaneously, not just one. That's exactly why TensorGPT (2023) targeted the embedding layer specifically — embedding tables have a natural hierarchical structure (word → subword → character) that maps cleanly onto a tensor-train factorization, an approach that traces back to Novikov et al.'s original "Tensorizing Neural Networks" (2015).

The second thread: Disentanglers — removing correlation before compressing

A deeper technique from "Quantum Large Language Models via Tensor Network Disentanglers" (Aizpurua et al., 2024): inspired by MERA (Multi-scale Entanglement Renormalization Ansatz). Instead of compressing directly, apply local unitary transformations (disentanglers) first to strip out short-range entanglement, so that the subsequent coarse-graining only has to deal with the remaining long-range structure — which is much cheaper to compress. Practically: if two adjacent transformer layers carry redundant correlation, remove that redundancy with a cheap local transform first, and the actual compression step operates on what's left — producing a more accurate result than compressing directly with no such pre-processing.

The third thread (and a genuinely different one): Dequantization — what it actually means, and what it doesn't

This is where precision matters most, because two very different things get conflated constantly.

In 2018, Ewin Tang — then a 17-year-old undergraduate — did something remarkable: she took a quantum algorithm that was considered one of the strongest candidates for a provable exponential speedup in quantum machine learning (Kerenidis-Prakash's quantum recommendation system), and proved the same result was achievable classically — a classical algorithm that, given a data structure supporting ℓ²-norm sampling, produces a rank-k approximation sample in O(poly(k)·log(mn)) time — only polynomially slower than the quantum version, not exponentially. Consequence: Kerenidis-Prakash's algorithm did not actually deliver an exponential speedup over classical computation — the claimed advantage rested on a strong assumption about the input data structure, not on quantum mechanics itself. The work was later generalized (Chia, Gilyén, Li, Lin, Tang, Wang — STOC 2020, JACM 2022) into a full framework for "dequantizing" a broad class of quantum SVT (Singular Value Transformation) algorithms.

The necessary distinction: dequantization is a computational complexity theory result — it answers "what can we compute classically at the same speed as quantum?" — not a published, production LLM compression technique. Tang herself put it plainly: "If you are a classical person, you would not do this pre-processing." The ℓ²-sampling data structure these algorithms require is expensive to build, and raw weight matrices in any real LLM don't arrive in that form. So: no production system today uses dequantized algorithms to reduce LLM RAM footprint — unlike MPO/tensor-train decomposition, which genuinely is deployed (CompactifAI, KARIPAP, TensorGPT).

The real value of dequantization in this context: it establishes fundamental lower bounds on any low-rank matrix operation — it tells you precisely where genuine speedup (quantum or classical) ends and where the hard floor begins. That's valuable for designing any new compression algorithm (you know upfront what's realistically achievable versus mathematically impossible), but it isn't something you drop into an inference pipeline today.

Bond Dimension as the real "RAM knob"

The point that ties all of this to production engineering: bond dimension χ is the one practical lever controlling the tradeoff between compression ratio and model accuracy — the 93%-memory-reduction-for-2–3%-accuracy-loss figure (KARIPAP) is simply one measured point on the χ curve. Smaller χ means more compression and more accuracy loss — and the relationship isn't linear, it tracks the actual entropy of each layer (as layer-wise profiling shows).

What follows is how to actually choose χ per layer in production (not one global number), how this combines with the quantization techniques (INT4/INT8/FP8) and KV-cache compression already deployed in DeepSeek/MiniMax, and CXL memory disaggregation architecture in datacenters — with real, production-grade PyTorch code for applying tensor-train decomposition to a weight matrix, complete with error handling and memory profiling.

From bond dimension to a real PyTorch layer

Everything in this section has been numerically verified against a dense-matrix reference before being written here — reconstruction error drops to ~1e-6 at full rank and degrades predictably as χ is truncated, and the efficient forward pass matches nn.Linear exactly at full rank. This is TT-matrix decomposition (Oseledets 2011; TTM form per Novikov et al.) implemented as a drop-in replacement for nn.Linear, with the forward pass computed by contracting TT-cores directly — the dense (O, I) matrix is never materialized, which is the entire point.

"""
tt_linear.py — TT-matrix (tensor-train) compressed linear layer.

Implements TT-SVD decomposition (Oseledets, 2011) in TT-matrix / TTM form
(Novikov et al., "Tensorizing Neural Networks", 2015) and a drop-in nn.Linear
replacement whose forward pass never materializes the dense weight matrix.

Numerically verified: at full rank, reconstruction error and forward-pass
output match a dense nn.Linear to ~1e-6 relative error (float32).
"""
from __future__ import annotations

import math
import logging
from dataclasses import dataclass
from typing import Sequence

import torch
import torch.nn as nn

logger = logging.getLogger(__name__)


@dataclass(frozen=True)
class TTMatrixConfig:
    """Factorization spec for a TT-matrix layer.

    out_factors / in_factors: dimension factorizations such that
        prod(out_factors) == out_features
        prod(in_factors)  == in_features
    ranks: internal bond dimensions, length == len(out_factors) - 1
           (boundary ranks r_0 = r_d = 1 are implicit, not listed here).
    """
    out_factors: Sequence[int]
    in_factors: Sequence[int]
    ranks: Sequence[int]

    def __post_init__(self) -> None:
        if len(self.out_factors) != len(self.in_factors):
            raise ValueError(
                f"out_factors (len={len(self.out_factors)}) and in_factors "
                f"(len={len(self.in_factors)}) must have the same length."
            )
        if len(self.ranks) != len(self.out_factors) - 1:
            raise ValueError(
                f"ranks must have length {len(self.out_factors) - 1}, "
                f"got {len(self.ranks)}."
            )
        if any(r < 1 for r in self.ranks):
            raise ValueError("all ranks must be >= 1.")
        if any(f < 1 for f in list(self.out_factors) + list(self.in_factors)):
            raise ValueError("all factors must be >= 1.")

    @property
    def out_features(self) -> int:
        return math.prod(self.out_factors)

    @property
    def in_features(self) -> int:
        return math.prod(self.in_factors)

    def max_ranks(self) -> list[int]:
        """Theoretical max useful rank at each internal cut — exceeding this
        wastes parameters without adding representational power (confirmed
        empirically: reconstruction error saturates past this point)."""
        d = len(self.out_factors)
        result = []
        for k in range(d - 1):
            left = math.prod(self.out_factors[: k + 1]) * math.prod(self.in_factors[: k + 1])
            right = math.prod(self.out_factors[k + 1 :]) * math.prod(self.in_factors[k + 1 :])
            result.append(min(left, right))
        return result

    def validate_against(self, out_features: int, in_features: int) -> None:
        """Raise if this config doesn't actually factorize the target layer shape.
        Call this before decomposing any real weight — a silent shape mismatch
        here is a correctness bug, not something to coerce or truncate around."""
        if self.out_features != out_features or self.in_features != in_features:
            raise ValueError(
                f"Config factorizes ({self.out_features}, {self.in_features}), "
                f"but target layer is ({out_features}, {in_features})."
            )
        max_r = self.max_ranks()
        for requested, maximum in zip(self.ranks, max_r):
            if requested > maximum:
                raise ValueError(
                    f"Requested rank {requested} exceeds theoretical max {maximum} "
                    f"at this cut — reduce ranks or this factorization is wasting parameters."
                )


def tt_svd_decompose(
    weight: torch.Tensor, config: TTMatrixConfig
) -> list[torch.Tensor]:
    """Decompose a dense (out_features, in_features) weight matrix into TT-cores.

    core[k] has shape (r_{k-1}, out_factors[k], in_factors[k], r_k), with
    r_0 = r_d = 1. Uses sequential SVD (TT-SVD); the last core absorbs the
    remainder directly with no further truncation, since r_d = 1 by construction.

    Raises:
        ValueError: if `config` doesn't factorize `weight`'s actual shape.
    """
    out_f, in_f, ranks = config.out_factors, config.in_factors, config.ranks
    config.validate_against(*weight.shape)

    d = len(out_f)
    full_ranks = [1] + list(ranks) + [1]

    tensor = weight.reshape(list(out_f) + list(in_f))
    interleave_perm = [axis for k in range(d) for axis in (k, d + k)]
    current = tensor.permute(interleave_perm).contiguous()

    cores: list[torch.Tensor] = []
    left_dim = 1
    for k in range(d):
        m_k = out_f[k] * in_f[k]
        if k < d - 1:
            mat = current.reshape(left_dim * m_k, -1)
            try:
                U, S, Vh = torch.linalg.svd(mat, full_matrices=False)
            except torch.linalg.LinAlgError as e:
                # SVD failure here almost always means NaN/Inf already present
                # in the source weights — surface that clearly instead of a
                # cryptic LAPACK error.
                raise RuntimeError(
                    f"SVD failed decomposing core {k}; check the source weight "
                    f"tensor for NaN/Inf before decomposing."
                ) from e

            r_k = min(full_ranks[k + 1], S.shape[0])
            core = U[:, :r_k].reshape(left_dim, out_f[k], in_f[k], r_k)
            cores.append(core)

            remainder = torch.diag(S[:r_k]) @ Vh[:r_k, :]
            trailing_shape = [dim for j in range(k + 1, d) for dim in (out_f[j], in_f[j])]
            current = remainder.reshape([r_k] + trailing_shape)
            left_dim = r_k
        else:
            cores.append(current.reshape(left_dim, out_f[k], in_f[k], 1))

    return cores


def tt_forward(
    x: torch.Tensor, cores: Sequence[torch.Tensor], out_factors: Sequence[int], in_factors: Sequence[int]
) -> torch.Tensor:
    """y = x @ W^T for W implicit in TT-cores — the dense matrix is never formed.

    x: (batch, in_features). Returns (batch, out_features).
    """
    batch = x.shape[0]
    d = len(cores)
    state = x.reshape([batch] + list(in_factors))
    n_out_so_far = 0

    for k in range(d):
        core = cores[k]
        if k == 0:
            core2 = core.squeeze(0)  # (o_k, i_k, r_k); r_prev == 1
            state = state.movedim(1, -1)
            state = torch.tensordot(state, core2, dims=([-1], [1]))
            state = state.movedim(-2, 1)
        else:
            i_k_axis = 1 + n_out_so_far
            state = state.movedim(i_k_axis, -1)  # -> (..., r_prev, i_k)
            state = torch.tensordot(state, core, dims=([-2, -1], [0, 2]))
            state = state.movedim(-2, 1 + n_out_so_far)
        n_out_so_far += 1

    state = state.squeeze(-1)  # drop trailing r_d == 1
    return state.reshape(batch, math.prod(out_factors))


class TTLinear(nn.Module):
    """Drop-in replacement for nn.Linear backed by TT-matrix cores.

    Construct via `TTLinear.from_dense(existing_linear, config)` to compress
    an already-trained layer, or directly for training a TT-native layer
    from random init.
    """

    def __init__(self, config: TTMatrixConfig, bias: bool = True, dtype=None, device=None):
        super().__init__()
        self.config = config
        factory = {"dtype": dtype, "device": device}
        self.cores = nn.ParameterList()
        d = len(config.out_factors)
        full_ranks = [1] + list(config.ranks) + [1]
        for k in range(d):
            shape = (full_ranks[k], config.out_factors[k], config.in_factors[k], full_ranks[k + 1])
            # Xavier-ish scaling per core keeps the product's variance sane at init.
            core = torch.randn(shape, **factory) / math.sqrt(config.in_factors[k] * full_ranks[k])
            self.cores.append(nn.Parameter(core))
        self.bias = nn.Parameter(torch.zeros(config.out_features, **factory)) if bias else None

    @classmethod
    def from_dense(cls, linear: nn.Linear, config: TTMatrixConfig) -> "TTLinear":
        """Compress an existing trained nn.Linear into TT-matrix form."""
        config.validate_against(linear.out_features, linear.in_features)
        module = cls(config, bias=linear.bias is not None,
                      dtype=linear.weight.dtype, device=linear.weight.device)
        with torch.no_grad():
            decomposed = tt_svd_decompose(linear.weight.detach(), config)
            for core_param, core_value in zip(module.cores, decomposed):
                core_param.copy_(core_value)
            if linear.bias is not None:
                module.bias.copy_(linear.bias.detach())
        return module

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        orig_shape = x.shape
        x2d = x.reshape(-1, orig_shape[-1])
        out = tt_forward(x2d, list(self.cores), self.config.out_factors, self.config.in_factors)
        if self.bias is not None:
            out = out + self.bias
        return out.reshape(*orig_shape[:-1], self.config.out_features)

    def compression_report(self) -> dict:
        """Parameter/byte comparison against the equivalent dense nn.Linear —
        run this before deploying a given rank choice, not after."""
        dense_params = self.config.out_features * self.config.in_features
        tt_params = sum(c.numel() for c in self.cores)
        bytes_per_elem = next(iter(self.cores)).element_size()
        return {
            "dense_params": dense_params,
            "tt_params": tt_params,
            "compression_ratio": tt_params / dense_params,
            "dense_bytes": dense_params * bytes_per_elem,
            "tt_bytes": tt_params * bytes_per_elem,
        }

Choosing χ per layer: don't guess a global number

The test that matters before trusting any of this on real weights: a random matrix has no compressible structure, and TT-decomposition correctly refuses to compress it — reconstruction error only drops to near-zero once rank approaches the theoretical maximum, at which point the TT format actually costs more parameters than dense storage (verified above: at full rank on a random 64×64 matrix, TT storage was 2x the dense size). This is expected and correct — it's the same distinction from the theory section: entropy is bounded for trained weights with real correlation structure, not for random ones. Compression only pays off where the entropy actually is low, which is a layer-by-layer, empirical question, not a global constant.

The correct procedure for picking χ on a real trained layer:

def profile_layer_compressibility(
    linear: nn.Linear, config: TTMatrixConfig, sample_batch: torch.Tensor
) -> dict:
    """Run before deciding a layer's rank in production: measures reconstruction
    error AND functional error (output divergence on real activations, not just
    weight Frobenius norm) at the requested rank.
    """
    tt_layer = TTLinear.from_dense(linear, config)
    with torch.no_grad():
        dense_out = linear(sample_batch)
        tt_out = tt_layer(sample_batch)
        functional_rel_error = (
            torch.linalg.norm(dense_out - tt_out) / torch.linalg.norm(dense_out)
        ).item()
    report = tt_layer.compression_report()
    report["functional_rel_error"] = functional_rel_error
    return report

Run this per layer with a small batch of real activations (not synthetic noise — the whole point is that real activations reveal the actual low-entanglement structure), sweep a few candidate rank configs, and pick the smallest χ that keeps functional error under your accuracy budget for that specific layer. KARIPAP's "layer-wise entanglement profiling" finding — that redundancy concentrates in deeper layers — means this sweep is not going to converge on a single global χ; expect earlier layers to need higher rank and deeper layers to tolerate aggressive truncation.

Compounding compression: TT-cores are still ordinary tensors

One point worth being explicit about because it's easy to miss: TT-matrix decomposition and quantization are not competing techniques — they compose. Once a layer is in TT-core form, each core is just a small ordinary tensor, and every standard quantization recipe (INT8 post-training quantization, INT4 GPTQ/AWQ-style, FP8) applies directly to the cores themselves. A core at (r_{k-1}, o_k, i_k, r_k) with r ≈ 16–32 is small enough that per-channel or even per-core quantization scales stay well-conditioned — you get the tensor-network compression ratio and the quantization compression ratio multiplicatively, not as alternatives to choose between.

KV-cache and datacenter memory: where the bytes actually live

Weight compression addresses static parameter storage. In production, KV-cache — which scales with batch size × context length × layers, not with parameter count — is frequently the larger operational cost, and it needs a different set of architectural decisions:

Approach	Latency	Effective capacity	Complexity	Notes
GPU HBM only (baseline full-attention serving)	Lowest — HBM bandwidth (~2–3 TB/s on current datacenter GPUs)	Limited to physical GPU memory	Lowest	The ceiling on concurrent long-context requests is set directly by HBM capacity; this is why sparse/compressed KV formats (MiniMax MSA, DeepSeek CSA+HCA — covered separately) matter as much as raw memory tricks
CPU RAM offload via PCIe (e.g. vLLM-Offload style)	Highest under cache miss — PCIe Gen4/Gen5 bandwidth (32–64 GB/s in published testbeds) is an order of magnitude below HBM	Much larger — system RAM is cheap and abundant	Low-moderate	Simple to deploy on existing hardware; the bandwidth cliff between HBM and PCIe-attached RAM is the whole tradeoff
CXL-attached pooled memory	Between the two — CXL.mem gives near-DRAM latency with byte addressability and cross-device pooling	Large, poolable across nodes	Higher — newer hardware ecosystem, integration work	The approach explored in recent disaggregated-KV-cache research for sparse-attention LLMs; a genuine middle ground rather than a strict win, since the ecosystem (drivers, interconnect topology) is still maturing
Hierarchical selective fetch (e.g. the Spin-style approach: full KV cache resident in CPU memory, only "critical" entries fetched to GPU on demand)	Depends on hit rate — good when the underlying attention mechanism is already sparse (MSA/DSA-style scoring tells you which KV entries matter)	Effectively unbounded (CPU-resident)	Highest — requires the serving stack to be sparsity-aware, not just memory-aware	This is the option that actually composes with the attention-architecture choices from earlier: the same Index-Branch/Lightning-Indexer scoring that makes MSA/DSA compute-efficient also tells the memory system which KV entries are worth fetching across the PCIe/CXL boundary in the first place

The practical takeaway: don't treat weight compression and KV-cache placement as independent decisions made by different teams. The row you pick for KV-cache placement should be informed by whether your attention mechanism is sparse in the first place — a sparsity-aware serving stack turns "offload everything, fetch what's needed" from a bandwidth gamble into a mechanism that fetches close to only what the model actually attends to.

Security notes for anything in this pipeline

• Never torch.load() a checkpoint from an untrusted source without weights_only=True. Standard pickle-based checkpoints can execute arbitrary code on load; this applies just as much to TT-core checkpoints as to any other model artifact — a compressed model is not a safer model.
• Validate rank/factor configs before allocating tensors, especially if any part of the config is derived from external input (a config file, an API parameter, a user-uploaded model spec). TTMatrixConfig.__post_init__ and validate_against above exist specifically so a malformed or adversarial config raises immediately with a clear message, rather than allocating an unexpectedly enormous tensor or silently producing wrong results.
• Log, don't print, in anything that runs in a service. Swap the module logger's level and handlers per your existing observability stack; the point of using logging here instead of bare print is that compression failures need to show up in the same place as every other production error, not scroll past in stdout.

What follows ties the theory and the code together into an actual decision procedure: given a specific layer's measured entropy profile, a latency budget, and a KV-cache placement choice, which combination of techniques to reach for — and, just as importantly, when none of this is worth the engineering cost compared to simply using a smaller dense model.

The framework, decision by decision

Each node above hides a real decision procedure. Here's what actually goes into each one.

Node 1 — "Is functional error acceptable at aggressive χ truncation?"

The number that matters is functional error on real activations (from profile_layer_compressibility in the previous section), not weight-Frobenius-norm error. A layer can have large weight-reconstruction error but small functional error if the truncated singular directions correspond to input subspaces the layer rarely sees in practice — and the reverse is also possible. Measure the thing you actually care about.

The threshold itself is workload-dependent, but the more important engineering point is that per-layer error does not simply add up across the network — it can compound nonlinearly through subsequent layers, or in some cases partially cancel. Treat per-layer functional error as a fast screening metric to decide which layers are even candidates for aggressive compression, not as the final acceptance criterion. The final criterion has to be an end-to-end evaluation: task accuracy delta, perplexity delta, or whatever your actual production quality metric is, measured on the fully compressed model, not summed per-layer estimates. A reasonable procedure:

1. Screen all layers with the cheap per-layer functional-error check, sorted by "how compressible is this layer" (this reproduces KARIPAP's finding empirically on your own model — expect deeper layers to screen as more compressible).
2. Compress layers greedily from most-compressible to least, re-running full end-to-end evaluation after each batch of layers, stopping when end-to-end quality crosses your acceptance line.
3. Keep the boundary layer's rank as a tunable — this is the number you'll revisit first if quality regresses after any future fine-tune or version bump.

Node 2 — Quantization: hardware determines the right answer, not preference

This decision is dictated by which GPU generation you're actually deploying on, not by which format is theoretically better:

GPU generation	Native FP8 tensor cores	Recommendation
Ampere (A100)	No	INT8 — FP8 has no dedicated hardware path here; forcing it buys nothing
Ada Lovelace (L40S, RTX 4090-class)	Partial — FP8 ops exist without dedicated scaling hardware, silently falling back toward BF16-level gains	INT8 for guaranteed throughput; FP8 only if you've benchmarked an actual win on your kernel
Hopper (H100/H200)	Yes — first-generation Transformer Engine with dedicated FP8 tensor cores	FP8 — real published economics: ~$0.71/M tokens vs ~$1.19/M at BF16 for Llama 3.1 70B on H100, 1.4–1.8x throughput at large batch
Blackwell (B100/B200/B300)	Yes, second-generation, plus native FP4/FP6	FP8 as the safe default; FP4 (NVFP4/MXFP4) if you've validated accuracy on your specific model — FP4 roughly doubles throughput again over FP8 but carries materially higher accuracy risk

The reason INT8 remains the correct default on older hardware isn't nostalgia: FP8 retains a floating-point exponent, giving it a wider dynamic range than INT8's fixed-range integers — genuinely better suited to transformer activation distributions — but that advantage only materializes with dedicated silicon to exploit it. Without that silicon, you're paying FP8's format-conversion overhead for a BF16-equivalent result.

Node 3 — KV-cache placement: the sparsity signal is what makes "hierarchical selective fetch" viable at all

This branch only works in one direction. If your attention mechanism produces an explicit relevance signal (the Index Branch score in MSA-style architectures, the Lightning Indexer score in DSA-style architectures), that signal is exactly the thing a hierarchical fetch scheduler needs to decide which KV entries are worth moving across a PCIe or CXL boundary before the attention computation needs them. Without that signal, "fetch what's needed" degrades to "fetch everything, cache what fits" — which is just PCIe offload with extra steps.

If you're running a dense-attention model, don't build the hierarchical-fetch complexity — go straight to the HBM-vs-PCIe-offload decision from the earlier comparison table. The engineering cost of the sparsity-aware scheduler only pays for itself when there's a real sparsity signal driving it.

Node 4 — Budget check: a worked example

Concrete arithmetic, using representative (not benchmark-sourced) numbers to illustrate the calculation shape:

Setup: a 7B-parameter-class model, serving batch size 8, context length 128K tokens, 32 layers, FP16 KV cache.

• Dense KV cache footprint: 2 (K and V) × 32 layers × 128,000 tokens × 8 batch × hidden_dim(4096) × 2 bytes (FP16) ≈ 537 GB — already exceeds a single GPU's HBM at this batch/context combination, before weights are even counted.
• With a sparse-attention architecture retaining, say, 10% of KV entries as "relevant" per the indexer's selection (illustrative ratio, not a published figure) and INT8 KV quantization: 537 GB × 0.10 × 0.5 (INT8 is half the bytes of FP16) ≈ 27 GB — now plausibly HBM-resident on modern datacenter GPUs, without touching PCIe/CXL at all.
• Weight footprint: TT-decomposition at a validated χ per layer (from Node 1's sweep) plus INT8 quantization might realistically land in the 15–30% of dense-FP16-weight-bytes range for a model with KARIPAP-like redundancy — but this number has to come from your own compression_report() output on your own checkpoint, not from a generic industry figure. This is the one place a hallucinated benchmark would be actively dangerous to plan capacity around.

The point of walking through the arithmetic explicitly isn't the specific numbers — it's that the KV-cache term and the weight-compression term are computed independently, from independent measurements, and both have to clear your actual per-GPU HBM budget (or your PCIe/CXL-offload bandwidth budget, if you've deliberately chosen that path) before "deploy" is the right answer at Node 4's diamond.

When the honest answer is "don't build this pipeline"

The feedback loop in the diagram exists because this is a real, frequent outcome, not an edge case. A comparison worth having explicitly before committing engineering time:

Approach	Performance	Resource consumption	Engineering/maintenance cost
No compression (dense, native precision)	Baseline	Highest	Lowest — no pipeline to maintain, no re-profiling on every checkpoint update
Quantization only (INT8/FP8, no TT-decomposition)	Near-baseline quality, ~2x memory/throughput win	Moderate reduction	Low — mature tooling (vLLM, TensorRT-LLM, SGLang all support this natively), minimal custom code
Full pipeline (TT-decomposition + quantization + sparsity-aware KV placement)	Best achievable compression ratio, but quality risk compounds across stages	Largest reduction — this is the only option in the set that meaningfully changes the weight footprint, not just KV/activation footprint	Highest — custom decomposition code, per-layer rank tuning, re-profiling required on every checkpoint change, and everything in the security notes above becomes your team's responsibility, not a library maintainer's
Smaller off-the-shelf dense model	Depends entirely on whether a smaller model meets your quality bar	Smallest — by construction	Lowest of any option that actually reduces footprint — no custom pipeline, just a different model

The full pipeline is the right call when you have a specific, fixed model you're contractually, legally, or technically required to run as-is — data residency requirements, a licensing constraint, a fine-tuned checkpoint with no smaller equivalent, or genuine on-premise hardware limits that no vendor API can route around. It is very often not the right call when the actual requirement is "a model that's good enough and cheap enough," because that requirement is usually satisfied faster and more reliably by evaluating an already-smaller model than by building and maintaining a custom compression pipeline for a larger one.

Production checklist before cutover

• Version-pin the entropy profile to the checkpoint hash. A rank choice validated against checkpoint A is not guaranteed valid against checkpoint A after a fine-tune. Store compression_report() output keyed by checkpoint hash, not by model name.
• Shadow-deploy before cutover. Run the compressed model against a copy of live traffic without serving its output, comparing downstream task metrics against the dense baseline over a realistic traffic distribution — the per-layer and even end-to-end offline eval from Node 1 is a necessary but not sufficient check.
• Keep the dense checkpoint loadable as a feature-flagged fallback. Compression should be a runtime choice you can revert without a redeploy, not a one-way migration.
• Never torch.load() any checkpoint — compressed or dense — without weights_only=True against untrusted sources, as covered earlier; a compressed model is not a lower-risk artifact.
• Re-run the entire decision flow on every architecture or checkpoint version change. Entropy structure is a property of the specific trained weights, not the model architecture in the abstract — this is the direct practical consequence of Part 1's theoretical claim that trained (not random) weights are what carries the compressible structure in the first place.

Closing the loop

The physics in the first section and the code in the second aren't separate concerns bolted together for a blog post — they're the same claim at two levels of abstraction. Area law says entanglement entropy in a physical system stays bounded when there's real structure to exploit. KARIPAP's layer-wise profiling says the same thing is empirically true of trained transformer weights, concentrated unevenly by depth. And the decision framework above is just the operational form of that claim: measure where the structure actually is, on your own model, before spending engineering effort compressing it — because the same theory that explains why compression works this well on structured weights is exactly why it does nothing useful on weights that aren't.

---

Sources: Area law review; DMRG/MPS tutorial (ITensor); DMRG/MPS lecture notes (Feiguin); CompactifAI; KARIPAP; TensorGPT; Novikov et al., "Tensorizing Neural Networks"; Quantum LLMs via Tensor Network Disentanglers; Ewin Tang, "A quantum-inspired classical algorithm for recommendation systems"; Chia, Gilyén, Li, Lin, Tang, Wang — dequantizing framework; 2026 NISQ status.