TurboQuant: How a Simple Spin Saves Gigabytes of GPU Memory

Bharath Kadaluri — Wed, 08 Apr 2026 19:24:45 +0000

Start With a Restaurant

Before we talk about AI, let me tell you about a busy restaurant

Imagine you manage a popular restaurant. Every order gets written down in full

Order #1: "One Chicken Biryani with extra raita and no onions, 
           One Butter Naan, Two Mango Lassi, Table 5, 7:30 PM"

Order #2: "Two Margherita Pizza with extra cheese and thin crust,
           One Veg Burger with no mayo, One Coke, Table 12, 7:45 PM"

Order #3: "Three Chicken Burger with extra spicy sauce,
           Two Masala Dosa, One Filter Coffee, Table 8, 8:00 PM"

Each order takes ~150 characters.On a busy Friday night with 500 orders, that's 75k characters, which is pages and pages of scribbled notes. The kitchen is drowning in paper.

Now, what if the kitchen agrees on a code system?

Codebook (shared between waiter and kitchen):

  Food:                  Extras:              Drinks:
  CB = Chicken Biryani   +R  = extra raita    ML = Mango Lassi
  MP = Margherita Pizza  +C  = extra cheese   CK = Coke
  VB = Veg Burger        +SP = extra spicy    FC = Filter Coffee
  KB = Chicken Burger    -O  = no onions
  BN = Butter Naan       -M  = no mayo
  MD = Masala Dosa       TC  = thin crust

Now those same orders become:

Order #1:  1×CB+R-O  1×BN  2×ML  T5  730P
Order #2:  2×MP+C,TC  1×VB-M  1×CK  T12  745P
Order #3:  3×KB+SP  2×MD  1×FC  T8  800P

The kitchen decodes instantly: "1×CB+R-O" -> "One Chicken Biryani, extra raita, no onions." No ambiguity, no lost information, just shorter.

Full orders:   ~150 characters each × 500 orders = 75,000 characters
Coded orders:  ~40 characters each  × 500 orders = 20,000 characters

Compression: 3.75x smaller. Same food gets cooked.

This is exactly what TurboQuant does to AI models.

Instead of storing the full number(0.4567689.. 16/32 bits). It stores the KV cache in (1010 4 bits) that maps to a value in shared codebook. The codebook is agreed in advance. This becomes the same for every layer and every conversation

The Trick: Spin, Snap, Pack

TurboQuant compresses each K (or V) vector in the below steps. Let me walk through the entire process on how the compression and decompression happens with a real 4-dimensional vector you can verify by hand.

We'll use 2-bit quantization (4 grid levels) to keep things small. Real models use something like 4-bit (16 levels) and 256 dimensions, but the math is identical.

Our example vector

K = [1.2, -0.8, 0.5, -1.1]

A normal balanced vector, no outliers, keeping it simple to understand

so 4 numbers assuming 4 bytes(float32) each
= 16 bytes
Lets compress it to 3 bytes and then try to bring it back(there will be slight errors/residual)

Step 1 : Measure the norm and Normalize

Measure the magnitude/norm of the vector

norm = sqrt(1.2**2 + 0.8**2 + 0.5**2 +1.1**2)
     = 1.8815

we save the norm as a float16 number which 2 bytes

Now divide the vector by norm to get a unit vector

unit = K / 1.8815 = [+0.6378, -0.4252, +0.2657, -0.5846]

Step 2 : Spin (Random Rotation)

Multiply the unit vector by a random 4×4 rotation matrix R:

Rotation matrix R (generated once from seed=42, never changes):

  [+0.8395, +0.4120, -0.2597, +0.2411]
  [+0.2956, -0.5448, -0.4830, -0.6185]
  [-0.3277, +0.7138, -0.3802, -0.4884]
  [-0.3171, -0.1547, -0.7448, +0.5664]

Each rotated value is a dot product of one row of R with the unit vector:

rotated[0] = (+0.8395 × +0.6378) + (+0.4120 × -0.4252)
           + (-0.2597 × +0.2657) + (+0.2411 × -0.5846)
           = +0.5355 - 0.1752 - 0.0690 - 0.1410
           = +0.1503

rotated[1] = (+0.2956 × +0.6378) + (-0.5448 × -0.4252)
           + (-0.4830 × +0.2657) + (-0.6185 × -0.5846)
           = +0.1886 + 0.2317 - 0.1283 + 0.3615
           = +0.6534

rotated[2] = (-0.3277 × +0.6378) + (+0.7138 × -0.4252)
           + (-0.3802 × +0.2657) + (-0.4884 × -0.5846)
           = -0.2090 - 0.3035 - 0.1010 + 0.2856
           = -0.3280

rotated[3] = (-0.3171 × +0.6378) + (-0.1547 × -0.4252)
           + (-0.7448 × +0.2657) + (+0.5664 × -0.5846)
           = -0.2023 + 0.0658 - 0.1979 - 0.3312
           = -0.6655

Before and after Rotation

Before (unit vector):   [+0.638, -0.425, +0.266, -0.585]
After (rotated):        [+0.150, +0.653, -0.328, -0.666]

Step 3: Snap, Building the Grid

We have 4 rotated numbers: [+0.150, +0.653, -0.328, -0.666]

We need to replace each one with the nearest "allowed" value from a small set. With 2 bits, we get to pick 4 allowed values. But which 4?

Lloyd-Max is the algorithm that finds this smart placement to find the 4 grid values (skipping the math intentionally)

This generates a spread in 4 values

Final codebook:                                                                                                                                                                                                                                
    Level 0: -0.674                                                                                                                                                                                                                              
    Level 1: -0.219                                                                                                                                                                                                                              
    Level 2: +0.219                                                                                                                                                                                                                              
    Level 3: +0.674

Step 4: Snap to Grid

Now snap each rotated value to the nearest codebook level

R-Vector: [+0.150, +0.653, -0.328, -0.666]

CodeBook: [-0.674, -0.219. +0.219, +0.674]
Indices: 0 1 2 3

Lets map the values in the R-vector to the closest vectors

value0 = +0.150 -> nearest Level 2(+0.219) error = 0.069
value1 = +0.653 -> nearest Level 3(+0.674) error = 0.021 <- tiny!
value2 = -0.328 -> nearest Level 1(-0.219) error = 0.109
value3 = -0.666 -> nearest Level 0(-0.674) error = 0.009 <- almost exact!

The R-Vector gets transformed to indices

[+0.150, +0.653, -0.328, -0.666] --> [2,3,1,0]

Step 5 Pack

We need to store this indices --> [2,3,1,0] --> hex 0x1E
which needs 1 byte (00000000) to store the indices 0 to 3

so total bytes = 2 bytes (for storing norm) + 1 byte for indices

Decompression/unpacking

indices to numbers from the codebook [2,3,1,0] --> [+0.219, +0.674, -0.219, -0.674]
unrotate/unspin to get the unrotated vector by multiplying R^T(transpose)

unit_hat[0] = (+0.8395 × +0.219) + (+0.2956 × +0.674)
            + (-0.3277 × -0.219) + (-0.3171 × -0.674)
            = +0.184 + 0.199 + 0.072 + 0.214
            = +0.669

unit_hat[1] = (+0.4120 × +0.219) + (-0.5448 × +0.674)
            + (+0.7138 × -0.219) + (-0.1547 × -0.674)
            = +0.090 - 0.367 - 0.156 + 0.104
            = -0.329

unit_hat[2] = (-0.2597 × +0.219) + (-0.4830 × +0.674)
            + (-0.3802 × -0.219) + (-0.7448 × -0.674)
            = -0.057 - 0.326 + 0.083 + 0.502
            = +0.203

unit_hat[3] = (+0.2411 × +0.219) + (-0.6185 × +0.674)
            + (-0.4884 × -0.219) + (+0.5664 × -0.674)
            = +0.053 - 0.417 + 0.107 - 0.382
            = -0.639

scale back with norm value

K_hat = unit_hat × 1.8815
K_hat = [+1.259, -0.620, +0.382, -1.202]

The Verdict: Original vs Reconstructed

         Original    Reconstructed    Error     Relative
         ────────    ─────────────    ─────     ────────
dim 0:    +1.200      +1.259          0.059      4.9%
dim 1:    -0.800      -0.620          0.180     22.6%
dim 2:    +0.500      +0.382          0.118     23.6%
dim 3:    -1.100      -1.202          0.102      9.3%

Overall relative MSE: 1.7%

Look at that! We crushed a 16-byte vector down to 3 bytes and got it back with 1.7% error, but still there is some difference

Now, lets do the math with 4 bit (16 levels) instead of 2

        Original    Reconstructed    Error     Relative
        ────────    ─────────────    ─────     ────────
dim 0:    +1.200      +1.175          0.025      2.1%
dim 1:    -0.800      -0.716          0.084     10.5%
dim 2:    +0.500      +0.433          0.067     13.4%
dim 3:    -1.100      -1.081          0.019      1.7%

Overall relative MSE: 0.4%

As this is a toy example we have taken, we are seeing almost good results, applying this on a real model with ~256 levels give much more better results

Real Numbers: How Much Memory Does This Save?

I implemented TurboQuant from scratch and tested it on real Gemma 4 models. Here's what we measured:

Gemma 4 E2B (5.1 billion parameters)

At 128K token context:

Without TurboQuant:
  Model weights:  10.25 GB
  KV cache:        2.25 GB
  Total:          12.50 GB

With TurboQuant (4-bit):
  Model weights:  10.25 GB  (unchanged — we only compress the cache)
  KV cache:        0.58 GB
  Total:          10.83 GB

Saved: 1.67 GB of KV cache memory

Which is 3.9x

Extrapolating the numbers

Extending this 3.9x as we increase the context length and the baseline KV, these are approximations

Context Length	Baseline KV	TurboQuant KV	Saved	Compression
8K tokens	2.0 GB	0.5 GB	1.5 GB	3.9x
32K tokens	8.1 GB	2.0 GB	6.0 GB	3.9x
128K tokens	32.2 GB	8.2 GB	24.0 GB	3.9x
256K tokens	64.4 GB	16.4 GB	48.0 GB	3.9x

P.S I have skipped the QJL(Quantized Johnson-Lindenstrauss) for handling the residuals

Wrapping up

Rotate, snap, pack , three operations, 3.9x less KV cache memory, same answers.

References:

DEV Community: Bharath Kadaluri