I tried gzip-compressing weights for STM32 inference. It saved Flash, but it didn’t solve the actual inference problem.

As you might have guessed, when working with microcontrollers, two of the biggest things we care about are Flash and SRAM size , i mean we so also care about cpu cycles but out major constraints are mostly memory when doing inference .
But if that is true, why not just gzip the weights have even smaller weights stored in Flash break them into chunks decompress them one by one and then use them for our computation?
Answer —
Why is quantization done in this case?
The STM32F411CEU6 has about 512 KB of Flash and 128 KB of SRAM. Nor does it have PyTorch sitting on it to directly load the .pth files that most of us make when training our homemade AI http://models.So in order to do inference on it you need to export the weights into a smaller and much simpler representation that:

1. fit inside the 512 KB Flash of the microcontroller.
2. can be used when computation such that its can do the computations easily. The STM32F411 also only has a single CPU core which is another problem in terms of speed so we do quantisation But why though?Why is it that we always have to do what other people say we have to do? So I decided to try something different. Instead of just going with quantization like a normal person, I decided to try "lossless compression — gzip" And here are the reasons why it is worse for this particular use case, even if gzip manages to give us good Flash compression.
3. The decompression creates unnecessary extra steps. these images are normal int8 quantization and the gzip are respectively like this -

So before the STM32 can even do the actual neural-network math, it now has another job: decompression and thats not all the next thins is

1. The decoded data still needs to become FP32 values i mean that is the whole point right without FP32 whats the point ? gzip does not change the numerical representation of the weights. It compresses the bytes representing those numbers. which sounds good and more intuitive on paper . So eventually: The STM32 still has to recover the original numerical representation before normal FP32 computation can happen.

1. We are trading Flash for CPU cycles. Yeah, I get that in embedded systems we often trade Flash or memory for CPU cycles. And sometimes that is completely fine because we don't really care about inference speed that much i mean in my project it takes a lot of time to print each characters and we may allow that but small delays to add up faster than we anticipate . but gzip takes it further cause now you need to do the decompression to get the original value this maybe not seem like a lot but do this like 100s of time and time jumps from 1 sec roughly to like 3 or even 4 sec. If I stream the model instead of decompressing the whole thing into RAM, then every inference becomes something like the image shown:

So I fix the SRAM problem but now its just so much more work and its like bound to add like a lot of time and frankly cooding it is a hassle .

1. Memory access becomes a lot weirder. Normally, with my INT8 quantization**, I can do something like this: acc += xv[i] * (float)w[i]; and then: out[o] = acc * scale + bias[o]; The full code looks something like this:

static inline float dot_x_int8(const float *xv, const int8_t *w, int n) {
    float acc = 0.0f;

    for (int i = 0; i < n; i++) {
        acc += xv[i] * (float)w[i];
    }

    return acc;
}
static void linear(...) {
    float acc = dot_x_int8(xv, &W[o * in_dim], in_dim);
    out[o] = acc * scale + bias[o];
}

here we know exactly what the offset must be.
every INT8 weight occupies exactly 1 byte.
pretty straight forward right ? i mean all you have to do is just offset the pointer to a fixed memory acess and boom you got the value right

something like -

W[0]
W[1]
W[2]
W[3]

we basically know their positions right ?
Conceptually:
address = base + index
and for a row of a matrix:
&W[o * in_dim]
is straightforward.
but gzip cannot do that .DEFLATED output is a compressed stream containing variable-length encoded information and LZ77 references.
So you generally cannot say:
weight 1000 = compressed_base + 1000 × constant_size
because there is no fixed compressed size for every individual weight.
You can solve some of this by creating independently compressed blocks and indexes, but now you are adding even more machinery just to access the model and fankly you would run in a lot of out of memory access errors before you nail this one .

If you still don't get what I am trying to say, here is a simple example.
FP32 is :-
weight = 0.21739182
32 bits
INT8 quantization says something like:
weight ≈ q × scale
scale = 0.005
q = round(0.21739182 / 0.005)
q ≈ 43

So what I actually store for this weight is:
43
which is an 8-bit integer, while the scale is shared across many weights.
During my inference:

gzip says something completely different:

we can keep the weights compressed in Flash decompress small chunks into SRAM use them throw them away and continue.
That would work.
But now every inference also involves running a DEFLATE decoder, maintaining decompression state, recovering the original bytes, and only then doing the actual neural-network computation.
With INT8 quantization, every weight stays one byte, every weight has a predictable location, and my inference loop can consume that representation directly with a cheap cast and a shared scale operation.
So even though gzip sounds like an obvious solution when your first thought it isnt in this usecase .