Zero-Allocation INT8 Quantization & Fast Inference on Embedded MCUs

TouvetTech — Wed, 04 Nov 2026 10:41:04 +0000

Deploying deep neural networks to microcontrollers (like ARM Cortex-M4/M7/M55 or RISC-V cores) is rarely bottlenecked by model architecture design—it is almost always constrained by memory bandwidth, cache line misses, and dynamic memory overhead.

When porting floating-point (FP32) networks down to integer execution (INT8), naive implementations often stumble into severe latency traps. In this article, we will look at the core mathematics of affine quantization, how to eliminate runtime zero-point overhead, and examine a deterministic, zero-allocation C++ execution kernel.

1. The Math: Symmetric vs. Asymmetric Quantization

Affine quantization maps real-valued floating-point numbers in the range [alpha, beta] to signed 8-bit integers q in the range [-128, 127]:

where the scale factor and zero-point are defined as:

The Latency Trap: Zero-Point Arithmetic Overhead

When executing an integer vector dot product between activations X and weights W under asymmetric quantization:

Notice what happens:

The first summation is our core integer multiply-accumulate (MAC):

The weight sum term can be precomputed offline during compile or weight conversion time:

However, the runtime activation sum requires dynamic accumulation on the fly across every inference layer:

If your weight distribution is symmetric around zero (Z_w = 0), that dynamic runtime activation accumulation vanishes completely:

This single optimization eliminates 20–25% of runtime ALU cycles on embedded cores that lack dedicated hardware NPU matrix coprocessors.

2. Zero-Allocation C++17 Implementation

Embedded systems operating under real-time constraints (RTOS) or safety standards (IEC 61508 / ISO 26262) forbid dynamic memory allocations (malloc or new) on inference hot paths due to heap fragmentation and non-deterministic latency.

Here is a clean, dependency-free C++17 implementation featuring 4-way superscalar loop unrolling:

#include <cstdint>
#include <cstddef>
#include <algorithm>

namespace EdgeML {

// High-throughput 4-way unrolled INT8 dot product -> INT32 accumulator
inline int32_t dot_product_int8(const int8_t* a, const int8_t* b, size_t length) {
    int32_t acc = 0;
    size_t i = 0;

    // 4-way unrolled core loop for instruction pipelining
    for (; i + 3 < length; i += 4) {
        acc += static_cast<int32_t>(a[i])     * static_cast<int32_t>(b[i]);
        acc += static_cast<int32_t>(a[i + 1]) * static_cast<int32_t>(b[i + 1]);
        acc += static_cast<int32_t>(a[i + 2]) * static_cast<int32_t>(b[i + 2]);
        acc += static_cast<int32_t>(a[i + 3]) * static_cast<int32_t>(b[i + 3]);
    }

    // Scalar cleanup tail
    for (; i < length; ++i) {
        acc += static_cast<int32_t>(a[i]) * static_cast<int32_t>(b[i]);
    }

    return acc;
}

// Fixed-point requantization (avoids floating-point hardware calls)
inline int8_t requantize(int32_t val, int32_t multiplier, int32_t shift, int32_t zero_point) {
    // 64-bit integer product followed by right shift
    int64_t scaled = (static_cast<int64_t>(val) * multiplier) + (1LL << (shift - 1));
    int32_t result = static_cast<int32_t>(scaled >> shift) + zero_point;
    return static_cast<int8_t>(std::clamp(result, -128, 127));
}

} // namespace EdgeML

3. Memory Hierarchy: Why NCHW Kills MCU Performance

On desktop GPUs, tensor data is frequently laid out in NCHW (Batch, Channel, Height, Width). However, on microcontrollers with 32-byte cache lines or tightly coupled memories (TCM):

In NCHW format, moving along the channel dimension to perform an inner-product dot-product requires striding across large memory addresses (H * W), triggering continuous cache line evictions.
In NHWC format, all channels for a given spatial pixel are contiguous in physical SRAM. Loading 4 adjacent channels fills a single 32-bit register directly, enabling SIMD parallel loads without cache thrashing.

4. Open Source Kernels & Further Reading

The complete open-source kernel suite is available on GitHub:

👉 touvettech/edgeml on GitHub

For engineers building production edge inference runtimes from scratch, we have also compiled an in-depth reference architecture guide covering:

Graph operator fusion (Conv + BatchNorm + ReLU in a single pass)
DMA ping-pong double buffering algorithms to hide flash latency
Quantization-Aware Training (QAT) vs Post-Training Quantization (PTQ) error bounds
SNR verification benchmarks