Edge ML: Practical Quantization, Pruning, and Latency Trade-Offs

Deploying machine learning models to microcontrollers and low-power embedded systems is fundamentally an exercise in resource accounting. Unlike server environments where compute is bounded by cost or thermal design power, edge devices operate under hard physical walls. An ARM Cortex-M4 or Cortex-M7 microcontroller often offers less than 1 MB of SRAM and a few megabytes of non-volatile flash storage. Even embedded Linux platforms, such as the Raspberry Pi Zero or entry-level NXP i.MX boards, face strict memory bandwidth bottlenecks and tight power limits.

To make deep neural networks execute reliably under these constraints, we rely on model compression techniques—primarily quantization and pruning. However, these techniques introduce non-linear trade-offs between precision, flash footprint, runtime memory usage, and operational latency.

The Memory Hierarchy on Microcontrollers

When optimizing a model for embedded hardware, distinguishing between static storage and dynamic memory is essential.

Flash memory holds the static model parameters (weights, biases) and the compiled program code. Read operations from flash are relatively fast, but writing during inference is not an option.

SRAM holds the runtime activations, the working tensor arena, and the stack/heap. SRAM is often the primary bottleneck. If a model's peak activation layer exceeds the allocated tensor arena size, the runtime aborts with an out-of-memory error. Because dynamic memory allocation (malloc) is discouraged or entirely disabled in real-time embedded environments, the entire activation arena must be allocated statically ahead of time.

This means that shrinking weights alone does not guarantee that a model can run on a microcontroller. The maximum activation tensor generated during the forward pass must fit within available SRAM alongside input/output buffers and peripheral drivers.

Quantization: INT8 vs FP32

Quantization maps continuous 32-bit floating-point values ($FP32$) to lower-bit representations, most commonly 8-bit integers ($INT8$).

The standard uniform affine quantization scheme maps a real-valued number $x$ to an integer $q$ via a scale factor $S$ and a zero-point $Z$:

$$q = \text{round}\left(\frac{x}{S}\right) + Z$$

Where $S$ is a positive floating-point number, and $Z$ is an integer of the same type as $q$.

Post-Training Quantization (PTQ)

In PTQ, a fully trained $FP32$ network is converted to $INT8$ using a small calibration dataset. The calibration run profiles the activation distributions across intermediate layers to determine optimal values for $S$ and $Z$. PTQ is straightforward and avoids the cost of full retraining. However, highly non-linear networks or models with narrow activation dynamic ranges often suffer noticeable accuracy degradation under PTQ alone.

Quantization-Aware Training (QAT)

QAT models the quantization noise during the training process itself. Forward passes simulate lower precision using fake-quantization nodes that clamp and round values, while backpropagation uses standard floating-point operations with a straight-through estimator (STE). QAT almost always recovers accuracy lost in PTQ, especially for lightweight architectures like MobileNetV2 or custom depthwise-separable CNNs, which are sensitive to weight clipping.

Here is a standard example of generating an INT8 full-integer quantized model using TensorFlow Lite for Microcontrollers:

import tensorflow as tf

def representative_dataset():
    for data in calibration_data_loader:
        # Yield normalized input batches for calibration
        yield [data.astype(np.float32)]

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset

# Enforce full INT8 execution across all operations
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

quantized_tflite_model = converter.convert()

with open("model_quantized_int8.tflite", "wb") as f:
    f.write(quantized_tflite_model)

Moving to full integer inference provides two distinct hardware benefits:

1. Model size decreases by roughly $4\times$, directly reducing flash usage.
2. Integer matrix multiplication instructions (like ARM CMSIS-NN SIMD instructions __SMLAD) execute significantly faster than software-emulated or even hardware single-precision floating-point units.

Pruning: Structured vs Unstructured

Pruning eliminates parameters that contribute minimally to the network's predictive output. We categorize pruning into two forms:

1. Unstructured Pruning: Individual weights below a given threshold are set to zero anywhere in the weight tensor. While this achieves high theoretical sparsity (often $70\text{--}90\%$), it creates arbitrary zero-patterns. Standard microcontrollers and embedded runtimes cannot leverage random sparsity without specialized sparse-matrix runtime engines. In practice, unstructured pruning usually bloats memory metadata and fails to reduce execution latency.

2. Structured Pruning: Entire channels, filters, or attention heads are removed. Because the pruned network retains dense matrix structures, it executes directly on commodity inference engines without custom sparse libraries. The resulting layer tensors are simply smaller dense arrays, leading to verified decreases in both activation size and latency.

If the target deployment platform lacks hardware acceleration for sparse matrices, structured pruning is the only reliable choice for achieving measurable latency gains.

A Worked Memory and Latency Budget

Consider an embedded sound-classification model deployed on an ARM Cortex-M4 running at 64 MHz, with 512 KB Flash and 128 KB SRAM.

Assume the architecture has three 2D convolutional layers followed by global average pooling and a fully connected output:

• Total parameters: 95,000 weights.
• Input shape: 1 channel, $32 \times 32$ spectrogram ($1{,}024$ samples).
• Layer 1: 16 filters, $3 \times 3$, stride 1, padding 'same'. Output: $32 \times 32 \times 16 = 16{,}384$ values.
• Layer 2: 32 filters, $3 \times 3$, stride 2, padding 'same'. Output: $16 \times 16 \times 32 = 8{,}192$ values.
• Layer 3: 64 filters, $3 \times 3$, stride 2, padding 'same'. Output: $8 \times 8 \times 64 = 4{,}096$ values.

Footprint Comparison

FP32 Baseline:

• Flash requirement: $95{,}000 \times 4\text{ bytes} \approx 380\text{ KB}$. This consumes $74\%$ of total flash.
• Peak activation buffer: Layer 1 requires $16{,}384 \times 4\text{ bytes} = 65.5\text{ KB}$.
• Double buffering for layer transitions (Layer 1 output + Layer 2 output): $65.5\text{ KB} + 32.7\text{ KB} = 98.2\text{ KB}$.
• SRAM overhead: $98.2\text{ KB}$ exceeds the safe SRAM budget when operating systems, peripheral buffers, and stacks need memory within the remaining $29.8\text{ KB}$.

INT8 Quantized:

• Flash requirement: $95{,}000 \times 1\text{ byte} \approx 95\text{ KB}$ (plus quantization parameters). Flash usage drops to under $20\%$.
• Peak activation buffer: Layer 1 requires $16{,}384 \times 1\text{ byte} = 16.4\text{ KB}$.
• Double buffering requirement: $16.4\text{ KB} + 8.2\text{ KB} = 24.6\text{ KB}$.
• SRAM headroom: $24.6\text{ KB}$ leaves more than $100\text{ KB}$ of SRAM available for sensor ingestion, circular audio buffers, and communication stacks.

Execution Latency

On the Cortex-M4, an FP32 multiply-accumulate loop takes roughly 4 to 6 clock cycles per operation when using software floating-point emulation or simple single-issue FP instructions. With CMSIS-NN and an INT8 SIMD instruction set, two 8-bit multiply-accumulate operations occur per cycle.

For a layer executing 1.5 million MAC operations:

• $FP32$: $\approx 1.5\text{M} \times 5\text{ cycles} = 7.5\text{ million cycles} \approx 117\text{ ms}$ at 64 MHz.
• $INT8$: $\approx (1.5\text{M} / 2) \times 1.2\text{ cycles} = 900{,}000\text{ cycles} \approx 14\text{ ms}$ at 64 MHz.

Quantization in this scenario changes the inference from unusable to well within a real-time 50 ms audio polling loop.

Benchmarking Considerations on Physical Boards

Simulator profiles frequently diverge from real-world behavior on bare-metal systems. Three issues consistently emerge:

1. Flash Wait States: Microcontroller clock frequencies often exceed internal flash access speeds. Reading weights directly from flash introduces wait states unless an internal cache or prefetch buffer is present. Benchmarking must occur on hardware running at target clock rates.
2. Memory Alignment: ARM architectures penalize unaligned 32-bit accesses. Ensure tensor arenas are explicitly aligned to 4-byte or 16-byte boundaries using compiler attributes such as alignas(16).
3. Fallback Operations: If a quantization converter encounters an unsupported layer, it may quietly insert a float conversion step. This causes intermediate conversions between INT8 and FP32, dramatically degrading latency without raising a build-time failure.

Key takeaways

• SRAM capacity limits the maximum activation tensor size, not just total parameter count; profile peak layer outputs early.
• Post-training INT8 quantization yields an immediate $4\times$ reduction in weight footprint and unlocks SIMD instruction acceleration on modern microcontrollers.
• Unstructured pruning provides minimal benefits on standard edge runtimes; rely on structured pruning to achieve true latency reductions.
• Always verify on-device operator support to prevent covert FP32 runtime fallbacks in compiled computation graphs.