DEV Community

Anguo
Anguo

Posted on

The workaround generalizes: Llama 3.2 1B on QCS6490 — 1,024 tok/s prefill, 16.4 tok/s decode

The workaround generalizes: Llama 3.2 1B on QCS6490 — 1,024 tok/s prefill, 16.4 tok/s decode

Yesterday I published the first public GenAI numbers for Qualcomm's QCS6490: Qwen3-0.6B at 1,555 tok/s prefill and 24.6 tok/s decode. Getting there took a hand-built chain, because the official AI Hub export script passes a flag the compiler rejects (--output_names is not supported for AIMET ONNX models, upstream issue #366, fix promised in v0.67.0 but not out yet).

The obvious next question: was that a one-off rescue, or does the same manual chain carry other models? Today it carried the best-known small model of all — Llama 3.2 1B Instruct.

Same chain, no new surprises

The recipe is identical: take the pipeline's own compile parameters from a failed export job, drop the one bad flag, pin the input shapes the script forgot to pin (that omission killed my first attempt on Qwen3 — dynamic shapes, dead on arrival), and submit the compile jobs through the Hub API directly. Six compile jobs, one link job, six profile jobs on a real Dragonwing RB3 Gen 2 board. Everything passed. The bug is still in the script; the chain still goes around it.

The numbers

Llama 3.2 1B Instruct, w4a16, context length 512, measured on RB3 Gen 2 (QCS6490):

  • Prefill: 1,024 tok/s
  • Decode: 16.4 tok/s

Method, stated plainly because it matters: these are derived from per-graph profiles on the real device — the same method as yesterday's Qwen3 numbers, which is exactly why the two rows are comparable. They are not a Genie end-to-end run; an end-to-end figure would include runtime overheads the per-graph numbers don't. I'd rather publish the method with the numbers than a bigger number with no method.

Model Quant Context Prefill (tok/s) Decode (tok/s)
Qwen3-0.6B w4a16 512 1,555 24.6
Llama 3.2 1B Instruct w4a16 512 1,024 16.4

To my knowledge this is the first public measurement of Llama 3.2 1B on QCS6490. If you have an earlier one, I'd genuinely like to see it — the whole point of publishing these is that the table should exist somewhere public.

Why a second row matters

One model through a workaround is an anecdote. Two models, from two different families, through the same chain with no new surprises is a method. The gradient is the real product here: 0.6B and 1B are now filled in, and the same chain is queued for Qwen3-1.7B and Llama 3.2 3B, so the curve of "what actually runs on this chip, and how fast" stops being extrapolation and becomes measurement.

All data, including the failures, lives in the open benchmark repo (qcs-bench) and on the model card:

Llama 3 license applies to the model; what's published here is measurements and the path that produced them.

Top comments (0)