DEV Community

Cover image for NUMALLAMA update: Qwen3-30B-A3B at ~90 tok/s on a dual-socket ARM server, no GPU
Miloserdev
Miloserdev

Posted on

NUMALLAMA update: Qwen3-30B-A3B at ~90 tok/s on a dual-socket ARM server, no GPU

What changed since last time

  • Faster prompt processing. Activation packing borrowed from Arm KleidiAI’s layout, balanced work split across threads, and larger prefill blocks: Llama 3.1 8B Q8 went from ~620 to ~800 tok/s. The output is bit-identical to before.
  • Ready-made models on Hugging Face, no conversion step: Qwen3-30B-A3B, Qwen3.8-27B, Qwen2.5, Llama 3.1/3.2, each with the license of the original. → huggingface.co/miloserdev
  • OpenAI-compatible server: streaming, tool calls, KV prefix cache. Any OpenAI client works.

Numbers on this box (2 sockets)

Model Decode Prefill
Qwen3-30B-A3B Q4 ~90 tok/s ~600 tok/s
Llama 3.1 8B Q8 39 tok/s ~800 tok/s
Qwen3.8-27B Q4 17 tok/s ~175 tok/s

Caveats

Tuned for ARM64 with dotprod. x86 runs, but not fast.
One request at a time.
Its own file format, not GGUF.

hf download miloserdev/Qwen3-30B-A3B-Instruct-2507-numallama --local-dir models/qwen3-30b
./runq models/qwen3-30b/qwen3-30b-a3b-instruct-2507-q4.bin -z models/qwen3-30b/tokenizer.bin -m server
Enter fullscreen mode Exit fullscreen mode

Repo: https://github.com/miloserdev/numallama
Feedback welcome, especially from anyone with Graviton or Ampere hardware.

Top comments (0)