What changed since last time
- Faster prompt processing. Activation packing borrowed from Arm KleidiAI’s layout, balanced work split across threads, and larger prefill blocks: Llama 3.1 8B Q8 went from ~620 to ~800 tok/s. The output is bit-identical to before.
- Ready-made models on Hugging Face, no conversion step: Qwen3-30B-A3B, Qwen3.8-27B, Qwen2.5, Llama 3.1/3.2, each with the license of the original. → huggingface.co/miloserdev
- OpenAI-compatible server: streaming, tool calls, KV prefix cache. Any OpenAI client works.
Numbers on this box (2 sockets)
Model Decode Prefill
Qwen3-30B-A3B Q4 ~90 tok/s ~600 tok/s
Llama 3.1 8B Q8 39 tok/s ~800 tok/s
Qwen3.8-27B Q4 17 tok/s ~175 tok/s
Caveats
Tuned for ARM64 with dotprod. x86 runs, but not fast.
One request at a time.
Its own file format, not GGUF.
hf download miloserdev/Qwen3-30B-A3B-Instruct-2507-numallama --local-dir models/qwen3-30b
./runq models/qwen3-30b/qwen3-30b-a3b-instruct-2507-q4.bin -z models/qwen3-30b/tokenizer.bin -m server
Repo: https://github.com/miloserdev/numallama
Feedback welcome, especially from anyone with Graviton or Ampere hardware.
Top comments (0)