I wrote the missing Apple Silicon runtime for NVIDIA's Nemotron Omni

#mlx #apple #machinelearning #opensource

NVIDIA's Nemotron-3-Nano-Omni-30B-A3B is an open-weights model that sees, hears and reasons. There is already a 4-bit MLX quantization of it on Hugging Face, done by yayr. But as that model card says plainly, only the text backbone loads with standard MLX tooling:

The vision and audio towers require a multimodal runtime that implements the C-RADIO ViT-H and Parakeet Conformer forward passes.

Nobody had written that runtime. So the model could talk on a Mac, but it could not see or hear.

I wrote it. It is pure MLX: the vision tower, the audio tower, the processor, and the multimodal token splicing, all ported from NVIDIA's reference implementation.

Verified, not asserted

The thing I actually care about here is not that it runs. It is that I can prove it runs correctly.

Every component is tested against NVIDIA's PyTorch reference on the same inputs with the same weights, in fp32 on CPU so the comparison is honest. pytest tests/ — 23 of 23 passing.

Component	Compared against	Result
Audio tower	`transformers` ParakeetEncoder + NVIDIA `SoundProjection`	cos 0.99999130 (min/frame)
Audio frontend	log-mel features	max abs delta 8.1e-6
Vision tower	`nvidia/C-RADIOv4-H` via `trust_remote_code`	cos 0.99996227 (min/token)
Vision tower, MLX CPU stream	same	1.00000000 — graph-exact

A port that is almost right is worse than no port at all, because you spend weeks chasing quality problems that are really numerical drift in a tower you never checked. Writing the parity harness first was the single best decision in this project.

Speed and memory

Measured on an M5 Max MacBook Pro, running the 4-bit quantized language model with both towers in bf16:

Image:  67.7 tok/s · 22.1 GB peak
Audio:  147  tok/s · 21.0 GB peak
Text:   152  tok/s · 17.9 GB peak

Wifi off the whole time. Nothing leaves the machine.

That 22.1 GB peak on the image path is the number I would pay attention to if you are deciding whether to bother. It suggests this fits on a 32 GB Mac. I cannot verify that, because the M5 Max is the only machine I have. If you run it on something smaller I would genuinely like to hear what happens — that is the most useful thing anyone could send me right now.

Why bother doing this locally

The obvious question is why not just call an API. A few reasons that matter to me.

The model is open weights. Someone should be able to run open weights on their own hardware, and if the only path requires a vendor's cloud, the openness is partly decorative.

Apple Silicon is genuinely fast enough now. 67 tokens a second while reading an image, on a laptop, is not a compromise.

And there are workloads where the data cannot leave the building at all. I do work that touches NDA and compliance-sensitive material, and "it runs offline" is not a nice-to-have there, it is the requirement.