Ollama vs llama.cpp vs vLLM: Throughput on M1/RTX 4090

#llm #vllm #llamacpp #ollama

The Setup That Revealed the Gap

I needed to benchmark three different inference engines — Ollama, llama.cpp, and vLLM — because the "just use vLLM" advice kept popping up in discussions, but nobody was showing apples-to-apples numbers. So I grabbed a Llama 2 7B model, ran it on my M1 MacBook Pro (16GB) and a cloud RTX 4090 instance, and measured tokens per second under identical conditions.

The results weren't what I expected. vLLM crushed both alternatives on the 4090 (112 tok/s vs 68 tok/s for llama.cpp), but on the M1, llama.cpp pulled ahead by 30%. Ollama sat in the middle on both platforms, which makes sense given it's a wrapper around llama.cpp with extra layers.

Here's the benchmark setup: batch size 1, context length 2048, FP16 precision (no quantization), temperature 0.7. I ran each inference 50 times and took the median to avoid outliers from thermal throttling or background processes.