DEV Community

Vishnu Hari Dadhich
Vishnu Hari Dadhich

Posted on Originally published at vishctl.dev on

Running llama.cpp on a 32 GB MacBook Air: A Direct Comparison with Ollama

#ai #localllm #llamacpp #ollama

In the previous post, I ran Ornith 1.5 9B on my 32 GB MacBook Air using Ollama and recorded baseline token-generation speeds on short prompts. Ollama is great for getting up and running quickly, but under the hood, its inference engine is built on llama.cpp.

In this post, we go one level down: building and running llama.cpp directly, offloading inference to Apple Silicon’s Metal GPU, and comparing performance numbers side by side with Ollama on the exact same model and quantization level.

Step 1: Build llama.cpp from Source

Building llama.cpp from source on macOS is fast and straightforward. On Apple Silicon, CMake automatically enables Metal support (GGML_METAL=ON) and compiles GPU compute kernels tailored for Apple’s unified memory architecture.

Clone the repository and compile the release binaries using all available CPU cores:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)
Enter fullscreen mode Exit fullscreen mode

Once the build finishes, the binaries live in build/bin. Add them to your current session’s $PATH:

export PATH="$(pwd)/build/bin:$PATH"
Enter fullscreen mode Exit fullscreen mode

To make this permanent across terminal sessions, append it to your shell configuration:

echo "export PATH=\"$(pwd)/build/bin:\$PATH\"" >> ~/.zshrc
source ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

Confirm that the build succeeded:

llama-server --version
Enter fullscreen mode Exit fullscreen mode

Step 2: Serve the Model (Out-of-the-Box Baseline)

llama.cpp can download GGUF models directly from Hugging Face Hub using the -hf flag, caching weights under ~/.cache/huggingface/hub:

llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99
Enter fullscreen mode Exit fullscreen mode

Here is what the flags do:

  • -hf ornith-ai/Ornith-1.5-9B-GGUF: Resolves the model on Hugging Face and downloads the default Q4_K_M quant.
  • -ngl 99: Offloads all 99 model layers to the GPU (Metal on Apple Silicon unified memory).
  • --port 8080: Binds the OpenAI-compatible HTTP server to port 8080.

Starting llama-server and downloading Ornith 1.5 9B from Hugging Face
Starting llama-server with automatic Hugging Face download and full Metal GPU offload.

During startup, llama.cpp downloads both the multimodal vision projector (mmproj-Ornith-1.5-9B-BF16.gguf) and the main model weights (Ornith-1.5-9B-Q4_K_M.gguf), offloads the layers into Metal, initializes inference slots, and listens on port 8080.

We test this default setup with a short greeting prompt through the OpenAI-compatible API:

curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "ornith-1.5-9b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
Enter fullscreen mode Exit fullscreen mode

Calling the llama-server chat completions API
First chat completion request via curl, returning detailed server-side timing breakdowns.

The response returns a complete timing breakdown in the timings object:

prompt eval time = 496.21 ms / 12 tokens (24.18 tokens per second)
eval time = 5527.17 ms / 96 tokens (17.19 tokens per second)
Enter fullscreen mode Exit fullscreen mode

Compare that to Ollama’s numbers on the same model and quant from the earlier post:

Metric Ollama (Default) llama.cpp (Default, no -fa)
Prompt Eval Speed 37.13 tokens/s 24.18 tokens/s
Generation Speed 16.68 tokens/s 17.19 tokens/s

Generation speed is practically neck-and-neck (~17 tokens/s). However, prompt evaluation (prefill latency) was noticeably slower on vanilla llama.cpp out of the box (24.18 tokens/s vs. Ollama’s 37.13 tokens/s).

Ollama enables FlashAttention and sensible batch sizes by default, while raw llama.cpp keeps conservative baseline settings unless configured explicitly.

Step 3: Verifying the Quants

Before comparing benchmark numbers, it is critical to confirm that both engines are executing the exact same quantization format. Comparing different quants (e.g. Q4_K_M vs Q8_0) would invalidate any performance conclusions.

Check the quant llama.cpp loaded:

curl -s http://localhost:8080/v1/models | jq


"ftype": "Q4_K - Medium"
Enter fullscreen mode Exit fullscreen mode

Check the quant Ollama is using:

curl -s http://localhost:11434/api/show -d '{"model": "ornith-1.5:9b"}' | jq '.details.quantization_level'


"Q4_K_M"
Enter fullscreen mode Exit fullscreen mode

Both runtimes are confirmed to be executing Q4_K_M. Precision and model size are strictly identical.

Step 4: Leveling the Playing Field with Flash Attention & Batch Tuning

To give llama.cpp parity with Ollama’s runtime optimizations, we restart llama-server with FlashAttention enabled and explicit batch sizes configured:

llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99 -fa on -b 2048 -ub 512
Enter fullscreen mode Exit fullscreen mode

Here is what these tuning flags configure:

  • -fa on (or --flash-attn on): Enables FlashAttention kernels for Apple Silicon Metal. This significantly accelerates prompt evaluation (prefill) and reduces memory bandwidth overhead.
  • -b 2048 (--batch-size): Sets the logical batch size for prompt evaluation.
  • -ub 512 (--ubatch-size): Sets the physical micro-batch size dispatched to Metal compute passes, keeping the GPU pipelines fully saturated.

Step 5: Side-by-Side Benchmark on Longer Generation

Both engines were given the same prompt:

"give a 25 line story"

A longer generation produces several hundred tokens, giving a much more reliable measurement of steady-state generation speed than short one-liners.

Here are the side-by-side results on the 32 GB MacBook Air:

Metric Ollama (ornith-1.5:9b) llama.cpp (-fa on -b 2048 -ub 512)
Prompt Eval Count 17 tokens 18 tokens
Prompt Eval Speed 46.01 tokens/s 48.68 tokens/s
Generation Count 496 tokens 436 tokens
Generation Speed 16.42 tokens/s 17.07 tokens/s
Total Duration 30.58 s 25.85 s

With FlashAttention and batch tuning enabled, llama.cpp matches or slightly edges out Ollama across both prompt evaluation ( 48.68 vs. 46.01 tokens/s ) and token generation speed ( 17.07 vs. 16.42 tokens/s ).

Ollama Command & Stats 

ollama run ornith-1.5:9b --verbose
>>> give a 25 line story
Thinking...
...done thinking.

[story output omitted]

total duration: 30.582007541s
load duration: 2.821208ms
prompt eval count: 17 token(s)
prompt eval duration: 369.481ms
prompt eval rate: 46.01 tokens/s
eval count: 496 token(s)
eval duration: 30.204706s
eval rate: 16.42 tokens/s
Enter fullscreen mode Exit fullscreen mode

llama.cpp Server Startup & Timing Logs

Server startup and live slot execution log:

➜ ~ llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99 -fa on -b 2048 -ub 512
0.00.729.018 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.729.664 W srv llama_server: -----------------
0.00.729.667 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.729.668 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.729.668 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.729.669 W srv llama_server: -----------------
0.00.731.435 I srv load_model: loading model 'ornith-ai/Ornith-1.5-9B-GGUF'
0.01.177.317 W model has unused tensor blk.32.attn_norm.weight (size = 16384 bytes) -- ignoring
0.01.177.324 W model has unused tensor blk.32.post_attention_norm.weight (size = 16384 bytes) -- ignoring
0.01.177.329 W model has unused tensor blk.32.attn_q.weight (size = 18874368 bytes) -- ignoring
0.01.177.331 W model has unused tensor blk.32.attn_k.weight (size = 2359296 bytes) -- ignoring
0.01.177.333 W model has unused tensor blk.32.attn_v.weight (size = 3440640 bytes) -- ignoring
0.01.177.338 W model has unused tensor blk.32.attn_output.weight (size = 9437184 bytes) -- ignoring
0.01.177.340 W model has unused tensor blk.32.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.01.177.341 W model has unused tensor blk.32.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.01.177.343 W model has unused tensor blk.32.ffn_gate.weight (size = 28311552 bytes) -- ignoring
0.01.177.345 W model has unused tensor blk.32.ffn_down.weight (size = 41287680 bytes) -- ignoring
0.01.177.347 W model has unused tensor blk.32.ffn_up.weight (size = 28311552 bytes) -- ignoring
0.01.177.350 W model has unused tensor blk.32.nextn.eh_proj.weight (size = 18874368 bytes) -- ignoring
0.01.177.353 W model has unused tensor blk.32.nextn.enorm.weight (size = 16384 bytes) -- ignoring
0.01.177.354 W model has unused tensor blk.32.nextn.hnorm.weight (size = 16384 bytes) -- ignoring
0.01.177.359 W model has unused tensor blk.32.nextn.shared_head_norm.weight (size = 16384 bytes) -- ignoring
0.01.799.972 I cmn init: llama threadpool init, n_threads = 4
0.01.959.375 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.01.959.377 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.01.959.377 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

0.02.172.437 I srv load_model: loaded multimodal model, '/Users/vishnuhd/.cache/huggingface/hub/models--ornith-ai--Ornith-1.5-9B-GGUF/snapshots/abdd624b12ebf020b767fff532ff44fe552b28c3/mmproj-Ornith-1.5-9B-BF16.gguf'
0.02.399.797 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.02.402.615 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.02.402.621 I srv llama_server: model loaded
0.02.402.623 I srv llama_server: listening on http://127.0.0.1:8080
0.02.402.623 W srv llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.02.402.623 W srv llama_server: ref: https://github.com/ggml-org/llama.cpp/pull/26508

0.36.386.802 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.36.386.826 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.42.586.497 I slot print_timing: id 3 | task 0 | n_gen = 100, tg = 16.98 t/s, tg_3s = 17.15 t/s
0.45.633.605 I slot print_timing: id 3 | task 0 | n_gen = 152, tg = 17.01 t/s, tg_3s = 17.07 t/s
0.48.679.586 I slot print_timing: id 3 | task 0 | n_gen = 204, tg = 17.03 t/s, tg_3s = 17.07 t/s
0.51.718.658 I slot print_timing: id 3 | task 0 | n_gen = 256, tg = 17.04 t/s, tg_3s = 17.11 t/s
0.54.746.036 I slot print_timing: id 3 | task 0 | n_gen = 308, tg = 17.07 t/s, tg_3s = 17.18 t/s
0.57.778.672 I slot print_timing: id 3 | task 0 | n_gen = 360, tg = 17.08 t/s, tg_3s = 17.15 t/s
1.00.817.109 I slot print_timing: id 3 | task 0 | n_gen = 412, tg = 17.08 t/s, tg_3s = 17.11 t/s
1.02.233.148 I slot print_timing: id 3 | task 0 | prompt eval time = 369.74 ms / 18 tokens ( 20.54 ms per token, 48.68 tokens per second)
1.02.233.155 I slot print_timing: id 3 | task 0 | eval time = 25476.17 ms / 436 tokens ( 58.57 ms per token, 17.07 tokens per second)
1.02.233.157 I slot print_timing: id 3 | task 0 | total time = 25845.92 ms / 454 tokens
1.02.233.159 I slot print_timing: id 3 | task 0 | graphs reused = 434
1.02.233.187 I slot release: id 3 | task 0 | stop processing: n_tokens = 453, truncated = 0
Enter fullscreen mode Exit fullscreen mode

And the raw curl request (with story output trimmed):

curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "ornith-1.5-9b",
    "messages": [{"role": "user", "content": "Give me a 25 line story"}]
  }'


{
  "choices": [{
    "finish_reason": "stop",
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "[story text omitted]",
      "reasoning_content": "[thinking omitted]"
    }
  }],
  "created": 1788423730,
  "model": "ornith-ai/Ornith-1.5-9B-GGUF",
  "system_fingerprint": "b10712-daef7b687",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 436,
    "prompt_tokens": 18,
    "total_tokens": 454,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "id": "chatcmpl-dLoeYkd6rvuFieOvUSr9dBxB798twvZi",
  "timings": {
    "cache_n": 0,
    "prompt_n": 18,
    "prompt_ms": 369.742,
    "prompt_per_token_ms": 20.54,
    "prompt_per_second": 48.68,
    "predicted_n": 436,
    "predicted_ms": 25476.174,
    "predicted_per_token_ms": 58.57,
    "predicted_per_second": 17.07
  }
}
Enter fullscreen mode Exit fullscreen mode

Key Takeaways

  1. Ollama is not faster than llama.cpp : Under the hood, Ollama is llama.cpp. Its initial out-of-the-box advantage in prompt evaluation comes entirely from default runtime tuning (FlashAttention and batch sizes), not a secret runtime or different quantization.
  2. FlashAttention is essential on Apple Silicon : Enabling -fa on doubled prompt evaluation speed from 24.18 tokens/s to 48.68 tokens/s , drastically cutting time-to-first-token.
  3. Consistent throughput : At 9B parameters with Q4_K_M quantization, Apple Silicon unified memory sustains a rock-solid ~17.1 tokens/sec.
  4. Why run llama.cpp directly? : Ollama provides exceptional developer ergonomics for local apps and testing. But direct llama.cpp gives you full control over context allocation, slot limits, KV cache quantization (-ctk, -ctv), and immediate access to upstream features and bugfixes.

In the next post, we will explore KV cache quantization and memory profiling to see how far we can stretch long-context windows on a 32 GB machine.

Top comments (1)

Collapse
 
p_o_26e854a54d851cd606f08 profile image
P O •

Nice comparison. The warm-model and longer-prompt numbers make the tuning point much clearer than a short hello benchmark, and I’d also watch memory pressure during multimodal runs since it can change the result.