Run a private OpenAI-compatible LLM endpoint on Apple Silicon with one Rust binary

Jinxuan Che — Wed, 02 Sep 2026 03:12:46 +0000

Disclosure: I maintain Ferrum, an MIT-licensed local LLM inference server written in Rust.

If you want a private OpenAI-compatible endpoint on an M-series Mac without setting up Python or a container, this is the shortest path I currently recommend.

The commands below were tested end to end with the published Ferrum v0.8.3 Homebrew build on an M1 Max.

1. Install Ferrum

brew tap sizzlecar/ferrum
brew install ferrum

2. Run the preflight check

ferrum doctor qwen3.5:4b-q4_k_m

ferrum doctor checks the local setup before inference. In one real install it caught missing Xcode Command Line Tools before the user reached a harder-to-diagnose failure.

3. Start an interactive chat

The first run downloads about 2.55 GiB. The terminal can look quiet during that download, so give it time before assuming it has hung.

ferrum run qwen3.5:4b-q4_k_m --disable-thinking

Qwen3.5 emits verbose reasoning by default. --disable-thinking gives a more conventional first-chat experience. Omit the flag when you want the reasoning behavior.

4. Start the OpenAI-compatible API

ferrum serve \
  --model qwen3.5:4b-q4_k_m \
  --served-model-name ferrum \
  --disable-thinking \
  --port 8000

In another terminal:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ferrum",
    "messages": [{"role": "user", "content": "Reply exactly: ferrum-ok"}],
    "max_tokens": 32
  }'

A working setup returns HTTP 200 with a non-empty assistant response.

What I am looking for

I am looking for failure-oriented feedback from Apple Silicon users:

Which step was unclear?
Did Homebrew, doctor, the model download, or the first API request fail?
Which OpenAI-compatible client should be tested next?
What would make you use this again instead of returning to your current local inference setup?

Repository and issue tracker: github.com/sizzlecar/ferrum-infer-rs