Run a private OpenAI-compatible LLM endpoint on Apple Silicon with one Rust binary

Jinxuan Che — Wed, 02 Sep 2026 03:12:46 +0000

Disclosure: I maintain Ferrum, an MIT-licensed local LLM inference server written in Rust.

If you want a private OpenAI-compatible endpoint on an M-series Mac without setting up Python or a container, this is the shortest path I currently recommend.

The model walkthrough below was tested with Ferrum v0.8.3 on an M1 Max. The one-command installation was checked with the published v0.8.8 macOS Metal build on September 8.

1. Install Ferrum

curl -fsSL https://ferrum.pandaailabs.com/install.sh | sh

The installer downloads the latest stable macOS Metal build, checks its SHA-256 checksum, and adds Ferrum to PATH. Open a new terminal before continuing. Run the same install command again to upgrade.

2. Run the preflight check

ferrum doctor qwen3.5:4b-q4_k_m

ferrum doctor checks the local setup before inference. In one real install it caught missing Xcode Command Line Tools before the user reached a harder-to-diagnose failure.

3. Start an interactive chat

The first run downloads about 2.55 GiB. The terminal can look quiet during that download, so give it time before assuming it has hung.

ferrum run qwen3.5:4b-q4_k_m --disable-thinking

Qwen3.5 emits verbose reasoning by default. --disable-thinking gives a more conventional first-chat experience. Omit the flag when you want the reasoning behavior.

4. Start the OpenAI-compatible API

ferrum serve \
  --model qwen3.5:4b-q4_k_m \
  --served-model-name ferrum \
  --disable-thinking \
  --port 8000

In another terminal:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ferrum",
    "messages": [{"role": "user", "content": "Reply exactly: ferrum-ok"}],
    "max_tokens": 32
  }'

A working setup returns HTTP 200 with a non-empty assistant response.

What I am looking for

If you try it, I’d like to hear which model and client you connect, and whether you reached your first useful response. If something gets in the way, include your Mac model and the step so I can investigate.

Repository and issue tracker: github.com/sizzlecar/ferrum-infer-rs