DEV Community

Jinxuan Che
Jinxuan Che

Posted on Edited on

Run a private OpenAI-compatible LLM endpoint on Apple Silicon with one Rust binary

#showdev #ai #rust

Disclosure: I maintain Ferrum, an MIT-licensed local LLM inference server written in Rust.

If you want a private OpenAI-compatible endpoint on an M-series Mac without setting up Python or a container, this is the shortest path I currently recommend.

The model walkthrough below was tested with Ferrum v0.8.3 on an M1 Max. The one-command installation was checked with the published v0.8.8 macOS Metal build on September 8.

1. Install Ferrum 

curl -fsSL https://ferrum.pandaailabs.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

The installer downloads the latest stable macOS Metal build, checks its SHA-256 checksum, and adds Ferrum to PATH. Open a new terminal before continuing. Run the same install command again to upgrade.

2. Run the preflight check 

ferrum doctor qwen3.5:4b-q4_k_m
Enter fullscreen mode Exit fullscreen mode

ferrum doctor checks the local setup before inference. In one real install it caught missing Xcode Command Line Tools before the user reached a harder-to-diagnose failure.

3. Start an interactive chat

The first run downloads about 2.55 GiB. The terminal can look quiet during that download, so give it time before assuming it has hung.

ferrum run qwen3.5:4b-q4_k_m --disable-thinking
Enter fullscreen mode Exit fullscreen mode

Qwen3.5 emits verbose reasoning by default. --disable-thinking gives a more conventional first-chat experience. Omit the flag when you want the reasoning behavior.

4. Start the OpenAI-compatible API 

ferrum serve \
  --model qwen3.5:4b-q4_k_m \
  --served-model-name ferrum \
  --disable-thinking \
  --port 8000
Enter fullscreen mode Exit fullscreen mode

In another terminal:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ferrum",
    "messages": [{"role": "user", "content": "Reply exactly: ferrum-ok"}],
    "max_tokens": 32
  }'
Enter fullscreen mode Exit fullscreen mode

A working setup returns HTTP 200 with a non-empty assistant response.

What I am looking for

If you try it, I’d like to hear which model and client you connect, and whether you reached your first useful response. If something gets in the way, include your Mac model and the step so I can investigate.

Repository and issue tracker: github.com/sizzlecar/ferrum-infer-rs

This post was drafted with assistance from OpenAI Codex. The commands and stated behavior were checked against the published build; no comparative performance claim is being made here.

Top comments (0)