DEV Community

Jinxuan Che
Jinxuan Che

Posted on

Run a private OpenAI-compatible LLM endpoint on Apple Silicon with one Rust binary

#ai #showdev #rust

Disclosure: I maintain Ferrum, an MIT-licensed local LLM inference server written in Rust.

If you want a private OpenAI-compatible endpoint on an M-series Mac without setting up Python or a container, this is the shortest path I currently recommend.

The commands below were tested end to end with the published Ferrum v0.8.3 Homebrew build on an M1 Max.

1. Install Ferrum 

brew tap sizzlecar/ferrum
brew install ferrum
Enter fullscreen mode Exit fullscreen mode

2. Run the preflight check 

ferrum doctor qwen3.5:4b-q4_k_m
Enter fullscreen mode Exit fullscreen mode

ferrum doctor checks the local setup before inference. In one real install it caught missing Xcode Command Line Tools before the user reached a harder-to-diagnose failure.

3. Start an interactive chat

The first run downloads about 2.55 GiB. The terminal can look quiet during that download, so give it time before assuming it has hung.

ferrum run qwen3.5:4b-q4_k_m --disable-thinking
Enter fullscreen mode Exit fullscreen mode

Qwen3.5 emits verbose reasoning by default. --disable-thinking gives a more conventional first-chat experience. Omit the flag when you want the reasoning behavior.

4. Start the OpenAI-compatible API 

ferrum serve \
  --model qwen3.5:4b-q4_k_m \
  --served-model-name ferrum \
  --disable-thinking \
  --port 8000
Enter fullscreen mode Exit fullscreen mode

In another terminal:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ferrum",
    "messages": [{"role": "user", "content": "Reply exactly: ferrum-ok"}],
    "max_tokens": 32
  }'
Enter fullscreen mode Exit fullscreen mode

A working setup returns HTTP 200 with a non-empty assistant response.

What I am looking for

I am looking for failure-oriented feedback from Apple Silicon users:

  • Which step was unclear?
  • Did Homebrew, doctor, the model download, or the first API request fail?
  • Which OpenAI-compatible client should be tested next?
  • What would make you use this again instead of returning to your current local inference setup?

Repository and issue tracker: github.com/sizzlecar/ferrum-infer-rs

This post was drafted with assistance from OpenAI Codex. The commands and stated behavior were checked against the published build; no comparative performance claim is being made here.

Top comments (0)