Introducing Superfluid: the local-AI native LLM server for multi-agent inference

Prabod Rathnayaka — Tue, 13 Oct 2026 03:33:23 +0000

The future of local AI is agentic and requires multiple agents to share one machine: one agent summarising your emails overnight, one watching your health data and one doing the shopping, while you are still able to chat and ask questions in between. Today's inference servers were built for throughput on data centre racks, not for one person sharing a laptop with eight agents. Without prioritisation, your question waits behind whatever is already running and without logging, a restart takes every session with it.

Superfluid is an open-source (Apache-2.0) server for running LLMs on your own machine. Our server supports the OpenAI, Anthropic and Ollama APIs and various runtimes: llama.cpp for GGUF models, MLX for MLX directories, the BaseRT engine for .base bundles.

Superfluid was built for multi-agent workloads on one machine:

curl -fsSL https://superfluid.sh/install.sh | sh
superfluid serve unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Eight agents working on 7.5k-token documents, then a chat question arrives after two seconds. Superfluid produces an answer in 0.36s, llama-server takes over 10 seconds. That's a 30x faster response for the same model on the same Mac.

What you get

Your chat is answered while agents work. Interactive requests preempt batch work at the next scheduler tick instead of waiting behind it.
Nothing is lost when a process dies. Every token is written to a log before you see it. A crashed worker or a restarted server picks its sessions up where it left off.
A shared prompt is computed once. 32 requests with the same system prompt cost one prefill.
One server for every runtime. The same APIs, tool calling, structured output and chat templates on llama.cpp, MLX and BaseRT.
Multi-device inference You add additional devices to your private cloud with a single command over an encrypted link.

Try it

Install for macOS (Apple silicon) or Linux (x86-64, arm64). On Apple silicon and Linux, the BaseRT engine is installed per default, while llama.cpp and MLX are installed automatically when needed.

curl -fsSL https://superfluid.sh/install.sh | sh

Or cargo install superfluid.

Serve a GGUF from Hugging Face. The tag picks the quantization:

superfluid serve unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Call it with anything that speaks the OpenAI API. Anthropic's SDK and Ollama clients pointed at the server address equally work:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8453/v1", api_key="not-needed")
stream = client.chat.completions.create(
    model="unsloth/Qwen3.8-27B-GGUF",
    messages=[{"role": "user", "content": "Explain KV caching in two sentences."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

An MLX model works the same way: superfluid serve mlx-community/Qwen3-4B-4bit.

Run a coding agent

superfluid launch claude --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

launch starts the server if none is running and runs the agent pointed at it. It works for Claude Code, Codex, pi, OpenCode, Hermes Agent, Cline, OpenClaw and the Ollama CLI.

Run multi-device inference

One machine is the head server. Create a token there and serve with a listener for nodes:

superfluid fleet token
superfluid serve unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M --http 0.0.0.0:8453 --fleet-listen 0.0.0.0:8454

On each other machine:

superfluid node join --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M --token <the token>

The node finds the head on the local network, pulls the model, and serves it over a link encrypted with the token. The head places whole sessions on nodes and only sampled tokens come back. If a node dies mid-stream, its session is replaced and no token is delivered twice. Nodes can run different runtimes for the same model: BaseRT on a Mac next to llama.cpp on a Linux box.

Performance

We measured performance with the native superfluid/tools/suite on an Apple M5 Pro with Qwen3.8-27B at 4 bits, where each server deploys the same runtime.

	`superfluid`	`llama-server`	Ollama	`mlx_lm.server`
TTFT for a chat request behind 8 agents:	1.5 s	291 s	380 s	612 s
Cache hits for 32 requests sharing a 4k-token prompt:	97%	75%	97%	0%
Time to serve after inference worker killed mid-stream	1.8 s	n/a	1.8 s, stream truncated	n/a

The serving layer adds no overhead, but increases performance in several scenarios: at one and eight concurrent requests, llama.cpp served through Superfluid matches llama-server on tokens per second. Superfluid batches MLX requests that mlx_lm.server serves less efficiently one at a time. Find every method and evaluation on the website: superfluid.sh/latest/performance/benchmarks.

Try it, break it, and tell us what you find.