Almost every local LLM setup is chasing the same thing: an endpoint on localhost that speaks the OpenAI API, so the editor plugin, agent framework or script you already have can stop sending tokens to a paid cloud key. Ollama got popular because it turned that into one command.
The company behind it is in a different spot now. It announced an $88M raise on July 9, 2026, says 8.9 million developers use it, and its pricing page sells cloud plans from $20 to $500 a month. We also timed it. Serving the identical model file on an M3 Pro MacBook Pro, Ollama read prompts about 7 percent slower than llama.cpp and generated about 9 percent slower. This is the condensed version of a longer comparison on DevToolLab, which carries the full benchmark script and a screenshot of every tool.
What Ollama actually is, and the three reasons people leave
Ollama is a CLI, a model library and an HTTP API on port 11434, sitting on top of inference engines it did not write. The macOS build of v0.40.4 (October 12, 2026) bundles license files for both llama.cpp and Apple's MLX, and in our run its server log showed a library model loading on the MLX engine. It is MIT licensed and had 183,101 GitHub stars on October 14, 2026.
The reasons to look elsewhere tend to be these:
-
Silent truncation. Ollama's FAQ says the default context window is 4,096 tokens unless
OLLAMA_CONTEXT_LENGTHis set. A longer prompt gets cut with no error. - The engines are available on their own. Anything Ollama runs, you can run directly and skip the wrapper's overhead.
- The business is moving to the cloud. Pro is $20 a month with $60 of usage credits, Max is $100 with $300, Team is $500, and all of it has been billed per token since August 31, 2026.
The first one is cheap to rule out before you migrate anything:
OLLAMA_CONTEXT_LENGTH=16384 ollama serve
Timing the wrapper on one model file
The test machine was an M3 Pro MacBook Pro with 18 GB of memory, running Ollama v0.40.4 and llama.cpp build b11609. Both served Qwen3.5-0.8B in Q8_0 from the ggml-org/Qwen3.5-0.8B-GGUF repo, with the SHA-256 checked against Hugging Face. A small Python script sent each server a 545-token prompt and asked for 128 tokens back, one warm-up run and then five timed runs. The first line of the prompt changed every run so neither server could serve it from a prompt cache.
| Setup | Prompt processing | Generation |
|---|---|---|
llama.cpp llama-server, Q8_0 GGUF |
2,600.9 tok/s | 108.6 tok/s |
llama.cpp llama-server, second pass at the end |
2,570.9 tok/s | 106.3 tok/s |
| Ollama, the same Q8_0 GGUF imported | 2,426.7 tok/s | 98.7 tok/s |
Ollama library qwen3.5:0.8b (MLX engine, mxfp8) |
2,264.6 tok/s | 95.3 tok/s |
The second llama.cpp pass landed within about 2 percent of the first, so thermal throttling does not explain the gap. Ollama's own library build, running on MLX, was no faster, and it was the noisiest: prompt throughput ranged from 1,729.8 to 2,970.3 tokens per second across runs.
One 0.8B model on one Mac is a small sample. The complete script is in the original post, and it takes a couple of minutes to run against your own model and hardware, which is the number that actually matters.
llama.cpp: drop the wrapper, keep the engine
llama.cpp is the C/C++ engine behind Ollama's GGUF path. Running it yourself removes exactly the layer measured above.
One command downloads a model from Hugging Face and starts an OpenAI-compatible server:
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
Context size, GPU layers and every other engine setting are plain command-line flags rather than a Modelfile. What you give up is model management: no curated library, no background service. Also note that llama-server now defaults to port 9931, so configs written for an older default need a fix. MIT, 131,123 stars, v0.6.0 on October 5, 2026, nightly b11609 on October 14.
LM Studio: a GUI first, a server second
LM Studio, now branded LM Studio Bionic, is a desktop app that runs models on llama.cpp and MLX. It suits people who want a model browser and a chat window instead of a terminal. The local server starts from the Developer tab or with lms server start, and it exposes both OpenAI-compatible and Anthropic-compatible endpoints.
The app itself is closed source; only the lms CLI is MIT. It has also started selling cloud inference (Bionic+ at $20 a month, Pro at $100), while running models locally stays free. Version at the time of testing: Bionic 1.1.8.
vLLM: for when a whole team hits the endpoint
vLLM is where you go when a local model turns into a shared internal service. PagedAttention, continuous batching and prefix caching let one GPU juggle many concurrent requests, and tensor, pipeline and expert parallelism spread the load across several. It speaks the OpenAI API and the Anthropic Messages API on NVIDIA, AMD and Intel GPUs.
It is not a laptop tool. The target is a Linux GPU box in a data center or us-east-1; Apple Silicon support is a plugin, not part of the core. If you are comparing, SGLang (Apache 2.0, v0.5.21) is its nearest open source peer. vLLM is Apache-2.0, 93,753 stars, v0.31.0 on October 5, 2026.
Jan: LM Studio, but open source
Jan, built by Menlo Research, is a desktop chat app that pulls models from Hugging Face and runs them on your machine. It serves an OpenAI-compatible API at localhost:1337, supports MCP for agent tools, and lets you mix cloud models from OpenAI, Anthropic or Groq into the same window. Its site claims more than 7 million downloads.
Jan is built for the desktop. For a headless server, llama.cpp or vLLM is the better fit. Apache-2.0, 44,938 stars, v0.8.6 on October 9, 2026.
LocalAI: one binary that pretends to be OpenAI
LocalAI puts an OpenAI-compatible API in front of several engines, so client code written against OpenAI keeps working on your own hardware. Its selling point is breadth: text, voice, vision, image and video models from one binary, and it will run CPU-only on a small VM or an old desktop with no GPU.
The cost of that breadth is operational. LocalAI is configured like a server, and more backends means more things to keep updated than with a single engine. MIT, 49,519 stars, v4.11.0 on October 2, 2026.
Docker Model Runner: models shipped like images
Docker Model Runner adds docker model run and pulls models from Docker Hub or any OCI registry. It runs inside Docker Desktop, Docker Engine, or as a standalone dmr binary, and models are pulled, tagged and pushed the same way your container images are.
Under the hood it uses llama.cpp, plus vLLM and Diffusers on Linux machines with NVIDIA GPUs. The API on port 12434 answers OpenAI, Anthropic and Ollama clients, so an existing Ollama integration only needs a new base URL. The weak spots: its repo has 659 stars, a small community next to the engines it wraps, and model coverage depends on what is in a registry. Apache-2.0, v1.2.8 on August 12, 2026, and it needs Docker Desktop 4.40 or later on macOS.
MLX LM: Apple's route on Apple Silicon
MLX LM is Apple's Python package for running and fine-tuning language models on MLX, the same framework Ollama itself now uses on Macs. It loads Hugging Face models in one command, quantizes them, and does LoRA or full fine-tuning on the same machine. mlx_lm.server exposes an API modeled on OpenAI's chat endpoint, on port 8080 by default.
It only runs on Apple Silicon, and its own docs say the server is not recommended for production. MIT, 7,297 stars, v0.32.0 on October 1, 2026.
The eight tools in one table
| Tool | Pick it for | Local API | License |
|---|---|---|---|
| Ollama | The one-command start | OpenAI-compatible, port 11434 | MIT |
| llama.cpp | Speed and every flag exposed | OpenAI-compatible | MIT |
| LM Studio | A desktop GUI | OpenAI and Anthropic compatible | Proprietary app |
| vLLM | Many users on GPU servers (Mac via plugin only) | OpenAI and Anthropic compatible | Apache-2.0 |
| Jan | An open source desktop app | OpenAI-compatible, port 1337 | Apache-2.0 |
| LocalAI | A drop-in API, including CPU-only boxes | OpenAI-compatible | MIT |
| Docker Model Runner | Teams already on containers | OpenAI, Anthropic and Ollama compatible, port 12434 | Apache-2.0 |
| MLX LM | Fine-tuning on a Mac | OpenAI-like, port 8080 | MIT |
Choosing without migrating twice
Before you switch, do these in order:
-
Rule out the context window. If quality falls apart on long prompts, set
OLLAMA_CONTEXT_LENGTHfirst. The 4,096-token default may be the entire problem. - Benchmark your own model. A single-digit gap on a 0.8B model on one Mac tells you nothing certain about a 70B model on an NVIDIA card.
- Count the people hitting the endpoint. One developer is a desktop question. A team sharing one URL is a vLLM question.
- Keep your client on the OpenAI API. Every tool here speaks it or something close, so changing servers is mostly a base URL change. A smoke test like the one below is easy to turn into client code with the cURL to Code Converter.
- Decide whether you care more about the files or the app. GGUF files move freely between llama.cpp, LM Studio, Jan and Ollama, and the GGUF Viewer shows a file's quantization, context length and chat template before you import it. MLX weights stay on Apple Silicon.
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "qwen3.5:0.8b", "messages": [{"role": "user", "content": "ping"}]}'
If you are still sizing hardware, the companion guide on local LLM VRAM requirements covers how much memory each quantization needs.
Short answers
-
Ollama without the wrapper: llama.cpp. It was the fastest server in our test, and
llama serve -hfhandles the download-and-serve step thatollama rundoes. - A window instead of a terminal: LM Studio, or Jan if you want open source.
- A team about to share one endpoint: vLLM on a Linux GPU server.
- A stack that already lives in containers: Docker Model Runner, which even answers Ollama's API.
- Fine-tuning on a MacBook: MLX LM.
Wrapping up
Ollama is still the easiest way to get a model running locally, and for plenty of people that is enough. It is also now a wrapper over engines you can run directly, made by a company putting more of its effort into cloud plans. Fix the context length, run the benchmark on your hardware, and if the gap shows up, the engine underneath is one command away.
Top comments (0)