Originally published at https://tekmag.thsite.top/omlx-the-open-source-menu-bar-llm-inference-server-trending-on-github/
oMLX is an open-source LLM inference server built for Apple Silicon that runs from your macOS menu bar, using tiered SSD caching and continuous batching to make local coding agents practical.
jundot/omlx recently hit 18.8k stars on GitHub, trending to #4 on GitHub Trending on May 10, 2026. It targets developers running Coding agents like Claude Code, Cursor, and OpenClaw locally on Mac.
Most local inference Tools cache KV state in RAM only. When a coding agent shifts context mid-session, the cache invalidates and the model recomputes from scratch. oMLX persists every cache block to SSD in safetensors format, so previously seen prefixes restore in milliseconds rather than seconds.
Key takeaways: oMLX runs local LLM inference from your macOS menu bar. It persists KV cache to SSD so coding agents don't recompute context on every shift. Continuous batching delivers up to 4.14x speedup on M3 Ultra 512GB. Supports OpenAI and Anthropic APIs with MCP tool calling. 18.8k GitHub stars and counting.
Menu bar control, web dashboard
The app lives in your menu bar. Start, stop, and monitor the server from there. A full web dashboard opens at http://localhost:8000/admin for model management, chat, benchmarks, and per-model settings. The dashboard supports English, Korean, Japanese, Chinese, French, Russian, Spanish, and Portuguese. All CDN dependencies are vendored, so everything works offline after the first run.
The macOS app is signed and notarized with in-app auto-update. It's not Electron. The Homebrew formula and source install are also available.
Continuous batching with measured speedups
oMLX uses mlx-lm's BatchGenerator for concurrent request handling. Benchmarks on an M3 Ultra 512GB show concrete gains:
With Qwen3-Coder-Next-8bit at 1024 prompt tokens and 128 generated tokens, 8x batching delivers 243.3 token/s, a 4.14x speedup over single-request throughput of 58.7 token/s.
Qwen3.5-122B-A10B-4bit on the same machine reaches 190.2 token/s at 8x concurrency, up from 56.6 token/s single-request. MiniMax-M2.5-8bit scales from 34.0 to 126.3 token/s. GLM-5-4bit moves from 16.7 to 60.3 token/s.
All measurements come from the official benchmark page. No cache reuse was applied during these tests.
Tiered KV cache architecture
The cache stack has two tiers. Hot blocks stay in RAM with write-back policy. Cold blocks move to SSD in safetensors format. The paged cache manager uses copy-on-write and prefix sharing across requests.
A ProcessMemoryEnforcer tracks total memory limits and TTL checks. An LRU eviction policy handles multi-model serving when RAM fills up.
This means long conversations with coding agents don't force full recomputation. The agent can circle back to earlier context and oMLX restores it from disk instead of re-running the prompt through the model.
Multi-model serving and API compatibility
oMLX serves LLM, vision-language, embedding, and reranker models simultaneously. It reads from the standard Hugging Face cache at ~/.cache/huggingface/hub, so models already downloaded by Transformers, MLX, vLLM, or llama.cpp work without re-downloading. It also picks up LM Studio folders and custom directories.
The API layer supports both OpenAI-compatible endpoints at /v1/chat/completions and native Anthropic endpoints at /v1/messages. Tool calling formats include JSON, Qwen, Gemma, GLM, and MiniMax. MCP integration is available with configurable tool result trimming for oversized outputs.
The admin dashboard generates the exact config Command for each supported client. Select a model, copy the command, paste into terminal. Works with Claude Code, OpenClaw, Codex, Cursor, and any OpenAI-compatible tool.
Install options and requirements
Requires macOS 15.0+ (Sequoia) and Apple Silicon (M1 through M4). 16GB RAM is the minimum, but 64GB+ is recommended for larger models. The sweet spot for daily coding work is an M-series Pro or Max with 64GB.
Three install paths:
macOS app: Download the DMG from Releases, drag to Applications. The welcome screen guides you through model directory, server start, and first download.
Homebrew: brew install omlx runs as a background service with auto-restart on crash. Logs go to $(brew --prefix)/var/log/omlx.log and ~/.omlx/logs/server.log.
Source: pip install -e . from the cloned repo. Building with native custom kernels for GLM-5.2 or MiniMax M3 requires full Xcode, not just Command Line Tools.
A plain pip install -e . does not build custom kernels. GLM-5.2 fused DSA prefill runs roughly 30x faster with kernels (845 vs ~29 tok/s on M3 Ultra). The fallback also uses more memory.
Why local Mac inference matters now
Coding agents invalidate KV cache frequently. Every context shift triggers full recomputation with most tools. oMLX changes that by persisting cache blocks to SSD across requests and server restarts.
The result is TTFT dropping from 30-90 seconds to under 5 seconds on long contexts, according to the project's benchmarks. A GitHub comment from a user running Qwen3.5 models describes the speed as making local AI on Mac "worthwhile," noting faster performance than LMStudio and more reliable tool calling.
The project started from vllm-mlx v0.1.0 and has evolved with multi-model serving, tiered KV caching, VLM support with full paged cache, an admin panel, and the macOS menu bar app. It uses Apple's MLX and mlx-lm, Blaizzy's mlx-vlm and mlx-embeddings, and dflash-mlx for block diffusion speculative decoding.
oMLX is Apache 2.0 licensed. The source is at github.com/jundot/omlx.
Conclusion
oMLX proves that local inference on Apple Silicon has moved past the hobby stage. Persisting KV cache to SSD fixes the biggest practical pain of running coding agents locally: context shifts no longer force full recomputation, and continuous batching turns a single M3 Ultra into a server that can keep up with concurrent requests. If you run Claude Code, Cursor, or any OpenAI-compatible tool on a Mac, the menu bar app is the lowest-friction way to test whether tiered SSD caching is worth swapping your current inference backend for.
Frequently asked questions
How is oMLX different from Ollama or LM Studio?
Ollama and LM Studio cache KV state in RAM only. When context shifts mid-session, the entire cache invalidates and recomputes from scratch. oMLX persists every cache block to SSD in safetensors format, so previously cached prefixes restore across requests and server restarts without recomputation.
What hardware do I need?
Apple Silicon (M1 or later) with macOS 15+. 16GB RAM is the minimum. 64GB+ is recommended for comfortable use with larger models. An M-series Pro or Max with 64GB is the sweet spot for daily coding work.
Does it work with Claude Code, Cursor, and other tools?
Yes. oMLX provides both OpenAI-compatible and Anthropic-compatible API endpoints. The web dashboard generates the exact config command for each supported client. It works as a drop-in backend for Claude Code, OpenClaw, Cursor, Codex, and any OpenAI-compatible tool.
Do I need to re-download models I already have?
No. oMLX reads the standard Hugging Face cache at ~/.cache/huggingface/hub, shared by Transformers, MLX, vLLM, and llama.cpp. It also picks up your LM Studio folder and custom directories. The admin dashboard includes a built-in HuggingFace downloader for new models.
What models are supported?
Any MLX-format model from HuggingFace. This includes Qwen, LLaMA, Mistral, Gemma, DeepSeek, MiniMax, GLM, and more. Reasoning models get automatic <thinking> tag handling. Vision-Language Models are supported since v0.2.0 with the same paged SSD caching.
Sources:
jundot/omlx GitHub repository |
oMLX official site |
TrendShift repository insights |
Lobste.rs discussion |
daily.dev coverage
{"@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{"@type": "Question", "name": "How is oMLX different from Ollama or LM Studio?", "acceptedAnswer": {"@type": "Answer", "text": "Ollama and LM Studio cache KV state in RAM only. When context shifts mid-session, the entire cache invalidates and recomputes from scratch. oMLX persists every cache block to SSD in safetensors format, so previously cached prefixes restore across requests and server restarts without recomputation."}}, {"@type": "Question", "name": "What hardware do I need?", "acceptedAnswer": {"@type": "Answer", "text": "Apple Silicon (M1 or later) with macOS 15+. 16GB RAM is the minimum. 64GB+ is recommended for comfortable use with larger models. An M-series Pro or Max with 64GB is the sweet spot for daily coding work."}}, {"@type": "Question", "name": "Does it work with Claude Code, Cursor, and other tools?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. oMLX provides both OpenAI-compatible and Anthropic-compatible API endpoints. The web dashboard generates the exact config command for each supported client. It works as a drop-in backend for Claude Code, OpenClaw, Cursor, Codex, and any OpenAI-compatible tool."}}, {"@type": "Question", "name": "Do I need to re-download models I already have?", "acceptedAnswer": {"@type": "Answer", "text": "No. oMLX reads the standard Hugging Face cache at ~/.cache/huggingface/hub, shared by Transformers, MLX, vLLM, and llama.cpp. It also picks up your LM Studio folder and custom directories. The admin dashboard includes a built-in HuggingFace downloader for new models."}}, {"@type": "Question", "name": "What models are supported?", "acceptedAnswer": {"@type": "Answer", "text": "Any MLX-format model from HuggingFace. This includes Qwen, LLaMA, Mistral, Gemma, DeepSeek, MiniMax, GLM, and more. Reasoning models get automatic <thinking> tag handling. Vision-Language Models are supported since v0.2.0 with the same paged SSD caching."}}]}
Top comments (0)