Sreeraj Sreenivasan

Posted on Jul 19

The Complete Guide to Local LLM Inference Tools in July 2026: llama.cpp, Ollama, vLLM, SGLang, and Beyond

#ai #llm #opensource #devtools

Nine tools, three layers, one decision framework. Everything you need to run open-source models in 2026.

Why This Guide Exists

The local LLM inference ecosystem has quietly matured into one of the most consequential layers of the open-source AI stack. In 2026, you can run Qwen3-235B on a Mac Studio, serve DeepSeek V4 to a hundred concurrent users from a single H100, or deploy Gemma 3 on a Raspberry Pi — all without a cloud API, without a subscription, and without sending a single token to a third-party server.

But choosing the wrong tool for your workload doesn't just cost performance. It determines whether your architecture even works. Running vLLM on a MacBook won't go well. Running Ollama for a team of fifty concurrent users won't scale. Running llama.cpp when you need structured JSON output from an agent loop is friction you don't need.

The single most important framing: These tools do not occupy the same layer of the stack. Some are raw inference engines. Some are experience wrappers around those engines. Some are production-grade serving systems. Choosing "the best one" without specifying your workload is like asking whether a hammer or a drill is better.

The Architecture Map

Before the tool list, here's how everything relates:

┌─────────────────────────────────────────────────────┐
│              LAYER 1: Developer UX                  │
│   Ollama · LM Studio · Jan · GPT4All · Open WebUI  │
│   (wrap engines below; optimised for ease of use)  │
├─────────────────────────────────────────────────────┤
│              LAYER 2: Inference Engines             │
│   llama.cpp · Apple MLX · ExLlamaV3 · MLC-LLM     │
│   (run the model; all others are built on these)   │
├─────────────────────────────────────────────────────┤
│           LAYER 3: Production Serving               │
│   vLLM · SGLang · LMDeploy · Aphrodite            │
│   (multi-user concurrency; GPU-optimised batching) │
├─────────────────────────────────────────────────────┤
│           LAYER 4: Datacenter / Scale               │
│   TensorRT-LLM + Triton (NVIDIA-only)              │
│   (maximum throughput; 28-min compile step)        │
└─────────────────────────────────────────────────────┘

⚠️  TGI (HuggingFace Text Generation Inference)
    → Moved to maintenance mode: March 21, 2026
    → Officially redirects new users to vLLM, SGLang,
      llama.cpp, and MLX. Migrate existing deployments.

Open Source Status at a Glance

Tool	License	Truly Open Source?
llama.cpp	MIT	✅ Yes
Ollama	MIT	✅ Yes
Jan	Apache 2.0	✅ Yes
GPT4All	MIT	✅ Yes
SGLang	Apache 2.0	✅ Yes
vLLM	Apache 2.0	✅ Yes
LMDeploy	Apache 2.0	✅ Yes
Aphrodite Engine	AGPL-3.0	✅ Yes (copyleft)
Apple MLX / mlx-lm	MIT	✅ Yes
MLC-LLM	Apache 2.0	✅ Yes
TensorRT-LLM	Apache 2.0	✅ Yes (NVIDIA-only runtime)
LM Studio	Proprietary	❌ Closed source
~~TGI~~	Apache 2.0	⚠️ Maintenance mode

This is a remarkable story: almost everything in the local LLM inference stack is fully open source under permissive licenses. LM Studio is the lone proprietary tool in common use, and Jan exists specifically as its open-source alternative.

Layer 1: Developer UX Tools

Start here. Zero to inference in minutes.

🔥 llama.cpp

GitHub: ggml-org/llama.cpp | Stars: 85,000+ | License: MIT

The foundation of the entire local LLM ecosystem. llama.cpp is a pure C/C++ inference engine with no external dependencies that runs GGUF-format quantized models on virtually any hardware — NVIDIA CUDA, AMD ROCm, Apple Metal, CPU-only, and even Raspberry Pi.

When people say "run a model locally," the odds are high that llama.cpp is doing the actual computation underneath, even if they're using Ollama, LM Studio, or Jan as the interface.

What makes it special:

GGUF format — the open standard for quantized model distribution; ~70% of community model releases use it
Widest hardware support of any inference engine: x86, ARM, Apple Silicon, CPU-only, embedded, air-gapped
10–25% faster than Ollama on identical hardware (no wrapper overhead)
llama-server binary provides a built-in OpenAI-compatible REST API when you need it
Full control over every inference parameter: context length, batch size, GPU layers, quantization level, threads

What it lacks:

No model management — you download and manage GGUF files manually from Hugging Face
No built-in model registry, chat UI, or automatic updates
Not optimised for multi-user concurrent serving (sequential request handling)

Build and run:

# Build from source (one-time, ~10-15 min)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build -j$(nproc)

# Run a model
./build/bin/llama-server \
  -m ./models/qwen3-8b-q4_k_m.gguf \
  --port 8080 \
  --ctx-size 32768 \
  -ngl 99  # GPU layers: 99 = all on GPU

Best for: Embedded deployments, air-gapped servers, maximum single-user inference speed, weird hardware nobody else supports, production pipelines where you own every layer.

⚡ Ollama

GitHub: ollama/ollama | Stars: 130,000+ | License: MIT

Ollama is the Docker of local LLMs. It wraps llama.cpp (or Apple MLX on Apple Silicon since v0.19, March 2026) in a Go binary with a model registry, automatic GPU detection, and an OpenAI-compatible REST API — all accessible from a single command.

It is the right first install for most developers. The whole agentic tooling ecosystem — Cursor, Continue, Aider, Open WebUI, LangChain, LlamaIndex — targets Ollama's API by default.

What makes it special:

ollama run qwen3:8b — pulls a quantized model and starts inference in under 5 minutes, zero config
OpenAI-compatible API at localhost:11434/v1 — works as a drop-in replacement for api.openai.com in most frameworks
On Apple Silicon, now uses MLX backend natively — the fastest Mac inference path, not llama.cpp
Serve multiple models simultaneously; Ollama manages memory and swaps on demand
Model library covers all major open-weight models: Qwen3, Llama 4, DeepSeek, Gemma, Mistral, Phi, and more

What it lacks:

10–20% slower than raw llama.cpp (wrapper overhead — unnoticeable in interactive chat, matters in batch jobs)
GGUF only — no HuggingFace native safetensors, no AWQ or GPTQ
Not designed for multi-user concurrent serving; queues requests sequentially under load

# Install
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run any open-weight model
ollama run qwen3:8b          # 6GB VRAM
ollama run qwen3:32b         # ~19GB Q4_K_M
ollama run deepseek-v3:7b    # Great for coding + reasoning
ollama run llama4:scout      # 10M context, 17B active

# Use the OpenAI-compatible API
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Hello"}]}'

Best for: Solo developers, prototyping, building agentic apps locally, anyone who wants to go from zero to inference in 5 minutes. The default starting point for 80% of developers.

🔓 Jan

GitHub: janhq/jan | Stars: 42,000+ | License: Apache 2.0 | Downloads: 5.3M+

Jan is the open-source answer to the question: "What if I want LM Studio's GUI but with full source code, zero telemetry, and a license I can audit?"

Built with Tauri (Rust) instead of Electron — leaner RAM footprint and better performance than most desktop AI apps. It wraps llama.cpp under the hood, serves an OpenAI-compatible API on localhost:1337, and ships an extension system that lets you add new model providers or workflows without touching the core app.

What makes it special:

Fully Apache 2.0 open source — every line of code is auditable
Zero telemetry by default — runs completely offline, no account required, no data ever leaves your device
MCP (Model Context Protocol) support — plug Jan into agentic frameworks natively
Extension system — add new model providers, remote API connections (OpenAI, Anthropic, Gemini), or custom workflows
Dual mode — local models and cloud APIs in the same interface; switch per conversation
Passes CMMC Level 1 and HIPAA technical safeguard reviews for regulated deployments
Windows, macOS (Apple Silicon + Intel), Linux

What it lacks:

Fewer advanced GPU tuning controls than LM Studio
RAG support limited to direct file attachment (no built-in vector store)
Less scriptable than Ollama for automation workflows

# Install via package manager or download from jan.ai
# macOS
brew install --cask jan

# API server runs on port 1337 by default
curl http://localhost:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Hello"}]}'

Best for: Privacy-first users, regulated industries (healthcare, legal, finance), teams that need an auditable open-source codebase, developers who want a full GUI desktop app without the proprietary overhead of LM Studio.

🌐 GPT4All

GitHub: nomic-ai/gpt4all | Stars: 73,000+ | License: MIT

GPT4All is the most non-technical-user-friendly entry in this list. Built by Nomic AI, it's a desktop app (Windows, Mac, Linux) designed for people who want a local ChatGPT without any command-line interaction at all. It also ships a Python SDK for developers who want GPT4All as an embedded inference library.

What makes it special:

Easiest possible onboarding for non-developers
CPU-first design — runs on laptops without a dedicated GPU (just slowly)
LocalDocs feature: attach a folder of PDFs or text files and query them in a local RAG pipeline — no setup required
Python SDK: from gpt4all import GPT4All — embed local inference in any Python app in two lines
Model ecosystem covers Llama, Mistral, Qwen, Falcon, and more in pre-optimised GGUF format

What it lacks:

Not designed for production serving or multi-user scenarios
Less control over inference parameters vs llama.cpp or Ollama
Slower model updates than the Ollama model library

# Python SDK
from gpt4all import GPT4All

model = GPT4All("Llama-3.2-3B-Instruct.Q4_0.gguf")
with model.chat_session():
    print(model.generate("Explain MoE architecture in one paragraph"))

Best for: Non-technical users who want a private local AI desktop assistant, developers who want to embed local inference in Python apps with zero setup, and anyone who needs CPU-only operation as a hard requirement.

Layer 2: Raw Inference Engines

Under the hood — what everything above is built on.

🍎 Apple MLX / mlx-lm

GitHub: ml-explore/mlx | Stars: 21,000+ | License: MIT

On Apple Silicon, the old framing of "Ollama vs MLX" has collapsed: Ollama 0.19+ uses MLX as its backend on M-series Macs automatically. But mlx-lm as a standalone Python library gives you capabilities Ollama doesn't expose — particularly local fine-tuning.

What makes it special:

Native Metal GPU acceleration — fastest inference on Apple Silicon hardware
The Qwen3-235B MoE runs at 5.5+ tok/s on an M4 Max with 128GB unified memory
LoRA and QLoRA fine-tuning on your Mac — tune a model on your own data without cloud GPU access
Unified memory architecture on M-series makes large models viable without VRAM constraints

pip install mlx-lm

# Run inference
python -m mlx_lm.generate \
  --model mlx-community/Qwen3-8B-4bit \
  --prompt "Explain radix attention in two paragraphs"

# Fine-tune locally
python -m mlx_lm.lora \
  --model mlx-community/Llama-4-Scout-17B-4bit \
  --train --data ./my_data

Best for: Apple Silicon developers who want to push past Ollama's API surface — specifically for fine-tuning, custom quantization, or scripted batch inference on Mac hardware.

Layer 3: Production Serving Frameworks

Multi-user, multi-GPU, OpenAI-compatible APIs at scale.

🚀 vLLM

GitHub: vllm-project/vllm | Stars: 50,000+ | License: Apache 2.0

vLLM is the production standard for multi-user LLM serving. Its PagedAttention algorithm treats GPU KV cache like virtual memory pages — the same technique that made OS virtual memory efficient in the 1970s, applied to GPU memory fragmentation in 2023. The result: 16–20× Ollama's concurrent throughput at peak load.

Note that the gap collapses to near-zero at one user. vLLM's advantage lives entirely at concurrency. A single developer running queries sequentially will see no benefit over Ollama, and will feel the slower cold start and more complex setup.

What makes it special:

PagedAttention — near-zero GPU memory waste from KV cache fragmentation; enables larger batch sizes and more concurrent users
Continuous batching — new requests join in-flight batches without waiting for previous requests to complete
Native HuggingFace safetensors model format — no quantization required (run full-precision FP16 or BF16)
Full function calling, structured outputs, streaming
Multi-GPU tensor parallelism: --tensor-parallel-size 4 splits a model across 4 GPUs
OpenAI-compatible API: drop-in replacement for api.openai.com

What it lacks:

NVIDIA CUDA required (AMD ROCm support exists but incomplete)
16GB+ VRAM minimum practical; plan for 20–30% more VRAM than model base size due to paging buffers
Slow cold start: minutes on first run (CUDA kernel compilation)
Cannot serve multiple models from one process (run separate vLLM processes per model)

pip install vllm

# Serve a model
vllm serve Qwen/Qwen3-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1

# Multi-GPU serving (4 GPUs)
vllm serve meta-llama/Llama-4-Scout-17B \
  --tensor-parallel-size 4 \
  --max-model-len 1000000  # 1M context

Best for: Production APIs serving 10+ concurrent users, internal AI platforms, multi-GPU datacenter deployments, any workload where throughput under concurrency is the primary constraint.

⚡ SGLang (Structured Generation Language)

GitHub: sgl-project/sglang | Stars: 18,000+ | License: Apache 2.0 | Runs on: 400,000+ GPUs worldwide

SGLang is the fastest-growing production serving framework in 2026, and the one most relevant to the agentic AI workflows that dominate modern development. Built by the LMSYS team at Berkeley, it powers trillions of tokens per day in production deployments.

Its core architectural breakthrough is RadixAttention — a prefix-caching scheme that reuses KV cache computations across requests that share a common prefix. In RAG pipelines where system prompts account for 60–80% of request tokens, RadixAttention skips that computation entirely on repeated requests.

The results are significant: SGLang beats vLLM by 29% on overall throughput on H100 GPUs, and delivers up to 6× acceleration in RAG scenarios specifically.

What makes it special:

RadixAttention — automated KV cache reuse for shared prefixes; transformative for RAG, chatbots, and agent loops
Multi-model serving from a single process (vLLM can't do this)
Structured output native — JSON schema enforcement, function calling, and constrained generation are first-class citizens in the architecture, not afterthoughts
Hardware breadth: NVIDIA, AMD, Intel Xeon, Google TPU, and Ascend NPU
Hugging Face and OpenAI API compatible
The fastest open-source framework for DeepSeek V3/V4 serving — the DeepSeek community has converged on SGLang as the reference implementation

pip install sglang[all]

# Serve with RadixAttention (prefix caching enabled by default)
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --mem-fraction-static 0.9

# Multi-model on same port
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3 \
  --tp 4  # 4-GPU tensor parallel

When SGLang beats vLLM:

RAG pipelines with shared system prompts (6× faster due to RadixAttention)
Multi-turn chatbots with long conversation history (prefix caching compounds)
Agent loops with repeated tool descriptions and schemas
Workloads requiring structured JSON output reliability
Multi-model serving from one process

Best for: AI agent deployments, RAG pipelines, any production workload with shared prefixes or structured output requirements. If you are building an agentic system in 2026, SGLang deserves evaluation before vLLM.

🔬 Aphrodite Engine

GitHub: PygmalionAI/aphrodite-engine | License: AGPL-3.0

Aphrodite is built on vLLM's PagedAttention foundation but extends it with the widest quantization support in any single serving framework — it handles GGUF, ExLlamaV3, GPTQ, AWQ, AQLM, BitNet, Bitsandbytes, MXFP4, TurboQuant, and more in one runtime.

The AGPL-3.0 license is worth noting: if you serve Aphrodite over a network in a commercial product, you may be required to open-source your server code. Check your compliance requirements before deploying commercially.

What makes it special:

Largest quantization format support of any single serving engine
Notably, ExLlamaV3/EXL2 support — a large chunk of the community quantization ecosystem on HuggingFace uses these formats and historically required a separate runtime
Extended sampler options (Mirostat, DRY, XTC, and more) — useful for creative/generative workloads

Best for: Teams with diverse quantization format requirements, community model ecosystems using EXL2/ExLlamaV3, or research workloads needing experimental sampler configurations.

🏭 LMDeploy

GitHub: InternLM/lmdeploy | Stars: 6,000+ | License: Apache 2.0

LMDeploy is OpenMMLab's production-grade inference toolkit, particularly strong for vision-language models and INT4 quantization on A100/A800 hardware. It supports multi-model serving from a single process and has one of the fastest time-to-first-token (TTFT) metrics on low-precision workloads.

What makes it special:

Best-in-class TTFT at INT4 precision on A100/A800
Multi-model serving from a single process
Optimised for InternLM, Qwen, Llama, and multimodal models
Prefill optimisation — reduces time waiting for first token on long prompts

Best for: Vision-language model serving, INT4 quantization workloads, and teams deploying on Chinese AI infrastructure (A100/A800 Ampere GPUs).

Layer 4: Datacenter Scale

🏔️ NVIDIA TensorRT-LLM + Triton

GitHub: NVIDIA/TensorRT-LLM | License: Apache 2.0

The highest-throughput option in the ecosystem — but with a meaningful cost: every model must be compiled into a TensorRT engine before first use, which takes 15–30 minutes. After that compilation, TensorRT-LLM leads at every concurrency level tested on H100 hardware.

What makes it special:

Fastest raw throughput at scale on NVIDIA H100/B200
FP8 and NVFP4 precision support — leverages Hopper/Blackwell hardware capabilities that other engines don't yet fully exploit
Triton Inference Server integration provides the production API surface, load balancing, and multi-model routing

What it costs you:

NVIDIA-only — AMD, Intel, and Apple Silicon are not supported
28-minute model compilation step per model version (one-time, then cached)
Most complex setup and maintenance overhead in this list
TensorRT-LLM leaves the API surface to Triton — you need to configure both

Best for: Datacenter-scale NVIDIA deployments where your team has dedicated ML engineers, you're running a fixed set of models at maximum throughput, and the compilation overhead is a one-time acceptable cost.

The Decision Framework

By workload:

Are you a solo developer prototyping?
  → Ollama (fastest start, widest framework support)

Do you prefer a GUI over the terminal?
  → Jan (fully open source, Apache 2.0) 
  → or LM Studio (proprietary but polished)

Do you need CPU-only or no-GPU inference?
  → llama.cpp directly, or GPT4All

Are you on Apple Silicon and want maximum Mac performance?
  → mlx-lm (standalone) or Ollama 0.19+ (uses MLX automatically)

Are you serving 10+ concurrent users?
  → vLLM (baseline production choice)

Are you serving a RAG pipeline or agentic workflows?
  → SGLang (RadixAttention gives 20-30% cost reduction in practice)

Do you need to serve multiple models from one process?
  → SGLang or LMDeploy (vLLM can't do this)

Do you have diverse quantization formats (EXL2, ExLlamaV3, GGUF, AWQ)?
  → Aphrodite Engine

Are you on NVIDIA datacenter hardware at scale?
  → TensorRT-LLM + Triton

Do you need everything fully auditable and open source?
  → Jan (GUI) or llama.cpp (engine) — both MIT/Apache 2.0 with no proprietary components

Quantization Format Quick Reference

Understanding formats matters because they determine which tools can load which models:

Format	Who supports it	Notes
GGUF	llama.cpp, Ollama, Jan, LM Studio, GPT4All, Aphrodite	Open standard; ~70% of community releases
Safetensors (FP16/BF16)	vLLM, SGLang, LMDeploy, TensorRT-LLM	HuggingFace native; full precision
AWQ	vLLM, SGLang, Aphrodite	4-bit, fast on NVIDIA
GPTQ	vLLM, Aphrodite	4-bit, older standard
EXL2 / ExLlamaV3	Aphrodite (primary), ExLlamaV3 runtime	Popular for community chat-tuned models
FP8	vLLM, SGLang, TensorRT-LLM	Hopper+ hardware; best efficiency at H100/B200
MLX	mlx-lm, Ollama on Mac	Apple Silicon only

The practical rule: start with GGUF Q4_K_M. It covers 95–98% of full-precision quality on most benchmarks, works on any hardware, and loads in every major tool. Only move to other formats when you have a specific reason.

Full Comparison Table

Tool	License	Open Source	Layer	Backend	Hardware	Multi-User	Best For
llama.cpp	MIT	✅	Engine	C++ native	Any (CUDA, ROCm, Metal, CPU)	❌	Max speed, any hardware
Ollama	MIT	✅	Dev UX	llama.cpp / MLX	Any	⚠️ Limited	Solo dev, prototyping
Jan	Apache 2.0	✅	Dev UX	llama.cpp	Any	❌	Privacy-first, open GUI
GPT4All	MIT	✅	Dev UX	llama.cpp	Any (CPU-first)	❌	Non-technical users
mlx-lm	MIT	✅	Engine	Apple MLX	Apple Silicon only	❌	Mac fine-tuning + inference
vLLM	Apache 2.0	✅	Production	CUDA/ROCm	NVIDIA (AMD limited)	✅	Multi-user production APIs
SGLang	Apache 2.0	✅	Production	CUDA/ROCm/TPU	NVIDIA, AMD, TPU, Ascend	✅	RAG, agents, structured output
Aphrodite	AGPL-3.0	✅ (copyleft)	Production	CUDA	NVIDIA	✅	Wide quantization formats
LMDeploy	Apache 2.0	✅	Production	CUDA	NVIDIA	✅	VLMs, INT4, low TTFT
TensorRT-LLM	Apache 2.0	✅	Datacenter	TensorRT	NVIDIA only	✅	Max datacenter throughput
LM Studio	Proprietary	❌	Dev UX	llama.cpp / MLX	Any	❌	Polished GUI
~~TGI~~	Apache 2.0	⚠️ Retired	—	—	—	—	Migrate to vLLM/SGLang

The One-Line Summary Per Tool

Tool	The one line
llama.cpp	The engine under everything — use it when you need maximum speed or unusual hardware.
Ollama	Docker for local LLMs — the right first install for most developers.
Jan	Ollama's open-source GUI alternative — fully auditable, zero telemetry, Apache 2.0.
GPT4All	Local AI for non-technical users — works CPU-only, zero terminal required.
mlx-lm	The fastest Mac-native inference path — the only tool that also lets you fine-tune locally on Apple Silicon.
vLLM	The production standard — 16–20× Ollama's concurrent throughput via PagedAttention.
SGLang	vLLM's smarter sibling for agentic workloads — RadixAttention makes RAG pipelines 6× faster.
Aphrodite	vLLM fork with the widest quantization format support in any single engine.
LMDeploy	Best for vision-language models and INT4 on A100/A800 hardware.
TensorRT-LLM	Maximum NVIDIA throughput — accept the 28-minute compile step for the best raw numbers.

Conclusion: Open Source Has Won the Inference Layer

This is what makes the 2026 local inference ecosystem genuinely exciting: almost everything in it is fully open source, permissively licensed, and community-maintained. The MIT and Apache 2.0 licenses that cover llama.cpp, Ollama, Jan, vLLM, SGLang, and mlx-lm mean you can inspect every line, fork freely, deploy commercially, and contribute back without a legal department signing off.

The one meaningful proprietary holdout — LM Studio — has Jan as a mature Apache 2.0 alternative. And the one closed-source research team that used to control the serving layer, HuggingFace with TGI, has gracefully stepped back and pointed users toward the open community alternatives.

The right tool depends entirely on your workload. But the right answer is almost certainly open source.

Versions and benchmark data verified as of July 2026. Tool capabilities and licenses evolve rapidly — check each project's GitHub README before making infrastructure decisions.

What's your current local inference stack? Drop it in the comments.

Top comments (1)

Luis Cruz • Jul 19

I appreciate how the article highlights the importance of understanding the different layers of the local LLM inference stack, particularly the distinction between raw inference engines like llama.cpp and experience wrappers like Ollama. The fact that llama.cpp can run on a wide range of hardware, from NVIDIA CUDA to Raspberry Pi, is a testament to its versatility. I've had experience with deploying llama.cpp on both CPU and GPU setups, and I can attest to its efficiency and ease of use. One aspect that I think would be interesting to explore further is optimizing llama.cpp for specific use cases, such as real-time language processing or low-latency inference, by fine-tuning its configuration and leveraging its flexibility. How do you think the community can contribute to further optimizing and expanding the capabilities of llama.cpp?