Comparing LLM Runtimes: 12 Self-Hosted Tools, When to Pick Each One

You have 12 viable options to run LLMs locally. The difference between them is not whether they work—all 12 are mature and offline-capable. The difference is hardware cost, ease of setup, speed, and what you're trying to build. This guide cuts through the noise and tells you which one fits your actual situation.

Start Here: Three Questions That Narrow It Down

Before you read specs, answer these:

1. Do you have a GPU? This split is the biggest. CPU-only tools work but are slower. GPU tools demand 8–16GB VRAM and exact driver setup.
2. Do you want a UI or an API? Desktop app users need GPT4All or text-generation-webui. API integrations need Ollama, LocalAI, or vLLM.
3. Are you serving many requests or just yourself? Batch/throughput matters if you're building a service. Solo use doesn't need it.

These three answers eliminate most of your options. The rest is picking between similar tools.

CPU-Only Path: 4GB–8GB RAM, No GPU

llama.cpp is the foundation. It's a single C++ inference engine for GGUF-format models. 4GB RAM minimum, runs on CPU, MIT licensed, very active development. Most other tools wrap or build on top of it. If you want the lightest possible setup, llama.cpp is fastest—but requires command-line comfort.

Ollama wraps llama.cpp in a simple CLI and REST API. 8GB RAM for 7B-parameter models. Same speed as llama.cpp, much easier to use: ollama run mistral and you're talking to a model. MIT license, mature. Pick this if you want CPU-only simplicity and don't mind a wrapper layer.

koboldcpp is llama.cpp plus a built-in web UI in a single executable. 4GB RAM, no install needed, AGPL-3.0 licensed. If you want a browser interface without fumbling with separate tools, this bundles it. Slower iteration than raw llama.cpp but faster setup than text-generation-webui.

llamafile is an Apache-2.0/MIT licensed single executable with no install needed. 4GB RAM, CPU-only. Run one file and it starts a server. Newer than the others, less battle-tested, but genuinely frictionless. Use if you hate setup work and want to try local LLMs in 30 seconds.

GPT4All is a desktop application. 8GB RAM, no GPU, MIT licensed, fully local chat. It bundles model downloads and fine-tunes everything for ease. Slower inference than raw llama.cpp but the smoothest non-technical experience. Pick if you're not a coder and want a clickable app.

LocalAI is an OpenAI-API-compatible local server. 8GB RAM, CPU-only OK, MIT licensed. Build it if your code expects OpenAI's API shape but you need it to run offline. Setup is slightly more complex than Ollama, but identical if you're already using the OpenAI SDK.

GPU Path: 8GB–16GB VRAM

ExLlamaV2 is built for quantized models (GPTQ, EXL2 formats). 8GB+ VRAM required, MIT licensed, very active. Single fastest option for GPU inference on quantized weights. Use if you want raw speed and your model is already quantized. Setup is less beginner-friendly than Ollama.

text-generation-webui (oobabooga) is a Gradio UI that supports many backends. 8GB RAM, GPU optional, AGPL-3.0. Works with llama.cpp, ExLlamaV2, AutoGPTQ, and others—you can swap backends without changing models. Best for people who like experimenting with multiple inference engines. UI is polished but overhead is real.

vLLM is a production-grade high-throughput serving engine. 16GB+ GPU VRAM recommended, Apache-2.0 licensed, very active. Designed for batched requests and multi-user setups. If you're building an API that serves 10+ concurrent requests, this is faster than the others. Overkill for solo use.

text-generation-inference (Hugging Face) is also production-grade. 16GB+ GPU VRAM required, Apache-2.0 licensed. Similar niche to vLLM—throughput-focused, Hugging Face integration tight. Pick vLLM or this, not both; vLLM is more active.

FastChat is a training-plus-serving framework. GPU recommended (16GB VRAM), Apache-2.0 licensed. Originated the Vicuna model family. Use if you need fine-tuning or training as well as serving. Otherwise, it's slower than dedicated runtimes.

Mobile/Edge Path: 4GB RAM, Any Hardware

MLC-LLM compiles LLMs for edge devices, mobile, and GPU. 4GB+ RAM, Apache-2.0 licensed, active development. It generates optimized code for your target (CPU, mobile GPU, desktop GPU). Setup is more involved, but inference is efficient. Use if you're deploying to phones, Raspberry Pi, or unfamiliar hardware.

Decision Checklist

• GPU available? Yes: ExLlamaV2 (speed) or text-generation-webui (flexibility). No: Ollama or llama.cpp.
• Need an API, not UI? Yes: Ollama, LocalAI, or vLLM (high traffic). No: GPT4All or text-generation-webui.
• Want zero setup friction? llamafile or GPT4All.
• Building a production service? vLLM or text-generation-inference.
• Deploying to mobile/edge? MLC-LLM.
• Want to experiment with backends? text-generation-webui.

The Real Constraint: RAM and VRAM Matter More Than the Tool

All 12 of these tools are mature, offline-capable, open-source, and maintained. The actual bottleneck is your hardware. A 7B-parameter model needs 8GB RAM to run smoothly. A 13B model needs 16GB. A 70B model needs a GPU with 40GB+ VRAM or quantization tricks. Pick your model size first, then pick the runtime that fits your hardware budget. Switching tools is easy; upgrading RAM is not.

Originally published at Forged Goods. The ready-made version: Local-AI Stack Directory: 40 Self-Hosted LLM & Vector-DB Tools, Verified Specs.