Running a language model locally used to be a hobbyist experiment. In 2026, it's a viable engineering decision for a growing slice of real workloads — and the tooling has finally caught up. StorageReview just published a comprehensive roundup of what actually works, with one useful caveat upfront:
"Roughly a third of the tools you will find recommended in a search result today are dead, and most of the pages recommending them have not noticed."
That makes this kind of maintained, hardware-grounded list genuinely valuable. Here's what you need to know.
The hardware reality first
Before picking tools, pick your memory tier. This is the decision that determines what model class you can run:
- 16–32GB unified / 16GB VRAM → 7–8B models. Chat, summarization, single-file code. Most of what most people do.
- 24–32GB VRAM / 48GB unified → 30B class. Where agentic coding goes from demo to functional.
- 96–128GB unified → 100B+ models. Approaches frontier quality. This is M3 Ultra / high-end workstation territory.
Everything else is downstream of this. The right tool installed on the wrong tier is still the wrong tool.
The runtimes (what actually runs the model)
Ollama (~179K GitHub stars, MIT) is the default answer, and it's earned that position. One command pulls models, exposes an OpenAI-compatible endpoint on localhost, and since January 2026 it also speaks the Anthropic Messages API — which is how people are now routing Claude Code at local models. Everything else in the ecosystem talks to Ollama.
llama.cpp (~124K stars, MIT) is what Ollama and LM Studio are built on. It matters because its backend list is enormous — CUDA, ROCm, Metal, Vulkan, SYCL, CANN, OpenCL — and vendor engineers contribute optimisations directly. An Intel Arc improvement in one build delivered ~5x prefill speedup to every Ollama and LM Studio user automatically.
LM Studio (closed source, free for commercial use since July 2025) is the hardware benchmarking tool. It exposes GPU offload layers, quantization choice, context length, and multi-GPU controls — the levers you need when you're measuring a machine rather than just using one.
The apps that sit on top
Jan (Apache 2.0, ~44K stars) is the pick for offline-first desktop use. Bundles llama.cpp, works with the network cable pulled out, and is the one to hand to a colleague who will never open a terminal.
Open WebUI (~149K stars) is the self-hosted multi-user option. Docker-based, deeply configurable. Worth knowing: the license changed in April 2025 from BSD-3 to a custom non-OSI-approved license with a branding clause. Fine at small scale unmodified; check the terms before building a product on top.
AnythingLLM (MIT, ~64K stars) handles document RAG most turnkeyly — bundles vector DB, handles chunking without config. Important flag: a March 2026 critical vulnerability (CVSS 9.6, RCE triggered by streamed model output) was fixed in 1.11.2. Make sure you're on a patched version, and turn telemetry off in Settings.
Coding agents
Cline (Apache 2.0, ~63K stars) has done the most real engineering for local model compatibility — compact system prompts built for Ollama and LM Studio, native tool calling per model family. Needs 24GB VRAM or 36GB unified, 32K+ context.
Aider (Apache 2.0, ~48K stars) sidesteps the main failure mode of local agents by not using JSON tool calling at all. It parses diff and whole-file edit formats from plain text — much more robust on local models. Caveat: one author wrote 96% of commits and release cadence has slowed significantly in 2026.
OpenHands (MIT, ~84K stars) is the self-hosted agent platform. Its own docs are refreshingly honest: if the agent acts like a chatbot or fails tools constantly, the model is the limitation. Wants 32K context minimum.
The surprise: GitHub Copilot CLI goes local
Since April 2026, the Copilot CLI works against Ollama, vLLM, and Foundry Local. GitHub auth is optional, no subscription required. Set COPILOT_OFFLINE and all telemetry stops. Most comparison articles still list it as cloud-only — it isn't anymore. (Note: the IDE extension still routes inline completions to the cloud even under bring-your-own-key.)
What to do
- Starting out: Ollama + Jan is the no-friction local stack. Get a model running, see if it fits your workload.
- For coding agents: Cline or Aider on top of Ollama, but check your VRAM first. Under 24GB, you're working against yourself.
- For document RAG: AnythingLLM, post-1.11.2 update, telemetry off.
- For teams/multi-user: Open WebUI on Docker. Read the license change if you're building a product around it.
- For benchmarking new hardware: LM Studio. It has the controls that matter.
The local LLM stack is real infrastructure now — not just a curiosity. The gap to frontier models is real for agentic and hard reasoning work, but for bounded tasks (which is most of what most people ship), local is a legitimate choice in 2026.
Source: Best Local LLM Tools in 2026 — StorageReview
✏️ Drafted with KewBot (AI), edited and approved by Drew.
Top comments (0)