I spent last month moving as much of my AI work as possible off hosted APIs and onto a machine under my desk. Not out of ideology. I wanted to know where the line currently sits between "this runs fine on my own hardware" and "stop kidding yourself, call the API."
To get an answer I went through six long teardowns from people who benchmark this for a living: Tech With Tim's local AI walkthrough, the Syntax hardware session, IBM Technology's Ollama explainer, Alex Ziskind on llama.cpp throughput, Gary Explains testing Qwen 3.8 27B, and Zen van Riel's category-by-category tier list. Here is where they land on the same page.
Memory is the spec sheet that matters
Your RAM or VRAM ceiling decides which models you can run at all.
Tech With Tim and Syntax both give roughly the same ladder:
RAM / VRAM Model size you can run
8 GB 3B to 4B
16 GB 7B to 8B
32 GB 14B to 30B
Syntax adds a useful piece of context: the 14B to 35B range is the sweet spot, big enough to be genuinely useful and small enough that you are not shopping for server hardware.
The reason a 30B model fits in 32 GB at all is quantization. Weights get compressed from 16 or 32 bit down to 4 or 8 bit with surprisingly little quality loss. Zen van Riel and Syntax both cite the same example: a 70B model that normally wants 140 GB comes down to somewhere between 30 and 70 GB. Tech With Tim puts it bluntly: without quantization, ordinary computers could not run these models at all.
What actually runs well
I want to split this into two buckets, because the gap between them is still wide in 2026.
Genuinely good locally:
Code autocomplete. Zen van Riel rates this S tier, and the numbers back it up. Qwen 2.5 Coder 7B returns completions in under 100 ms even on a GPU with only a couple gigabytes of VRAM. That is faster than most network round trips to a hosted model.
Chat and drafting. A 7B or 8B Llama or Mistral model on 16 GB handles everyday questions fine. Syntax points out it gets better when you bolt on tool access, like web or docs search.
Image generation. Flux and SDXL turn out images in seconds on a decent GPU. Zen van Riel cites a blind test where Flux beat older Midjourney versions 71% of the time. Training a custom LoRA takes 15 to 20 images on consumer hardware.
Speech to text. Zen van Riel treats Faster Whisper with Large V3 Turbo as close to solved for English.
Text to speech. He calls this the most improved category of the past 18 months. Chatterbox from Resemble AI reportedly beat ElevenLabs in blind listening tests with over 60% preference, across 23+ languages.
Still not there:
Video generation. Even an RTX 5090 cannot run the full 14B Wan model usefully. You drop to the 5B version and lose quality. Zen van Riel's summary is blunt: disappointing and slow.
Agentic coding. This is the one that surprised me. Local models handle single-file edits well, but reading an entire codebase, writing code, running tests and iterating is a different game. Both Syntax and Zen van Riel say local models do not match frontier models like Claude Opus 4.6 here. Coding needs the context window to stay full, and you cannot clear it the way you would in a chat session.
Voice agents. Latency is fine. Pipecat hits sub-800 ms voice to voice on ordinary Mac hardware by chaining STT, an LLM and TTS. The responses themselves are noticeably weaker than what you get from a good hosted model.
Picking your runner
Almost everything in this space is a wrapper around llama.cpp. Tech With Tim makes that point twice in his video, which is why the tooling feels similar across products.
The choice comes down to who you are:
Non-technical, want a chat window. LM Studio. No terminal required.
Developer writing scripts. Ollama. One command to pull and run, and it exposes an OpenAI-compatible REST API so existing code mostly just works.
Containerized production. Docker Model Runner, per Tech With Tim's recommendation.
There is a real tradeoff on Ollama. Alex Ziskind measured it at 100 tokens/sec through Ollama versus 124 tokens/sec calling llama.cpp directly. You are paying roughly 20% for the convenience. For most people that is the right trade. If you are squeezing throughput out of a Mac Studio, it is not: Alex Ziskind got 1,226 tokens/sec by running 16 llama server instances with parallelism set to 64 and 1,024 concurrency, then putting Nginx in front as a round-robin balancer.
A note on hardware if you are buying: Syntax and Tech With Tim disagree here in a way that is worth understanding. Syntax favors unified memory (AMD Strix Halo, Apple M-series) because you get far more capacity per dollar, and a Strix Halo box can hand 108 GB to the GPU while leaving 20 GB for the OS. Tech With Tim counters that a dedicated GPU like an RTX 4090 is 2 to 3 times faster at inference, even with less total memory. So the choice is what you run more often: a bigger model at moderate speed, or a smaller one at maximum speed.
Where I landed
For my setup (32 GB, one GPU) the pattern that works is a 7B to 14B Qwen or Llama model through Ollama for chat and coding, Flux for images, and one hosted API subscription I reach for only when a task needs serious reasoning.
Six people who benchmark this independently reached nearly the same shape of answer. I put together the full hardware-to-model breakdown if you want the per-category tier ranking too.
If you are deciding right now: check your RAM first, pick the largest model in the 7B to 14B range that fits, and stop worrying about benchmarks after that point. New models ship every month or two and get better on the same hardware, so the model you pick today is not the model you will be running in six months.
Top comments (0)