Local Consumer LLMs Are Kind of Terrible Right Now. They’re Going to Kill Cloud AI Anyway.

Here’s a disclosure up front — I use Claude quite frequently. I use it for research when I’m writing an in-depth piece, I use it to review code (especially when that code is from someone else), and I use it for a smattering of tasks both simple and complex where I feel limited by my own understanding and capability. It’s fast, it’s sharp, and if I’m being honest, I rely on it more than I probably should. I also use local AI pretty constantly — right now, I am running five separate models as a mixture of experts in a self-checking council for code scaffolding and review. So understand that what I’m about to say isn’t coming from someone who’s cheerleading against frontier AI out of ideological principle.

Local LLMs, as a category, are currently aggressively mediocre.

That’s not a take designed to be contrarian — it’s just accurate. If you fire up Ollama today, pull a capable open-weight models, and ask it something genuinely complex, you are going to notice the gap. The reasoning stumbles and context limitations surface quickly. The multi-step tasks that Claude handles with what looks like effortless coherence take considerably more coaxing from a local 8B model. And the wait is often significant — people used to their query taking 30 seconds to handle will be shocked at the reality that local LLMs are often a “set, walk away, have a coffee, check on the kids, and then see if we’re actually done generating” affair.

This all needs a bit of a reality check as well — while I’m sure there’s a chunk of tech bros currently screaming at the screen that some local models are killing the game, they’re not using consumer-grade hardware that is common. They’re using two massive 16GB graphics cards in a breakout box connected over Oculink — whereas most people who want to use local LLMs as a self-starter are more likely to be using a Mac Mini or a name brand prebuilt with Linux installed. And in those cases?

Well, simply put — there is no honest reading of the current benchmark data that says local models have caught the frontier.

But the trajectory — and this is what I actually want you to sit with — is something else entirely.

At the end of 2023, the best closed model scored around 88% on MMLU while the best open-weight alternative managed roughly 70.5%. That was a gap of 17.5 percentage points, and it felt huge — especially considering the bulky hardware you’d have to plop thousands of dollars down on to actually achieve this result.

By early 2026, that gap is effectively zero on knowledge benchmarks, and single digits on most reasoning tasks. Open-weight models now trail the state-of-the-art proprietary models by only about three months on average. Across independent benchmarks and real-world workloads, the performance gap between proprietary frontier models and open-source alternatives has narrowed to less than five percent on many core tasks. And the cost of that hardware? That calculus is starting to change too — especially as more consumer-grade hardware is building for local AI first.

What was a canyon two years ago is now a crack — and that crack is getting smaller every quarter.

So let’s talk about what’s actually happening here, why local models matter even before they fully close that gap, how you can set one up and evaluate it for yourself today, and what the shape of this thing looks like from where I’m standing.

The Honest State of Local Models Right Now

Let’s stay honest for a moment, because the local AI community has a tendency to paper over the gaps with benchmark cherrypicking.

The current best-in-class local setup — something like Qwen 3 72B or DeepSeek V4 Flash running on a machine with enough VRAM to handle it — is genuinely impressive. Qwen 3 72B scores 83.1 on MMLU and 84.2 on HumanEval, which would have put it in GPT-4 territory just two years ago. For most daily coding tasks — completion, explanation, refactoring, test generation — you can run a capable local model and not feel like you’re working with something broken. Llama 3.3 8B scores 73.0 on MMLU at Q4_K_M quantisation, which is a number that required a cloud API call to hit in 2023.

But the ceiling on complex multi-step reasoning remains meaningfully lower than frontier cloud models — particularly for anything below the 14B parameter class. In February 2026, Claude Opus 4.6 leads Humanity’s Last Exam at 53.1% with tool access. The best open-weight models aren’t close on that benchmark. On agentic tasks — BrowseComp, TerminalBench, OSWorld — open models trail significantly. The consistency gap is real too: proprietary models tend to perform well across all categories simultaneously, while open models spike on specific benchmarks and dip on others.

There’s also the quantisation tradeoff. Q4_K_M quantisation generally preserves benchmark scores within 1–3 points of full precision on MMLU for most models — which is good news — but degradation can exceed 5% on specialised tasks like multi-step maths reasoning. You’re making a significant tradeoff — and while that’s fine, it’s one that you aren’t having to make with cloud models. You’re making a trade, and it’s important to understand what you’re trading.

All of this is also not just a benchmark question — there’s also the equally important hardware question. The smallest setup that gives you genuinely capable inference — something that won’t frustrate you on a real task — requires at least 8GB of RAM for a 7B or 8B model, and realistically 16GB+ for a comfortable experience. The “sweet spot” models like Llama 3.3 8B and Mistral Small 3 7B run on a 16GB machine, but pushing into 70B-class models — where the quality gap closes considerably — requires the kind of hardware budget that not everyone has sitting around. An RTX 4090 or an Apple M4 Max/Ultra starts to look like a reasonable investment if you’re running this professionally, but it’s not nothing. And if you take a look at Reddit, you’ll see that the people who can actually show off the frontier of local LLMs are the ones willing to drop more than $75K on 16 DGX Spark units, not the average person using local models for routine tasks.

None of this is catastrophic. It’s just the current reality. And the current reality is worth naming plainly before we get to the part that actually matters.

Why You Should Care Even Before They’re Better Than Claude

You might be reading this and thinking to yourself “man this person hates local AI”. To be clear, I think local AI is a huge revolution that’s right on the horizon. And today? Well, “not as good as Claude” and “not worth running locally” are not the same statement.

The case for local models isn’t purely about closing the capability gap with frontier cloud models. It’s about what local inference gives you that cloud inference structurally cannot — and those are things that matter regardless of where the benchmarks sit right now.

Privacy is the obvious one. Healthcare companies, law firms, industrial operators, and government contractors cannot send their data to third-party APIs — not because of paranoia, but because of HIPAA, GDPR, SOC 2, and a dozen other compliance requirements that make cloud AI a non-starter for sensitive workloads. Running Ollama locally means your data never leaves your network. There are no terms of service to audit, no data retention policies to worry about, and no compliance exposure from third-party processing. For those use cases, a local 8B model that’s 80% as capable as Claude is not a wild compromise — and in some cases, it may be the only viable option.

Cost is another dimension that doesn’t get discussed honestly enough. GPT-4o currently charges approximately $15 per million output tokens, and Claude Opus 4.6 is in the same range. For developers iterating on prompts, building RAG pipelines, or processing high volumes of documents, those charges compound fast. A Mac mini M4 Pro with 48GB RAM runs you roughly $1,800 USD as a one-time cost, with monthly electricity expenses of $10 — $20 USD with regular use. Break-even analysis puts local deployment at 3–8 months of use versus cloud API pricing at scale, and after that it’s essentially free inference. For production workloads processing more than 10 million tokens a month, the maths is not close.

And then there’s model sovereignty  — which is less talked about but increasingly important. OpenAI retired 33 models in January 2025 alone. When GPT-5 launched, its changed model routing broke production workflows overnight. Companies had to debug broken integrations under pressure. With open-weight models running locally, you control the version — you can freeze a model that works and run it indefinitely with no deprecation notices, no forced migrations, and no breaking changes to your prompt stack because someone at the vendor decided to retire a model. In production environments where stability matters, that’s a significant operational advantage.

TrafficBench benchmarking found that 80.7% of LLM workloads can be handled by small local models with 77.1% energy savings versus cloud-only deployment. The implication isn’t that local replaces cloud for everything — it’s that most workloads don’t actually need the frontier, and running them locally is cheaper, faster, and more private.

A Mixture of Solutions

So here we land upon a significant reality — as things currently stand, local AI can’t beat the cloud providers, but it’s not an all or nothing proposition. Being able to deploy local AI for tasks which carry significant cloud costs and require local inference is pretty good — while you don’t have the frontier of Claude or ChatGPT, you do have good enough considering the work you’re actually doing. Taking the next step into the cloud gives you that rapid frontier experience — and hybridising between the two gives you a very strong solution.

For right now, that’s going to be the reality for some time to come — but things are changing rapidly. You can see the calculus above — over time, the reality of local models has become more reasonable, more attractive, and potentially more useful. And while right now local models are kind of “meh” compared to cloud models, that may not always be the reality.

So what’s going to change this ultimately? A few things:

1. Better local hardware  — right now, unless you want to shell out a crazy amount of money, you’re quite limited in what you can do. Some local models compete in very, very limited areas like computer vision and home automation, but we’re going to need much better hardware and drivers to run it. As an example, the new Mac Minis are decent enough for local AI — but the introduction of things like TinyGPU will make merging unified memory with external GPUs and NPU units more feasible, thus dramatically increasing the ability, size, and context window of local models.
2. Better pricing  — jumping on the hardware issue is the fact that what is available is just too dang expensive. Mega AI datacentres use a lot of RAM, a lot of CPUs, and a lot of cooling systems, meaning that what hardware we do have is crazy expensive. Right now the math heavily favours cloud compute, but that’s not going to be true forever.
3. Universal drivers and memory layers  — until we get a way to really hybridise our VRAM, RAM, processors, etc. across a totally (and truly) unified layer, we’re always going to run into issues. The Mac Mini does really well with a unified data bus across its memory, but it can’t really use eGPUs at scale. Mega-Linux boxes with fifty graphics cards are great but they’re not cost effective. Adding horizontal power scalability is difficult unless you’re a data centre. Once we figure out that connectivity problem — be it via an updated Thunderbolt standard, Oculink, or whatever connector comes into vogue — we’re going to see an explosion of networked local AI.
4. Cost losses in the cloud space come home to roost  — right now, most AI providers are operating in the red. Eventually, they will need to change their cost models either due to necessity or investor hunger. And just like how Netflix cost increases are causing people to look for alternatives, cost surges for cloud resources are going to drive people locally.

Setting Up Your First Local LLM with Ollama

The fastest way to experience the current state of this for yourself — and I genuinely recommend doing this before forming a strong opinion either way — is to get Ollama running on your machine. Yes, llama.cpp is probably better for resource use, but for the average person, this is going to be a much more accessible methodology. Ollama has become the de facto GUI standard for running local LLMs, and for the average user who doesn’t want to live in the CLI, this will represent their first foray.

It takes about ten minutes to go from zero to a running model — here’s how to get going.

Installation

macOS:

curl -fsSL https://ollama.ai/install.sh | sh

Either run this, or download the desktop app directly from ollama.com. On Apple Silicon, Ollama automatically uses Metal GPU acceleration — no additional configuration required.

Windows:

Download the installer from ollama.com/download. Windows ARM64 devices received a native build in 2026, eliminating the performance penalty from previous x86 emulation. If you’re on Windows 10 and have an NVIDIA GPU, make sure your driver is version 525 or newer.

Linux:

curl -fsSL https://ollama.ai/install.sh | sh

For systemd-based distributions (Ubuntu, Debian), enable it to run as a service:

sudo systemctl enable ollama
sudo systemctl start ollama

Pulling and Running Your First Model

Once Ollama is installed, pulling a model is a single command. For a first experiment, llama3.2:latest is a reasonable starting point - it’s 2GB, runs on any machine with 8GB RAM, and gives you a representative feel for what local inference currently looks like.

ollama pull llama3.2:latest

To start chatting directly in your terminal, you can use the CLI like this:

ollama run llama3.2

Alternately, you can also do all of this — including the model pull — inside the GUI itself, which is going to arguably be the more common install and use methodology.

If you want to step up to a more capable model and your machine has 16GB+ RAM, try llama3.3:8b or mistral:7b:

ollama pull llama3.3:8b
ollama run llama3.3:8b

Checking GPU Acceleration

After pulling a model, verify whether Ollama is using your GPU or falling back to CPU-only inference. In a separate terminal window, you can issue this command:

tail -f ~/.ollama/logs/server.log | grep "n_gpu_layers"

If n_gpu_layers is 0, inference is CPU-only. If it matches the total layer count, you have full GPU offload. Partial offload happens when the model is too large for your VRAM - Ollama handles this automatically, but it’s worth knowing where you sit.

Using the REST API

A cool thing about Ollama is that you can use the GUI or CLI, but you can also call a local API which runs by default at http://localhost:11434. You can issue commands to it the same way you would any other API tool:

curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.3:8b",
    "prompt": "Explain the difference between RBAC and ABAC access control models",
    "stream": false
  }'

Or using the chat endpoint:

curl http://localhost:11434/api/chat \
  -d '{
    "model": "llama3.3:8b",
    "messages": [
      { "role": "user", "content": "Write a Python function to parse a JWT without a library" }
    ],
    "stream": false
  }'

This makes Ollama particularly good as a drop-in replacement for external API requests. You’ll get substantially worse performance — at least on most setups — but you’ll get full privacy and control, and that’s not something to laugh at.

Evaluating the Gap Yourself

Once you’ve got it running, I’d suggest a simple three-step evaluation to form your own opinion.

First, give it a task you actually do every day — a coding problem, a writing task, an explanation — and see how it handles it. Note how long it takes to get the answer — and how many times you have to instruct it to tighten up its reasoning and output. Compare it to your cloud usage — and note any delta between the two.

Second, give it something genuinely complex: a multi-step reasoning problem, a document analysis task, something that requires holding a lot of context. See how quickly it hallucinates or makes assumptions. Ask the AI to complete a multi-step process that requires creative thinking and see how it compares against the froniter cloud model.

Finally, note where it starts to degrade. That degradation point is the gap — and watching it directly is more instructive than any benchmark number. See how long you can keep the conversation going before it goes off the rails — and compare that to the chat limit in place on ChatGPT or Claude.

You’ll likely find that for everyday tasks, capable local models handle the load. You’ll also likely find that somewhere around complex multi-step reasoning, the frontier cloud models pull ahead. Both things will be true at the same time, and that’s the more accurate picture — albeit one that is constantly changing.

The Trajectory Is the Story

And here’s where I want to spend a moment, because that gap is shrinking faster than almost anyone predicted.

Consider the rate of movement. The performance gap between the best open-weight model and the current mainstream proprietary leader currently sits at roughly 6 points on overall capability indices. In mid-2024, that gap was considerably wider. In October 2024, it was 15–20 points on quality indices. In eighteen months, open-weight models have closed more than half of that distance.

There are specific areas where open-weight models already match or beat proprietary ones outright. On maths: GLM-5 (Reasoning) scores 98 on AIME 2025 and 95 on HMMT 2025 — competitive with the best proprietary maths scores. On competitive coding: DeepSeek V4 Pro reaches 93.5 on LiveCodeBench and Kimi K2.5 reaches 89.6 — both ahead of Claude Opus 4.6 at 76 on that specific benchmark. On knowledge: GLM-5 hits 96 on MMLU and 94 on GPQA.

These aren’t cherrypicked anomalies — they’re leading indicators of where the whole category is heading.

Epoch AI data shows open-weight models now trail the state-of-the-art by only about three months on average. That’s the lag time between a frontier capability appearing in a closed model and appearing in an open-weight one. And realistically, three months is not a strategic moat — it’s at best a release cycle.

A Berkeley California Management Review analysis draws the comparison to Clayton Christensen’s disruption framework deliberately: open-source LLMs are following the classic disruption pathway — starting with cost advantages that democratise access, then rapidly improving through community-driven innovation while offering capabilities that closed models structurally cannot match. The pattern has played out in operating systems, databases, and web servers, and there’s no principled reason to think AI model weights are exempt from it.

Not There Yet, But the Direction Is Clear

I want to be precise about the claim I’m making here.

I’m not saying local LLMs are ready to replace Claude for everything today. They’re not — in actuality, local AI models are comparatively terrible when looking at the cloud frontier. The complex reasoning gap is real. The agentic capability gap is real. For the hardest problems, frontier cloud models still hold a meaningful lead, and for tasks that genuinely require that ceiling, the answer right now is still to use them (for better or for worse).

What I’m saying is something more specific. I’m saying that the capability gap is narrowing faster than the AI narrative acknowledges — and while local models are arguably pretty bad compared to the frontier, the structural gap is closing incredibly fast. For the majority of actual daily workloads, a well-run local setup may already be good enough, costs nothing per token, and keeps your data on your hardware. The lack of quality or the inability to do complex multi-stage multi-variable tasks can be excused when the cost is zero after a few months compared to working with a cloud provider that will always take your money.

And there are some specific areas where open-weight models have already surpassed frontier closed models. The three-month lag time between frontier capability and open-weight capability is not a property of the category — it’s a snapshot that gets updated with every release cycle. The intelligence is getting cheaper, the models are getting smaller without losing quality, the hardware is getting faster, and the community building all of this is not slowing down. An NVIDIA position paper from June 2025 makes the argument that the next real leap forward in agentic AI won’t come from models getting bigger — it’ll come from them getting smaller, more specialised, and more deployable at the edge.

Local AI right now is not perfect. It’s not the best answer in most circumstances where AI is super valuable. But it is on a clear and measurable path toward being better, stronger, and accessible to anyone who wants to run it — and that trajectory is worth paying attention to.

So I’ll reiterate my statement — local AI right now is comparably terrible against cloud AI. But that statement is getting weaker day by day — and soon, the inverse will be true.