I Replaced All My Cloud AI With Local Models — Here's What Actually Broke
I write a lot about local AI. How it saves money. How it's private. How a $150 GPU can replace $200/month in API subscriptions. And all of that is true.
But here's what I don't talk about enough: some things still don't work. Not "kind of work" — actually fail. Like, I-open-ChatGPT-and-do-it-manually fail.
After 14 months of running a local-first AI setup (Mac Mini M4 + Windows PC with RTX 3060 + Ubuntu fallback), I want to be honest about the dealbreakers. The tasks where local models still lose. Because nobody's building the "my local LLM produced garbage and I wasted 40 minutes" blog post, and someone should.
My Setup (Context for the Numbers)
Before I list failures, here's what I'm working with:
| Machine | Hardware | Models | Role |
|---|---|---|---|
| Mac Mini M4 | 10-core CPU, 16GB RAM | Qwen 3.5 9B, Granite 3.2 8B | Orchestration, routing, writing |
| Windows PC | RTX 3060 12GB | Qwen 3 Coder 30B, DeepSeek R1 8B | Code generation, reasoning |
| Ubuntu box | Older GPUs, 128GB RAM | DeepSeek R1 (larger), MinCPM-V | Heavy tasks, vision |
Electricity cost: ~$12/month. API costs: $0 for most tasks.
It's good. It handles maybe 85% of what I throw at it. That other 15%? That's where the pain lives.
Dealbreaker #1: Complex Multi-Step Reasoning
This is the big one. The one everyone talks about and nobody wants to admit.
I had a bug in a distributed task queue — the kind where Worker A processes a job, Worker B picks up the result, but under load, B sometimes gets a stale version. Classic distributed systems problem.
I fed the entire codebase (~2000 lines across 6 files) to DeepSeek R1 8B on the RTX 3060. It gave me a confident, detailed answer about adding a version field to the message format.
The problem? The version field was already there. The bug was in the retry logic — when Worker B got a 409 conflict, it was retrying with the OLD payload instead of re-fetching. The model didn't catch this because it couldn't actually trace the execution flow through the code. It pattern-matched "distributed systems bug" and gave me the textbook answer.
I pasted the same code into Claude Sonnet. It found the retry bug in 30 seconds.
Time wasted on local: 40 minutes (trying the "fix," testing, realizing it was wrong, trying again)
Time with cloud: 2 minutes
Cost difference: $0 vs ~$0.03
That $0.03 would have saved me 38 minutes. Local models are bad at multi-step reasoning where you need to trace causation through code. They pattern-match instead of actually following the logic.
Dealbreaker #2: Anything Requiring Current Information
My local models don't have internet access. I could hook them up to RAG pipelines and web search tools, and I have for specific projects. But for general "what's the latest X?" questions, local models are blind.
Examples where I had to switch to cloud:
- "What's the current version of Ollama?" — Local told me 0.5.7. It was 0.6.2.
- "Does Qwen 3.5 support tool calling?" — Local said no. It does (since training data cutoff was before that release).
- "What's the best GPU under $300 in 2026?" — Local recommended the RTX 3060 (which I already own). But the RX 7700 XT dropped to $280 and has 12GB VRAM too — local didn't know.
If your task depends on current information, local models are a non-starter unless you invest in a RAG pipeline that fetches real-time data. And at that point, you're building infrastructure, not just running a model.
Dealbreaker #3: Long-Context Understanding
Here's a scenario: I have a 15-page API contract from a client. I need to extract the authentication flow, the error handling requirements, and the rate limits, then generate a Python SDK that matches all of it.
My 30B model can ingest the text. It produces something that looks like a Python SDK. But it misses things:
- The contract specifies rate limits per endpoint (not global) — the model made them global
- Error codes are hierarchical (4xx for client, 5xx for server, but also custom codes like 4201, 4302) — the model treated them as flat
- Authentication uses a rotating token with a 5-minute expiry — the model implemented static tokens
Each of these is a subtle but critical failure. The model "understood" the document in the sense that it could summarize it. But it didn't actually follow the specifications precisely.
For documents under ~3 pages, local models are fine. Beyond that, especially with technical specifications, you start losing details that matter.
Dealbreaker #4: Vision Tasks (On Consumer Hardware)
I run Granite 3.2 Vision 2B and MinCPM-V for image understanding. They're okay. They can tell you what's in a photo. They cannot:
- Read a complex diagram and extract the architecture it describes
- Understand a UI mockup and generate matching CSS
- Accurately transcribe handwritten notes with mixed languages
- Distinguish between visually similar but functionally different components in a circuit diagram
The 2B vision model runs fast (4 seconds on the Windows GPU). But it's 2B parameters. It sees a diagram and says "this is a flowchart." Great. I can see that too.
The larger vision model (MinCPM-V on Ubuntu) does better but takes 3 minutes per image. For my automated pipeline that processes 20+ images per article, that's an hour of GPU time. Cloud vision APIs do the same in seconds.
Where local vision works: "What's in this photo?" "Is this a cat or a dog?" "Extract the text from this screenshot."
Where it fails: Anything requiring nuanced visual understanding.
Dealbreaker #5: Generating Consistent Long-Form Content
I know, I know — I wrote an article about how my AI content pipeline works. And it does. For 1000-1500 word articles with a clear structure.
But give a local model a 3000-word target with a specific voice, specific technical examples, and a specific argument structure, and it starts to drift. Paragraph 3 sounds like me. Paragraph 7 sounds like a Wikipedia article. Paragraph 12 is repeating Paragraph 4 in different words.
This is why my content pipeline uses short articles. Not because I don't have enough to say — because the model's coherence window is about 1500 words before it starts repeating itself or losing the thread.
For this article, I'm writing it myself. The local model can handle the structure and individual sections. But threading a consistent argument through 2500 words? That's still a human job.
What I Actually Do Now (The Hybrid Setup)
I've stopped pretending local can do everything. Here's my actual workflow:
| Task | Where It Goes | Why |
|---|---|---|
| Quick questions, summaries | Local (Qwen 3.5 9B) | Fast, free, good enough |
| Code generation (< 200 lines) | Local (Qwen 3 Coder 30B) | Solid for most tasks |
| Code generation (complex, multi-file) | Cloud (Claude Sonnet) | Better at tracing logic |
| Short-form writing | Local (Granite 3.2 8B) | Good structure, decent tone |
| Long-form writing (> 1500 words) | Myself, with local for structure | Coherence window is the limit |
| Vision (simple) | Local (Granite Vision 2B) | "What's in this photo" — fine |
| Vision (complex) | Cloud (GPT-4o Vision) | Architecture diagrams, UI mocks |
| Current information queries | Cloud (Perplexity/GPT-4o) | Local doesn't know today's news |
| Data extraction | Local | Surprisingly good, even at 7B |
Monthly cloud spend now: ~$25 (down from $187)
Monthly local spend: ~$12 (electricity)
Total: ~$37/month for 85% local, 15% cloud
The Real Takeaway
Local AI is incredible for what it does well. I saved over $2,000 in a year. I run my entire content pipeline, my code review, my daily summaries, and my monitoring — all on hardware I own.
But if you're starting out and someone tells you "you don't need cloud AI at all," they're either lying or they don't do the kind of work that requires deep reasoning, current data, or complex visual understanding.
The honest answer is hybrid. Local for the 80-85% of tasks where it's good enough (and cheaper and faster and more private). Cloud for the rest. And being smart about which is which.
The biggest trap isn't the technology. It's the denial. When you've invested time and money into a local setup, you want it to work for everything. You start accepting lower quality outputs because "it's free." But free output that's wrong isn't free — it costs your time to fix it.
I still use local for the vast majority of my work. I just know when to stop.
Sam Hartley runs a multi-machine AI home lab and writes about the infrastructure that makes local AI actually usable. Yes, he still has a ChatGPT subscription. No, he's not proud of it.
→ Custom automation setups on Fiverr
→ Follow CelebiBots on Telegram
Top comments (0)