If you want to run LLMs locally, I recommend using Ollama to manage model weights and serve a local API. While you will need substantial hardware—typically a minimum of 16GB of VRAM or an Apple Silicon Mac with unified memory—this setup lets you run private, free agentic workflows on your own machine.
Let's be honest: if you want to run an LLM locally, your first major hurdle isn't writing code—it's convincing your workstation not to melt. Our new AI overlords have an insatiable appetite, and they have basically claimed ownership of our system memory. But if you have the hardware to feed them, running models locally is the ultimate way to reclaim your privacy and stop paying subscription fees.
What is the easiest way to run an LLM locally?
If you want to get started without losing your mind, I highly recommend downloading Ollama. It is a lightweight desktop application that handles downloading model weights and running them, while exposing a clean API that you can hook into other developer tools.
What I like about Ollama is that it sits quietly on your desktop and gives you a simple, ChatGPT-style text box interface. Under the hood, a model is really just a giant file filled with numbers—specifically, the weights. Ollama manages these files for you and, crucially, exposes local APIs.
This API support means you can download an open-source version of Claude Code (like OpenHands or other terminal agents), point it at Ollama, and run a fully autonomous coding agent right on your machine. Here is how simple it is to fire up a model once you have Ollama installed:
ollama run llama3
What are the hardware requirements for local LLMs?
In my experience, you are going to need at least 16GB of VRAM to get anything appreciable done with a local setup. However, if you have a Mac, I find their unified memory architecture to be a massive advantage because it allows the system to use regular RAM as VRAM.
I’ll be blunt: running local models requires a lot of experimentation to find the sweet spot for your specific machine. If your GPU runs out of VRAM, your system will swap to regular system memory, and your generation speeds will slow down to a crawl.
If you have a Mac with Apple Silicon, you are in luck. Because of how Apple designs its unified memory, your GPU can access a huge chunk of your system RAM directly. This means you can run slightly larger, more complex models that would typically require an incredibly expensive, enterprise-grade graphics card on a PC.
| Model Size | Minimum VRAM (PC/Linux) | Recommended Unified Memory (Mac) | Performance Expectation |
|---|---|---|---|
| 8B Parameters | 12GB - 16GB | 16GB - 24GB | Fast, great for basic tasks and completions. |
| 14B Parameters | 16GB - 24GB | 32GB | Balanced, capable of moderate reasoning. |
| 70B Parameters | 48GB+ | 64GB+ | Slow on consumer gear, but highly accurate. |
How do you connect local LLMs to coding agents?
To connect a local LLM to a coding agent, I configure the agent's endpoint to point to Ollama's default local API address (http://localhost:11434). This routes all of the agent's prompts to your own hardware instead of hitting external cloud endpoints.
Imagine you are building an autonomous agent to refactor a legacy project. Instead of running up a massive bill on proprietary cloud APIs, you can point your local agent framework to Ollama.
You do have to be very careful with how you configure smaller models here. A smaller, 8B model can get confused by complex agentic loops and start hallucinating or looping. It takes some trial and error to get them behaving successfully. But once you dial it in? It’s completely free—apart from your power bill.
FAQ
Can I run local LLMs on a standard laptop without a dedicated GPU?
You can, but I wouldn't recommend it for daily use. Running LLMs on a standard CPU means token generation will be incredibly slow—often slower than you can read—making it practical only for basic testing rather than actual development workflows.
How do quantized models help with hardware limitations?
Quantization compresses model weights (for example, from 16-bit to 4-bit precision), allowing them to fit into much smaller VRAM allocations. I find that using quantized models is the only realistic way for most developers to run powerful models on standard consumer hardware.
Is my code and data private when using Ollama?
Yes, absolutely. Because the model weights are downloaded and executed locally on your own CPU and GPU, nothing is ever sent back to a cloud server. This is why I recommend local models for developers working on proprietary or highly sensitive codebases.
Top comments (0)