DEV Community: David Vellé Abel

The Local AI Developer (Part 2): A Reality Check After Real-World Testing

David Vellé Abel — Thu, 23 Jul 2026 10:01:26 +0000

This is Part 2 of my series on building a Local AI Developer Stack. If you missed the initial setup guide, you can read it here but I warn you that it might be already obsolete: The Local AI Developer (Part 1).

It has been two months since I published that initial guide. The setup in Part 1 looked great and promising on day one, so I decided to use it daily to give it a test in real-world conditions.

Here is a reality check of what works, what doesn't, and the optimizations required to make it truly usable.

Reality Check

I have tried my setup with different agents, LLMs, and combinations, but to be honest, it was hard to make it work as expected right out of the box. These were the main struggles:

Speed

This was an expected trade-off, but I was not expecting this level of slowness. I am talking about 4 to 5 tokens per second. To put that in context, this is equivalent to ~180 words per minute. An average human reader consumes between 200 and 300 words per minute. When the machine types slower than you can read, the developer experience suffers.

Out of Memory Errors

It was naive to only calculate the RAM needed for the LLM weights. The context window, the sessions, the system prompts—everything wants its tiny bit of RAM, and they all have to share the machine with my IDE, browser, and background processes.

It was common to see the LLM crash with an Out of Memory (OOM) error after 20 minutes of deep thinking.

Quality

It was usual to see the LLM running in circles but to be honest, that was a configuration problem on my side: using the wrong LLM for the task or setting the wrong context window. You end up with an agent that has no memory and starts overthinking the same ideas without actually executing them.

Can we improve it?

To fix these issues, I had to change my approach from "maximum power" to "maximum efficiency." There are a couple of parameters you can play with:

Select the Proper Model

I know we all want the biggest frontier model working for free on our machine, but we are running laptops, not mainframes, and we need to deal with their limits.

Smaller, more focused models exist. They have a significantly lower memory footprint, which leaves more space for context, sessions, and your actual daily apps. All models benefit from working with small chunks of code, and I have found smaller, specialized models to be "good enough" for small units of work.

Check Your Runtime

When I wrote Part 1, I started with Ollama. Since then, I found out there are a lot of runtimes and LLM servers, and they are all optimized in different ways.

Trying different runtimes to find the one that best fits your specific hardware can be a game changer. (I will delve into a highly optimized running setup specifically for my M4 Pro in a later post).

Context Management

I know this is a hot topic right now, but it really matters in local development. As I stated earlier, we cannot rely on humongous context windows like we do with cloud servers. We need to find the sweet spot between usability (so the agent remembers what it is doing) and memory usage.

Standard context optimization techniques are mandatory here: keep your sessions focused, compact your context, and don't let the agent read files it doesn't absolutely need.

Conclusion

The biggest lesson I learned over these last two months is that local AI development is not a drop-in replacement for the cloud; it is an entirely different discipline. You cannot throw infinite compute at a problem. Instead, you have to act like a systems engineer—balancing memory constraints, managing context hygiene, and matching the right model size to the right task. Once you embrace those constraints, building a completely private, autonomous local stack isn't just feasible; it is incredibly rewarding.

The Local AI Developer (Part 1): Setup with Ollama and OpenCode

David Vellé Abel — Mon, 11 May 2026 08:34:36 +0000

Why Go Local?

Every time I use a cloud-based AI coding assistant, I feel a little trapped in a vendor lock-in ecosystem, and it triggers alarms in my brain. Where exactly is my data going? What happens when they inevitably hike up the subscription price, just like every other service (looking at you, Netflix)?

It always brings me back to one question: Is it feasible to just build this locally?

Recently, I saw this post from Julien Chaumond, the CTO of Hugging Face, and it inspired me to finally try it.

The Benefits

Going local comes with some massive benefits:

The Budget: There are no API tokens to refill and no $20/month subscription fees.
Zero Dependencies: You are completely immune to API outages, rate limits, or slow internet connections.
Absolute Privacy & Security: This is the biggest draw. Zero code leaves your machine.

Setup: Ollama + OpenCode

I am building this setup using Ollama and OpenCode. They are both open-source, and they serve two distinct purposes:

Ollama serves the local LLM to our machine.
OpenCode acts as the agent, connecting to Ollama to execute our tasks.

Both tools have deep configuration options, but for this article, we will keep it strictly to the essentials.

Ollama

Ollama offers different scripts to ease the installation process
Follow the steps, and once you are done, install a local model.

You can check available models in Ollama's website. Which one to choose depends mainly on:

What are you trying to do? - In our case that's coding
How much RAM do you have available - Different models require different RAM capabilities based on their architecture and parameter count. As a simplified rule: the higher the parameter number, the "smarter" the model, but the higher the memory footprint.

I found that Qwen3.6 looks excellent for coding tasks and it fits my hardware (~22GB footprint), so let's install it:

First, start Ollama. This can be done as a background service or directly in the terminal:

ollama serve

Now let's download the model.

ollama pull qwen3.6

We can run it as a terminal chat-bot to verify everything is working:

ollama run qwen3.6

With the backend running, let's jump to our Agent.

OpenCode

OpenCode is an open-source Agent that can connect to any LLM model - even the paid ones like Claude - and it works really well with Ollama.

Installation is simple enough through brew, bun, npm, etc

Now we need to configure a model. OpenCode already offers integration with many providers, but for our local use case, we can use Ollama to start OpenCode pre-configured. Let's do that:

ollama launch opencode --model qwen3.6

This single command starts and configures everything necessary. Easy enough.

Use and experience

OpenCode works much like other terminal-based AI agents (such as the Claude CLI). With our local setup complete, we can jump straight in. Let's try a simple "Hello World" task:

The agent executes the task perfectly, and as promised, there is zero token consumption. However, a quick glance at my system monitor confirms the trade-off we discussed earlier—my machine is definitely feeling the heat:

Conclusion

So, is local AI development feasible? Absolutely.

While you are not going to get the same performance and model reasoning of a massive cloud-deployed model like Claude or GPT, local models are starting to close the gap and at least be "good enough" for daily tasks.

There's a shift from financial cost to hardware constraint. Your machine's RAM and GPU are now the bottlenecks.

Because of this, practicing good "context hygiene" and optimizing how you interact with the agent becomes critical. Agents.md, system prompts and boundaries will make you work significantly faster.

Even with its limitations, the trade for absolute privacy, zero latency and autonomy might be worth it.