This article was written by AgentOne Research.
Ollama is the easiest way to run AI models on your own computer. You install it, run one command, and you're chatting with a model that never touches the cloud.
The problem is that chatting is all it does. If you ask it to check a website or clean up your Downloads folder, it'll explain how you could do it yourself. It won't actually do it.
In this post, I'll show you how to turn Ollama into an AI agent that can search the web, read and write files, run commands, and connect to your apps. No code, no API keys, and it's completely free.
Why Can't Ollama Use Tools on Its Own?
A lot of open models support tool calling (sometimes called function calling). Instead of answering in plain text, the model can reply with something like "call web_search with the query 'ollama release notes'."
Ollama supports this, but it doesn't have any tools of its own. When the model asks for a tool, Ollama just hands that request back to the app that sent the message. Something else has to:
- Tell the model which tools it can use
- Run the tool when the model asks for it
- Send the result back to the model
- Keep going until the task is done
That loop is basically what an "agent" is. If you're a developer and want to build it yourself, the Ollama tool calling docs are a good place to start. If you just want an agent that works, you can let an app handle it. I'll be using AgentOne, since it supports Ollama out of the box and has the tools built in.
What You'll Need
- Ollama, from https://ollama.com/download (Windows, macOS, and Linux)
- A model that supports tools (more on that below)
- AgentOne, from https://www.agent-one.dev/download
- A decent GPU or a Mac with Apple Silicon. 8 GB of GPU memory is enough to get started, but more is better.
Step 1: Pull a Model That Supports Tools
Not every model can call tools. On ollama.com, you can filter the model library to only show the ones that can: https://ollama.com/search?c=tools
Once you've picked one, pull it from your terminal:
ollama pull bonsai-2:27b
Here's what I'd try based on how much GPU memory you have (download sizes are from ollama.com):
| GPU memory | Models to try | Good for |
|---|---|---|
| 8 GB |
bonsai-2:9b (3.2 GB), ornith-1.5:9b (6.6 GB) |
Simple tasks, like one search or one file |
| 12-16 GB |
bonsai-2:27b (8.3 GB) |
Tasks with a few steps |
| 24 GB+ |
qwen3.8:27b (18 GB), muse-glimmer:30b (18 GB), flint (23 GB) |
Research, coding, and longer tasks |
Bonsai 2 is worth calling out. It uses ternary weights, so the 27B version is only 8.3 GB, which means you can run a much bigger model than you'd normally fit. If you have a 24 GB card, Muse Glimmer is built specifically for local agents and tool use, and Flint is built for coding agents.
Smaller models are faster, but they're also a lot more likely to call the wrong tool or make up arguments. If your agent keeps getting confused, try a bigger model before anything else.
Step 2: Increase the Context Length
This one is easy to miss, and it's the reason a lot of local agents seem dumb.
Every time an agent sends a message, it also sends the model a description of every tool it can use, plus the results of every tool it has already run. That adds up quickly. If the context window is too small, the model loses track of the start of the conversation and forgets what it was doing.
Ollama picks a default context length based on how much GPU memory you have. If you have less than 24 GB, you only get 4K tokens, which isn't nearly enough for agent work. Ollama's own docs recommend at least 64K for agents.
The easiest way to change it is the context length slider in the Ollama app's settings. If you run Ollama from the terminal instead, set it when you start the server:
OLLAMA_CONTEXT_LENGTH=64000 ollama serve
On Windows, you can set it permanently from PowerShell, then quit Ollama and open it again:
[Environment]::SetEnvironmentVariable("OLLAMA_CONTEXT_LENGTH", "64000", "User")
To make sure it worked, run a model and check the CONTEXT column in ollama ps. A bigger context uses more memory, so if the PROCESSOR column shows part of the model running on your CPU, try 32K instead.
Step 3: Set Up AgentOne
Install AgentOne and open it. You don't need an account to use local models.
When AgentOne asks you to set up your account, click I want to use my own API keys instead. On the next screen, look at the Local section. Ollama is already turned on and set to its default address (http://127.0.0.1:11434), so there's nothing to fill in. Just click Continue.
AgentOne checks Ollama for your downloaded models every time it starts, so the model you pulled in Step 1 should already be there. If you pull another model later, you can find it under Settings > Providers > Ollama.
Step 4: Give It Something to Do
Click the model selector next to the message box and pick your Ollama model.
Now ask it to do something a normal chatbot can't. Here are a few ideas:
- "Search the web for the latest Ollama release and tell me what changed."
- "Look in my Downloads folder and tell me which files are bigger than 500 MB."
- "Make a folder called
invoices-2026on my desktop and move every PDF with 'invoice' in the name into it." - "Every weekday at 8am, check the weather and give me a one-line forecast."
You'll see each tool call as it happens. Before AgentOne does anything that changes your system, like editing files or running a command, it asks for your approval first.
Step 5: Add More Tools
Out of the box, AgentOne can search and browse the web, work with files, run terminal commands, remember things between chats, and run tasks on a schedule. If you want more than that:
- Extensions: connect Gmail, Google Calendar, Notion, GitHub, Slack, and thousands of other apps.
- MCP servers: AgentOne supports the Model Context Protocol, so you can add any MCP server you want, local or remote.
One tip for local models: only turn on the tools you actually need. Every tool you enable gets sent to the model with every message. A small model with 80 tools to choose from will struggle, but the same model with 10 tools usually does fine.
Troubleshooting
The model replies in text instead of using tools.
It probably doesn't support tool calling. Make sure it has the Tools tag on ollama.com.
It calls tools that don't exist, or passes broken arguments.
The model is probably too small for the task. Try a bigger one, or split the task into smaller steps.
It forgets what it was doing halfway through.
Your context length is too small - see Step 2. AgentOne also summarizes long chats when they get close to the limit, but that only helps if the limit is reasonable to begin with.
My Ollama models don't show up in AgentOne.
Make sure Ollama is running, then restart AgentOne. If you changed Ollama's port or you're running it on another computer, update the Base URL in Settings > Providers > Ollama.
It's really slow.
Your model probably doesn't fit in your GPU memory, so part of it is running on your CPU. You can check with ollama ps. Try a smaller model.
Local or Cloud?
Local models have gotten a lot better, but they're still not as good as GPT-6 or Claude, especially for long tasks. In my opinion:
- Use a local model for private files, simple automations, and anything you'd run over and over. It's free and your chats stay on your computer.
- Use a cloud model for long research, complex coding, or anything that takes a lot of steps. The best cloud models are still a lot better at staying on track.
You don't have to pick just one, though. AgentOne lets you switch models per chat, so you can use Ollama most of the time and switch to a cloud model with your own API key when you need it.
If you're looking for other options, check out my list of the 7 best free AI agents in 2026. And if you get stuck, come ask in the AgentOne Discord!
Happy building!
Top comments (0)