We build an AI app builder, and one feature we wanted was "bring your own Ollama": point the agent at your own models and pay nothing for tokens. Getting a chat answer out of a local model is easy. Getting a multi-step coding agent to work reliably is not. These are the things that broke for us, and what fixed them.
1. "Coder" models that can't call tools
The first surprise: qwen2.5-coder has no native tool calling in Ollama. It writes nice code in chat, but an agent needs it to call read_file, write_file and run_command, and it simply doesn't.
Of the models we tried, the Qwen3 family handled tool calls best. llama3-groq-tool-use:8b worked for one call and then fell back to plain text on the next turn, which kills a multi-step loop.
Lesson: pick a model for tool calling first and coding second.
2. Thinking mode fights tool calls
Qwen3's thinking mode would reason at length and then fail to produce the tool call. Turning thinking off fixed it. For Qwen3 that meant appending /nothink; other models take a think: false API parameter.
3. Tool calls that arrive as text
Small models sometimes write the tool call as JSON inside a normal text reply instead of a structured tool call. We added a parser fallback for JSON blocks, <tool_call> tags and a few other formats.
Ollama also doesn't support tool_choice: "required", only "auto". So when the model replied with text only, we retried with a short nudge: "You must use a tool now."
4. A 4GB GPU is mostly a prefill problem
On a GTX 1050 with 4GB of VRAM, the model spills into system RAM and generates at roughly 4–10 tokens a second. That's slow but usable.
The real killer was prefill. A 3,000-token prompt took 30–60 seconds before the first token appeared. An agent sends a big prompt on every step, so the timeouts piled up.
What helped:
- Compacting history without another model call. Older steps get replaced with a short structured summary: files read, files written, commands run, errors seen. Roughly 3,000 tokens became about 100.
- A "tiny model" mode. For very small models we drop retrieval context, the file tree and long instructions from the system prompt, and cap tool output at a few hundred characters.
- Fewer turns in memory. A tiny model gets the last 6 messages, not the whole conversation.
5. Cloud agent, local model
If the agent runs in the cloud, it can't see localhost:11434 on your laptop. You need a tunnel or a public address, and if it's public, a key in front of it. Ollama Cloud avoids that, since it's already reachable.
What we'd tell anyone trying this
- Test multi-turn tool calling, not single answers. Some models pass the first call and fail on the next.
- Budget for prefill, not just tokens per second.
- Expect the smallest models to chat fine and still fail full-stack builds. Use the biggest coding model your hardware or plan allows.
Disclosure: I work on NoCoder, where you can connect your own Ollama server or Ollama Cloud key and build full-stack apps with it, including on the free plan. The lessons above apply to any agent you run on Ollama.
Top comments (0)