Building a Local AI Agent (Part 1): Six Technical Challenges

Florian Zielasko — Wed, 29 Apr 2026 18:57:14 +0000

I've been building Reiseki (霊石) — a fully local AI agent that runs on your machine via Ollama, even if you do not have more than 8-10 GB of RAM. The agent uses a ReAct loop (Reason → Act → Observe) to handle file operations, document generation, reminders, and more.

Reiseki is open source: github.com/Flo1632/reiseki

Along the way I ran into six technical problems that aren't obvious. Here's what I've encountered and learned.

In Part 2 I'll cover the UX and design challenges — my goal was to make a local AI agent feel understandable and usable for someone who has never touched a terminal.

The Six Problems and How I Solved Them

(Other Suggestions Welcome)

1. The agent forgot everything on restart

The ReAct loop uses a Python list for session history — it lives in memory and disappears when the server restarts. If you are used to ChatGPT or Claude, this is very confusing. You actually want it to remember what you discussed with it.

The fix was a chat_log table in SQLite. Every user and assistant message is written there as it happens. At the start of each new request, the last 10 turns are fetched from the database and prepended to the message history so the model has continuity across sessions. It does not remember everything, but at least the last conversation.

2. The agent loop needs a hard iteration cap - especially for small models like Qwen 2.5-coder 7b

Without a limit, a confused model or a buggy tool result can cause the agent to loop forever — calling the same tool repeatedly, getting the same error, never stopping. On a local device with limited RAM, that's a serious problem.

The fix is a hard cap of 10 iterations. In practice, most tasks finish in 1-3 iterations. The cap is a safety net.

3. Sending all tools on every request wastes context

With 15+ tools defined, sending the full list on every request fills a meaningful chunk of the context window — and confuses the model. When it sees create_chart and analyse_data alongside a simple question like "what time is it?", it sometimes reaches for tools it doesn't need.

The fix was dynamic tool selection based on relevance scoring. Core tools (read file, write file, list directory, document generators) are always included. Specialized tools (e.g. charts, data analysis) are only added when the query text is relevant to them — scored via string similarity between the query and each tool's description and keywords.

In practice, this makes the model more focused and reduces unnecessary tool calls.

After testing I found that if you have a model with a larger context window, it might be better to enable at least the tool calls you would use regularly.

4. The context window grows with every tool call

In a ReAct loop, the message history grows by at least two entries per tool call — one for the assistant's decision, one for the tool result. A task like "read these five files and summarize them" can easily hit the model's context limit before it finishes.

This was handled with three layers:

Context compression — after every four tool calls in a single turn, the middle portion of the message history is summarized by the model itself, then replaced with that summary
Cross-turn cap — the in-memory session history is capped at 20 messages
Persistent log cap — the SQLite chat log is capped at 2000 rows with a rolling delete

The compression approach: the model summarizes its own previous steps in 2-3 sentences, which gets injected back as a single message. It loses detail but keeps the agent on track without blowing the context limit. If the summarization call itself fails, it falls back to a hard truncation of the last few messages.

5. Local models don't always return structured tool calls

The Ollama SDK has a proper structured field for tool calls — but not every model actually uses it. Gemma and Qwen sometimes serialize tool calls as plain JSON text in the response content instead. If you only handle the structured case, the agent silently ignores half its tool calls and you just receive a message in JSON format claiming it called a tool.

The fix was a layered fallback parser: try structured first, then parse the content as JSON, then scan for embedded JSON objects anywhere in the text, then try newline-by-newline. It's more code than it should be, but it makes the agent reliable across different models.

6. Injecting past turns into the system prompt is a security risk

My first approach was to paste previous messages directly into the system prompt as a block of text. But a security audit flagged this.

The system prompt has operator-level trust — the model treats it like instructions from a developer. Injecting user messages there effectively promotes them to the same level. A past message like "ignore previous instructions" would now carry the same authority as your actual configuration. And because the history is baked into the prompt text, clearing the session doesn't actually reset it.

The fix is to inject past turns as regular user/assistant entries in the message array, not as text in the system prompt. The model treats them with user-level trust, they stay isolated from the system context, and clearing the log actually resets them. It's a small structural change but an important one.

This project was built entirely with Claude Code. The technical decisions and design goals are mine; Claude handled the implementation.

What technical problems have you run into building local AI agents? Curious whether others have found better approaches.

Part 2 — UX and Design Challenges: coming soon.

DEV Community: Florian Zielasko