Build a Free AI Agent From Scratch, Then Harden It

The first agent most people build works beautifully for ten minutes and then loops forever, burns tokens, or quietly does the wrong thing at 3 a.m. I have shipped enough unattended agent systems to know the demo is only a small part of the real work. This guide builds a tool-using agent from first principles with a local model and zero paid services, then covers what changes when nobody is watching it.

Step 1: Pick a local model that can actually call tools

A local model needs native tool-calling support for a hand-written agent loop to be reliable, because parsing tool requests out of free text is where most home-built agents die. Ollama's docs state that it supports tool calling, so a local model can request tool calls and use the results in its replies (Ollama tool calling).

Two practical details helped me early:

• A model run locally through Ollama needs no account or API key from a hosted provider, because it runs on your own machine (IBM tutorial). That is what makes this genuinely free.
• Ollama's OpenAI-compatible endpoint supports tools too, so code written against the OpenAI client can point at a local model (Ollama blog). I write my loop against that interface so I can swap in a hosted model later without rewriting anything.

For a model, OpenAI's gpt-oss in the 20B size is listed in the Ollama library with native function calling and structured outputs (gpt-oss). Memory needs vary by source and machine, and a 16 GB machine can be tight, so check the model page for your hardware rather than trusting my numbers. Smaller tool-capable models are fine for learning the loop.

ollama pull gpt-oss:20b

Step 2: Write the agent loop yourself, before any framework

An agent is a loop: send messages and tool definitions to the model, execute any tool calls it requests, append the results, and repeat until the model answers or a stopping condition hits. Anthropic's "Building effective agents" says the same thing, that an agent ends when the task is done or a stopping condition triggers, such as a maximum number of iterations (summary). That iteration cap is not optional. I treat it as the first line of the loop, not an afterthought.

The Ollama Python library lets you pass plain Python functions as tools and builds the schema from type annotations and a Google-style docstring (Ollama blog). That keeps the whole agent under 60 lines:

import ollama

def read_file(path: str) -> str:
    """Read a text file.

    Args:
        path: Absolute path to the file.
    """
    with open(path) as f:
        return f.read()[:8000]  # cap output size

TOOLS = {"read_file": read_file}
MAX_STEPS = 8

def run(task: str) -> str:
    messages = [{"role": "user", "content": task}]
    for step in range(MAX_STEPS):
        resp = ollama.chat(model="gpt-oss:20b",
                           messages=messages,
                           tools=list(TOOLS.values()))
        msg = resp.message
        messages.append(msg)
        if not msg.tool_calls:
            return msg.content
        for call in msg.tool_calls:
            fn = TOOLS.get(call.function.name)
            try:
                result = fn(**call.function.arguments) if fn else "unknown tool"
            except Exception as e:
                result = f"tool error: {e}"
            messages.append({"role": "tool",
                             "tool_name": call.function.name,
                             "content": str(result)})
    return "stopped: step limit reached"

Notice what the loop does with failure: tool errors go back to the model as text instead of crashing the run. In production, that single decision is the difference between an agent that self-corrects and one that pages you.

If you stream responses, there is a trap. Ollama's docs say you must accumulate the partial fields (thinking, tool call, tool result) to keep conversation history correct (Streaming). I build and debug the loop non-streaming first, then add streaming once the logic is solid.

Step 3: Spend your effort on tools, not the prompt

Tool design moves reliability more than prompt wording does. Anthropic reported that while building their SWE-bench agent they spent more time optimizing tools than the overall prompt, and that requiring absolute file paths removed the model's path mistakes (Anthropic).

I apply that principle constantly. My rules for tools in unattended systems:

• Make wrong usage impossible. Absolute paths, enums instead of free strings, required fields instead of optional ones.
• Return small, structured output. Truncate (note the 8000-char cap above) and say when you truncated.
• Return errors the model can act on. "File not found: /data/x.txt, did you mean /data/y.txt?" beats a stack trace.
• One job per tool. Two narrow tools beat one flexible one, because the model picks between them more accurately.

When I see an agent misbehave, my first move is to read the tool call logs, not to rewrite the system prompt. In my experience, the tool interface most often turns out to be what invited the mistake.

Step 4: Add memory in two layers

Agent memory needs two separate layers: the message history for the current run, and durable state outside the model for anything that must survive a restart. Conflating them is the most common design mistake I see.

Layer	Lives in	Purpose	Failure mode if missing
Working memory	The messages list	Reasoning within one run	Agent forgets what it just did
Durable state	SQLite or Postgres	Task status, results, what was already processed	Re-does work after a crash or restart

For free and local, SQLite is enough. In my scheduled workers, the database is the source of truth and the model is stateless: each run reads the task row, does the work, and writes the outcome. If the process dies mid-run, the next run picks up from the row, not from a half-remembered context. Keep a cap on history length too, because long tool outputs quietly eat the context window and degrade tool-call accuracy before you ever hit an error.

For semantic recall (retrieving past notes or documents), the pattern I use in local-LLM RAG pipelines is pgvector with hybrid search, but do not add that until the agent actually needs it. A plain table with status columns handles most real automation.

Step 5: Guardrails are an architecture, not a prompt

Guardrails that live only in the system prompt are suggestions. Real guardrails are enforced in code, outside the model. Anthropic's guidance recommends the simplest solution that works, adding complexity only when needed, and warns that agents cost more and can compound errors, which is why it recommends sandboxed testing and guardrails (Simon Willison's summary).

OWASP lists Excessive Agency as LLM06:2025 in its Top 10 for LLM Applications, with stated root causes of excessive functionality, excessive permissions and excessive autonomy. Its damage-limiting measures include logging and monitoring extension activity and rate-limiting (OWASP). Those three root causes map directly onto controls you can write today:

1. Least functionality: register only the tools this specific task needs. A summarizing agent does not get a delete tool.
2. Least permission: run tools with a scoped credential or a read-only mount, never your own admin keys.
3. Bounded autonomy: the step cap from Step 2, a wall-clock timeout, and a human approval gate for any irreversible action.
4. Log every call: tool name, arguments, result size, duration, to an append-only table. This is how you debug at 3 a.m. without reproducing the run.

A small wrapper makes the allowlist and logging structural:

def guarded_call(name, args, allowed, log):
    if name not in allowed:
        log(name, args, "DENIED")
        return "tool not permitted for this task"
    result = TOOLS[name](**args)
    log(name, args, f"ok len={len(str(result))}")
    return result

Step 6: What changes when it runs unattended

Moving from demo to production changes the questions you ask. In a demo you ask "did it work?" In production you ask "how will I know when it silently stops working?" These are the changes I make to every agent that leaves my laptop:

• Idempotency. Every task has an ID, and re-running it must not duplicate side effects. Crashes and retries are guaranteed, so design for them.
• Dead-letter handling. After N failed attempts a task is parked with its error and the full tool trace, not retried forever.
• Heartbeats. The scheduler records the last successful run; an alert fires on absence of success, not just presence of errors. A dead agent produces no errors.
• A cost and rate ceiling. With local models the cost is time and hardware rather than dollars, but the same discipline applies as a hard cap on runs per hour. If you later point the same loop at a hosted API, check current rate limits and tiers on the provider's own page, since they change (Gemini's, for example, are documented on Google's rate limits page, and free-tier numbers vary per project, so read yours in AI Studio rather than trusting a blog).
• Offline evals. A fixed set of tasks with known-good outcomes that you run after every change to a tool or prompt. Without it you are shipping on vibes.

These are the same patterns behind the multi-agent content system I run, where scheduled workers research, write and publish with no human in the loop. The model is the least interesting part. The state machine, the logging and the failure handling are what let it run for weeks.

Step 7: When to add a framework or MCP

Add a framework when you can name the specific problem it solves for you, not before. After building the loop by hand you will know exactly what you are buying: graph-style control flow, checkpointing, or tracing. Until then the 60-line version is easier to debug than any abstraction.

Tool protocols deserve the same pragmatism. MCP's 2026-07-28 specification, announced on July 28, 2026, makes the protocol core stateless by removing the initialize handshake and the Mcp-Session-Id header (MCP blog). It also introduces a formal lifecycle where a deprecated feature must stay available at least 12 months before removal, and deprecates Roots, Sampling, Logging, Dynamic Client Registration and the legacy HTTP+SSE transport in this release (Cloudflare). For me the takeaway is that the ecosystem is still moving. Wrap your tools behind your own thin interface (like the TOOLS dict above) so an MCP spec change touches one adapter, not your whole agent.

What I'd do

• Build the loop by hand once, with a step cap, error-as-text tool results, and a SQLite task table. Do this before touching any framework.
• As a personal rule of thumb, put far more of your effort into tool design than into prompt wording.
• Enforce permissions in code with an allowlist, scoped credentials and an approval gate for irreversible actions.
• Define "agent is healthy" as a recent success, not an absence of errors.
• Keep your model interface OpenAI-compatible so local-to-hosted is a config change.
• Write ten eval tasks before you write the eleventh tool.

The free stack is genuinely enough to learn on and to prototype on. The hard, valuable part is everything around the model, and that part costs nothing but care.

If you are building an agent that has to run unattended and want a second pair of eyes on the architecture, you can reach me at lazar-milicevic.com/#contact, or keep reading the other build notes on the blog.