Jaseph

Posted on Feb 8

I Built an Open-Source AI Assistant with 19 Tools That Actually Controls Your Computer

#ai #opensource #python #selfhosting

The Problem

ChatGPT is great for conversation, but it can't do anything on your computer. Cursor and Claude Code are great for coding, but they can't send emails or browse the web. I wanted one AI assistant that could do everything — on my machine, with my API key, under my control.

So I built SelfAgent.

What Is SelfAgent?

SelfAgent is an open-source AI-powered personal assistant that runs locally on your computer. You interact with it through a WhatsApp-style chat interface, and it can:

Browse the web and fill forms
Read and send emails
Write, run, and debug code
Build complete websites
Manage files and Git repos
Analyze data and create charts
Download files and YouTube videos
Schedule recurring tasks
And 11 more tools...

All running locally. Your data never leaves your machine.

The Architecture

User <-> Chat UI (WebSocket) <-> FastAPI Backend
                                      |
                              +-------+--------+
                              |   Agent Core   |
                              |  Think -> Plan |
                              |  Execute ->    |
                              |  Observe       |
                              +-------+--------+
                                      |
                    +-----------------+------------------+
                    |                 |                   |
              Memory System     Tool Registry        Task Manager

The agent follows a simple loop:

Think — understand the user's request
Plan — decide which tools to use
Execute — call the tools
Observe — check the results, iterate if needed

This runs for up to 25 iterations per request, so the agent can handle complex multi-step tasks.

5 AI Providers, 1 Interface

SelfAgent works with:

Provider	Cost	Best For
Ollama	Free (local)	Privacy, offline use
OpenRouter	Free tier available	Free cloud models (Llama 70B, DeepSeek)
Google	Free tier available	Gemini 2.5 Pro with vision
OpenAI	Paid	GPT-4o
Anthropic	Paid	Claude Sonnet/Opus

You can store API keys for all providers and switch between them instantly from the settings panel. No restart needed.

The Ollama Experience

This is where it gets interesting. With Ollama, you get:

Status monitoring — green/red dot shows if Ollama is running
Dynamic model list — see all installed models
Pull from UI — type a model name, click Pull, watch the progress bar
Zero configuration — no API key, no account, no internet needed

brew install ollama
ollama pull llama3.2
ollama serve

Select "Ollama (Local)" in SelfAgent settings. You now have a fully local AI assistant.

19 Built-in Tools

Here's every tool and what it does:

Web Browser — Playwright-based. Navigate, click, fill forms, screenshot, run JS. Supports CSS, text, and ARIA selectors.
Web Search — DuckDuckGo. No API key needed.
File Manager — CRUD operations, sandboxed to home directory. Search by filename.
Code Writer — Write and run code in Python, JavaScript, Go, Rust, Java, and more. 7 project templates included.
Website Builder — 8 templates (landing, portfolio, blog, e-commerce, etc.) with local preview server.
Terminal — Persistent shell sessions. Run long processes, manage multiple sessions.
Git — Full Git workflow + GitHub API. Clone, commit, push, create PRs and repos.
Email — IMAP/SMTP. Read inbox, search, send HTML-formatted emails, reply with attachments.
Database — SQLite queries, table inspection, CSV/JSON export/import.
API Tester — Like Postman. Send requests, assert responses, generate docs.
Screenshot — Full screen and window capture.
Clipboard — Read/write system clipboard.
System Control — System info, open apps, manage processes.
Image — Resize, crop, convert, compress, watermark, batch processing.
PDF — Create from markdown, read, merge, split, watermark.
Data Analyzer — Load CSV/JSON/Excel, statistics, 6 chart types, data cleaning.
Downloader — Direct files + YouTube videos/audio via yt-dlp.
Scheduler — One-time and cron-style recurring task automation.
Plugin System — Add custom tools by dropping a Python file in plugins/.

Memory That Actually Works

SelfAgent has a 3-layer memory system:

Layer 1: Short-term — in-memory buffer of recent messages
Layer 2: Long-term — SQLite for all conversations + ChromaDB for semantic search
Layer 3: User Profile — key/value store for name, email, preferences

The agent remembers who you are across sessions. Tell it your email once — it remembers forever.

Getting Started

git clone https://github.com/oezercet/SelfAgent.git
cd SelfAgent
./setup.sh
./run.sh

Open http://localhost:8765. Set your API key or select Ollama. Start chatting.

What's Next?

Multi-agent collaboration
Voice input
Mobile app
More community plugins

DEV Community