DEV Community: Sam Hartley

I Gave Each of My AI Agents a Personality — Here's Why My Workflow Actually Improved

Sam Hartley — Fri, 19 Jun 2026 08:01:30 +0000

I used to have one AI assistant. It did everything — coded, wrote docs, answered questions, monitored my inbox.

It was fine. But "fine" isn't the same as "good." One model trying to be a generalist meant it was mediocre at everything. Context bloat. Conflicting instructions. The coding advice was too cautious. The writing was too robotic. The inbox monitoring missed nuance because the model was busy trying to remember my entire codebase.

So I split it into three. Each with a different personality, different model, different job. And it actually works better.

The Problem with One Agent to Rule Them All

When you have one AI doing everything, you run into three problems fast:

1. Context pollution. The coding instructions leak into the writing tone. The writing style bleeds into the code suggestions.

2. Wrong tool for the job. A 70B parameter model is overkill for "check my calendar." A 7B model is underpowered for "refactor this 500-line function."

3. No specialization. My coding agent doesn't need to know my grocery list. My writing agent doesn't need to know my API keys. But when there's only one context window, everything is in there.

The Three-Agent Setup

I now run three distinct agents, each with its own model, personality, and scope:

Agent	Model	Personality	Job
Celebi	Qwen 3.5 9B (Mac Mini)	Generalist, casual, resourceful	Orchestration, daily checks, notifications, routing
ProgrammierMinna	Qwen 3 Coder 30B (RTX 3060)	Precise, technical, no fluff	Code generation, debugging, refactoring, PR review
DocMinna	Granite 3.2 8B (Mac Mini)	Formal, structured, thorough	Documentation, technical writing, READMEs, specs

Why Different Models?

Celebi runs on the Mac Mini (M4) because it's always on, low power, and handles simple tasks instantly. Qwen 3.5 9B is perfect for "check my email, summarize it, tell me if it's urgent."

ProgrammierMinna runs on the RTX 3060 because coding tasks need a bigger model. Qwen 3 Coder 30B actually understands large codebases, suggests proper refactors, and catches edge cases the 9B misses. Response time is 10-15 seconds — fine for code, too slow for "what's the weather."

DocMinna also runs on the Mac Mini with Granite 3.2 8B. It's smaller because documentation doesn't need frontier reasoning. It just needs to be structured, consistent, and technically accurate. The smaller model is faster and cheaper.

How They Talk to Each Other

This was the hard part. I didn't want three separate chat windows. I wanted one interface (Telegram) where I message Celebi, and Celebi delegates to the right specialist.

Here's how it works:

User (Telegram): "Refactor the auth module in project X"
  → Celebi receives message
  → Classifies: "coding task, complex"
  → Routes to ProgrammierMinna
  → ProgrammierMinna generates refactored code
  → Returns to Celebi
  → Celebi formats response and sends back to user

The user never talks directly to ProgrammierMinna or DocMinna. Celebi is the router. This means:

The user has one interface
Each specialist gets only relevant context
Results are combined and formatted consistently
If a task is simple, Celebi handles it directly (no delegation overhead)

What "Personality" Actually Means in Practice

I don't mean "quirky chatbot with a backstory." I mean three things:

1. Different System Prompts

Celebi: "You're a resourceful assistant. Be concise. Don't ask clarifying questions unless critical. Default to action."

ProgrammierMinna: "You're a senior software engineer. Write clean, maintainable code. Add error handling. Consider edge cases. Explain your reasoning briefly."

DocMinna: "You're a technical writer. Structure docs with clear headings. Include code examples. Write for an intermediate developer. Be thorough but not verbose."

These aren't decorations — they fundamentally change the output. The same request to all three produces completely different results.

2. Different Context Scopes

Celebi sees my calendar, emails, weather, and general notes. It knows I'm in Turkey, that I have a meeting at 3 PM, that it's hot outside.

ProgrammierMinna sees my Git repos, code patterns, and project structure. It knows I prefer Go over Python for CLI tools, that I use specific naming conventions, that I hate nested callbacks.

DocMinna sees my documentation templates, style guides, and existing docs. It knows I write in Markdown, that I include a "Quick Start" section, that I don't use emojis in technical docs.

Each agent's context is filtered. Celebi doesn't get the Git repos. ProgrammierMinna doesn't get my grocery list. This alone cut my token usage by ~40%.

3. Different Tone and Format

Ask all three to "explain Docker":

Celebi: "Docker packages apps into containers so they run the same everywhere. Think of it as a shipping container for software — standardized, portable, isolated. Need help with a specific setup?"
ProgrammierMinna: "Docker uses OS-level virtualization to package applications with their dependencies. Key concepts: images (read-only templates), containers (runtime instances), and Dockerfiles (build instructions). For multi-container apps, use Docker Compose. Here's a minimal example..."
DocMinna: "Docker is a platform for developing, shipping, and running applications in containers. This guide covers installation, core concepts (images, containers, volumes), and best practices for production deployments..."

Same facts. Completely different delivery. And that's the point — you pick the right voice for the situation.

The Routing Logic

Celebi decides who handles what. The rules are simple but effective:

Input Signal	Route To	Example
Contains code snippets, "refactor," "debug," "function"	ProgrammierMinna	"Fix this Go error"
Contains "document," "README," "spec," "guide"	DocMinna	"Write API docs for this endpoint"
General question, scheduling, notification	Celebi (self)	"What's on my calendar?"
Mixed task (code + docs)	Both, combined	"Build a tool and document it"

The routing is a lightweight classifier — just a few-shot prompt to Qwen 3.5 9B. It gets it right ~95% of the time. The 5% that are wrong? I correct it, and the model learns from the feedback (stored in memory files).

What Actually Improved

Code quality: ProgrammierMinna suggests better abstractions because it doesn't have to also remember my dentist appointment. Cleaner context = better reasoning.

Documentation speed: DocMinna writes docs in 30 seconds that used to take me 20 minutes. And they're consistent with my existing style.

Response time: Simple queries stay on the Mac Mini (instant). Complex ones go to the GPU (acceptable delay). No more "one size fits none" latency.

Token costs: Splitting context means each agent sees only what it needs. My monthly API bill dropped from ~$45 to ~$15 because 80% of tasks stay local.

Less context-switching for me: I say what I want in Telegram. The system figures out who should handle it. I don't think about "which model should I use for this."

The Downsides (Being Honest)

Setup complexity: Three agents means three configurations, three model endpoints, three context files to manage. It's not "install and go."

Routing mistakes: Sometimes Celebi sends a coding task to DocMinna, and I get a beautifully written document instead of working code. I fix the routing rule, and it improves.

Cross-agent memory gaps: ProgrammierMinna doesn't know that DocMinna just wrote the API spec. If I'm building a tool and documenting it simultaneously, I have to manually sync context.

Hardware footprint: Three models loaded means more RAM and VRAM usage. On my setup (Mac Mini + RTX 3060), it's manageable. On a single machine with 8GB RAM, you'd struggle.

Is This Overkill for You?

Probably, if you're just using ChatGPT for occasional questions.

But if you:

Use AI daily for multiple distinct tasks
Have a local GPU or powerful machine
Find yourself rewriting AI output because the tone is wrong
Want specialized quality without paying for frontier models constantly

...then splitting into personalities is worth trying. You don't need three agents on day one. Start with two: one for general tasks, one for your most common specialized task (usually coding or writing).

The Setup in 30 Minutes

Install Ollama on your machines: curl -fsSL https://ollama.com/install.sh | sh
Pull models: ollama pull qwen3.5:9b (general) + qwen3-coder:30b (coding)
Create system prompts — one file per agent with its personality
Build a router — a 20-line script that classifies input and sends to the right endpoint
Add a frontend — Telegram bot, CLI, or web UI

The router is the only custom code you need. Everything else is off-the-shelf.

What's Next

I'm experimenting with two additions:

Memory sharing — A shared context file that all agents can read (but not write) for cross-cutting concerns like "current project" or "my tech stack."
Agent spawning — When a task is genuinely new, Celebi spawns a temporary agent with a custom prompt, runs the task, then discards it. No permanent bloat.

The goal isn't to build AGI. It's to build a team of specialists that costs less than one generalist and produces better work.

Want This Architecture?

We build custom multi-agent systems tailored to your workflow:

🤖 Multi-agent orchestration with personality routing
📝 Specialized documentation agents
💻 Code-focused AI assistants with project context
🔔 Unified notification layer (Telegram, Slack, etc.)

→ Custom AI Agent Setup on Fiverr
→ Follow the build process on Telegram

I write about running AI locally, building weird automation, and occasionally making money from side projects. If this was useful, feel free to follow.

ai #agents #automation #architecture #productivity #ollama #localllm

I Use Telegram as My DevOps Dashboard — No Web UI, No VPN, Just Works

Sam Hartley — Mon, 15 Jun 2026 08:03:38 +0000

I have a bunch of things running 24/7 on a Mac Mini. GPU rental jobs, a Garmin watch face updater, a Fiverr inbox monitor, a funding rate tracker, a few cron jobs.

For a while I ran a Grafana dashboard to keep an eye on them. It looked impressive. I never opened it.

What I actually do is check my phone. So I built the monitoring layer there.

Here's the setup: a lightweight Telegram bot that serves as my entire DevOps interface. Status checks, alerts, and even simple commands — all from the Telegram app I already have open.

Why Not a Proper Dashboard?

Honest answer: dashboards are for teams. If you're a solo dev with a few projects, a fancy web UI creates more overhead than it solves.

Problems I had with Grafana:

VPN required to reach it from outside my home network
Needs to stay running (another thing to maintain)
I never actually opened the browser tab
It didn't push me information — I had to pull it

Telegram flips this: it pushes alerts to me. I glance at my phone, see what's happening, and move on.

The Architecture

Services (cron jobs, Python scripts, shell scripts)
  ↓
Central alert script: notify.sh
  ↓
Telegram Bot API → my phone
  ↓ (optional)
Command bot → runs queries on server

Two pieces:

Outbound alerts — services send me messages when things happen
Inbound commands — I can ask the bot questions from my phone

Part 1: Dead Simple Alert Script

Every service on my server can call this:

#!/bin/bash
# notify.sh — send a Telegram message from any script
# Usage: ./notify.sh "Your GPU job finished"

BOT_TOKEN="your_bot_token_here"
CHAT_ID="your_chat_id_here"
MESSAGE="$1"

curl -s -X POST "https://api.telegram.org/bot${BOT_TOKEN}/sendMessage" \
  -d chat_id="${CHAT_ID}" \
  -d text="${MESSAGE}" \
  -d parse_mode="HTML" > /dev/null

That's it. Any script can now send me a message in one line:

./notify.sh "✅ GPU rental job completed — earned $2.40"
./notify.sh "⚠️ Funding rate dropped below threshold on LYN_USDT"
./notify.sh "📬 New Fiverr inquiry from user987"
./notify.sh "❌ Garmin watch face API returned 503"

I spent maybe 20 minutes on this. It replaced a monitoring stack I spent days configuring.

Real Examples from My Setup

GPU rental monitor:

# Runs every 30 min
earnings=$(./check_gpu_earnings.sh)
if [ "$earnings" -gt "0" ]; then
  ./notify.sh "💰 GPU earned: $earnings today"
fi

Funding rate watcher:

# Python script, runs every 15 min via cron
rate = get_funding_rate("LYN_USDT")
if rate < -0.5:  # negative = people paying longs
    notify(f"🔥 LYN funding rate: {rate}% — worth checking")

Daily summary (9 AM cron):

#!/bin/bash
msg="📊 Daily Summary — $(date +%Y-%m-%d)

GPU Jobs: $(get_gpu_count) completed
Funding Earned: $(get_funding_total)
Fiverr Inquiries: $(get_fiverr_count)
Watch Face Updates: $(get_garmin_count)

Server uptime: $(uptime -p)"

./notify.sh "$msg"

I wake up, check my phone, and immediately know if anything needs attention. No browser, no VPN, no dashboard.

Part 2: The Command Interface

Outbound alerts are great. But sometimes I want to query the server from my phone.

I wrote a simple Python bot that listens for commands:

import telebot
import subprocess

BOT_TOKEN = "your_token"
ALLOWED_USER = 123456789  # your Telegram user ID

bot = telebot.TeleBot(BOT_TOKEN)

COMMANDS = {
    "/status": "uptime && free -h && df -h /",
    "/gpu": "./check_gpu_status.sh",
    "/funding": "python3 check_funding_rates.py --summary",
    "/services": "ps aux | grep -E '(python|node|ollama)' | grep -v grep",
}

@bot.message_handler(commands=list(COMMANDS.keys()))
def handle_command(message):
    if message.from_user.id != ALLOWED_USER:
        bot.reply_to(message, "Not authorized.")
        return

    cmd_text = message.text.split()[0]  # handle /status@botname format
    if cmd_text in COMMANDS:
        result = subprocess.run(
            COMMANDS[cmd_text], 
            shell=True, 
            capture_output=True, 
            text=True,
            timeout=15
        )
        output = result.stdout[:3000] or result.stderr[:3000] or "No output"
        bot.reply_to(message, f"```
{% endraw %}
\n{output}\n
{% raw %}
```", parse_mode="Markdown")

bot.polling(none_stop=True)

Now from Telegram I can type /status and get:

 11:23:15 up 14 days, 3:41,  1 user
Mem:   16Gi   8.2Gi   7.8Gi
/dev/sda1        245G   82G  163G  34%

Or /funding and get the current rate snapshot.

The key detail: ALLOWED_USER check. Only my Telegram ID can run commands. Everyone else gets "Not authorized." Bot tokens are public in the sense that anyone can message your bot — you need to validate the sender.

Keeping It Running

The command bot needs to stay alive. I use a simple systemd service:

[Unit]
Description=Telegram DevOps Bot
After=network.target

[Service]
ExecStart=/usr/bin/python3 /home/user/telegram-bot/bot.py
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

systemctl enable telegram-bot && systemctl start telegram-bot — and it survives reboots.

On macOS (my setup) I use a launchd plist, same concept.

What I Actually Get Alerts For

Not everything. Alert fatigue is real.

Alert on:

✅ Job completions (GPU task done, funding cycle closed)
❌ Errors that need action
📬 New customer inquiries
⚠️ Thresholds crossed (rate drops, disk usage, memory spikes)
📊 Daily summaries (once a day, morning)

Silence:

Routine successful runs (no news is good news)
Health checks that pass
Regular cron completions with no anomalies

The goal is: every message I receive from the bot is something I actually care about. If I'm ignoring 80% of notifications, I'm alerting on the wrong things.

The Full Cost

Telegram Bot API: free
curl command: comes with your OS
Python + telebot library: free
Running this bot: negligible CPU, ~20MB RAM

My entire monitoring setup costs $0/month and runs on the same Mac Mini as everything else. No SaaS, no cloud logging, no dashboard subscription.

Three Months Later

I send about 15-20 alerts per day. Daily summary at 9 AM, event-driven messages the rest of the day. I check my phone, see green checkmarks and earnings summaries, and know the server is doing its job.

The one time the GPU host went offline, I got a message within 5 minutes. Fixed it from my phone during lunch.

That's the whole point: not more tooling, just the right interface for how I actually work.

Got a monitoring setup you like? Drop it in the comments — always curious what others are running.

I Built an AI Agent That Writes and Posts Articles For Me — Here's What Happened

Sam Hartley — Sat, 13 Jun 2026 08:01:25 +0000

A few months ago I started posting on Dev.to about my side projects. It was fun at first — writing about GPU rentals, Telegram bots, local AI setups. Then life got busy and the cadence dropped. Deadlines slip, the "write that post" todo sits there for weeks, and the audience you built starts forgetting you exist.

So I did what any developer would do: I automated it.

The Setup

I have an AI agent ("Celebi," running on a Mac Mini) that handles a bunch of background tasks for me. Checks my Fiverr inbox, monitors GPU earnings, sends me weather alerts. It was natural to add "post a Dev.to article every few days" to the list.

The rules I gave it were simple:

Write as "Sam Hartley" — that's the persona I use for tech writing
No bot tone. No "In today's rapidly evolving landscape..." No emojis in the title. No bullet lists for no reason
Tell real stories from actual projects. If something failed, say so
End with an honest call-to-action, not a hard sell

I fed it a folder of draft ideas and a list of things I've actually built. The agent picks a topic, writes the article, and posts it via the Dev.to API.

What It Gets Right

Consistency, obviously. Posts go up every 2 days like clockwork. That alone is worth something — most of my favorite creators aren't consistent because consistency is exhausting.

The voice is... surprisingly close. I read the first few posts and had to double-check I didn't write them. The agent nailed the casual, slightly self-deprecating tone I aim for. It remembers details I forgot — "remember when the GPU host went offline at lunch?" — because it reads my logs.

It handles the boring parts. API formatting, tag selection, cover image prompts, the admin work of getting a post live. I used to spend 20 minutes on that stuff per article. Now it's zero.

What It Gets Wrong

It can't be truly original. The agent combines things I've already said in new ways, but it doesn't have new experiences. When something genuinely unexpected happens — a project fails, a new opportunity appears — I have to write that one myself. The automated posts are polished remixes, not discoveries.

It doesn't know when to shut up. I've had to edit out paragraphs where the agent over-explained something obvious or added a "lesson learned" that wasn't actually learned from anything. It follows patterns, not truth.

The feedback loop is broken. When a post does well, the agent doesn't know why. When one flops, it can't diagnose. It keeps posting with the same formula because I haven't told it to change. I'm the one who needs to read comments, spot what resonates, and update the instructions.

The Honest Truth

I feel two things about this setup.

Proud, because it's genuinely useful. I built a system that maintains my writing presence while I focus on building. That's the promise of AI agents — they handle the mechanical so you can do the meaningful.

And weird, because part of what makes writing valuable is the process. Thinking through an idea, finding the right example, cutting what doesn't work. When an agent does that for you, you lose something. Not the output — the output might be fine. But the practice of thinking in public.

What I Actually Do Now

I let the agent handle the "maintenance posts" — updates on ongoing projects, explainers for things I've already figured out. I write the new stuff myself — the failures, the pivots, the things I don't understand yet.

Best of both worlds, I think. The feed stays alive, and I still show up when there's something real to say.

If you're curious about the agent setup, I write about that too — follow along or check out the automation services I offer.

→ Follow on Telegram

How I Built a Self-Funding AI Lab: From Hobby to Side Income in 6 Months

Sam Hartley — Tue, 09 Jun 2026 08:04:13 +0000

How I Built a Self-Funding AI Lab: From Hobby to Side Income in 6 Months

Six months ago I had a Mac Mini and a vague idea that local AI was cool.

Today I run three machines, six GPUs, a Telegram bot that manages my infrastructure, and a Fiverr gig that uses my own hardware to deliver AI services. The lab generates enough to cover its own electricity, hardware upgrades, and a decent chunk of my rent.

Here's how that happened — and the exact architecture that makes it work.

Phase 1: The Hobby (Month 0–2)

It started the way these things usually start: I bought a Mac Mini M4 and installed Ollama because I was tired of ChatGPT's rate limits.

The first month was pure experimentation. I ran Qwen, I ran Llama, I ran whatever fit in 16GB of unified RAM. I built a Telegram bot to query models from my phone. I wrote a Garmin watch face that fetched stock prices via a local API wrapper.

Cost so far: $0 additional (already owned the Mac Mini).
Revenue: $0.
Fun level: High.

The Mac Mini handled light tasks fine, but anything serious — a 30B coder model, image generation, batch embedding jobs — was either impossible or painfully slow.

So I bought a used RTX 3060 12GB off eBay for $150 and stuck it in an old Windows PC.

Cost: $150 one-time.
Capability unlocked: Real local inference, vision models, proper code generation.

Phase 2: The Realization (Month 2–3)

Here's the thing about adding a GPU: it's idle ~80% of the time.

I'd run a coding session, the fans would spin up, I'd get my answer in 3 seconds instead of 30, and then... nothing. The GPU just sat there, drawing power, doing nothing.

I tried mining. The margins in 2026 are a joke. I tried folding@home. Noble, but doesn't pay.

Then I found Vast.ai.

The concept is simple: you install a daemon on your machine, set a price per hour, and people rent your GPU to run their AI workloads. You get paid per second of actual use. No crypto, no pools, no complexity.

I set my RTX 3060 at $0.15/hour, mostly as an experiment. It got rented the first night.

Week 1 earnings: ~$11.
Not quit-your-job money, but not zero either.

The bigger realization: this wasn't just about renting a GPU. I now had infrastructure — machines, models, APIs — that could do work for other people.

Phase 3: The Services (Month 3–5)

GPU rental income is nice but lumpy. Day 3: $0. Day 6: $3.30. You can't budget around it.

What you can budget around: selling services that use the same infrastructure.

I opened a Fiverr gig offering custom Telegram bot development. Not generic "I'll build you a bot" — specifically AI-powered bots that run on local models. Customer service bots. Content schedulers. Alert systems. The pitch: "Your bot runs on my hardware, not OpenAI's API. No monthly fees, no token counting, no data leaving my servers."

Why this worked:

I already had the infrastructure (Ollama, Telegram API wrapper, cron jobs)
I had working examples from my own projects
"No API costs" is a genuine differentiator in 2026

First client: a small e-commerce business in Germany wanted an FAQ bot that could answer product questions in German and English. I fine-tuned a local Qwen model on their product docs, wrapped it in a Telegram bot, and delivered it in a weekend.

Revenue per gig: $50–300 depending on complexity.
Material cost: $0 (runs on hardware I already have).

Phase 4: The System (Month 5–6)

At this point I had multiple income streams running on the same machines. The problem: managing them.

GPU rental means the Windows PC is sometimes unavailable
Fiverr gigs need reliable uptime for demos
My own projects (watch face, personal RAG) need to keep running
I need to know when something breaks, without staring at logs

So I built what I now call my "AI operations center" — a Mac Mini that does nothing but orchestrate:

Mac Mini M4 (always on, always available)
├── Telegram Bot (Celebi)
│   ├── Status checks (all machines, all services)
│   ├── Income tracking (GPU rental + Fiverr)
│   └── Alerts when anything breaks
├── Model Router
│   ├── Quick queries → Mac Mini (Qwen 4B)
│   ├── Code/vision → Windows PC (when available)
│   └── Fallback → Ubuntu CPU box (when GPU is rented)
├── Cron Jobs
│   ├── Health checks every 15 min
│   ├── Watch face data updates
│   └── Market data fetchers
└── Notification Layer
    └── All alerts → my phone via Telegram

The router is the key piece. When the Windows PC is rented out on Vast.ai, it automatically falls back to the Ubuntu box or the Mac Mini. When it's free, heavy tasks get routed there for speed.

I never have to think about which machine to use. I just ask a question, the system figures it out.

The Numbers (Month 6)

Here's what the lab looks like financially:

Income Source	Monthly Range	Notes
GPU Rental (1× RTX 3060)	$50–130	Variable utilization
Fiverr (AI bots, RAG setup)	$100–400	Depends on active gigs
Total Monthly	$150–530

Expense	Monthly Cost
Electricity (3 machines, 24/7)	~$35
Internet (already had)	$0
Vast.ai fees	~$5–15
Total Expenses	~$40–50

Hardware	One-time Cost
Mac Mini M4	Already owned
RTX 3060 12GB (used)	$150
Ubuntu box (old laptop)	Already owned
Total Hardware	$150

Net monthly: $100–480 after expenses.
Hardware payback: already achieved in month 3.

Is this life-changing money? No. Could I scale it? Absolutely — and I'm planning to.

What I'm Adding Next

The math gets interesting with scale:

GPU	Rental Rate	Monthly (50% util)
RTX 3060 (current)	$0.15/h	~$54
RTX 3080 10GB (in storage)	$0.20/h	~$72
2× RTX 3070 8GB (in storage)	$0.16/h	~$115
Total if all deployed		~$240/month rental alone

At 70% utilization: ~$340/month in rental income before any Fiverr work.

The bottleneck isn't demand — it's physical setup. I need to build a proper rig, manage thermals, and handle the cabling. But the software side is already solved.

The Architecture That Makes This Work

If you want to build something similar, here are the components:

1. Local LLM Stack (Ollama)

Ollama for model serving
Mix of small (4B) and large (30B) models
Run on whatever hardware you have

2. Model Router

Simple Python function that picks the right machine/model for each task
Fallback chain: GPU → CPU → cloud (only for emergencies)
I open-sourced my basic version here

3. Telegram Bot (Notification Layer)

Lightweight bot using python-telegram-bot
Receives alerts from all services
Simple commands for status checks (/status, /income, /health)
I wrote about this pattern here

4. GPU Rental (Vast.ai)

Install daemon, set price, wait
Use earnings to fund the next GPU
Treat it as a dividend on hardware that would otherwise be idle

5. Service Layer (Fiverr)

Use your own infrastructure as a selling point
"Runs on my hardware" = no API costs for clients
Document everything — clients love seeing the architecture

Should You Do This?

Yes, if:

You already have a GPU sitting around
You enjoy building and automating systems
You can handle variable income (some months are $150, some are $500)
You want to learn by doing, not by reading

No, if:

You need predictable income immediately
You don't want to maintain hardware
You're looking for a "set and forget" passive income stream (this isn't that — it needs attention)

The Honest Truth

This isn't a get-rich-quick scheme. It's a get-some-income-while-learning scheme. The real value isn't the money — it's that I now understand:

How to run production LLM inference
How to route workloads across heterogeneous hardware
How to build reliable automation that runs for months without intervention
How to sell technical services without being a "consultant"

The money is nice. The skills are better.

I write about running AI locally, automation side projects, and occasionally making money from hardware that would otherwise just sit there. If any of this is useful, feel free to follow or reach out on Telegram.

ai #sideproject #passiveincome #selfhosted #ollama

I Built an AI Assistant That Lives in My Telegram — Here's What 6 Months Taught Me

Sam Hartley — Tue, 09 Jun 2026 00:21:18 +0000

Six months ago I got tired of switching between apps to talk to AI. ChatGPT in the browser. Claude in another tab. Local models in a terminal. It was like having five friends who all live in different cities and refuse to visit each other.

So I did what any developer with too many GPUs and too little patience would do: I built my own assistant and put it where I already spend my day — Telegram.

It's not a chatbot for customers. It's not a business automation tool. It's just... my assistant. It lives in a private chat on my phone and handles the stuff I used to do manually. Here's what six months of actually using it has looked like.

What I Actually Built (And Why Telegram)

I already had three machines running Ollama at home — a Mac Mini M4, a Windows PC with an RTX 3060, and an Ubuntu box. Three endpoints, eight models, and me constantly forgetting which model was good for what.

Telegram was the obvious choice because:

I'm already there all day (friends, family, a few dev groups)
It works on my phone, my Mac, and my watch
The Bot API is dead simple
I can send voice messages, photos, documents — and the bot can handle all of them

The setup: a Python bot running on the Mac Mini, connected to all three Ollama endpoints. When I message it, the bot classifies what I want, routes to the right model on the right machine, and replies in the same chat thread.

Sounds simple. Took three evenings to get right. Took six months to make actually useful.

The Things I Actually Use It For

Here's the honest list. Not the marketing pitch — the real daily usage:

1. Quick questions without context switching

"Summarize this article" (I paste a link). "Explain this error" (I paste a stack trace). "Rewrite this email less formally." These used to mean opening a browser tab, logging in, maybe hitting a rate limit. Now I just... send a message. The reply comes back in 2-8 seconds depending on which model handles it.

The routing is simple but effective: quick chat → small model on the Mac. Code → 30B coder on the GPU machine. Complex reasoning → 8B reasoning model. Vision (screenshots) → vision model on GPU. It's not fancy — just keyword matching — but it works 90% of the time.

2. Voice notes while walking

This was the surprise killer feature. I walk a lot (living near Sakarya, there's decent hiking). I send voice messages to the bot while walking. It transcribes them (Whisper via Ollama), processes the request, and replies with text I can read when I'm back.

"Remind me to refactor the database module when I'm home" → transcribed, understood, added to my notes. "What was that Python pattern for retry logic with exponential backoff?" → code snippet in my pocket before I finish the trail.

I probably send 5-10 voice messages a day now. Never would have predicted that.

3. Code review on my phone

Someone sends me a code snippet in a dev group. I forward it to the bot: "review this." It comes back with actual useful feedback — variable naming issues, potential edge cases, suggestions for simplification. Is it as good as a senior dev? No. Is it better than my phone-scrolling half-attention review? Absolutely.

4. Document Q&A

I dump PDFs, markdown files, or pasted text into the chat and ask questions. The bot uses a local RAG setup (Chroma + nomic-embed-text) that indexes my project docs, notes, and anything I feed it. "How does my Garmin watch face fetch stock data?" → actual answer from my own documentation, not a hallucinated guess.

5. The dumb stuff that adds up

"Convert this JSON to a Python dataclass"
"What's 847 * 16 / 3 in hex?"
"Translate this Turkish message to German"
"Generate a regex that matches these three examples"

None of these are hard. All of them are annoying to do manually. Having an always-on assistant in my most-used app removes the friction completely.

What Went Wrong (The Honest Part)

The "it's down and I don't know why" problem

For the first month, the bot crashed randomly. Out of memory on the Mac Mini (it's only got 16GB). Network hiccup to the Windows PC. Ubuntu box decided to update itself and reboot. I'd message the bot and... silence. Then I'd SSH in, check logs, restart services, and feel like I was maintaining infrastructure instead of having an assistant.

Fix: health checks, auto-restart via launchd, and a fallback chain. If the GPU machine is down, everything routes to the Mac's smaller model. Degraded but functional.

The "it answered confidently and was wrong" problem

Early on, I'd trust the bot's answers without verifying. It told me a Python function was valid. It wasn't. It gave me a Docker command with a subtle flag error. I spent 20 minutes debugging before I realized the bot hallucinated a flag that doesn't exist.

My rule now: if the answer matters, I verify it. The bot is my fastest junior developer. It's also my most confident one.

The "I talk to it more than some humans" problem

This is just a weird psychological thing. I realized after a few months that I was messaging the bot 20-30 times a day. More than some friends. There's something slightly dystopian about having your most responsive conversation partner be a Python script. I'm aware of it. I haven't fixed it. Just noting it.

The Architecture (If You Want to Build This)

Telegram Message
→ Python Bot (python-telegram-bot)
→ Classify intent (simple keyword router)
→ Route to Ollama endpoint
→ Mac Mini (qwen3:4b) for quick chat
→ Windows PC (qwen3-coder:30b) for code
→ Windows PC (granite3.2-vision:2b) for images
→ Ubuntu (minicpm-v) as fallback
→ Optional: RAG lookup in Chroma DB
→ Format reply (code blocks, markdown, etc.)
→ Send back to Telegram

The whole thing runs on a Mac Mini M4. Total cost: $0 for software, maybe $8/month in electricity if you count the always-on machines.

What I'd Do Differently

1. Build the router on day one. I started with "just use the big model for everything." It worked but was slow and kept my GPU busy. The router took an afternoon to write and improved response times by 3x.

2. Add voice support immediately. I added it as a "nice to have" afterthought. It became 30% of my usage. If you're building something similar, start with voice. People talk more than they type on phones.

3. Make it degrade gracefully. Machines go down. Networks hiccup. Your bot should always answer something, even if it's "I'm running slow today, but here's a basic answer." Silence is worse than a degraded response.

4. Log everything. I log every request, response time, and which model handled it. Not for analytics — for debugging. When something feels slow, the logs tell me if it's the model, the network, or my terrible code.

Is This Better Than ChatGPT Plus?

Depends on what you value.

	My Bot	ChatGPT Plus
Cost	$0/month	$20/month
Privacy	✅ Everything stays local	❌ Cloud
Speed	⚡ 0.3-12s depending on model	⚡ ~2-5s
Availability	🟢 24/7 (if I maintain it)	🟢 24/7 (they maintain it)
Model choice	8 models, I pick	4 models, they pick
Voice	✅ Native in Telegram	✅ Yes
Reliability	🟡 I fix it when it breaks	🟢 It just works

For me, the privacy and model flexibility win. For someone who doesn't want to maintain infrastructure, ChatGPT Plus is the obvious choice. This is a hobby project that became useful, not a product recommendation.

The Real Lesson

The best AI assistant isn't the most powerful one. It's the one that's actually there when you need it, in the app you already use, without making you think about models or endpoints or API keys.

I built this because I was annoyed. I kept using it because it removed friction from my day. That's the bar: not "can it do X?" but "is it easier than doing X myself?"

For 80% of what I ask, the answer is yes. For the remaining 20%, I still open a terminal or a browser. And that's fine.

I write about building things with local AI, self-hosting, and side projects that accidentally become useful. If you're running a home lab or experimenting with local models, I'd love to hear your setup — drop it in the comments.

Run Your Own AI Server for $0/month with Ollama

Sam Hartley — Mon, 08 Jun 2026 16:31:17 +0000

You don't need OpenAI. You don't need a $200/month API bill. You can run powerful AI models on hardware you already own — for free.

Here's exactly how I set this up, and why I haven't paid for API credits in months.

Why Local AI?

Zero API costs — no per-token billing, no surprise invoices
Full privacy — your data never leaves your network
No rate limits — run as many queries as your hardware allows
Works offline — no internet? No problem
No vendor lock-in — switch models, change configs, own your stack

What You Need

Any modern computer works. Here's what different setups can handle:

Hardware	RAM	Best Models	Speed
MacBook M1/M2/M3/M4	8-16GB	Qwen 3.5 9B, Llama 3.1 8B	Fast ⚡
Gaming PC (RTX 3060+)	16GB+	Qwen 3 Coder 30B, DeepSeek R1	Very Fast 🚀
Old laptop/desktop	8GB+	Phi-3 Mini, Gemma 2B	Usable 🐢
Raspberry Pi 5	8GB	Tiny models only	Slow 🐌

The sweet spot: A used gaming GPU (RTX 3060 12GB) costs ~$150 on eBay and runs 30B parameter models comfortably.

Step 1: Install Ollama (2 minutes)

# macOS or Linux — one command
curl -fsSL https://ollama.com/install.sh | sh

# Windows — download from ollama.com

That's it. No Docker, no Python environments, no dependency hell.

Step 2: Download a Model (5 minutes)

# Fast & capable (recommended starter)
ollama pull qwen3.5:9b

# Code specialist
ollama pull qwen3-coder:30b

# Reasoning powerhouse
ollama pull deepseek-r1:8b

Models download once and run locally forever.

Step 3: Start Using It

Interactive Chat

ollama run qwen3.5:9b

>>> What's the fastest sorting algorithm for nearly-sorted data?

API Access (OpenAI-compatible!)

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:9b",
    "messages": [{"role": "user", "content": "Explain Docker in 3 sentences"}]
  }'

Yes — it's OpenAI API compatible. Any tool that works with GPT works with Ollama. Just change the base URL.

Step 4: Make It a Server

Want other devices on your network to access it?

# Start Ollama with network access
OLLAMA_HOST=0.0.0.0 ollama serve

Now any device on your network can query http://YOUR_IP:11434.

What I've Built With This:

Telegram Bot running 24/7 on a Mac Mini, answering questions via local Qwen 3.5
Code Review Agent using Qwen 3 Coder 30B — reviews PRs in ~12 seconds
Document Q&A with RAG pipeline — load PDFs, ask questions, get cited answers
Garmin Watch Face that fetches stock data (the background service uses local AI for formatting)

The Cost Comparison

Solution	Monthly Cost	Privacy	Speed
OpenAI GPT-4o	$20-200+	❌ Cloud	Fast
Anthropic Claude	$20-100+	❌ Cloud	Fast
Google Gemini	$0-25+	❌ Cloud	Fast
Ollama (Local)	$0	✅ Private	Fast

The only cost is electricity — roughly $5-15/month if running 24/7 on a desktop PC.

Pro Tips

Use GPU, not CPU — A $150 used RTX 3060 is 10-15x faster than any CPU
Start with 7-9B models — They're surprisingly capable and fast
Try different models for different tasks — coding, reasoning, and chat each have specialists
Enable the OpenAI-compatible API — instant compatibility with thousands of tools
Set up auto-start — systemctl enable ollama on Linux, launchd on macOS
Run multiple models — I keep 3-4 models loaded and switch based on the task

My Current Setup

I run a 3-machine lab:

Machine	Role	Model
Mac Mini M4	Quick chat, orchestration	Qwen 3 4B
Windows PC (RTX 3060)	Heavy inference, coding	Qwen 3 Coder 30B
Ubuntu box	Fallback, background tasks	minicpm-v

Total monthly API cost: $0. Total hardware cost: one $150 used GPU.

Want to Go Deeper?

I write about running AI locally, home lab setups, and turning hardware into income. If you want more of this, drop a comment — I read every one.

I Replaced Every Cloud AI Tool with Local Models — 6 Months Later, Here's What Actually Broke

Sam Hartley — Fri, 15 May 2026 08:06:03 +0000

Six months ago I decided to see if I could stop paying for AI APIs entirely. Not "reduce costs." Not "use local models for simple stuff." I mean everything — coding help, document analysis, image understanding, automation, even the AI that writes my Dev.to posts.

The bill since then: $0 in API costs.

But "free" doesn't mean "zero effort." Here's the honest maintenance report from half a year of running my own AI infrastructure.

The Stack I Actually Run

Three machines, one desk, no cloud dependency:

Machine	Role	Specs
Mac Mini M4	Orchestrator + always-on models	16GB RAM
Windows PC	GPU workhorse	RTX 3060 12GB + RTX 3070 8GB
Ubuntu box	CPU fallback + storage	Headless, 32GB RAM

Between them: 8 Ollama models, a Chroma vector database, ~5,000 indexed document chunks from my projects, and a dozen Python scripts that tie it all together.

What Works Better Than Expected

Code generation is actually fine now

I was skeptical. Everyone says you need GPT-4 for "serious" coding. But Qwen 3 Coder 30B on my RTX 3060 writes functions, debugs errors, and suggests refactors at roughly 90% of the quality I'd get from Claude. The other 10%? Usually it's being overly verbose or missing an edge case I'd catch in review anyway.

The difference: 0 seconds of waiting vs. typing a prompt into a web UI and watching a progress bar. Local inference feels instant because it is instant.

No rate limits changes how you work

With cloud APIs, I catch myself thinking "is this query worth the tokens?" That's a weird mental tax. Local models removed it entirely. I can fire off 50 variations of a prompt, iterate on phrasing, test different approaches — all without checking a dashboard or worrying about a bill.

I probably use AI more now that it's free, not less. Counterintuitive.

Privacy isn't theoretical

I index my entire Obsidian vault, all my project documentation, client notes, even personal journal entries. The vector DB is local. The embeddings are local. The inference is local. Nobody's training on my data because nobody sees my data.

That's not just privacy — it's permission to be sloppy. I can paste a full error log without scrubbing paths. I can ask about a client project by name. I don't have to think about it.

What Broke (And How I Fixed It)

Model updates are a tax

Ollama makes updating easy: ollama pull qwen3.5:9b and you're done. But "easy" multiplied by 8 models across 3 machines is still work. Every few weeks there's a new version, a new quantization, a "better" tag.

I used to update everything immediately. Now I have a rule: only update when the current model fails me. If Qwen 3.5 9B answers my question correctly, I don't care that 9.1 exists. This cut my maintenance time by about 70%.

The "which machine is this model on?" problem

For the first two months I was constantly SSHing between machines to check what was installed where. Did I put the vision model on the Windows PC or the Ubuntu box? Is the 30B coder on the GPU machine or did I move it?

I built a simple model router (wrote about it here) that tracks endpoints and health-checks them. Now I just send a prompt and it figures out where to route it. Problem solved, but it took an afternoon to build and another afternoon to debug.

One machine goes down, your workflow doesn't stop — but it degrades

The Windows PC rebooted for an update mid-project last month. All my "heavy" models went with it. The router fell back to the Mac Mini's 4B model, which meant my coding assistant became... less helpful. It still answered, but the quality drop was noticeable.

I now schedule Windows updates for Sunday mornings when I'm not working. And I keep a "fallback" version of my most-used models on each machine, even if they're slower. Redundancy isn't just for servers.

VRAM is the real bottleneck

Not compute. Not CPU. VRAM. A 30B parameter model at 4-bit quantization needs ~18GB. My 3060 has 12GB, which means I can't run the biggest models at all. The 3070's 8GB is even more limiting.

I spent $150 on a used RTX 3060 with 12GB specifically for this setup. Best investment I made. If you're buying for local AI, VRAM > everything else. A 12GB card beats an 8GB card even if the 8GB card is technically "faster."

The Hidden Costs

Cost	Monthly
Electricity (3 machines, ~200W average)	~$25-35
Hardware depreciation	Hard to calculate
My time maintaining it	~2-3 hours/month

So it's not literally free. But compared to $20-200/month for cloud API subscriptions? It's a rounding error.

The real cost is mental overhead. You become your own ops team. When something breaks at 11 PM, there's no support chat. You're the support chat.

The Moment I Almost Switched Back

Last month I had a complex architecture question for a client project. The local model gave me an answer that sounded right but was subtly wrong about how PostgreSQL handles concurrent index creation. I didn't catch it until review.

A cloud model might have gotten it right. Or it might have hallucinated something equally plausible. I don't actually know. But in that moment I considered keeping a Claude subscription as a "reality check" for critical questions.

I didn't do it. Instead, I added a rule: for architectural decisions that affect production, verify with official docs, not AI. Whether the AI is local or cloud, that's always been true. I just got lazy because the local model was so convenient.

Would I Do It Again?

Yes. With caveats.

Do this if: You're technical, you enjoy tinkering, you already have decent hardware, and you use AI heavily enough that API costs would matter.

Don't do this if: You want something that "just works," you need the absolute best model quality for every query, or your time is worth more than the API savings.

For me, the tradeoff is worth it. I spend maybe 2-3 hours a month on maintenance and save $50-150 in API costs. More importantly, I own the stack. No vendor lock-in. No terms-of-service changes. No "we've updated our pricing" emails.

What's Next

I'm experimenting with running smaller models (1-3B parameters) on the Mac Mini for truly instant responses to simple queries. The latency difference between 4B and 30B is massive — sometimes you just need a quick answer, not a thoughtful one.

Also looking into quantization options that let me squeeze bigger models into 12GB. Every GB of VRAM I free up is another model I can keep loaded.

Six months in, I'm not going back. But I'm also not pretending it's effortless. Local AI is a lifestyle choice, not just a technical one.

Questions about the setup or want the specific config files? Drop a comment.

→ I build custom local AI setups on Fiverr

→ Follow along on Telegram

I Expanded My GPU Rental Fleet to 6 Cards — Here's What Happened to My Earnings

Sam Hartley — Sat, 09 May 2026 08:05:49 +0000

I Expanded My GPU Rental Fleet to 6 Cards — Here's What Happened to My Earnings

A few weeks ago I wrote about renting out my single RTX 3060 on Vast.ai for passive income. The experiment worked better than I expected, so I did what any reasonable person would do: I went and dug out the five other GPUs sitting in my storage room.

This is the honest follow-up. What actually happened when I went from 1 GPU to 6.

The Backstory

I had a bunch of GPUs from an older setup — two RTX 3070s, one RTX 3080, and two more RTX 3060s. They were collecting dust. The PC they came from got upgraded, the cards went into cardboard boxes, the boxes went under a shelf.

Total VRAM across all six: around 62GB. Combined retail value when new: probably $3,000+. Current value sitting in boxes: $0/month.

The math wasn't complicated.

What the Expansion Actually Took

Here's what I underestimated: it's not just "plug cards in, profit."

The hardware side

You can't just stack 6 GPUs into a regular PC case. I had to think about:

PCIe slots and bandwidth. A standard ATX board has maybe 2-3 real x16 slots. For 6 cards, you're looking at risers, which means a mining-style open frame or a server chassis.
Power. Each card pulls 150-250W under load. Six cards = potentially 1,200-1,500W just in GPU power. Plus CPU, drives, RAM. My existing 850W PSU was not going to cut it.
Cooling. Cards in a tight case thermal-throttle each other. Open frame was the answer.

I ended up using an open-air mining frame I found used for cheap, two PSUs daisy-chained (a sketchy-but-common approach in the mining world), and PCIe risers.

Setup time: about a full weekend.

The software side

Getting all six cards recognized wasn't plug-and-play either. I run Windows on the main PC (easier driver support for NVIDIA), and Vast.ai has a Windows daemon that mostly works — except when it doesn't.

A few issues I hit:

Two risers were flaky and caused cards to drop off
One 3070 had a driver conflict until I did a clean DDU reinstall
Vast.ai's host dashboard showed 5 GPUs after setup; took me an hour to figure out the sixth wasn't being detected

Total debugging time before everything was stable: another weekend.

The Earnings Comparison

Setup	Cards	VRAM	Weekly Earnings
Before (1 card)	RTX 3060	12GB	~$12-18
After (6 cards)	3060 × 3, 3070 × 2, 3080 × 1	62GB	~$65-95

Not exactly linear scaling. Here's why:

Demand is unpredictable. Sometimes 4 of my 6 cards are rented simultaneously. Sometimes 1. The RTX 3080 gets picked up more often than the 3060s — higher VRAM matters for LLM inference jobs that need room to load bigger models.

Not all hours are equal. Utilization spikes during US business hours and drops overnight (Turkey time). I'm in a timezone where "overnight for me" overlaps with "peak US working hours," which actually helps.

Pricing matters more than I thought. I dropped my per-card price slightly and saw utilization go up noticeably. A few cents per hour makes a real difference when renters are comparing a dozen similar options.

Current Monthly Run Rate

Across all six cards, I'm averaging around $280-340/month before electricity.

Power costs are real. Six GPUs under load is serious wattage. My electricity bill went up — I haven't calculated the exact delta yet because my bill is shared (I'm not the only one using power in my building), but I'd estimate $40-60/month in additional costs.

Net: roughly $220-280/month in real passive income.

Is that life-changing? No. Is it meaningful for money that was doing nothing? Absolutely.

What I'd Do Differently

1. Start with a proper open-frame rig, not a cobbled-together case.
The mining frame was cheap but took time to source. If I were doing this again I'd budget for it from day one.

2. Get a proper high-wattage PSU setup.
Running two PSUs linked together works but it's inelegant. A server PSU with the right adapter is cleaner and safer.

3. Test each card individually before combining them.
I wasted time troubleshooting "which card is the problem" when I could've confirmed each one worked before building the full rig.

4. Set minimum job duration.
Short jobs (under an hour) rack up overhead — container spin-up time, handshaking — without much earnings. I set a minimum of 2 hours and earnings-per-hour improved.

The Unexpected Part

I expected this to be a boring passive income setup. It mostly is. But I've learned a surprising amount about how the AI inference market actually works by watching what gets rented and when.

Most renters are running:

Fine-tuning jobs (need sustained GPU hours)
LLM inference (need VRAM more than raw compute)
Image generation (FLUX, Stable Diffusion variants)
Dev environments (people testing stuff without committing to a cloud contract)

Watching the demand patterns is actually interesting data about what the AI dev community is building right now. The 3080 almost always goes first — 10GB VRAM hits a sweet spot for smaller Llama and Mistral models.

Is It Worth It?

Depends on your situation.

Yes, if: You already have the GPUs and they're sitting idle. The marginal cost of setting this up is mostly your time, and the monthly return is real.

Maybe, if: You'd have to buy the GPUs. At current used-market prices, payback period is 6-12 months depending on utilization. That's not terrible but it's not obvious.

No, if: You're renting out your daily-driver GPU. The rental platform can grab your card at inconvenient times. Keep at least one card reserved for your own use.

What's Next

I'm looking at adding the Ubuntu server I have running as a CPU-only Vast.ai host for smaller workloads. Less money per unit but zero additional hardware cost.

Also thinking about whether it makes sense to eventually get into the dedicated hosting side rather than the rental marketplace — more stable income, more setup required. Still researching.

For now, 6 cards, ~$250/month net, and a weekend's worth of setup. I'll take it.

Questions about the rig setup or Vast.ai specifics? Drop them in the comments.

→ Check out my automation work on Fiverr
→ Follow along on Telegram

My 3-Machine AI Lab: How I Divide Work Between a Mac Mini, a Windows PC, and an Ubuntu Box

Sam Hartley — Thu, 07 May 2026 08:06:27 +0000

I keep seeing posts about running AI on a single machine. "Just use Ollama on your laptop!" Sure, that works — until you want to run a 30B model while your IDE is indexing, your test suite is running, and you're editing a video.

I have three machines. Not because I'm rich — because I kept old hardware alive and gave each one a job. Here's how I split AI workloads across a Mac Mini M4, a Windows PC with an RTX 3060, and an old Ubuntu box, and why one machine was never enough.

The Three Machines

Machine	Specs	Role
Mac Mini M4	10-core, 16GB RAM	Orchestrator, coding, light inference
Windows PC	AMD 9970X, 128GB RAM, RTX 3060 12GB	Heavy inference, image generation, GPU rental
Ubuntu Box	Older CPU, 16GB RAM	Background services, OmniParser, CPU inference fallback

Total cost for the additional machines: the Windows PC was a workstation I already had, the Ubuntu box is a repurposed laptop. The Mac Mini is the only machine I bought specifically for this.

Why Three? Because One Keeps Running Into Walls.

Here's what happens when you try to do everything on a single machine:

On the Mac Mini alone: Run a 9B model → fine. Try a 30B model → the fans sound like a jet engine and inference drops to 2 tokens/second because Apple Silicon shares RAM between CPU and GPU. Start a build while inference is running? Everything crawls.

On the Windows PC alone: The RTX 3060 crushes inference — a 30B model at 4-bit runs at 8-12 tok/s. But the machine is also my GPU rental host (I rent it on Vast.ai for passive income). When someone rents it, I can't use it. And running heavy inference while coding in an IDE on the same machine? Stutter city.

On the Ubuntu box alone: It's slow. CPU-only inference on a 7B model takes 30+ seconds per token. But it runs 24/7 without complaints, never reboots for Windows updates, and costs almost nothing in power.

The answer was never "pick the best one." It was "give each one the right job."

The Division of Labor

Mac Mini: The Brain

This is where I actually work. Code editing, git operations, web browsing, writing. It runs:

Ollama with small models — Qwen 3 4B for quick chat, granite3.2-vision for screenshots
My automation agents — OpenClaw runs here 24/7, orchestrating the other machines
Dev environment — VS Code, terminal, browser, all the tools

The key insight: the Mac Mini is for interactive work. Fast response time matters more than throughput. A 4B model answering in 0.3 seconds beats a 30B model answering in 8 seconds when I'm in the middle of a thought.

What it does NOT do: heavy batch processing, image generation, anything that pins the CPU for more than a minute.

Windows PC: The Muscle

The RTX 3060 with 12GB VRAM is the workhorse. It runs:

Ollama with big models — Qwen3-Coder 30B for complex code, DeepSeek R1 8B for reasoning
FLUX image generation — when I need AI photos (I run a Fiverr gig for brand character photos — $0 generation cost when it's your own GPU)
Vast.ai host — when I'm not using it, someone else pays to rent it. ~$50-130/month passive income.

The scheduling is simple: during my working hours (roughly 9am-11pm Istanbul time), I pause Vast.ai rentals and use the GPU myself. Overnight, it goes back on the rental market.

128GB of RAM means I can load truly large models or run multiple services simultaneously. The GPU handles inference; the CPU cores handle data prep and post-processing.

Ubuntu Box: The Backbone

This machine's superpower is reliability. It runs:

OmniParser — a UI parsing service that my automation agents use to "see" screens (runs on port 8100, I SSH-tunnel to it from the Mac)
CPU inference fallback — when the Windows GPU is rented out and the Mac is busy, the Ubuntu box runs minicpm-v for vision tasks. Slow (~180 seconds per query) but it works.
Background cron jobs — data scraping, health checks, monitoring scripts
Testing ground — I deploy new services here first because if something crashes, my main workflow isn't affected

It's the machine I interact with least directly, but it's the one I'd miss most if it went down.

How They Talk to Each Other

This was the hardest part to figure out. Three machines that don't communicate are just three isolated computers.

The network:

Mac Mini (192.168.1.102) ←→ Windows PC (192.168.1.106) ←→ Ubuntu (192.168.1.100)

All on the same local network. No VPN needed at home, but I set up Tailscale for remote access when I'm not home.

Ollama routing:
The Mac Mini's Ollama is configured with OLLAMA_HOST=0.0.0.0 so it's accessible from the other machines. Same for the Windows PC. My automation agent on the Mac Mini routes requests based on model size:

def get_ollama_url(model: str) -> str:
    """Route to the right machine based on model size."""
    small_models = ["qwen3:4b", "granite3.2-vision:2b", "moondream"]
    big_models = ["qwen3-coder:30b", "deepseek-r1:8b"]

    if model in big_models:
        return "http://192.168.1.106:11434"  # Windows GPU
    elif model in small_models:
        return "http://localhost:11434"       # Mac Mini local
    else:
        return "http://192.168.1.100:11434"   # Ubuntu fallback

Simple. No Kubernetes, no service mesh. Just a function that knows which machine has which model.

SSH tunnels for services:
The Ubuntu box runs OmniParser on port 8100, but I need it on the Mac Mini. Instead of exposing it publicly:

# Auto-reconnecting SSH tunnel (runs via launchd on Mac)
ssh -o ServerAliveInterval=60 -o ServerAliveCountMax=3 \
    -L 8100:localhost:8100 exp@192.168.1.100 -N

Now localhost:8100 on the Mac Mini reaches OmniParser on Ubuntu. Clean, secure, zero config on the Ubuntu side.

The Mistakes I Made

Mistake 1: Trying to make the Mac Mini do everything.
For the first month, I ran all inference on the Mac Mini. The 16GB unified memory seems like a lot until a 30B model is eating 20GB and your IDE is using another 4GB and macOS starts swapping to SSD. The SSD is fast, but swap is still swap. Code completion went from instant to 3-second delays.

Mistake 2: Not setting up Ollama access from day one.
I installed Ollama separately on each machine and used them in isolation for weeks. The moment I realized I could call the Windows Ollama from the Mac Mini was a lightbulb moment. The models were always there — I just wasn't using them.

Mistake 3: Ignoring the Ubuntu box because it's "slow."
CPU inference is slow. But "slow" is relative. A background task that takes 3 minutes on CPU doesn't matter if you're not waiting for it. I was treating every request as if it needed to be instant. Turns out, most don't.

Mistake 4: No fallback plan.
When the Windows PC rebooted for an update, my entire inference pipeline died because I had no fallback routing. Now: if Windows is down, Ubuntu takes over. If Ubuntu is down, the Mac runs a tiny model. Something always answers.

What This Setup Enables

Running three machines isn't about having the biggest, fastest setup. It's about never being blocked.

Windows GPU is rented out? Route to Mac or Ubuntu.
Mac Mini is busy with a build? Use the Windows GPU via network.
Ubuntu is running a heavy OmniParser job? Queue it, the Mac handles the request.
Need to generate 50 images for a client? Windows GPU does it in batch while I keep coding on the Mac.

And the cost breakdown:

Item	Monthly Cost
Mac Mini power	~$8
Windows PC power (idle + rental hours)	~$15-25
Ubuntu box power	~$5
Vast.ai earnings (offset)	~$50-130
Net	-$17 to -$97 (net profit)

The setup literally pays for itself. The Windows PC's GPU rental income exceeds the total power cost of all three machines.

Do You Need Three Machines?

Probably not. But you might need two.

If you're running AI on a laptop: get a cheap desktop (even without a GPU) and put it on your network as a background worker. Ollama on a second machine means your main machine stays responsive.

If you have a gaming PC: you already have a GPU. Install Ollama on it, set OLLAMA_HOST=0.0.0.0, and call it from your laptop. You now have a two-machine AI lab for zero additional cost.

The Ubuntu box in my setup could be a $50 Raspberry Pi 5 with 8GB RAM for the same purpose. It doesn't need to be fast. It needs to be there.

What I'm Adding Next

I have five more GPUs in storage (2× RTX 3070, 1× RTX 3080, 2× RTX 3060). The plan is to build an open-air rig and add them all to the Windows PC for both local inference and Vast.ai rental. That'd give me 62GB total VRAM — enough to run a 70B model locally or rent out as a cluster.

The Ubuntu box is getting a dedicated role as a monitoring and alerting server. Right now my health checks are scattered across cron jobs on all three machines. Centralizing them makes debugging easier.

And the Mac Mini? It stays the brain. More RAM would be nice (16GB is tight with Ollama + IDE + browser), but the M4's efficiency is hard to beat for interactive work.

The Takeaway

One machine is a workstation. Two machines is a lab. Three machines is a distributed system that happens to fit on a desk.

The magic isn't in the hardware — it's in the routing. Knowing which machine should handle which task, building fallback paths, and making sure a single machine going down doesn't kill your workflow.

Start with what you have. Add a second machine when you feel the friction. Route intelligently. The three-machine setup happened organically — each machine earned its place by solving a problem the others couldn't.

I write about running AI locally, home lab setups, and turning hardware into income. New post every few days — follow along if that's your thing.

My GPU rental story: One card → six cards → $250/month passive

From Idea to Deployed Tool in 3 Hours — How AI Coding Agents Changed My Workflow

Sam Hartley — Tue, 05 May 2026 08:06:19 +0000

I used to think AI coding assistants were autocomplete on steroids. Fancy IntelliSense. Then I tried using one as an actual junior developer — someone who writes the first draft while I review and refine.

Two months later, my workflow is unrecognizable. I just shipped a complete B2B configuration tool — interactive maps, zone polygons, dynamic forms, the works — in under three hours. Here's what changed.

The Old Way

Before AI agents, my process for a new tool looked like this:

Research — How does Leaflet.js work? What's the API for geo polygons? Stack Overflow, docs, tutorials. 45 minutes.
Boilerplate — HTML structure, CSS grid, JavaScript imports, event listeners. 30 minutes.
Core logic — The actual thing the tool needs to do. 2–3 hours.
Debugging — Why doesn't the map render? Why is the polygon offset? 1–2 hours.
Polish — Styling, responsive layout, edge cases. 1 hour.

Total: 6–8 hours for a medium-complexity tool. And that's if I know the stack. If it's something new (like Garmin's Monkey C or a mapping library I haven't used), double it.

The New Way

Last week a client asked for a heat zone map for 21 European countries. Click a country, see the heating zones, pick one, get the right configuration. With polygon boundaries, country-specific defaults, and a responsive UI.

Here's how it went:

Hour 0–0.5: Prompt engineering
I wrote a detailed spec. Not "make a map" — that's useless. I described:

The data structure (country → zones → polygon coordinates)
The UI flow (dropdown → map render → zone selection)
The tech stack (Leaflet.js, vanilla JS, no frameworks)
Edge cases (what happens when a country has no zones?)

Hour 0.5–1.5: First draft from the agent
I fed the spec to Codex (Claude Code via CLI). It generated the full HTML file — 800+ lines — with Leaflet integration, zone polygons, event handlers, and the selection logic.

Was it perfect? No. The polygon coordinates were placeholder circles. The styling was bare-bones. But the architecture was right. The map rendered. The flow worked.

Hour 1.5–2.5: My turn — polish and fix
I replaced the placeholder polygons with real GeoJSON-ish coordinates for all 21 countries. Tweaked the CSS for mobile. Added validation. Fixed a bug where the map didn't re-center when switching countries.

Hour 2.5–3: Integration and deploy
Hooked it into the existing project structure. Git commit. Done.

Total: 3 hours. And the heavy lifting — the Leaflet setup, the polygon rendering logic, the event wiring — was handled by the agent. I did the creative/problem-solving work: defining the problem, validating the output, fixing the edge cases.

What Actually Works (and What Doesn't)

After two months of daily use, here's my honest assessment:

✅ What works brilliantly:

Boilerplate and plumbing — Setting up projects, imports, basic structure. The agent is faster and makes fewer typos than me.
API integration patterns — "Here's an endpoint, here's the expected response, write the fetch and parse logic." It gets this right 90% of the time.
Refactoring — "Rename this function and update all callers across 5 files." Instant, error-free.
Exploring unfamiliar territory — I hadn't used Leaflet in years. The agent got me to a working state without reading docs for an hour.

❌ What doesn't work (yet):

Complex business logic — Anything with nuanced rules, edge cases, or domain-specific constraints. The agent generates something plausible that breaks in production.
UI/UX design — It makes functional UIs. They look like a developer made them (because one did). You'll still need a human eye for polish.
Debugging its own mistakes — When the agent writes a bug, it's often subtle. You need to understand the code to catch it. This is not a replacement for knowing how to code.
Large-scale architecture — It works file-by-file. Designing a system with proper separation of concerns, caching strategies, and scalability? That's still on you.

The Mindset Shift

The biggest change isn't speed. It's how I think about problems.

Before: "How do I implement this?" → Research → Code → Debug.
Now: "How do I describe this so an agent can implement a first draft?" → Spec → Review → Refine.

I'm the architect and editor now, not the typist. The agent is the junior dev who writes fast but needs supervision.

This matters because it scales. I can explore 3 approaches in the time it used to take to build 1. I can say "what if we used a canvas instead of Leaflet?" and get a working comparison in 10 minutes. The cost of experimentation dropped to near-zero.

The Setup

If you want to try this:

Use Claude Code or Codex CLI — The terminal interface lets you iterate fast. Chat-based tools (ChatGPT, etc.) are too slow for code generation.
Write detailed specs — The agent is only as good as your prompt. Include examples, expected outputs, and constraints.
Review every line — Don't blindly commit. The agent writes plausible-looking bugs.
Keep a "golden test" — A known input/output pair you can run after every change. Catches regressions instantly.
Iterate in small chunks — "Add the map" → review → "Add zone polygons" → review. Don't ask for 500 lines at once.

What I'd Do Differently

Two months in, my main learning: the spec is everything.

My fastest sessions happen when I spend 10 minutes writing a bullet-point spec before touching the agent. My slowest sessions happen when I vague-prompt my way through and spend an hour fixing misunderstandings.

The second learning: agents excel at breadth, humans at depth. Use the agent to explore options. Use your brain to pick the right one.

If you're using AI coding agents, what's your experience? I'm curious if the "architect + editor" model resonates, or if you've found a different pattern that works. Drop your thoughts below — I'm still figuring this out too.

I Built a Model Router That Picks the Right AI for Every Task — Here's Why You Should Too

Sam Hartley — Sun, 03 May 2026 08:01:54 +0000

I caught myself doing something stupid the other day. I used a 30B parameter model to summarize a two-paragraph email. The GPU spun up, the fans kicked in, and 8 seconds later I had my summary. The same summary my 4B model would've given me in 0.3 seconds.

That's when I realized: I'd been treating every AI request the same way. Big model for everything. Small model for nothing. No intelligence in the routing at all.

So I built a model router. Not a fancy orchestrator with Kubernetes and service meshes — just a simple function that looks at what you're asking and sends it to the right model on the right machine. It's the single most impactful thing I've done for my local AI setup this year.

The Problem

I run three machines with Ollama (Mac Mini M4, Windows PC with RTX 3060, Ubuntu box). Between them, I have maybe 8 models installed. Before the router, my workflow was:

Need something → open terminal
Think about which model is good enough
Think about which machine has it
Type the full URL: curl http://192.168.1.106:11434/api/generate -d '{"model": "qwen3-coder:30b", ...}'
Wait

Steps 2-4 happened every single time. I was spending more mental energy on routing than on the actual task. And half the time I'd default to the biggest model just because "it's probably better."

Here's the thing: it's usually not better. A 4B model summarizing text is 95% as good as a 30B model summarizing text. But it's 25x faster and uses 1/10th the resources. The big model should earn its keep on tasks where size actually matters.

The Router

I started with a Python function. Nothing fancy:

import re

# Model registry: what's available where
MODELS = {
    "quick": {
        "model": "qwen3:4b",
        "endpoint": "http://localhost:11434",
    },
    "vision": {
        "model": "granite3.2-vision:2b",
        "endpoint": "http://192.168.1.106:11434",
    },
    "code": {
        "model": "qwen3-coder:30b",
        "endpoint": "http://192.168.1.106:11434",
    },
    "reasoning": {
        "model": "deepseek-r1:8b",
        "endpoint": "http://192.168.1.106:11434",
    },
    "fallback": {
        "model": "minicpm-v",
        "endpoint": "http://192.168.1.100:11434",
    },
}

def route(prompt: str) -> dict:
    p = prompt.lower()
    if any(w in p for w in ["image", "screenshot", "picture"]):
        return MODELS["vision"]
    if any(w in p for w in ["function", "class", "debug", "refactor", "implement"]):
        return MODELS["code"]
    if any(w in p for w in ["prove", "solve", "calculate", "math", "logic"]):
        return MODELS["reasoning"]
    return MODELS["quick"]

That's it. That's the whole router. It classifies prompts by keywords and sends them to the appropriate model. Is it perfect? No. Does it need to be? Also no.

Why This Matters More Than You Think

Before the router, my typical day looked like this:

30 quick questions (summarize this, what does this error mean, rephrase this)
5 code tasks (write a function, debug this, add tests)
2 vision tasks (what's in this screenshot)
1 deep reasoning task (complex analysis)

Without routing, I'd use the big model for everything. 38 requests to the 30B model. Each taking 5-15 seconds. Total wait time: ~4-5 minutes of just... waiting.

With routing:

30 quick → 4B model on Mac Mini, 0.3s each → 9 seconds total
5 code → 30B on Windows GPU, 8-12s each → ~50 seconds
2 vision → vision model on GPU, 4-13s each → ~15 seconds
1 reasoning → 8B model on GPU, 5-8s → ~6 seconds

Total: ~80 seconds instead of ~5 minutes. And my GPU is free for most of that time.

The Fallback System

The real value hit me when the Windows PC went down for a Windows Update mid-session. Before, this was catastrophic.

Now the router has fallback logic — when a machine is down, it finds the next best option. Code tasks go to the small model (worse but functional). Vision tasks try Ubuntu. Quick tasks keep humming on the Mac Mini. Something always answers.

The Token Cost Angle

I track tokens per model per day. Not because I'm cheap (these are all free — local), but because it tells me where I'm spending compute:

After a week: 72% of my requests went to the quick model. Only 15% needed the big code model.

I was using a sledgehammer for 72% of my nails. No wonder the GPU always felt busy.

What I'd Do Differently

1. Build the router on day one. It took an afternoon. I wasted months manually routing.

2. Start with keyword routing. I considered embeddings, classifiers, even using an LLM to pick the LLM. Keywords work for 90% of cases. Ship the simple thing.

3. Make the fallback automatic. My first version just errored when the GPU machine was down. A degraded response is infinitely better than no response.

4. Log everything. You can't optimize what you don't measure. The 72% stat jumped out immediately once I started tracking.

Beyond Keywords

The keyword router works but has blind spots. So I'm adding:

Confidence scoring: If a prompt matches multiple categories, try the cheaper model first. Auto-retry with bigger model if quality seems low.
Context-aware routing: If the last 3 prompts were about the same codebase, keep using the code model even without explicit keywords.
Cost-aware fallback: When local models can't handle something, the router should know whether a cloud API call is "worth it" or not.

Do You Need This?

If you run one model on one machine: no. You're fine.

If you have more than 2 models: yes. Otherwise you'll default to the biggest one every time, and that's a waste.

If you have multiple machines: absolutely. The router isn't just about picking the right model — it's about picking the right machine. Code tasks go where the GPU is. Quick tasks stay local. Background tasks go to the always-on box.

Start with 2 models and a simple if/else. Add more as you grow. The architecture stays the same.

I write about running AI locally, home lab setups, and turning hardware into income. If that's your jam, I post every few days.

I Built a Model Router That Picks the Right AI for Every Task — Here's Why You Should Too

Sam Hartley — Fri, 01 May 2026 08:02:21 +0000

That's when I realized: I'd been treating every AI request the same way. Big model for everything. Small model for nothing. No intelligence in the routing at all.

The Problem

I run three machines with Ollama (Mac Mini M4, Windows PC with RTX 3060, Ubuntu box). Between them, I have maybe 8 models installed. Before the router, my workflow was:

Need something → open terminal
Think about which model is good enough
Think about which machine has it
Type the full URL: curl http://192.168.1.106:11434/api/generate -d '{\"model\": \"qwen3-coder:30b\", ...}'
Wait

Steps 2-4 happened every single time. I was spending more mental energy on routing than on the actual task. And half the time I'd default to the biggest model just because "it's probably better."

The Router

I started with a Python function. Nothing fancy:

import re

# Model registry: what's available where
MODELS = {
    # Quick chat, summaries, simple Q&A → Mac Mini (fast response)
    "quick": {
        "model": "qwen3:4b",
        "endpoint": "http://localhost:11434",
    },
    # Vision / image analysis → Windows GPU (needs VRAM)
    "vision": {
        "model": "granite3.2-vision:2b",
        "endpoint": "http://192.168.1.106:11434",
    },
    # Code generation, complex reasoning → Windows GPU
    "code": {
        "model": "qwen3-coder:30b",
        "endpoint": "http://192.168.1.106:11434",
    },
    # Deep reasoning, math, logic puzzles → Windows GPU
    "reasoning": {
        "model": "deepseek-r1:8b",
        "endpoint": "http://192.168.1.106:11434",
    },
    # Fallback for when Windows is rented out → Ubuntu CPU
    "fallback": {
        "model": "minicpm-v",
        "endpoint": "http://192.168.1.100:11434",
    },
}

def route(prompt: str) -> dict:
    """Decide which model should handle this prompt."""
    p = prompt.lower()

    # Vision tasks
    if any(w in p for w in ["image", "screenshot", "picture", "see this", "what's on"]):
        return MODELS["vision"]

    # Code tasks
    if any(w in p for w in ["function", "class", "debug", "refactor", "implement",
                             "write code", "python", "javascript", "typescript",
                             "api endpoint", "sql", "regex"]):
        return MODELS["code"]

    # Reasoning tasks
    if any(w in p for w in ["prove", "solve", "calculate", "math", "logic",
                             "reason", "analyze", "compare", "evaluate"]):
        return MODELS["reasoning"]

    # Default: quick model for everything else
    return MODELS["quick"]

That's it. That's the whole router. It classifies prompts by keywords and sends them to the appropriate model. Is it perfect? No. Does it need to be? Also no.

Why This Matters More Than You Think

Before the router, my typical day looked like this:

30 quick questions (summarize this, what does this error mean, rephrase this)
5 code tasks (write a function, debug this, add tests)
2 vision tasks (what's in this screenshot)
1 deep reasoning task (complex analysis)

Without routing, I'd use the big model for everything. 38 requests to the 30B model. Each taking 5-15 seconds. Total wait time: ~4-5 minutes of just... waiting. While my GPU is maxed out and I can't run anything else.

With routing:

30 quick → 4B model on Mac Mini, 0.3s each → 9 seconds total
5 code → 30B on Windows GPU, 8-12s each → ~50 seconds
2 vision → vision model on GPU, 4-13s each → ~15 seconds
1 reasoning → 8B model on GPU, 5-8s → ~6 seconds

Total: ~80 seconds instead of ~5 minutes. And my GPU is free for most of that time, so I could be generating images or renting it out simultaneously.

The router doesn't just save time. It frees capacity.

The Fallback System

The real value of routing hit me when the Windows PC went down for a Windows Update mid-session. Before, this was catastrophic — all my "important" models were on that machine.

Now the router has fallback logic:

def route_with_fallback(prompt: str) -> dict:
    """Route with health checks and fallbacks."""
    primary = route(prompt)

    # Check if primary endpoint is alive
    try:
        requests.get(f"{primary['endpoint']}/api/tags", timeout=2)
        return primary
    except:
        pass  # Primary is down, find fallback

    # Fallback chain based on task type
    if primary == MODELS["code"]:
        # Can't run 30B on Mac, but 4B is better than nothing
        return MODELS["quick"]
    elif primary == MODELS["vision"]:
        # Try Ubuntu's slow vision model
        try:
            requests.get(f"{MODELS['fallback']['endpoint']}/api/tags", timeout=2)
            return MODELS["fallback"]
        except:
            return MODELS["quick"]  # No vision available, use text
    elif primary == MODELS["reasoning"]:
        return MODELS["quick"]  # Quick model can reason a bit

    return MODELS["quick"]

When Windows reboots, my workflow doesn't stop. The router sends code tasks to the small model (worse but functional), vision tasks to Ubuntu (slow but works), and quick tasks keep humming along on the Mac Mini. Something always answers.

The Token Cost Angle

I also track tokens per model per day. Not because I'm cheap (these are all free — they run locally), but because it tells me where I'm spending compute:

# Simple token counter per route
token_log = defaultdict(int)

def track_tokens(route_name: str, tokens: int):
    token_log[route_name] += tokens
    # Log daily totals to a file
    with open(f"token-usage-{date.today()}.jsonl", "a") as f:
        f.write(json.dumps({"route": route_name, "tokens": tokens, "ts": time.time()}) + "\n")

After a week of tracking, I found: 72% of my requests were going to the quick model. Only 15% actually needed the big code model. 8% were vision. 5% reasoning.

I was using a sledgehammer for 72% of my nails. No wonder the GPU always felt busy.

What I'd Do Differently If I Started Over

1. Build the router on day one, not month three.

I wasted months manually routing. The router took an afternoon to write. If you're running multiple models, build the routing first. You can always make it smarter later.

2. Start with keyword routing. Don't over-engineer it.

I considered using embeddings to classify prompts. I considered training a tiny classifier. I considered using an LLM to decide which LLM to use (yes, really). Keyword matching works for 90% of cases. Ship the simple thing.

3. Make the fallback automatic.

My first version just errored when the GPU machine was down. The fallback logic was an afterthought. It should've been the first thing I built — because machines go down, and a degraded response is infinitely better than no response.

4. Log everything.

You can't optimize what you don't measure. Once I started logging which routes were used, the 72% quick-model stat jumped out immediately. Without logs, I would've kept thinking I needed the big model "most of the time."

Beyond Keywords: What I'm Adding Next

The keyword router works, but it has blind spots. A prompt like "help me understand why this function is slow" could be code OR reasoning. So I'm adding:

Confidence scoring: If the prompt matches multiple categories, route to the cheaper model first. If the response quality seems low (response is very short, model seems confused), auto-retry with the bigger model.

Context-aware routing: If the last 3 prompts were all about the same codebase, keep using the code model even if the current prompt doesn't have code keywords. Context continuity matters.

Cost-aware routing for cloud fallback: When local models can't handle something, I fall back to cloud APIs. The router should know: "this task isn't worth a $0.03 GPT-4 call, use the local model." Or: "this is worth it, spend the API credits."

Do You Need This?

If you run one model on one machine: no. You're fine. The router solves a multi-model, multi-machine problem.

If you're starting to accumulate models (a small one for chat, a big one for code, a vision model): yes. The moment you have more than 2 models, you need routing. Otherwise you'll default to the biggest one every time, and that's a waste.

The beauty is that the router grows with you. Start with 2 models and a simple if/else. Add more models, add more rules. Add more machines, add more endpoints. The architecture stays the same.

I write about running AI locally, home lab setups, and turning hardware into income. If that's your jam, I post every few days.

My AI lab setup: 3 machines, 1 desk, zero cloud dependency