DEV Community

Gabriele Di Maria
Gabriele Di Maria

Posted on

How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)

#ai #automation #python #selfhosted

How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)

Published: 2026-09-25 | ~2,200 words | 11 min read

Why This Exists

I needed AI agents that could actually do things — run code, call APIs, browse the web, write files — not just chat. Cloud APIs got expensive fast, rate-limited unpredictably, and locked me into someone else's infrastructure.

So I built a local stack on my Windows machine (i7-8700K, 32GB RAM, RTX 3080). Total monthly cost: $0. Everything runs locally or on free tiers.

This article documents what actually works, what broke repeatedly, and the solutions that stuck.

The Stack (What's Actually Running) 

┌─────────────────────────────────────────────────────────────────┐
│                      ORCHESTRATION                               │
│  Windmill (localhost:8000) — 3 containers                       │
│  - windmill-postgres, windmill-server, windmill-worker          │
│  - Workspace: admins, NO_AUTH=true (dev only)                   │
│  - 13 scripts deployed: 7 pillars + 4 automations + 1 reviewer  │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      MODEL LAYER                                 │
│  Primary: nvidia/nemotron-3-super-120b-a12b                     │
│  Fallback 1: openai/gpt-oss-120b (Groq)                         │
│  Fallback 2: openai/gpt-oss-20b (Groq)                          │
│  Reviewer: nvidia/nemotron-3-ultra-550b-a55b (adversarial)      │
│                                                                  │
│  Local Proxy: 127.0.0.1:8001 (OpenAI-compatible)                │
│  - 10 NVIDIA API keys in rotation                               │
│  - 6-model cascade on failure                                   │
│  - Tool calling + SSE streaming support                         │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      AGENT INTERFACE                             │
│  Hermes Agent (Nous Research) — Telegram + local CLI            │
│  - Configured with custom provider → local proxy                │
│  - Fallback providers: Groq (gpt-oss-120b → gpt-oss-20b)       │
│  - Auxiliary model DISABLED (caused rate loops)                 │
│  - Bot Mode: 3 bots configured on Ultra 550B (see below)        │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      TOOLS & AUTOMATION                          │
│  Webwright (Playwright) — browser automation                    │
│  NVIDIA Reviewer — adversarial code review via Windmill         │
│  Edge-TTS — local voice synthesis (Microsoft, free)             │
│  FFmpeg — video assembly                                        │
│  Retry watcher — cron every 2 min, Telegram alerts              │
└─────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

The Model Layer: Getting Reliability from Free Tiers

The Core Problem

NVIDIA NIM free tier: 40 RPM. That's it. One agent loop can burn 10 requests in seconds. Groq free tier is generous but has different models. OpenRouter credits exhausted.

Solution 1: 10-Key Rotation (nvidia_key_rotation.py) 

# Simplified logic
class KeyRotator:
    def __init__(self, keys: list[str]):
        self.keys = keys
        self.state = {k: {"fails": 0, "cooldown_until": 0} for k in keys}
        self.current_idx = 0

    def get_key(self) -> str:
        now = time.time()
        for _ in range(len(self.keys)):
            key = self.keys[self.current_idx]
            if self.state[key]["cooldown_until"] <= now:
                return key
            self.current_idx = (self.current_idx + 1) % len(self.keys)
        raise Exception("All keys in cooldown")

    def report_result(self, key: str, status: int):
        if status == 429:
            self.state[key]["cooldown_until"] = time.time() + 30
            self.state[key]["fails"] += 1
        elif status in (401, 403, 410):
            self.state[key]["fails"] = 999  # permanent skip
        else:
            self.state[key]["fails"] = max(0, self.state[key]["fails"] - 1)
Enter fullscreen mode Exit fullscreen mode

Result: 429 errors become 30-second cooldowns instead of hard stops. Keys that return 401/403/410 (EOL models) are permanently skipped.

Solution 2: 6-Model Cascade (nvidia_cascade.py)

When a key works but the model fails (timeout, 500, bad output), cascade to the next model:

ACTIVE_MODELS = [
    "nvidia/nemotron-3.5-lightning-30b-a3b",   # fastest, good for simple
    "openai/gpt-oss-20b",                       # Groq fallback via proxy
    "nvidia/nemotron-3-super-120b-a12b",       # primary workhorse
    "openai/gpt-oss-120b",                      # Groq fallback via proxy
    "glm-5.2",                                  # backup
    "nvidia/nemotron-3-ultra-550b-a55b",       # heavy reasoning, reviewer
]
Enter fullscreen mode Exit fullscreen mode

Each model tried in order. First success wins. Logs show which model actually responded.

Solution 3: Local Proxy with Tool Calling + SSE (nvidia_proxy.py)

This was the hardest part. Hermes sends tool calls; the proxy must forward them to NVIDIA and return tool_calls in the response. Also must stream SSE chunks correctly.

Key fixes that took days:

  • Forward tools parameter in request body to NVIDIA
  • Parse NVIDIA's response: if tool_calls present, return as-is (don't wrap in text)
  • SSE: yield chunks with data: {json}\n\n, end with data: [DONE]\n\n
  • Handle finish_reason: tool_calls correctly

Without these, Hermes either got empty streams or couldn't execute tools.

Solution 4: Groq Fallback in Hermes Config 

# Hermes config.yaml (relevant section)
model: "nvidia/nemotron-3-super-120b-a12b"
provider: "custom"
custom:
  base_url: "http://127.0.0.1:8001/v1"
fallback_providers:
  - name: "groq"
    models:
      - "openai/gpt-oss-120b"
      - "openai/gpt-oss-20b"
Enter fullscreen mode Exit fullscreen mode

Behavior: if primary (proxy) fails, Hermes automatically retries on Groq. Next turn, it tries primary again.

The Orchestration Layer: Windmill

Why Windmill?

  • Self-hosted, no cloud dependency
  • Scripts as version-controlled Python files
  • Built-in retries, approval gates, schedules
  • Variables for secrets (API keys, tokens)
  • UI for monitoring job runs

The 10-Pillar Attempt (7 Real, 3 Archived)

Pillar Purpose Status
1 Approval Gate (human-in-the-loop) ✅ Working, tested end-to-end
2 Alternative Supervision ❌ Blocked — needs API keys
3 Self-Healing 📦 Archived — no Docker socket in Windmill CE
4 Resilience/Queue ✅ Restored via API (job 01a0d246)
5 Watchdog/Resume 📦 Archived — same as Pillar 3
6 Cleanup/Mutex ⚠️ Partial — cleanup paths wrong, mutex OK
7 Sandbox ⚠️ Partial — shell escape possible
8, 9, 10 — ❌ Never deployed

Reality: Only Pillars 1 and 4 are production-ready. The rest need Docker socket access (Windmill EE feature) or significant rework.

What Windmill Actually Runs

  • retry_watcher.py — cron every 2 min, checks failed jobs, retries, notifies Telegram
  • nvidia_reviewer.py — adversarial code review (called via webhook or manually)
  • Webhook comando-gabriele — receives signed Telegram commands, executes terminal/file ops

The Agent Interface: Hermes Agent

Configuration That Works 

# ~/.config/hermes/config.yaml (key sections)
model: "nvidia/nemotron-3-super-120b-a12b"
provider: "custom"
custom:
  base_url: "http://127.0.0.1:8001/v1"
  api_key: "not-needed"  # proxy handles auth

fallback_providers:
  - name: "groq"
    models:
      - "openai/gpt-oss-120b"
      - "openai/gpt-oss-20b"

auxiliary:
  title_generation:
    enabled: false  # critical: was causing rate loops
Enter fullscreen mode Exit fullscreen mode

Bot Mode (Actually Configured)

3 bots, all on nvidia/nemotron-3-ultra-550b-a55b (strongest model via proxy):

  • @direttore — strategic coordinator, custom SOUL.md
  • @analista — market/technical analysis, custom SOUL.md
  • @critico — adversarial review, custom SOUL.md

Group chat "Progetto ReportForge" completed 3 rounds, converged on positioning. Cron on @direttore runs every 24h.

Limitation: Bots share the same proxy/key pool. Heavy concurrent use hits rate limits. Serial execution (max 3 rounds) mitigates this.

Real Problems & Solutions

1. Rate Limits → Proxy + Rotation + Cascade + Fallback

Before: 429 errors killed agent loops. Manual key switching.
After: Automatic. 10 keys, 30s cooldown on 429, cascade across 6 models, Groq fallback. Uptime >99% on free tiers.

2. Hallucinations → NVIDIA Reviewer + Model 550B

Problem: Nemotron-Super-120b hallucinates function signatures, file paths, config options when context grows.
Fix:

  • Keep sessions short (one task per conversation)
  • Use /new frequently
  • Run adversarial review on critical code via nvidia_reviewer (Ultra 550B, temp 0.1)
  • Reviewer prompt: "Find errors, risks, hidden assumptions. Don't be nice. Be concise."

3. Tool Calling Broken → Proxy Rewrite

Problem: Proxy returned text instead of tool_calls. Hermes waited forever.
Fix: Proxy now passes tools param to NVIDIA, returns raw tool_calls array. Verified with Hermes tool execution.

4. Empty Streams → SSE Implementation

Problem: "Empty stream" errors. NVIDIA sends SSE; proxy wasn't forwarding chunks correctly.
Fix: Proper chunk parsing, data: prefix, [DONE] terminator. Hermes now receives stream correctly.

5. Model EOL (410 Gone) → Active List Maintenance

Burned: meta/llama-3.1-*, meta/llama-3.3-* (Aug 26, 2026), openai/gpt-oss-120b on NVIDIA (Sep 3, 2026).
Process: Before using any model, curl test. 410 = remove from cascade. Current active list maintained in nvidia_cascade.py.

6. BOM in JSON → UTF-8 Without BOM

Problem: PowerShell Out-File -Encoding UTF8 adds BOM (EF BB BF). Python json.loads fails silently.
Fix: Always write with [System.IO.File]::WriteAllText($path, $content, $utf8NoBom) where $utf8NoBom = New-Object System.Text.UTF8Encoding $false.

7. Windmill DB via psql → API Only

Mistake: Direct psql inserts. Windmill cache didn't refresh. Scripts invisible in UI.
Rule: Only official Windmill API (CLI or HTTP). If API fails, stop and ask.

Numbers That Are Real

Metric Value Verified
NVIDIA API keys 10 API_key.txt
Active models in cascade 6 nvidia_cascade.py
Fallback models (Groq) 2 Hermes config
Windmill containers 3 healthy docker ps
Deployed scripts 13 Windmill UI
Pillar 1 test PASS Job 01a0d0cf
Pillar 4 restored PASS Job 01a0d246
NVIDIA Reviewer test PASS Found bug in sum(a,b): return a-b
Webwright + Reviewer integration PASS 10 issues found in git-filter-branch script
Monthly cost $0 No paid services used
Disk C: free 36% df -h
Disk D: free 35% df -h
Disk E: free 66% df -h

What Works Well

  1. Local proxy + key rotation — transforms brittle free tier into reliable service
  2. Windmill for orchestration — scripts as code, visible runs, retries built-in
  3. Hermes + custom provider — full agent capability (tools, files, terminal) on local models
  4. NVIDIA Reviewer (Ultra 550B) — catches real bugs, costs $0
  5. Webwright — browser automation that LLMs can actually drive (writes Playwright code)
  6. Bot Mode — multi-perspective analysis without human moderation
  7. Retry watcher + Telegram — self-healing for transient failures

What Doesn't Work / Needs Work

  1. Pillars 2, 3, 5, 7, 8, 9, 10 — mostly need Docker socket (EE) or redesign for CE
  2. Sandbox isolation — shell escape possible in current pillar 8
  3. Hermes context overflow — long sessions degrade quality; /new required frequently
  4. Concurrent bot usage — shares key pool, hits rate limits
  5. No schedules in Windmill — 0 created, retry watcher is external cron
  6. CBAPI MCP — package has no CLI entry point; REST API works but ClickBank-focused (not SaaS)
  7. AgentFuse affiliation — pending approval (24h), MCP ready to install

What I'd Do Differently

Decision Would Change To
Started with 10-pillar design Start with 2-3 pillars that solve immediate pain
Built custom proxy from scratch Use existing OpenAI-compatible proxy if one supported tool calling + SSE
Put all models on same key pool Separate key pools per model tier
Used Windmill CE for self-healing Accept CE limits; build external watchers instead
Tried to automate everything Keep human-in-the-loop (Pillar 1) for critical gates

Operational Checklist: Replicate This Stack

  1. Hardware: Any machine with 16GB+ RAM, decent CPU. GPU optional (NIM runs on CPU).
  2. Docker Desktop → WSL2 backend → docker-compose up for Windmill
  3. NVIDIA API keys → 10 free keys from build.nvidia.com → API_key.txt
  4. Proxy → nvidia_proxy.py on 127.0.0.1:8001 (tool calling + SSE)
  5. Rotation → nvidia_key_rotation.py + nvidia_cascade.py
  6. Hermes → Configure custom provider → proxy URL, add Groq fallbacks
  7. Windmill Variables → Store Telegram token, NVIDIA reviewer key as secrets
  8. Webwright → pip install webwright → skill for Hermes
  9. NVIDIA Reviewer → Windmill script calling NIM Ultra 550B with adversarial prompt
  10. Retry watcher → Windows Task Scheduler or cron → Telegram alerts

What's Next

  1. AgentFuse affiliation — if approved, install MCP, generate tracked links for SaaS tools developers actually use (Cursor, Vercel, Beehiiv, ConvertKit, Linear, PostHog)
  2. Dev.to article — this one, published honestly
  3. Micro-tool packaging — ReportForge (GitHub/Jira weekly reports) ready for Gumroad at €29/€39
  4. Video engine — same stack, packaged as CLI (faceless video automation)
  5. Simplify pillars — archive unused, harden the 2 that work

Resources

Closing

This stack exists because I refused to pay $500+/month for agent infrastructure that randomly rate-limits or changes APIs. It's not magic — it's plumbing. Lots of plumbing. But it works, it's mine, and it costs nothing.

If you build something similar: test every model before trusting it, log every fallback, and keep sessions short. The free tiers are generous but unforgiving.

Built on Windows 11, Docker Desktop, WSL2. Zero cloud dependencies. Zero subscriptions.

Discuss on Dev.to | GitHub | Telegram

Top comments (0)