How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)

Gabriele Di Maria — Fri, 25 Sep 2026 16:00:28 +0000

How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)

Published: 2026-09-25 | ~2,200 words | 11 min read

Why This Exists

I needed AI agents that could actually do things — run code, call APIs, browse the web, write files — not just chat. Cloud APIs got expensive fast, rate-limited unpredictably, and locked me into someone else's infrastructure.

So I built a local stack on my Windows machine (i7-8700K, 32GB RAM, RTX 3080). Total monthly cost: $0. Everything runs locally or on free tiers.

This article documents what actually works, what broke repeatedly, and the solutions that stuck.

The Stack (What's Actually Running)

┌─────────────────────────────────────────────────────────────────┐
│                      ORCHESTRATION                               │
│  Windmill (localhost:8000) — 3 containers                       │
│  - windmill-postgres, windmill-server, windmill-worker          │
│  - Workspace: admins, NO_AUTH=true (dev only)                   │
│  - 13 scripts deployed: 7 pillars + 4 automations + 1 reviewer  │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      MODEL LAYER                                 │
│  Primary: nvidia/nemotron-3-super-120b-a12b                     │
│  Fallback 1: openai/gpt-oss-120b (Groq)                         │
│  Fallback 2: openai/gpt-oss-20b (Groq)                          │
│  Reviewer: nvidia/nemotron-3-ultra-550b-a55b (adversarial)      │
│                                                                  │
│  Local Proxy: 127.0.0.1:8001 (OpenAI-compatible)                │
│  - 10 NVIDIA API keys in rotation                               │
│  - 6-model cascade on failure                                   │
│  - Tool calling + SSE streaming support                         │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      AGENT INTERFACE                             │
│  Hermes Agent (Nous Research) — Telegram + local CLI            │
│  - Configured with custom provider → local proxy                │
│  - Fallback providers: Groq (gpt-oss-120b → gpt-oss-20b)       │
│  - Auxiliary model DISABLED (caused rate loops)                 │
│  - Bot Mode: 3 bots configured on Ultra 550B (see below)        │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      TOOLS & AUTOMATION                          │
│  Webwright (Playwright) — browser automation                    │
│  NVIDIA Reviewer — adversarial code review via Windmill         │
│  Edge-TTS — local voice synthesis (Microsoft, free)             │
│  FFmpeg — video assembly                                        │
│  Retry watcher — cron every 2 min, Telegram alerts              │
└─────────────────────────────────────────────────────────────────┘

The Model Layer: Getting Reliability from Free Tiers

The Core Problem

NVIDIA NIM free tier: 40 RPM. That's it. One agent loop can burn 10 requests in seconds. Groq free tier is generous but has different models. OpenRouter credits exhausted.

Solution 1: 10-Key Rotation (`nvidia_key_rotation.py`)

# Simplified logic
class KeyRotator:
    def __init__(self, keys: list[str]):
        self.keys = keys
        self.state = {k: {"fails": 0, "cooldown_until": 0} for k in keys}
        self.current_idx = 0

    def get_key(self) -> str:
        now = time.time()
        for _ in range(len(self.keys)):
            key = self.keys[self.current_idx]
            if self.state[key]["cooldown_until"] <= now:
                return key
            self.current_idx = (self.current_idx + 1) % len(self.keys)
        raise Exception("All keys in cooldown")

    def report_result(self, key: str, status: int):
        if status == 429:
            self.state[key]["cooldown_until"] = time.time() + 30
            self.state[key]["fails"] += 1
        elif status in (401, 403, 410):
            self.state[key]["fails"] = 999  # permanent skip
        else:
            self.state[key]["fails"] = max(0, self.state[key]["fails"] - 1)

Result: 429 errors become 30-second cooldowns instead of hard stops. Keys that return 401/403/410 (EOL models) are permanently skipped.

Solution 2: 6-Model Cascade (`nvidia_cascade.py`)

When a key works but the model fails (timeout, 500, bad output), cascade to the next model:

ACTIVE_MODELS = [
    "nvidia/nemotron-3.5-lightning-30b-a3b",   # fastest, good for simple
    "openai/gpt-oss-20b",                       # Groq fallback via proxy
    "nvidia/nemotron-3-super-120b-a12b",       # primary workhorse
    "openai/gpt-oss-120b",                      # Groq fallback via proxy
    "glm-5.2",                                  # backup
    "nvidia/nemotron-3-ultra-550b-a55b",       # heavy reasoning, reviewer
]

Each model tried in order. First success wins. Logs show which model actually responded.

Solution 3: Local Proxy with Tool Calling + SSE (`nvidia_proxy.py`)

This was the hardest part. Hermes sends tool calls; the proxy must forward them to NVIDIA and return tool_calls in the response. Also must stream SSE chunks correctly.

Key fixes that took days:

Forward tools parameter in request body to NVIDIA
Parse NVIDIA's response: if tool_calls present, return as-is (don't wrap in text)
SSE: yield chunks with data: {json}\n\n, end with data: [DONE]\n\n
Handle finish_reason: tool_calls correctly

Without these, Hermes either got empty streams or couldn't execute tools.

Solution 4: Groq Fallback in Hermes Config

# Hermes config.yaml (relevant section)
model: "nvidia/nemotron-3-super-120b-a12b"
provider: "custom"
custom:
  base_url: "http://127.0.0.1:8001/v1"
fallback_providers:
  - name: "groq"
    models:
      - "openai/gpt-oss-120b"
      - "openai/gpt-oss-20b"

Behavior: if primary (proxy) fails, Hermes automatically retries on Groq. Next turn, it tries primary again.

The Orchestration Layer: Windmill

Why Windmill?

Self-hosted, no cloud dependency
Scripts as version-controlled Python files
Built-in retries, approval gates, schedules
Variables for secrets (API keys, tokens)
UI for monitoring job runs

The 10-Pillar Attempt (7 Real, 3 Archived)

Pillar	Purpose	Status
1	Approval Gate (human-in-the-loop)	✅ Working, tested end-to-end
2	Alternative Supervision	❌ Blocked — needs API keys
3	Self-Healing	📦 Archived — no Docker socket in Windmill CE
4	Resilience/Queue	✅ Restored via API (job 01a0d246)
5	Watchdog/Resume	📦 Archived — same as Pillar 3
6	Cleanup/Mutex	⚠️ Partial — cleanup paths wrong, mutex OK
7	Sandbox	⚠️ Partial — shell escape possible
8, 9, 10	—	❌ Never deployed

Reality: Only Pillars 1 and 4 are production-ready. The rest need Docker socket access (Windmill EE feature) or significant rework.

What Windmill Actually Runs

retry_watcher.py — cron every 2 min, checks failed jobs, retries, notifies Telegram
nvidia_reviewer.py — adversarial code review (called via webhook or manually)
Webhook comando-gabriele — receives signed Telegram commands, executes terminal/file ops

The Agent Interface: Hermes Agent

Configuration That Works

# ~/.config/hermes/config.yaml (key sections)
model: "nvidia/nemotron-3-super-120b-a12b"
provider: "custom"
custom:
  base_url: "http://127.0.0.1:8001/v1"
  api_key: "not-needed"  # proxy handles auth

fallback_providers:
  - name: "groq"
    models:
      - "openai/gpt-oss-120b"
      - "openai/gpt-oss-20b"

auxiliary:
  title_generation:
    enabled: false  # critical: was causing rate loops

Bot Mode (Actually Configured)

3 bots, all on nvidia/nemotron-3-ultra-550b-a55b (strongest model via proxy):

@direttore — strategic coordinator, custom SOUL.md
@analista — market/technical analysis, custom SOUL.md
@critico — adversarial review, custom SOUL.md

Group chat "Progetto ReportForge" completed 3 rounds, converged on positioning. Cron on @direttore runs every 24h.

Limitation: Bots share the same proxy/key pool. Heavy concurrent use hits rate limits. Serial execution (max 3 rounds) mitigates this.

Real Problems & Solutions

1. Rate Limits → Proxy + Rotation + Cascade + Fallback

Before: 429 errors killed agent loops. Manual key switching.
After: Automatic. 10 keys, 30s cooldown on 429, cascade across 6 models, Groq fallback. Uptime >99% on free tiers.

2. Hallucinations → NVIDIA Reviewer + Model 550B

Problem: Nemotron-Super-120b hallucinates function signatures, file paths, config options when context grows.
Fix:

Keep sessions short (one task per conversation)
Use /new frequently
Run adversarial review on critical code via nvidia_reviewer (Ultra 550B, temp 0.1)
Reviewer prompt: "Find errors, risks, hidden assumptions. Don't be nice. Be concise."

3. Tool Calling Broken → Proxy Rewrite

Problem: Proxy returned text instead of tool_calls. Hermes waited forever.
Fix: Proxy now passes tools param to NVIDIA, returns raw tool_calls array. Verified with Hermes tool execution.

4. Empty Streams → SSE Implementation

Problem: "Empty stream" errors. NVIDIA sends SSE; proxy wasn't forwarding chunks correctly.
Fix: Proper chunk parsing, data: prefix, [DONE] terminator. Hermes now receives stream correctly.

5. Model EOL (410 Gone) → Active List Maintenance

Burned: meta/llama-3.1-*, meta/llama-3.3-* (Aug 26, 2026), openai/gpt-oss-120b on NVIDIA (Sep 3, 2026).
Process: Before using any model, curl test. 410 = remove from cascade. Current active list maintained in nvidia_cascade.py.

6. BOM in JSON → UTF-8 Without BOM

Problem: PowerShell Out-File -Encoding UTF8 adds BOM (EF BB BF). Python json.loads fails silently.
Fix: Always write with [System.IO.File]::WriteAllText($path, $content, $utf8NoBom) where $utf8NoBom = New-Object System.Text.UTF8Encoding $false.

7. Windmill DB via psql → API Only

Mistake: Direct psql inserts. Windmill cache didn't refresh. Scripts invisible in UI.
Rule: Only official Windmill API (CLI or HTTP). If API fails, stop and ask.

Numbers That Are Real

Metric	Value	Verified
NVIDIA API keys	10	`API_key.txt`
Active models in cascade	6	`nvidia_cascade.py`
Fallback models (Groq)	2	Hermes config
Windmill containers	3 healthy	`docker ps`
Deployed scripts	13	Windmill UI
Pillar 1 test	PASS	Job 01a0d0cf
Pillar 4 restored	PASS	Job 01a0d246
NVIDIA Reviewer test	PASS	Found bug in `sum(a,b): return a-b`
Webwright + Reviewer integration	PASS	10 issues found in git-filter-branch script
Monthly cost	$0	No paid services used
Disk C: free	36%	`df -h`
Disk D: free	35%	`df -h`
Disk E: free	66%	`df -h`

What Works Well

Local proxy + key rotation — transforms brittle free tier into reliable service
Windmill for orchestration — scripts as code, visible runs, retries built-in
Hermes + custom provider — full agent capability (tools, files, terminal) on local models
NVIDIA Reviewer (Ultra 550B) — catches real bugs, costs $0
Webwright — browser automation that LLMs can actually drive (writes Playwright code)
Bot Mode — multi-perspective analysis without human moderation
Retry watcher + Telegram — self-healing for transient failures

What Doesn't Work / Needs Work

Pillars 2, 3, 5, 7, 8, 9, 10 — mostly need Docker socket (EE) or redesign for CE
Sandbox isolation — shell escape possible in current pillar 8
Hermes context overflow — long sessions degrade quality; /new required frequently
Concurrent bot usage — shares key pool, hits rate limits
No schedules in Windmill — 0 created, retry watcher is external cron
CBAPI MCP — package has no CLI entry point; REST API works but ClickBank-focused (not SaaS)
AgentFuse affiliation — pending approval (24h), MCP ready to install

What I'd Do Differently

Decision	Would Change To
Started with 10-pillar design	Start with 2-3 pillars that solve immediate pain
Built custom proxy from scratch	Use existing OpenAI-compatible proxy if one supported tool calling + SSE
Put all models on same key pool	Separate key pools per model tier
Used Windmill CE for self-healing	Accept CE limits; build external watchers instead
Tried to automate everything	Keep human-in-the-loop (Pillar 1) for critical gates

Operational Checklist: Replicate This Stack

Hardware: Any machine with 16GB+ RAM, decent CPU. GPU optional (NIM runs on CPU).
Docker Desktop → WSL2 backend → docker-compose up for Windmill
NVIDIA API keys → 10 free keys from build.nvidia.com → API_key.txt
Proxy → nvidia_proxy.py on 127.0.0.1:8001 (tool calling + SSE)
Rotation → nvidia_key_rotation.py + nvidia_cascade.py
Hermes → Configure custom provider → proxy URL, add Groq fallbacks
Windmill Variables → Store Telegram token, NVIDIA reviewer key as secrets
Webwright → pip install webwright → skill for Hermes
NVIDIA Reviewer → Windmill script calling NIM Ultra 550B with adversarial prompt
Retry watcher → Windows Task Scheduler or cron → Telegram alerts

What's Next

AgentFuse affiliation — if approved, install MCP, generate tracked links for SaaS tools developers actually use (Cursor, Vercel, Beehiiv, ConvertKit, Linear, PostHog)
Dev.to article — this one, published honestly
Micro-tool packaging — ReportForge (GitHub/Jira weekly reports) ready for Gumroad at €29/€39
Video engine — same stack, packaged as CLI (faceless video automation)
Simplify pillars — archive unused, harden the 2 that work

Resources

Windmill: https://windmill.dev (self-hosted docs)
Hermes Agent: https://hermes-agent.nousresearch.com
NVIDIA NIM: https://build.nvidia.com (free tier: 40 RPM)
Groq: https://console.groq.com (free tier: generous)
Webwright: https://github.com/microsoft/webwright
Edge-TTS: pip install edge-tts (Microsoft, free)

Closing

This stack exists because I refused to pay $500+/month for agent infrastructure that randomly rate-limits or changes APIs. It's not magic — it's plumbing. Lots of plumbing. But it works, it's mine, and it costs nothing.

If you build something similar: test every model before trusting it, log every fallback, and keep sessions short. The free tiers are generous but unforgiving.

Built on Windows 11, Docker Desktop, WSL2. Zero cloud dependencies. Zero subscriptions.

Discuss on Dev.to | GitHub | Telegram

DEV Community: Gabriele Di Maria

How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)

How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)

Why This Exists

The Stack (What's Actually Running)

The Model Layer: Getting Reliability from Free Tiers

The Core Problem

Solution 1: 10-Key Rotation (nvidia_key_rotation.py)

Solution 2: 6-Model Cascade (nvidia_cascade.py)

Solution 3: Local Proxy with Tool Calling + SSE (nvidia_proxy.py)

Solution 4: Groq Fallback in Hermes Config

The Orchestration Layer: Windmill

Why Windmill?

The 10-Pillar Attempt (7 Real, 3 Archived)

What Windmill Actually Runs

The Agent Interface: Hermes Agent

Configuration That Works

Bot Mode (Actually Configured)

Real Problems & Solutions

1. Rate Limits → Proxy + Rotation + Cascade + Fallback

2. Hallucinations → NVIDIA Reviewer + Model 550B

3. Tool Calling Broken → Proxy Rewrite

4. Empty Streams → SSE Implementation

5. Model EOL (410 Gone) → Active List Maintenance

6. BOM in JSON → UTF-8 Without BOM

7. Windmill DB via psql → API Only

Numbers That Are Real

What Works Well

What Doesn't Work / Needs Work

What I'd Do Differently

Operational Checklist: Replicate This Stack

What's Next

Resources

Closing

Solution 1: 10-Key Rotation (`nvidia_key_rotation.py`)

Solution 2: 6-Model Cascade (`nvidia_cascade.py`)

Solution 3: Local Proxy with Tool Calling + SSE (`nvidia_proxy.py`)