How to run Hermes, a self-improving personal AI agent, fully local with QVAC

What Hermes is, and why it is different

Most AI agents you have seen are task tools. You give them a job, they do it, they forget you. Hermes Agent, from Nous Research, is built on a different idea: an agent that is yours, that remembers you, and that gets better the longer you use it.

Three things make it stand out:

• It has a persona. A single file, SOUL.md, defines how it talks. Edit it and the agent changes character on the next message.
• It remembers. Conversations, facts, and context persist across sessions in a local memory, so it is a companion, not a one-shot.
• It learns. Hermes has a library of skills (research, notes, GitHub, documents, smart home, and many more) and a background curator that creates and refines skills from experience. The agent that finishes your week is not the one that started it.

All of that personality, memory, and learned skill is data about you. Which raises the obvious question: where does the model behind it run?

Why local is the right default for a personal agent

When a model only answers questions, where it runs is mostly about privacy. When a model is your personal agent, holding your persona, your memory, and skills it learned from your own habits, it becomes about ownership. If the brain lives in someone else's datacenter, then your assistant's memory of you, and everything it learns from you, flows through a system you do not control: its uptime, its terms, its access policy, all of which can change without you.

Running the model locally changes the property, not just the policy:

• Your context stays on the machine. Your persona, your memory, the skills the agent builds: none of it has to leave your disk to reach the model.
• It works offline. Once the model is downloaded, you can disconnect entirely and keep talking to your agent. Turn on airplane mode and it still answers.
• No per-token bill. A personal agent you talk to all day, that runs scheduled jobs in the background, would meter up fast against a cloud API. Locally it costs nothing per token.
• No silent model swaps. The weights are a file on your disk. Your agent's "brain" does not change under you between sessions.

The trade is latency and raw capability: a model that fits on a laptop is smaller than a frontier cloud model. For a personal agent that lives with your data, that trade is easy, and it keeps getting better as small models improve.

QVAC is the piece that makes the local half practical. It is an open-source SDK and CLI from Tether that runs models on your own device across every major GPU backend (NVIDIA, AMD, and Intel through Vulkan, Apple Silicon through Metal), and it ships an OpenAI-compatible server. That server is the bridge Hermes plugs into.

How it fits together

Two long-running pieces and you:

1. The QVAC server holds the model in memory and answers OpenAI-compatible requests on a local port. Start it once and leave it running.
2. Hermes runs the agent: persona, memory, skills, tools. You point it at the QVAC server once, then just talk to it.
3. Your messages go to Hermes, which thinks using the local model and acts with its tools.

The model inference never leaves the machine. (Some skills, like web research, still use the internet for that specific tool, the same way a browser does. The brain stays on-device.)

Which model to run

Hermes is tool-heavy: it sends a large set of tools on every turn, so it pays to run a capable model. The setup below uses Qwen3.6-35B-A3B, a mixture-of-experts model: 35 billion parameters of knowledge, only about 3 billion active per token, so it stays fast for its size. To switch models, change the constant in the config file (step 2 of the setup).

Use case	Model	Recommended RAM	Required storage	Config name
Lightest, runs almost anywhere	Qwen3.5-4B (4-bit)	8 GB	~3 GB	QWEN3_5_4B_MULTIMODAL_Q4_K_M
Balanced	Qwen3.5-9B (4-bit)	16 GB	~6 GB	QWEN3_5_9B_MULTIMODAL_Q4_K_M
Recommended for the full agent	Qwen3.6-35B-A3B MoE (4-bit)	36 GB	~22 GB	QWEN3_6_35B_A3B_MULTIMODAL_Q4_K_M

Hermes leans on tool calls, and smaller models are weaker at that, so on the 4B or 9B keep the active skill set lean (see "What to try next"). The 35B-A3B MoE handles the full toolset most reliably.

Be realistic: these are small local models, and tool-heavy agent work is where they struggle most. The 4B is for trying the setup, the 9B mainly adds reliability over the 4B rather than smarts, and the 35B-A3B MoE is the only one that drives the full toolset dependably, so it is what we recommend. It is a capable on-device assistant, not a frontier cloud model.

A GPU helps a lot but is not required. QVAC uses Apple Metal on Apple Silicon and Vulkan on NVIDIA, AMD, and Intel. Run qvac doctor to see what your hardware supports. On the 35B-A3B, expect the first turn to take around 20 seconds (Hermes sends a large system prompt plus its tools), and faster turns after. For a snappier agent, narrow the active skills (see "What to try next").

Set it up

The walkthrough is copy-paste. Run each step in order. The only slow part is the first model download, which is cached after that.

Requirement: Node.js 22.17 or newer (node --version, from nodejs.org). Disk and RAM per the model table above.

1. Install the QVAC CLI (all platforms)

npm install -g @qvac/cli

2. Create a model config. Makes a folder and writes one small config file.

macOS and Linux:

mkdir -p ~/qvac-hermes && cd ~/qvac-hermes
cat > qvac.config.json <<'EOF'
{
  "plugins": ["@qvac/sdk/llamacpp-completion/plugin"],
  "serve": {
    "models": {
      "qwen3.6-moe": {
        "model": "QWEN3_6_35B_A3B_MULTIMODAL_Q4_K_M",
        "type": "llamacpp-completion",
        "default": true,
        "preload": true,
        "config": { "tools": true, "toolsMode": "static", "ctx_size": 32768, "gpu_layers": -1, "reasoning_budget": 0 }
      }
    }
  }
}
EOF

Windows (PowerShell):

mkdir "$HOME\qvac-hermes"; Set-Location "$HOME\qvac-hermes"
@'
{
  "plugins": ["@qvac/sdk/llamacpp-completion/plugin"],
  "serve": {
    "models": {
      "qwen3.6-moe": {
        "model": "QWEN3_6_35B_A3B_MULTIMODAL_Q4_K_M",
        "type": "llamacpp-completion",
        "default": true,
        "preload": true,
        "config": { "tools": true, "toolsMode": "static", "ctx_size": 32768, "gpu_layers": -1, "reasoning_budget": 0 }
      }
    }
  }
}
'@ | Out-File -Encoding utf8 qvac.config.json

The two settings that matter: "tools": true and "toolsMode": "static". Static is what lets an external client like Hermes run the model's tool calls itself. (With "dynamic" the tool calls never execute, which looks like the model is broken.)

3. Start the QVAC server (all platforms). Run it in that folder and leave the terminal open. First run downloads the model once (about 22 GB for the MoE). Ready when you see QVAC API server listening.

qvac serve openai --model qwen3.6-moe

It listens on http://localhost:11434/v1. That is the whole command: --model qwen3.6-moe is the alias you defined in step 2, and running from the ~/qvac-hermes folder lets QVAC auto-detect that qvac.config.json (which is also what sets toolsMode: static). Run it somewhere without that config and it will error that the alias is unknown.

4. Install Hermes (new terminal). The installer sets up Python, the global hermes command, and its dependencies.

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

On Windows, run that installer inside WSL (Windows Subsystem for Linux). The QVAC server from step 3 runs natively on Windows, and Hermes in WSL reaches it over localhost either way.

5. Point Hermes at QVAC. Tell Hermes to use your local server as a custom OpenAI-compatible provider.

hermes config set model.provider custom
hermes config set model.base_url "http://localhost:11434/v1"
hermes config set model.default qwen3.6-moe
hermes config set model.api_key qvac

qwen3.6-moe must match the alias in your config (the key under serve.models). The api-key value is a placeholder: the local QVAC server does not check it, but Hermes wants the field filled, so any non-empty value works. (You can also run the interactive hermes model wizard and pick "custom" instead of these four commands.)

6. Talk to your local agent.

hermes

That opens an interactive chat. Try turning off your network and asking it whether it runs in the cloud or on your machine. It keeps answering.

What you just built, and what to try next

You now have a personal AI agent running entirely on your own machine. A good first test is the obvious one: ask it whether it is in the cloud or local, then enable airplane mode and ask again. It keeps answering, because the inference was never leaving your device.

From there, the things that make Hermes Hermes all run on the same local model:

• Give it a personality. Edit ~/.hermes/SOUL.md, then send a message. The persona is loaded fresh each time, so the change is immediate.
• Let it remember. Tell it something about how you work, start a new session later, and it recalls it. The memory lives in ~/.hermes/.
• Use a skill. Run hermes skills to see the library, then ask it for something real: take structured notes, organize a project on its task board, summarize a folder of documents. Pick local skills (notes, files, board) to stay fully offline; web skills will use the internet for that tool.
• Let it improve. Over time the curator refines skills from what you do. That learning happens on your hardware and stays there.

For speed, narrow the active skills to what you actually use (hermes skills, hermes tools); a smaller toolset means a smaller prompt and faster turns on a local model.

Notes and limits

• A model that fits on a laptop is not a frontier cloud model. Expect it to need clearer instructions and to be slower than a cloud API today. That gap is closing as local hardware speeds up and small models improve.
• "Local" here means the model runs on your device. A given skill may still call the internet for its own job (web research, for example), the same as any app would. The intelligence, your persona, and your memory stay on the machine.
• QVAC is Apache 2.0 and free. Hermes Agent is from Nous Research (github.com/NousResearch/hermes-agent). The model is downloaded once and cached locally. QVAC source and docs: github.com/tetherto/qvac and docs.qvac.tether.io.