DEV Community: Vivek Shetye

How I Built a Fully Local AI Coding Agent with Qwen 3.8, LM Studio, and Pi (No Paid APIs)

Vivek Shetye — Tue, 13 Oct 2026 15:33:47 +0000

What if you could have an AI coding agent that reads your files, writes code, executes commands, and fixes errors without sending a single model request to the cloud?

No OpenAI API. No Anthropic API. No per-token billing.

Just an open-weight AI model running on your own hardware.

I recently decided to test exactly how practical this setup has become.

Using Qwen 3.8 27B, LM Studio, and Pi Coding Agent, I built a fully local AI coding workflow and challenged it to create an entire web application from scratch.

And the results were surprisingly good.

The agent built a React + TypeScript application, executed commands, tested its work, identified errors, and fixed them without requiring my intervention.

All while running a 27-billion-parameter model locally on my MacBook Pro.

But there were some interesting trade-offs, especially around inference speed, hardware requirements, and context management.

In this article, I’ll walk through the complete setup, explain why I chose these tools, and share what actually happened when I put the agent to work.

🎥 Full video walkthrough

In the video, I cover:

Installing LM Studio and downloading Qwen 3.8 27B.
Understanding 4-bit quantization.
Starting the local inference server.
Installing Pi Coding Agent.
Connecting Pi to LM Studio.
Testing local tool calling.
Building the Mission Control dashboard.
Reviewing the results, performance, and limitations.

🚀 What Did I Build?

Before getting into the setup, here’s what I asked my local AI coding agent to create.

An AI Agent Mission Control dashboard for monitoring and managing a team of autonomous AI agents.

The application needed several features:

Agent Overview: Monitor active agents, running tasks, completed tasks, and failures.
Agent Management: View agents, their roles, current tasks, and statuses.
Kanban Task Board: Organize tasks across Backlog, In Progress, Review, and Completed.
Live Activity Feed: Track agent activity and task execution.
Analytics Dashboard: Visualize token usage, estimated costs, success rates, and task performance.
Modern UI: A responsive dark-themed interface with charts, animations, and polished interactions.

And I didn’t want a static HTML mockup.

I wanted a working React application with realistic sample data.

The interesting part?

The entire coding workflow ran locally.

The AI model wasn’t hosted by OpenAI, Anthropic, or another cloud inference provider.

It was running directly on my computer.

🧠 The Architecture: Three Tools, One Local AI Coding Agent

The architecture is relatively simple.

           Developer
               |
               v
       +----------------+
       | Pi Coding Agent|
       |                |
       | Read Files     |
       | Write Code     |
       | Execute Bash   |
       | Edit Files     |
       +----------------+
               |
               | OpenAI-Compatible API
               v
       +----------------+
       |   LM Studio    |
       |                |
       | Local Inference|
       | Server         |
       +----------------+
               |
               v
       +----------------+
       |  Qwen 3.8 27B  |
       |                |
       | 4-bit Quantized|
       | Local AI Model |
       +----------------+
               |
               v
        Your Computer

Each component has a specific responsibility.

1. Qwen3.8 27B — The Brain

Qwen 3.8 27B is the language model responsible for understanding instructions, reasoning about coding tasks, and generating code.

For this experiment, I used a 4-bit quantized version of the model, approximately 16GB in size.

This makes it much more practical to run a large model on consumer hardware.

2. LM Studio — The Local Inference Server

LM Studio makes running open-weight AI models locally much easier.

Instead of manually setting up inference frameworks, Python environments, or model-serving infrastructure, you get a desktop application where you can:

Discover and download models.
Load models into memory.
Chat with models directly.
Configure inference settings.
Expose models through an OpenAI-compatible API.

That last feature is particularly important because it allows external applications to communicate with your locally running model.

3. Pi Coding Agent — The Hands

A language model alone cannot automatically inspect your project files, modify code, or execute terminal commands.

That’s where Pi comes in.

Pi is a lightweight, extensible terminal-based AI coding agent.

It provides the tools and agent loop needed to interact with your development environment.

Instead of simply generating code in a chat window, Pi can use the model to work through actual coding tasks.

And because Pi is relatively minimal, it’s particularly interesting for local model experiments.

⚙️ Step 1: Install LM Studio and Download Qwen 3.8 27B

First, download LM Studio from its official website:

LM Studio: https://lmstudio.ai/

Install the application for your operating system and hardware.

Once installed, open LM Studio and navigate to the model search section.

Search for:

Qwen 3.8 27B

You’ll notice that multiple versions of the same model may be available.

For this experiment, I selected a 4-bit quantized version.

Why 4-bit Quantization?

Running a 27-billion-parameter model at full precision can require substantial memory.

Quantization reduces the precision used to store model weights.

For example:

Precision	Approximate Raw Weight Storage
FP16 (16-bit)	54 GB
INT8 (8-bit)	27 GB
4-bit	13.5 GB

These are theoretical weight-storage estimates. Actual model files and runtime memory requirements can be higher because of quantization metadata, model architecture, KV cache, and other overhead.

The 4-bit model I downloaded was approximately 16GB.

That’s a significant reduction compared with FP16.

Of course, quantization comes with trade-offs.

Reducing precision can affect model accuracy, especially on tasks involving complex reasoning or code generation.

However, modern quantization techniques can often preserve much of the original model’s capabilities while substantially reducing memory requirements.

For running larger models locally, this is extremely useful.

Load the Model

After downloading:

Open My Models in LM Studio.
Select your downloaded Qwen model.
Click Load Model.
Configure the context length according to your available memory.
Load the model.

For my experiment, I configured a context length of 200,000 tokens.

Keep in mind that configuring a large context window doesn’t mean your hardware can efficiently process that entire context. Actual memory usage and performance depend on the model, runtime, and KV-cache configuration.

Once the model is loaded, you can test it using LM Studio’s built-in chat interface.

Try sending a simple message to verify that inference is working.

Now we need to make this model accessible to our coding agent.

🔌 Step 2: Enable LM Studio’s Local API Server

LM Studio can expose your local model through an OpenAI-compatible API.

This is what allows Pi Coding Agent to communicate with Qwen.

Inside LM Studio:

Navigate to the Developer tab.
Enable the local server.
Verify that your model is available.

By default, LM Studio uses port 1234.

The local API endpoint is:

http://localhost:1234/v1

You can verify that the server is responding by running:

curl http://localhost:1234/v1/models

This should return information about the models exposed by your LM Studio server.

At this point, your computer is acting as its own AI inference server.

And the best part is that you don’t need to send inference requests to an external model provider.

Note: Keep the server bound to localhost unless you intentionally need remote access. Exposing an inference endpoint to your network introduces additional security considerations.

🛠️ Step 3: Install Pi Coding Agent

Now that Qwen is running locally, we need an agent that can actually work with our codebase.

For this, I’m using Pi Coding Agent.

Official project:

https://pi.dev/

Pi is designed around a relatively small set of core capabilities.

Out of the box, it provides four essential tools:

Tool	Purpose
`read`	Read files from your project
`write`	Create files and write content
`edit`	Modify existing files
`bash`	Execute terminal commands

That’s it.

And these four tools are enough to handle a surprising amount of coding work.

Pi also supports extensions, allowing you to add functionality when needed.

Install Pi

Follow the installation instructions on Pi’s official website.

After installation, open your terminal and run:

pi

If a new Pi session opens, the installation is working.

Now let’s connect it to our local model.

🔗 Step 4: Connect Pi to LM Studio

There are multiple ways to configure Pi to use a locally hosted model.

You can manually configure a custom provider, but I used a community extension called pi-lmstudio.

The extension discovers models exposed by LM Studio and makes them available inside Pi.

You can find it through Pi’s package directory.

Install the extension using the command provided on its package page.

Once installed, restart Pi:

pi

Then open the model selector:

/model

You should see the model currently exposed by LM Studio.

Select your Qwen 3.8 27B model.

And that’s it.

Pi is now using your locally running Qwen model instead of a hosted inference API.

The workflow looks like this:

Your Prompt
    |
    v
Pi Coding Agent
    |
    | Sends model request
    v
LM Studio API (localhost:1234)
    |
    v
Qwen 3.8 27B
    |
    | Returns response / tool calls
    v
Pi Coding Agent
    |
    | Executes tools
    v
Your Local Project

One important distinction: Pi executes tools locally, while LM Studio handles model inference.

The model generates the instructions and tool calls, but Pi is responsible for carrying them out.

Verify That Tool Calling Works

Before assigning a complicated task, I recommend testing the setup with something simple.

For example, create a Python file:

print("Hello, World!")

Then ask Pi:

Read the Python file in this directory, explain what it does, and create a document containing the explanation.

This tests whether the model can:

Understand the instruction.
Select the appropriate tools.
Read a local file.
Generate a summary.
Write a new file.

In my experiment, Pi successfully completed this test.

Now it was time for something more challenging.

💻 Step 5: Can a Local 27B Model Build an Entire Web Application?

This was the part I was most interested in.

Getting a model to answer questions is one thing.

Getting it to work through a multi-step software development task is something else entirely.

I started with an empty project directory.

Then I asked Pi to build an AI Agent Mission Control dashboard.

Here’s a condensed version of the prompt:

Build a modern, polished AI Agent Mission Control dashboard for managing and monitoring a team of autonomous AI agents.

The dashboard should include:

* Overview cards for Active Agents, Running Tasks, Completed Tasks, and Failed Tasks
* A list/grid of AI agents with names, roles, status, current task, and model being used
* A Kanban-style task board with Backlog, In Progress, Review, and Completed columns
* A Live Activity Feed showing agents starting tasks, using tools, completing work, and handing tasks to other agents
* System metrics such as token usage, estimated cost, success rate, and average task duration
* A sidebar with Overview, Agents, Tasks, Activity, Analytics, and Settings

Use realistic mock data for a small AI-agent team such as Research Agent, Coding Agent, Browser Agent, Content Agent, and QA Agent.

Make it feel like a premium SaaS mission-control product: clean dark theme, excellent typography, subtle animations, status indicators, charts, responsive layout, and polished interactions.

Build the complete working website, not just a static mockup.

Then I let the agent work.

What Happened?

The agent created a React + TypeScript project and started implementing the application.

It performed multiple tool calls, created files, wrote components, and executed terminal commands.

But what impressed me most wasn’t just the generated code.

It actually attempted to test and fix its own work.

During the process, the agent:

Created the application structure.
Generated React components.
Implemented the dashboard interface.
Ran the application locally.
Encountered errors.
Inspected the results using screenshots.
Made additional changes to address issues.

And it completed this workflow without requiring me to manually fix the errors it encountered.

The final result included the agent overview, task board, activity feed, analytics dashboard, and settings interface.

To be clear, this was a demonstration application using sample data, not a production-ready autonomous agent management platform.

But as an experiment in local AI-assisted software development, it was encouraging.

📊 The Results: Performance, Hardware, and Cost

Here’s the actual setup I used.

Component	Configuration
Computer	MacBook Pro
Chip	Apple M5 Max
Memory	48GB unified memory
AI Model	Qwen3.8 27B
Quantization	4-bit
Model Size	Approximately 16GB
Inference Runtime	LM Studio
Coding Agent	Pi
Observed Speed	10–15 tokens/second
Task Duration	A few hours
Paid Model API Cost	$0

The agent managed to build the application, but it wasn’t particularly fast.

At approximately 10–15 tokens per second, complex coding tasks can take quite a while.

And when the agent needs to perform multiple rounds of reasoning, tool execution, and error correction, that time adds up.

Still, the fact that this entire workflow ran on consumer hardware is interesting.

There was no per-token API bill.

However, zero API cost doesn’t mean zero cost.

You still need suitable hardware, electricity, and time.

And depending on your workload, a hosted model might still be more economical or productive.

🧩 Why I Chose Pi Instead of a Larger Coding Agent

This is one of the more interesting technical aspects of the experiment.

When running AI models locally, context management becomes especially important.

Every instruction, conversation message, tool definition, and tool result consumes tokens.

And those tokens aren’t free from a computational perspective.

Even when you’re not paying an API provider, your hardware still needs to process them.

For local models, large contexts can increase:

Memory consumption.
Time to process prompts.
KV-cache requirements.
Overall task execution time.

This is where Pi’s design becomes interesting.

A Minimal Agent Harness

Pi starts with four core tools:

read
write
edit
bash

Rather than beginning with a large collection of built-in capabilities, you can extend Pi as needed.

This gives you more control over the tools and functionality available to your local model.

For example, you might eventually add:

Browser automation.
Web search.
Custom development tools.
Project-specific workflows.
Additional integrations.

But you don’t necessarily need all of those capabilities loaded from the beginning.

A smaller tool surface can help keep the setup simpler.

Important: This doesn’t automatically mean Pi uses fewer tokens or performs better than Claude Code, Codex, or other coding agents in every scenario.

That would require a controlled benchmark using the same model, tasks, and evaluation criteria.

For my experiment, Pi was simply a good fit because I wanted a lightweight, extensible agent environment.

⚠️ What Are the Limitations of Running AI Coding Agents Locally?

As promising as the results were, there are some important limitations.

1. Hardware Requirements

A 27B model, even when quantized, requires a significant amount of memory.

And the model weights are only part of the equation.

You also need memory for inference overhead, context processing, and the operating system.

Not everyone has a machine capable of comfortably running models this large.

2. Inference Can Be Slow

My setup generated approximately 10–15 tokens per second.

That’s usable, but considerably slower than many hosted inference services.

For complicated tasks involving hundreds of tool calls or repeated debugging cycles, performance becomes a major consideration.

3. Tool-Calling Reliability Varies

An AI coding agent needs more than good code generation.

It must reliably decide when to use tools, generate valid tool calls, interpret results, and continue working.

A model might perform well on a single coding prompt but struggle with a long sequence of tool interactions.

This is something I want to evaluate more systematically in future experiments.

4. Local Models Aren’t Always Competitive With Frontier Models

Strong hosted models still have advantages on many complex software engineering tasks.

Smaller local models may struggle with:

Large codebases.
Complex architectural changes.
Long debugging sessions.
Multi-step reasoning.
Maintaining consistency across many files.

And quantization can introduce additional accuracy trade-offs.

So I wouldn’t consider this a universal replacement for cloud-based coding agents just yet.

5. Local Inference Doesn’t Automatically Make an Agent Safe

Running the model locally helps keep inference data under your control.

But an agent that can execute shell commands and modify files still has significant permissions.

A local model can make mistakes, delete files, or execute unintended commands.

For sensitive projects, consider using isolated development environments, restricted permissions, version control, and approval mechanisms.

Also remember that extensions or tools with network access may still communicate externally.

A fully local inference endpoint is not, by itself, a guarantee that every part of an agent workflow stays offline.

🔮 Are Fully Local AI Coding Agents Becoming Practical?

I think we’re getting closer.

Not because local models have suddenly become better than every cloud model.

But because the overall ecosystem has improved significantly.

We now have:

Open-weight models capable of increasingly sophisticated coding tasks.

Local inference tools like LM Studio that make downloading and running models accessible.

Lightweight agent frameworks like Pi that can turn those models into useful coding assistants.

And hardware capable of running models that would have been impractical for many developers just a few years ago.

What’s particularly interesting is that these components are independent.

You can swap the model without necessarily replacing your coding agent.

You can experiment with different quantization levels.

You can add or remove agent capabilities.

And you can choose how much of your development workflow runs locally.

For developers who care about privacy, experimentation, offline workflows, or avoiding recurring API costs, that’s a compelling direction.

The next challenge isn’t just making local models generate better code.

It’s making them reliable enough to complete long-running software engineering tasks with minimal human intervention.

And that requires evaluating the entire agent workflow, not just the model’s benchmark scores.

💬 Final Thoughts

My biggest takeaway from this experiment wasn’t that a local 27B model could replace Claude Code or Codex.

It was that a complete AI coding workflow can now run on a personal computer without relying on paid model APIs.

And that opens up some interesting possibilities.

Local-first development environments.

Private AI coding assistants.

Offline software development workflows.

And customizable agents that developers can adapt to their own hardware and requirements.

There’s still plenty of room for improvement, particularly around speed, reliability, and long-context performance.

But I’m excited to see how far this ecosystem progresses.

What if you could use Claude Code for FREE? 👀 I connected Claude Code OmniRoute free AI models from OpenRouter & NVIDIA. Here’s the full setup, trade-offs, and how it works 👇

Vivek Shetye — Tue, 06 Oct 2026 15:22:18 +0000

Vivek Shetye

Oct 6

How to Use Claude Code for Free with OmniRoute

#ai #opensource #agents #claude

6 min read

How to Use Claude Code for Free with OmniRoute

Vivek Shetye — Tue, 06 Oct 2026 15:01:32 +0000

Claude Code is one of my favorite AI coding agents.

It can inspect your codebase, edit files, execute terminal commands, use tools, and work through multi-step coding tasks directly from the terminal.

But normally, the intelligence behind that experience comes from Anthropic’s Claude models.

What if we kept Claude Code as the coding agent harness, but replaced the model underneath it?

That’s what I wanted to test.

The result looks like this:

Claude Code
     ↓
 OmniRoute
     ↓
Free AI Models
     ↓
OpenRouter / NVIDIA / Other Providers

Instead of sending model requests directly to Anthropic, Claude Code sends them to OmniRoute, which acts as an AI gateway and routes them to the models and providers you configure.

And yes — that means you can run Claude Code using free AI models. 🆓

But there’s an important distinction.

We are NOT getting Claude Sonnet or Claude Opus for free.

We’re using Claude Code as the agent harness while replacing the underlying model with free models from other providers.

Let’s set it up.

🎥 Full video walkthrough

I recorded the entire process step-by-step, including:

✅ Installing OmniRoute locally
✅ Connecting OpenRouter
✅ Connecting NVIDIA
✅ Importing free AI models
✅ Configuring Claude Code
✅ Fixing the authentication issue I encountered
✅ Launching Claude Code through OmniRoute
✅ Running Claude Code with free models

🧠 Claude Code Is More Than the Model

An important idea behind this experiment is separating the agent harness from the LLM powering it.

Claude Code provides the agent experience around the model.

It can:

inspect files
understand your project
modify code
execute terminal commands
use tools
iterate through coding tasks

Normally, Claude Code communicates with Anthropic’s models.

But if we place an AI gateway between Claude Code and the model, the architecture becomes much more flexible.

┌─────────────────┐
│   Claude Code   │
│  Coding Agent   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│    OmniRoute    │
│   AI Gateway    │
└────────┬────────┘
         │
         ▼
┌─────────────────────────┐
│ OpenRouter │ NVIDIA │ … │
└─────────────────────────┘
         │
         ▼
┌─────────────────┐
│ Free AI Models  │
└─────────────────┘

Claude Code remains the interface and coding agent.

OmniRoute controls where the model requests go.

🔀 That separation is what makes this experiment interesting.

⚙️ Step 1: Install OmniRoute

The first thing we need is OmniRoute.

OmniRoute is an open-source AI gateway that can sit between applications such as Claude Code and different AI model providers.

Install OmniRoute using the installation instructions for your operating system.

Once installed, start it:

omniroute

After a few seconds, the OmniRoute server should start.

Open the dashboard URL shown in your terminal.

You now have a local AI gateway running on your machine.

🔌 Step 2: Connect an AI Provider

Next, we need models.

For my setup, I connected two providers:

OpenRouter
NVIDIA

Let’s start with OpenRouter.

Inside the OmniRoute dashboard, navigate to:

Providers → OpenRouter → Add Connection

Generate an API key from your OpenRouter account and add it to the connection.

If you only want free models, enable the option to:

Only import free models

Save the connection.

OmniRoute will import the available free models associated with that provider.

You can also run health checks from the dashboard to see which models are currently available. 🩺

🚀 Step 3: Add NVIDIA Models

I also connected NVIDIA to give OmniRoute another source of models.

The process is similar.

Navigate to:

Providers → NVIDIA → Add Connection

Generate an NVIDIA API key, paste it into OmniRoute, validate it, and save the connection.

Again, I configured the connection to import the free models available through the provider.

Now the architecture starts becoming more interesting:

                    ┌──→ OpenRouter → Free Models
Claude Code → OmniRoute
                    └──→ NVIDIA ───→ Free Models

Instead of Claude Code being tied directly to one model provider, the gateway can expose models from multiple providers.

💻 Step 4: Connect Claude Code to OmniRoute

Now for the important part.

If you don’t already have Claude Code installed, install it using Anthropic’s official installation instructions.

Once Claude Code is available, OmniRoute can configure it automatically.

Run:

omniroute setup-claude

OmniRoute creates profiles that allow Claude Code to communicate through the gateway.

However, I ran into an issue here.

🐛 The Authentication Problem I Encountered

My OmniRoute API was protected with authentication.

Normally, you should be able to create an access token and pass it while configuring Claude Code.

In my testing, however, the API key parameter wasn’t being respected correctly and Claude Code continued returning an authentication error.

So I used a temporary workaround.

I disabled API authentication in OmniRoute.

⚠️ Important Security Warning

I would only consider doing this when OmniRoute is:

running locally
bound appropriately
not exposed publicly
not accessible from an untrusted network

Do not expose an unauthenticated AI gateway to the internet.

After disabling authentication locally, I ran the Claude setup again and the connection succeeded.

🎯 Step 5: Launch Claude Code with a Free Model

Now we can actually launch Claude Code through OmniRoute.

I used OmniRoute’s coding profile:

omniroute launch --profile auto-coding

The auto-coding profile allows OmniRoute to select among the coding models available in the configuration.

And that’s it.

Claude Code starts normally.

Except now the architecture is:

Claude Code
     ↓
OmniRoute
     ↓
Alternative AI Model

Instead of:

Claude Code
     ↓
Anthropic Model

🎉 Claude Code is now running using the models routed through OmniRoute.

So… Is Claude Code Actually Free Now?

Kind of.

And this distinction is important.

You’re not getting Anthropic’s Claude Sonnet or Opus models for free.

You’re using the Claude Code agent experience with alternative models that may have free access tiers or free routes.

Think of it as separating two pieces:

Claude Code = Agent Harness
Free Model = Intelligence
OmniRoute = Routing Layer

This means the quality of the experience depends heavily on the model you choose.

And that brings us to the trade-offs.

⚡ Trade-Off #1: Free Models Have Limits

Free doesn’t mean unlimited.

Free model providers commonly impose things like:

request limits
token quotas
rate limits
temporary availability restrictions

A model that works today may also become unavailable or limited later.

For experimentation and personal projects, this can still be extremely useful.

For production workloads, you’ll want something more predictable.

🧩 Trade-Off #2: Not Every Model Works Well with Claude Code

This is probably the biggest issue.

Claude Code isn’t simply sending a prompt and displaying some text.

The model needs to participate in an agentic coding workflow.

That means capabilities such as:

Understand task
      ↓
Inspect files
      ↓
Call tools
      ↓
Modify code
      ↓
Run commands
      ↓
Observe results
      ↓
Continue reasoning

Some models handle these workflows surprisingly well.

Others don’t.

A model might be excellent at generating code but struggle with:

tool calling
instruction following
long coding tasks
maintaining context
structured outputs
multi-step agent workflows

So don’t evaluate a model purely by its coding benchmark score.

Agent compatibility matters too.

🔐 Trade-Off #3: Your Code Goes to the Upstream Provider

This one is important.

When you route Claude Code through another provider, your prompts and potentially parts of your codebase are being sent to that provider.

Before using this setup with private repositories or sensitive code, check the provider’s:

privacy policy
data retention policy
model training policy
API terms

I would be particularly careful with proprietary company code, credentials, customer information, and production secrets.

📜 Trade-Off #4: Check Provider Terms

You should also check the terms for every provider you connect.

Some providers may have restrictions around:

automation
proxies
redistribution
free API usage
rate limits
agent workloads

A few minutes reading the provider’s terms can save you from account problems later.

💡 Why This Is More Interesting Than Just “Claude Code for Free”

The cost savings are interesting.

But I think the architecture is the bigger story.

We’re increasingly separating the AI application from the AI model.

Instead of:

AI Application
      ↓
One Model
      ↓
One Provider

we can build:

                    ┌──→ Model A
                    │
AI Agent → Gateway ─┼──→ Model B
                    │
                    ├──→ Model C
                    │
                    └──→ Model D

Now the gateway becomes an abstraction layer between the agent and the models.

That opens up some interesting possibilities.

🧠 Use Different Models for Different Coding Tasks

Imagine routing:

Simple code edits
      ↓
Small / Free Model
Complex debugging
      ↓
Powerful Reasoning Model
Documentation
      ↓
Fast Cheap Model
Large Refactor
      ↓
Frontier Coding Model

Instead of asking:

“Which AI model should power my coding agent?”

the better question may eventually become:

“Which model should handle this particular task?”

That’s a very different architecture.

Final Thoughts

Claude Code doesn’t necessarily have to be coupled to a single model provider.

By introducing an AI gateway like OmniRoute, you can separate the coding agent experience from the model powering it.

That gives you room to experiment with:

🔀 multiple providers
🆓 free models
💰 cheaper models
🧠 stronger reasoning models
⚡ faster models
🛠️ models optimized for specific tasks

Free models won’t replace Claude Sonnet or Opus for every coding task.

But that’s not really the point.

The interesting part is being able to choose.

And as AI coding agents become more capable, I suspect the question will increasingly shift from:

“Which coding agent has the best model?”

to:

“Which models should my coding agent use for each task?”

JEV Explained: Why It Could Matter for AI Agents

Vivek Shetye — Wed, 30 Sep 2026 19:20:27 +0000

What if one of the most interesting new AI models doesn’t generate text at all?

No chatbot responses.

No code generation.

No essays.

Instead, it is designed to do something much narrower:

Make decisions for software.

That’s the idea behind JEV, a new model from TypeSafe AI.

And the numbers TypeSafe AI is reporting immediately caught my attention:

⚡ Decisions in milliseconds
💰 Around $0.042 per million input tokens
📤 No charge for output tokens
📊 On selected workflows, TypeSafe reports nearly 200× faster and 400× cheaper performance compared with the LLMs it tested

Those are TypeSafe AI’s own workflow comparisons, not universal benchmarks. Different workloads could produce very different results.

But the more interesting question isn’t whether JEV can beat an LLM on every benchmark.

It’s:

Why are we using generative LLMs for decisions that don’t require generating anything?

That question becomes especially interesting when you start building AI agents.

🎥 Full video walkthrough

🧠 What Is JEV?

The easiest way to understand JEV is to compare it with a traditional Large Language Model.

Imagine a customer sends this message:

“My package was supposed to arrive yesterday, but tracking hasn’t updated.”

You want your system to decide which team should handle it.

Possible options:

billing
shipping
other

Ask an LLM and it might generate something like:

This request should be routed to the shipping department because
the customer is asking about a delayed package.

That works.

But your application doesn’t necessarily need an explanation.

It needs:

shipping

Or, even better, probabilities associated with the available decisions.

Conceptually:

shipping: 0.94
billing: 0.02
other: 0.04

Your software can then decide what happens next.

That’s the fundamental difference.

LLMs generate tokens.

JEV evaluates predefined decisions.

The output is primarily intended for software to act on, rather than for a human to read as a conversation.

⚡ Why Remove Text Generation?

Modern LLMs are incredibly capable.

They can reason through complex problems, understand large codebases, write software, analyze documents, interact with tools, and orchestrate workflows.

But that flexibility comes with computational cost.

Now consider a typical AI agent.

The agent might need a powerful model to:

understand a repository
debug an application
reason through a complicated problem
generate a patch
plan a multi-step workflow

But between those large reasoning tasks, agents constantly make smaller decisions.

For example:

Which model should handle this task?
Did the previous step succeed?
Should I retry?
Does this command require approval?
Which tool should run next?
Should this result be escalated?

Today, we often solve these problems with…

another LLM call.

And that works.

But we’re invoking a generative model when the actual output we need might simply be:

retry

or:

needs_approval

or:

use_stronger_model

That is the category of problem TypeSafe AI is targeting with what it calls System One Models.

JEV is its first public model.

🔄 JEV vs LLMs: The Important Difference

Think about the two approaches like this:

Traditional LLM	JEV
Generates text	Evaluates decisions
Predicts the next token	Scores predefined options
Great for reasoning and generation	Designed for bounded judgments
Flexible output	Structured output
Human-readable responses	Software-actionable decisions
Useful for open-ended problems	Useful for constrained choices

This distinction matters.

JEV isn’t necessarily trying to become the model inside your AI coding agent that understands an entire repository and writes a feature.

Instead, it could potentially sit beside that model.

A powerful LLM handles:

reasoning
coding
planning
generation

JEV handles:

classification
routing
scoring
triage
bounded decisions

In other words:

The LLM generates. JEV decides.

🤖 Why JEV Could Be Interesting for AI Agents

This is where I think things get particularly interesting.

Consider an agent built using something like Hermes Agent, Claude Code, or another agent framework.

A simplified architecture today might look like:

User
 ↓
Agent
 ↓
LLM
 ↓
Tools
 ↓
LLM
 ↓
Tools
 ↓
LLM
 ↓
Result

The LLM becomes responsible for almost everything.

But imagine separating different responsibilities:

             ┌── Retrieval / Memory
             │
User → Agent ├── JEV → Fast decisions
             │
             ├── Powerful LLM → Reasoning + Coding
             │
             └── Tools → Actions

Now different components handle different types of work.

That’s closer to the architecture I would want to experiment with.

Here are three workflows I’d test first.

1. 🔀 Intelligent Model Routing

This might be the most obvious use case.

Suppose your AI agent has access to multiple models.

Maybe you have:

Fast/Cheap Model
        +
Powerful/Expensive Model

A simple task arrives:

Rename this variable across these files.

Do you really need your most expensive reasoning model?

Probably not.

But then another task arrives:

Debug this race condition across multiple distributed services.

That’s a very different problem.

Instead of sending everything to the strongest model, JEV could potentially classify the incoming task.

For example:

Task
 ↓
JEV
 ↓
┌─────────────────┐
│ SIMPLE          │ → Fast/Cheap Model
│ COMPLEX         │ → Powerful Model
│ UNCERTAIN       │ → Powerful Model
└─────────────────┘

This could make model routing much more deliberate.

And importantly, you’d want to measure whether the routing mistakes cost more than the model savings.

Cheap routing isn’t useful if important tasks constantly go to the wrong model.

That’s why I’d want to test:

Cost
Latency
Routing accuracy
Escalation rate
Task success rate

rather than looking only at token pricing.

2. 🛡️ Risk Detection Before Tool Execution

This one gets even more interesting.

AI agents increasingly interact with real tools.

They can:

execute shell commands
modify files
interact with APIs
deploy applications
access databases
manage infrastructure

Before executing an action, you might want another layer that evaluates what kind of action is being attempted.

Imagine an agent generates:

rm -rf ./build

versus:

rm -rf /

Those commands clearly shouldn’t be treated the same way.

A decision model could evaluate questions such as:

Does this affect production?
Does this modify sensitive files?
Could this expose credentials?
Is this destructive?
Should a human approve this?

The architecture might look like:

AI Agent
   ↓
Proposed Tool Action
   ↓
JEV
   ↓
Risk Classification
   ↓
Policy Engine
   ↓
ALLOW / BLOCK / REQUIRE APPROVAL

There’s an important distinction here:

JEV shouldn’t necessarily control permissions.

Your application policy should.

JEV could provide a classification or score, while deterministic code decides what actually happens.

That keeps the security boundary outside the model.

3. 🔎 Agent Output Triage

Imagine you have an autonomous agent processing hundreds or thousands of tasks.

Most results might be perfectly normal.

Some won’t be.

Today, you could send every result through another powerful LLM for verification.

But that potentially means:

1000 tasks
+
1000 verification LLM calls

Instead, a smaller decision model could potentially triage those outputs.

For example:

Agent Result
     ↓
    JEV
     ↓
 ┌───────────────┐
 │ Looks Normal  │ → Finish
 │ Uncertain     │ → Stronger LLM
 │ Suspicious    │ → Human Review
 └───────────────┘

Now your expensive reasoning model only gets involved when necessary.

Again, whether this actually improves the system depends on the accuracy of that triage.

But architecturally, it’s an interesting pattern.

🏗️ What Could the Future AI Agent Stack Look Like?

This leads to the bigger idea behind JEV.

Over the last few years, we’ve increasingly treated the LLM as the center of everything.

Need classification?

Use an LLM.

Need routing?

Use an LLM.

Need scoring?

Use an LLM.

Need validation?

Use another LLM.

Need to judge that LLM?

Use another LLM. 😅

That architecture works because modern models are incredibly flexible.

But flexibility doesn’t necessarily mean every task should use the same model.

A future AI agent stack could instead look something like:

┌─────────────────────────────┐
│      Agent Orchestration    │
├─────────────────────────────┤
│                             │
│ Retrieval → Context         │
│                             │
│ JEV → Fast Decisions        │
│                             │
│ LLM → Reasoning + Coding    │
│                             │
│ Tools → Actions             │
│                             │
├─────────────────────────────┤
│       Policy / Control      │
└─────────────────────────────┘

Each component has a more specialized responsibility.

The reasoning model doesn’t need to make every tiny routing decision.

The decision model doesn’t need to understand and rewrite an entire codebase.

And tools remain responsible for actually interacting with the outside world.

💰 What About JEV’s Cost and Speed Claims?

This is where it’s important to separate an interesting architecture from benchmark hype.

TypeSafe AI reports pricing around:

$0.042 per million input tokens, with no output-token charge.

It also reports latency in the millisecond range.

And on selected company workflows, TypeSafe says JEV achieved results approaching:

200× faster

and

400× cheaper

than the LLM approaches it compared against.

Those numbers sound impressive.

But they should be interpreted in context.

These aren’t universal statements that JEV is “400× better than LLMs.”

They’re results from specific workflows and comparisons.

The test I’d really like to see is much simpler.

Take the same real-world decision workload and run it through:

JEV
vs.
Small LLM
vs.
Frontier LLM

Then compare:

Accuracy
False positives
False negatives
Latency
Cost
Reliability

Because if JEV costs almost nothing but makes significantly more bad routing decisions, the token savings don’t matter.

On the other hand, if it can handle those decisions reliably at a fraction of the latency and cost…

then things get interesting very quickly.

🧩 Specialized Models vs One Giant Model

And this is the part of JEV that interests me most.

Not whether JEV itself becomes the dominant solution.

Not whether every AI agent suddenly needs it.

And definitely not whether it “replaces LLMs.”

It probably shouldn’t.

The interesting idea is that not every problem inside an AI agent needs to become another LLM generation.

We might eventually build agent systems where:

Retrieval handles context.
Decision models handle bounded judgments.
Reasoning models handle difficult problems.
Coding models handle software development.
Tools handle actions.
Policy code controls permissions.
Agents orchestrate everything.

Instead of asking one increasingly powerful model to do absolutely everything, we could build systems from specialized intelligence components.

And JEV is an interesting early example of what that architecture could look like.

Run Hermes Agent Inside Docker: A Safer Setup for Autonomous AI Agents 🐳

Vivek Shetye — Tue, 22 Sep 2026 16:35:22 +0000

Giving an AI agent unrestricted access to your entire computer is probably not a great idea.

Especially when that agent can:

Run terminal commands
Install packages
Create and modify files
Access external services
Work autonomously through multi-step tasks

Those capabilities are exactly what make Hermes Agent powerful.

But they also raise an important question:

Where should all of those actions actually run?

Instead of running Hermes Agent directly on my host machine, I decided to run it inside a Docker container and explicitly control what the agent can access.

The goal isn’t to make Hermes magically “secure” by putting it inside Docker.

The goal is much simpler:

Give the agent access to what it needs, not your entire computer.

In this tutorial, I’ll walk through the complete setup.

We’ll configure:

Hermes Agent inside Docker
Persistent storage for memory, sessions, and configuration
A limited filesystem boundary
Hermes Gateway
Hermes Dashboard
Dashboard authentication
Hermes Desktop connected to the container
Some important security considerations when containerizing autonomous AI agents

🎥 Prefer watching instead?

I walk through the complete setup step-by-step in the video:

Watch the full Hermes Agent Docker tutorial on YouTube →

🤖 Why Run Hermes Agent Inside Docker?

Hermes isn’t just a chatbot.

It can interact with your environment and perform actions on your behalf.

For example, depending on the tools you enable, Hermes can work with files, execute terminal commands, install dependencies, use external services, and autonomously work through tasks.

That’s incredibly useful.

But if the agent runs directly on your main machine, you need to think carefully about what files, credentials, and system resources it can reach.

That’s where Docker becomes useful.

Instead of letting Hermes operate directly across my host environment, I can put it inside a container and explicitly decide what crosses that boundary.

The architecture we’ll build looks roughly like this:

┌──────────────────── HOST MACHINE ────────────────────┐
│                                                      │
│  Docker                                              │
│                                                      │
│  ~/.hermes-docker  ───────────────┐                  │
│                                   │                  │
│                    ┌──────────────▼──────────────┐   │
│                    │      Docker Container       │   │
│                    │                             │   │
│                    │      Hermes Agent           │   │
│                    │                             │   │
│                    │  /opt/data                  │   │
│                    │  Gateway                    │   │
│                    │  Dashboard                  │   │
│                    └─────────────────────────────┘   │
│                                                      │
│  Personal files / SSH keys / other directories       │
│              NOT mounted into container              │
│                                                      │
└──────────────────────────────────────────────────────┘

Hermes gets a dedicated environment and a directory for its persistent data.

The rest of the machine stays outside that boundary unless I explicitly expose something.

🐳 Step 1: Install Docker

On macOS or Windows, install Docker Desktop and make sure the Docker engine is running.

On Linux, you can install Docker Engine using the instructions for your distribution.

You can verify Docker is available with:

docker --version

Once Docker is working, we can create an isolated environment for Hermes.

📁 Step 2: Create a Dedicated Directory for Hermes

First, create a directory specifically for the Dockerized Hermes instance:

mkdir -p ~/.hermes-docker

This directory is important for two reasons.

1. Persistence

Docker containers are disposable.

If we remove and recreate the Hermes container, we don’t want to lose things like:

Configuration
Sessions
Memory
Skills
Profiles
Logs
Other persistent agent data

We’ll store those outside the container in ~/.hermes-docker.

2. It becomes part of our trust boundary

Anything inside this directory is intentionally available to Hermes.

Instead of mounting my entire home directory, I’m giving the agent one dedicated location.

That distinction matters.

⚙️ Step 3: Run the Hermes Setup Inside Docker

Now we can start the official Hermes Agent image and launch the setup process:

docker run --rm -it \
  -v ~/.hermes-docker:/opt/data \
  nousresearch/hermes-agent setup

The most important part here is:

-v ~/.hermes-docker:/opt/data

The -v option creates a bind mount between a directory on the host and a directory inside the container.

Think of it like this:

HOST                           CONTAINER
~/.hermes-docker   ───────▶   /opt/data

The left side:

~/.hermes-docker

is the directory we just created on the host.

The right side:

/opt/data

is where that directory appears inside the Docker container.

So when Hermes saves configuration, sessions, memory, skills, or other persistent data under /opt/data, those files are actually stored in ~/.hermes-docker on the host.

That means we can destroy and recreate the container without losing the Hermes environment.

More importantly, we’re sharing one dedicated directory instead of the entire host filesystem.

🧠 Step 4: Configure Hermes Agent

The first run downloads the required Docker image and launches the Hermes setup wizard.

From there, follow the normal Hermes configuration process.

In my setup, I:

Use the quick setup
Authenticate with Nous Portal
Select a model
Keep the terminal backend local
Skip messaging platform configuration for now

Once setup finishes, the configuration is stored in our persistent directory.

That means future containers using the same mount can reuse it.

🌐 Step 5: Start the Hermes Gateway and Dashboard

Now we can start Hermes as a background container:

docker run -d \
  --name hermes-docker \
  --restart unless-stopped \
  -v ~/.hermes-docker:/opt/data \
  -p 18642:8642 \
  -p 19119:9119 \
  -e HERMES_DASHBOARD=1 \
  nousresearch/hermes-agent gateway run

There are a few important things happening here.

Run in the background

-d

starts the container in detached mode.

Give the container a name

--name hermes-docker

makes it easier to manage later:

docker logs hermes-docker
docker stop hermes-docker
docker start hermes-docker

Keep the persistent directory

We’re mounting the same directory again:

-v ~/.hermes-docker:/opt/data

So Hermes sees the configuration we created during setup.

Enable the dashboard

-e HERMES_DASHBOARD=1

enables the Hermes Dashboard.

🔌 Understanding the Port Mapping

You’ll notice I’m using:

-p 18642:8642
-p 19119:9119

Docker port mappings follow this pattern:

HOST_PORT:CONTAINER_PORT

So:

18642 → 8642
19119 → 9119

The ports on the right are the ones Hermes uses inside the container.

The ports on the left are the ones exposed on my Mac.

I’m deliberately using different host ports because I already have another Hermes instance running locally and don’t want the ports to clash.

If you’re running only the Docker instance, you can use the default ports 8642 and 9119 instead of 18642 and 19119.

✅ Step 6: Verify the Container Is Running

Run:

docker ps

You should see the hermes-docker container running.

If something isn’t working, one of the first places to look is the container logs:

docker logs hermes-docker

This becomes particularly useful when configuring the dashboard.

🔐 Step 7: Configure Dashboard Authentication

When I initially opened the dashboard, it didn’t start correctly.

Instead of hiding that part from the tutorial, I kept it in because it’s a useful troubleshooting example.

Checking:

docker logs hermes-docker

showed that the dashboard needed an authentication provider.

Hermes provides authentication options for securing dashboard access.

I configured mine using Nous Portal authentication.

Once authentication was configured, the dashboard started successfully.

With my port mapping, I can access it at:

localhost:19119

And now the Hermes Dashboard is running while Hermes itself remains inside the container.

💬 Step 8: Chat With Hermes Inside the Container

You can also enter the running container directly:

docker exec -it hermes-docker bash

Then start Hermes:

hermes

At this point you’re interacting with Hermes from a terminal inside the Docker container, rather than running Hermes directly on the host.

That’s useful for testing.

But I don’t necessarily want to enter the container every time I want to use the agent.

There’s a better option.

🖥️ Step 9: Connect Hermes Desktop to the Docker Instance

Hermes Desktop can connect to the Gateway running inside our container.

Open Hermes Desktop and navigate to:

Settings → Gateway → Remote Gateway

Then configure it to use the Gateway exposed by your Docker container.

In my setup, that’s my url:

http://localhost:19119

After authenticating, save the configuration and reconnect.

Now Hermes Desktop communicates with the Dockerized Hermes instance.

So from the user’s perspective, I still get the convenient desktop interface.

But the agent itself is operating inside the container.

🛡️ What Does Docker Actually Protect?

This is probably the most important part of the tutorial.

Running an AI agent inside Docker does not automatically make it safe.

Docker gives us a useful isolation boundary, but that boundary depends heavily on how the container is configured.

For example, imagine doing this:

Host machine
      │
      ▼
Entire home directory
      │
      ▼
Docker container
      │
      ▼
Hermes Agent

You’ve technically containerized Hermes.

But you’ve also exposed a huge portion of your machine to it.

That’s not the setup I want.

Instead:

Host machine
      │
      ├── Personal files       ❌
      ├── SSH keys             ❌
      ├── Other projects       ❌
      ├── Docker socket        ❌
      │
      └── ~/.hermes-docker     ✅
                │
                ▼
          Hermes Container

The container should get access only to the resources required for the task.

🚫 1. Don’t Mount Your Entire Home Directory

If you mount your complete home directory into the container, Hermes could potentially access whatever the container’s permissions allow within that mount.

That might include personal files, source code, configuration files, credentials, or other sensitive data.

That’s why I’m using:

~/.hermes-docker

as a dedicated location instead.

If Hermes doesn’t need something, don’t expose it.

🔑 2. Be Careful With Credentials

The same principle applies to API keys and tokens.

Only provide the credentials Hermes actually needs.

Avoid unnecessarily exposing things such as:

~/.ssh

or unrelated cloud credentials and secrets.

Containerization doesn’t help much if you put every sensitive credential inside the container anyway.

⚠️ 3. Don’t Expose the Docker Socket Without a Very Good Reason

Another thing I’m deliberately not mounting is:

/var/run/docker.sock

Giving a container access to the host Docker daemon can dramatically expand what that container is capable of doing.

For an autonomous agent, that’s an especially important boundary to think about.

If Hermes doesn’t need to control Docker on the host, don’t give it that capability.

🧩 4. Docker Doesn’t Solve Prompt Injection

There’s another important limitation.

An AI agent can still encounter malicious or adversarial content.

For example, content from a webpage, repository, document, or other external source could attempt to manipulate the agent into performing unintended actions.

Putting the agent inside Docker doesn’t eliminate that problem.

The difference is what happens after the agent attempts the action.

If Hermes only has access to:

/opt/data

then we’ve limited the environment available to it.

If we’ve mounted our entire computer, exposed sensitive credentials, and given it access to the Docker daemon, the potential impact is very different.

That’s why I think containerization is better understood as blast-radius reduction, not a magic AI security solution.

🎯 The Principle: Least Privilege for AI Agents

As AI agents become increasingly capable, I think one traditional security principle becomes even more important:

Least privilege.

An agent should receive the minimum access required to complete its task.

Not:

Agent
  │
  └── Everything on my computer

But:

Agent
  │
  ├── Required files
  ├── Required credentials
  ├── Required network services
  └── Required tools

Nothing more.

This becomes increasingly important as agents gain longer-running autonomy, terminal access, browser access, external integrations, scheduled execution, and the ability to coordinate multiple tools.

Docker vs VM for AI Agents

Docker is not the only way to isolate an autonomous agent.

You could also use:

A virtual machine
A dedicated VPS
A sandboxed execution environment
A separate physical machine
More restrictive container runtimes

A VM generally provides a stronger isolation boundary than a standard application container because it virtualizes a larger portion of the environment.

Docker, however, is lightweight and convenient for local development.

For my Hermes setup, it gives me a useful middle ground:

Direct Host Execution
        ↓
Docker Container
        ↓
Virtual Machine / Dedicated Environment

The appropriate boundary ultimately depends on what capabilities you’re giving the agent and what resources it can access.

Final Architecture

After everything is configured, my setup looks roughly like this:

                   ┌─────────────────────────────┐
                   │        Host Machine         │
                   │                             │
                   │      Hermes Desktop         │
                   └──────────────┬──────────────┘
                                  │
                           Remote Gateway
                                  │
                                  ▼
              ┌─────────────────────────────────┐
              │       Docker Container          │
              │                                 │
              │       Hermes Gateway            │
              │       Hermes Dashboard          │
              │       Hermes Agent              │
              │                                 │
              │       /opt/data                 │
              └───────────────┬─────────────────┘
                              │
                         Bind Mount
                              │
                              ▼
                    ~/.hermes-docker
                              │
              ┌───────────────┴──────────────┐
              │ Config • Memory • Sessions   │
              │ Skills • Profiles • Logs     │
              └──────────────────────────────┘
       Rest of host filesystem → NOT MOUNTED
       SSH keys                → NOT EXPOSED
       Docker socket           → NOT EXPOSED

Hermes still has the environment it needs to work.

But we’re being much more deliberate about what crosses the boundary.

Final Thoughts

Autonomous AI agents are becoming capable of doing much more than generating text.

They’re increasingly able to:

⚡ Execute commands
🗂️ Manipulate files
🌍 Interact with external systems
🧰 Use specialized tools
🔄 Run multi-step workflows autonomously

That’s exciting.

But the more capable the agent becomes, the more important its execution environment becomes.

Running Hermes Agent inside Docker doesn’t solve every security problem.

It does, however, give us a practical way to control what the agent can access and reduce the potential blast radius when something unexpected happens.

The rule I follow is simple:

Only give an AI agent access to what it actually needs.

Everything else should stay outside the boundary.

🚀 What if your AI agent wasn’t tied to a single model provider? I connected Hermes Agent to OmniRoute to route across free AI models from multiple providers. Here’s the full setup 👇 #AI #AIAgents #OpenSource

Vivek Shetye — Tue, 15 Sep 2026 15:14:45 +0000

Vivek Shetye

Sep 15

How I Run Hermes Agent on FREE AI Models with OmniRoute 🚀

#ai #hermes #agents #opensource

7 min read

How I Run Hermes Agent on FREE AI Models with OmniRoute 🚀

Vivek Shetye — Tue, 15 Sep 2026 15:05:20 +0000

Free AI models are great.

Until your agent hits a quota halfway through a task.

Then suddenly you’re:

Searching for another provider
Generating another API key
Changing endpoints
Switching models
Restarting your workflow

For a simple chat application, that might just be annoying.

For an AI agent executing multi-step tasks, it becomes a much bigger problem.

So instead of connecting Hermes Agent directly to a single AI provider, I tried a different architecture:

                 ┌── OpenRouter ──→ Free Models
                 │
Hermes Agent ─→ OmniRoute
                 │
                 └── NVIDIA ──────→ Available Models

Hermes talks to one endpoint.

OmniRoute sits between the agent and the model providers and handles the routing.

The result is a surprisingly useful setup for experimenting with Hermes Agent on free AI models without tightly coupling the agent to one provider.

In this tutorial, I’ll walk through exactly how I set it up.

🎥 Prefer the Video?

I created a complete step-by-step video showing the setup, configuration, and a real Hermes Agent task running through OmniRoute.

Watch: How I Run Hermes Agent on FREE AI Models

If you’d rather understand the architecture and follow the setup at your own pace, keep reading.

The Problem With Depending on One Free AI Provider

There are now plenty of ways to experiment with capable AI models without immediately paying for API usage.

That’s great for:

Learning agent development
Building prototypes
Testing AI workflows
Experimenting with different models
Running personal automation
Comparing model capabilities

But there’s an obvious limitation:

Free access is rarely unlimited or guaranteed.

A provider may impose request limits.

A particular model might become unavailable.

Traffic might affect access.

A model could disappear from the free catalog.

Or you might simply exhaust your quota.

If your architecture looks like this:

Hermes Agent
     │
     ▼
Provider A
     │
     ▼
Model

your agent is directly dependent on that provider.

Once Provider A stops serving your requests, your workflow stops too.

Instead, I wanted something closer to this:

                     ┌── Provider A
                     │
                     ├── Provider B
Hermes Agent ──→ Gateway
                     ├── Provider C
                     │
                     └── Provider D

That’s where OmniRoute comes in.

🧠 What Is Hermes Agent?

Hermes Agent is an open-source agent harness from Nous Research for building AI agents capable of doing much more than generating chat responses.

Depending on the tools and configuration available to it, Hermes can perform tasks involving things like:

User Request
     ↓
Hermes Agent
     ↓
Reasoning
     ↓
Tool Calls
     ↓
Files / Terminal / Web / Other Tools
     ↓
Tool Results
     ↓
More Reasoning
     ↓
Final Result

This makes model availability particularly important.

A multi-step agent workflow might require many model calls before the task is finished.

If your model becomes unavailable halfway through, that can interrupt the entire workflow.

🚦 What Is OmniRoute?

Think of OmniRoute as a traffic director for AI models.

Rather than forcing your application to manage every model provider individually, OmniRoute provides a gateway between your application and those providers.

Instead of:

Application → Provider

you get:

Application → OmniRoute → Providers

For this experiment, our application happens to be Hermes Agent.

So the architecture becomes:

                       ┌──────────────┐
                       │  OpenRouter  │
                       └──────┬───────┘
                              │
┌──────────────┐      ┌───────▼───────┐
│ Hermes Agent │ ───→ │   OmniRoute   │
└──────────────┘      └───────┬───────┘
                              │
                       ┌──────▼──────┐
                       │   NVIDIA    │
                       └─────────────┘

OmniRoute is open source, can run locally, and supports a large number of AI providers.

More importantly for this experiment, it allows us to expose a single endpoint to Hermes while managing multiple model/provider connections behind it.

⚡ Why Put an AI Gateway Between Hermes and the Models?

You could simply connect Hermes directly to OpenRouter, NVIDIA, or another provider.

So why add another layer?

Because it separates two responsibilities.

Hermes Agent handles the task.

OmniRoute handles model access and routing.

Conceptually:

Hermes:
"What should I do next?"

        ↓

OmniRoute:
"Which configured model should serve this request?"

That separation becomes increasingly useful as you add more providers.

Instead of reconfiguring the agent every time your model strategy changes, Hermes continues talking to the same gateway.

You manage providers behind it.

🛠️ Step 1: Install OmniRoute Locally

The first thing I did was install OmniRoute locally.

After following the installation instructions, OmniRoute can be started from the terminal.

Once the server starts, it provides access to its local dashboard.

From there you can manage things like:

Providers
Models
Endpoints
API Keys
Routing
Analytics

This becomes the control plane for our model infrastructure.

Important security note 🔐

If you’re running a gateway locally or exposing it beyond your machine, don’t leave default credentials unchanged.

Create proper API keys and configure authentication appropriate for your environment.

🔌 Step 2: Connect OpenRouter

The first provider I added was OpenRouter.

Inside OmniRoute:

Providers
   ↓
OpenRouter
   ↓
Add Connection

You’ll need an OpenRouter API key.

Once the connection is configured, OmniRoute can import the models available through that account.

For this experiment, I enabled the option to import only free models.

The result looks conceptually like:

OpenRouter
     │
     ├── Free Model A
     ├── Free Model B
     ├── Free Model C
     └── ...

You can also perform health checks to see whether configured models are currently available.

That’s already useful but we’re still relying on one provider.

So let’s add another.

🟢 Step 3: Add NVIDIA as Another Provider

I repeated essentially the same process with NVIDIA.

Generate an API key through NVIDIA’s model/API platform and add the connection inside OmniRoute.

Now our gateway has multiple provider connections:

                OmniRoute
                   │
          ┌────────┴────────┐
          ▼                 ▼
     OpenRouter           NVIDIA
          │                 │
      Free Models      Available Models

And this is where the architecture starts becoming interesting.

Hermes doesn’t need separate configurations for both providers.

As far as Hermes is concerned, there is still only:

One API endpoint

OmniRoute handles what happens behind it.

🤖 Step 4: Connect Hermes Agent to OmniRoute

Now we need to point Hermes toward the gateway.

Hermes supports configuring a custom model endpoint.

Using the Hermes model configuration flow, I selected the custom endpoint option and provided:

API Base URL → OmniRoute local endpoint

API Key → OmniRoute API key

Compatibility → Auto Detect

For the model configuration, I used OmniRoute’s:

auto/best-coding

This is where the model selection gets abstracted away from Hermes.

Instead of telling Hermes:

Always use Model X

we’re effectively saying:

Send the request to OmniRoute.

Let the routing layer determine the appropriate
configured model for the request.

Once that configuration is saved, OmniRoute becomes a model endpoint available to Hermes Agent.

🧪 Step 5: Give Hermes a Real Agent Task

A simple “Hello World” wouldn’t tell us much.

So I gave Hermes something closer to an actual agent workflow.

The task was essentially:

Find the latest and most important AI news from reliable sources, summarize the key developments, and create a polished single-page visual report presenting the major stories, trends, and sources.

That means Hermes needs to do much more than answer a single prompt.

Conceptually:

Research
   ↓
Collect Information
   ↓
Evaluate Sources
   ↓
Summarize
   ↓
Identify Trends
   ↓
Generate Web Page
   ↓
Write Files

And here’s the important part:

Hermes was sending its model requests through OmniRoute throughout the workflow.

Hermes Task
    │
    ▼
OmniRoute
    │
    ▼
Best Available Configured Model
    │
    ▼
Response
    │
    ▼
Hermes Continues Task

After a minute or two, Hermes completed the task and generated the visual AI news report.

📊 Step 6: See Which AI Models Actually Handled the Requests

This was one of my favorite parts of the experiment.

After Hermes finished, I went back into OmniRoute.

Under its usage analytics, I could inspect information about the requests being sent through the gateway.

That included things such as:

Requests
Input tokens
Output tokens
Model usage
Which models actually served requests

So instead of the model-routing layer being a complete black box, I could inspect what happened after the task.

The workflow effectively becomes:

             ┌───────────────┐
             │ Hermes Agent  │
             └───────┬───────┘
                     │
                     ▼
             ┌───────────────┐
             │   OmniRoute   │
             └───────┬───────┘
                     │
            Routing Decision
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
      Model A                Model B
          │                     │
          └──────────┬──────────┘
                     ▼
                 Response

                     +

              Usage Analytics

For agent development, that observability is useful.

It gives you visibility into what your model-routing infrastructure is actually doing rather than only seeing the final agent output.

🧩 Why This Architecture Matters Beyond Free Models

The interesting part isn’t actually the word free.

It’s the abstraction.

Without a gateway:

Agent → Specific Provider → Specific Model

With a gateway:

Agent → Model Gateway → Routing Strategy → Provider/Model

That means the agent doesn’t necessarily need to care whether the underlying request eventually reaches one provider or another.

This becomes useful when experimenting with:

Multi-provider AI infrastructure

Different providers can sit behind one gateway.

Model fallback strategies

Your application doesn’t have to own every routing decision.

Cost optimization 💰

Different models can potentially be selected based on your infrastructure strategy.

Model experimentation

You can change the models behind the gateway without redesigning your entire agent integration.

Observability

A centralized gateway can give you a better view of model usage across your agent workflows.

And these patterns aren’t limited to Hermes.

The broader architecture is applicable to many AI applications:

AI Application
      ↓
AI Gateway
      ↓
Routing / Policies
      ↓
Multiple Models & Providers

As AI systems become more agentic, this layer becomes increasingly interesting.

⚠️ But There’s a Catch: “Free” Doesn’t Mean Unlimited

This is important.

Using multiple free-model providers does not magically create unlimited free inference.

Every provider has its own policies and constraints.

For example:

OpenRouter free models have limits.

NVIDIA access can vary depending on the model, account, availability, and traffic.

Free models may also:

Respond more slowly
Become temporarily unavailable
Change their limits
Disappear from catalogs
Have different context windows
Behave differently across tasks

And there’s another important consideration:

Always read the provider’s terms.

Some services have specific rules around:

Automation
API usage
Processing
Redistribution
Rate limits
Free-tier usage

A few minutes checking those requirements before building automation around a provider can save you problems later.

Think of this architecture as a way to use available resources more flexibly, not as a way to bypass provider limits.

🏗️ The Bigger Lesson: Decouple Your Agent From Your Model Provider

This experiment reinforced something I’ve increasingly found useful when working with AI agents:

The agent and the model infrastructure don’t have to be the same layer.

Your agent should focus on:

Reasoning
Tools
Memory
Tasks
Workflows

Your model gateway can focus on:

Providers
Models
Routing
Availability
Usage
Policies
Observability

Put those together and you get a much cleaner architecture:

┌──────────────────────────────┐
│          AI Agent            │
│                              │
│ Tools • Memory • Workflows   │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│          AI Gateway          │
│                              │
│ Routing • Models • Analytics │
└──────────────┬───────────────┘
               │
       ┌───────┼───────┐
       ▼       ▼       ▼
   Provider Provider Provider
      A        B        C

For small experiments, this might feel like an extra layer.

But once you’re testing multiple models and providers—or building longer-running agent workflows—the separation starts making much more sense.

🎯 Final Thoughts

If you’re learning AI agent development and want to experiment without immediately spending heavily on API calls, Hermes Agent + OmniRoute + free AI model providers is an interesting setup to explore.

You get:

✅ One endpoint for Hermes Agent

🔀 Multiple model/provider options behind the gateway

🧠 Automatic model selection strategies

📈 Centralized usage analytics

🛠️ The flexibility to add or change providers later

Just remember that the free tiers still have their own quotas, availability constraints, and terms.

The real value here isn’t “unlimited free AI.”

It’s building an agent architecture that isn’t unnecessarily coupled to one model provider.

Hermes Agent goes far beyond basic prompting. 🤖 I broke down /loop, /goal, Cron, Delegation, Bot Mode, Kanban & Mixture of Agents and when each actually makes sense. A practical mental model for building more autonomous AI agents. 👇

Vivek Shetye — Tue, 08 Sep 2026 15:29:15 +0000

Hermes Agent Challenge Submission: Write About Hermes Agent

Vivek Shetye

Sep 8

🤖 Hermes Agent Advanced Features Explained: Loop, Goal, Cron, Delegation, Bot Mode, Kanban & Mixture of Agents

#ai #agents #hermesagentchallenge #opensource

6 min read

🤖 Hermes Agent Advanced Features Explained: Loop, Goal, Cron, Delegation, Bot Mode, Kanban & Mixture of Agents

Vivek Shetye — Tue, 08 Sep 2026 15:03:20 +0000

Using Hermes Agent as a normal AI assistant is pretty straightforward.

You give it a task. It reasons about the problem, uses tools when necessary, and gives you a result.

But Hermes gets much more interesting when you ask a different question:

How do I get an AI agent to keep working without constantly prompting it?

Hermes Agent gives you several ways to do this.

You can create recurring loops, give the agent a persistent goal, schedule Cron Jobs, delegate work to sub-agents, create persistent specialist bots, coordinate multiple agents through Kanban, or even use multiple LLMs through Mixture of Agents.

The confusing part?

These features all increase agent autonomy, but they solve very different problems.

I recently published a video breaking down how these advanced Hermes Agent features work under the hood and, more importantly, when you should use each one.

🎥 Watch the full video:

Here’s the mental model I use to distinguish them.

🔁 1. Loop: Repeat Something Inside the Current Session

The easiest way to understand Hermes Agent’s /loop command is:

“Do this again later while I’m working in this session.”

A loop takes a prompt and reruns it on a recurring cadence.

For example:

Check my deployment every 5 minutes and tell me when it goes live.

The key distinction is that Loop is time-driven and session-scoped.

Use it for monitoring, repeated checks, polling, or other temporary recurring work associated with your current Hermes session.

🎯 2. Goal: Keep Working Until the Objective Is Complete

/goal solves a completely different problem.

Instead of saying:

“Do this again later.”

You’re saying:

“Keep working until this is actually done.”

Some tasks can’t reliably be completed in a single agent turn.

Maybe Hermes needs to modify code, run tests, inspect failures, make another change, and repeat the process.

That’s where Goal becomes useful.

After each continuation, the result can be evaluated to determine whether the objective has actually been completed.

Hermes also supports a completion contract, which can describe:

The desired outcome
How success should be verified
Constraints that must remain true
What is inside or outside the task’s scope
When Hermes should stop and ask for help

Instead of:

Fix authentication.

You can define something closer to:

Fix authentication so all tests pass.

Keep the existing login API response unchanged.

Only modify the authentication service and its tests.

Stop and ask me if a database migration becomes necessary.

Now both execution and evaluation have a much clearer definition of “done.”

One important misconception:

Goal does not automatically create an agent team, Kanban workflow, or independent background process.

It keeps the current Hermes session focused on a standing objective.

So:

Loop → Do this again later.

Goal → Keep working until this is done.

⏰ 3. Cron Jobs: Scheduled, Unattended AI Agent Workflows

Both Loop and Goal are connected to an active Hermes session.

But what if you want Hermes to wake up every morning and perform some work even when you’re not interacting with it?

That’s where Cron Jobs come in.

For example:

Every morning at 9 AM:

- Check my news feeds
- Find important AI developments
- Summarize them
- Send me a report on Telegram

When the scheduled time arrives, Hermes can launch a fresh agent session specifically for that run.

This makes Cron useful for things like:

📊 Daily reports
🔍 Recurring research
🛡️ Security audits
🧪 CI health checks
📰 News monitoring
📨 Scheduled summaries

The fundamental distinction is:

Loop
↓
Keep checking while this session is active.

Cron
↓
Wake up independently on a schedule and run the task.

If I’m watching a deployment for the next 20 minutes, I’d probably use Loop.

If I want a report generated every morning whether I’m there or not, I’d use Cron.

🧩 4. Sub-Agent Delegation: Temporary Parallel Workers

Now we move from one autonomous agent to multiple agents working simultaneously.

Suppose I ask Hermes:

Compare three approaches for implementing authentication
and recommend the best one.

The main agent could investigate each approach sequentially.

Or it could delegate:

Parent Agent
     │
     ├── Sub-Agent A → Research approach A
     ├── Sub-Agent B → Research approach B
     └── Sub-Agent C → Research approach C

The sub-agents can perform those tasks independently and return their findings to the parent.

This provides two major advantages.

⚡ Parallelism

Independent work can happen simultaneously rather than forcing the main agent to perform everything sequentially.

🧱 Context Isolation

Large research, coding, refactoring, and analysis tasks can consume significant context.

Delegating them means much of the intermediate work stays inside the sub-agent’s context rather than flooding the parent’s conversation.

The important distinction is that delegation is temporary.

A sub-agent exists to perform a focused task, return its result, and finish.

If you want a specialist identity that persists over time, you need something else.

🤖 5. Profiles & Bot Mode: Persistent AI Specialists

Imagine having separate Hermes agents for:

👨‍💻 Coding
🔬 Research
✍️ Content
📈 Analysis

You probably don’t want your research agent’s accumulated context and workflows constantly mixed with your coding agent.

That’s what Hermes Profiles help solve.

A profile acts like a separate Hermes environment with its own configuration and state.

This allows specialist agents to develop independently.

Bot Mode builds a more intuitive multi-agent experience on top of these persistent profiles.

The key distinction from delegation is persistence:

Sub-Agent
↓
Temporary worker for a task

Bot / Profile
↓
Persistent specialist identity

Multiple bots can also communicate with one another, allowing you to create workflows involving specialist agents.

But once those workflows become complicated, messages alone aren’t enough.

📋 6. Kanban: Durable Multi-Agent Orchestration

Suppose you have three persistent Hermes specialists:

Researcher
    ↓
Analyst
    ↓
Writer

You want the researcher to collect evidence.

Only after the research is finished should the analyst evaluate it.

And only after the analysis is complete should the writer produce the final report.

You could coordinate everything through agent messages.

But what happens when the workflow introduces:

Dependencies
Failures
Retries
Artifacts
Structured handoffs
Human review
Work lasting hours or days

Messages alone become fragile.

That’s where Hermes Kanban becomes useful.

Instead of the workflow existing only inside agent conversations, the work becomes structured, persistent task state.

Conceptually:

Research Topic
      ↓
Analyze Evidence
      ↓
Human Review
      ↓
Write Final Report

If Analyze Evidence depends on Research Topic, it waits.

Once the research task is completed, the dependent task can proceed.

This makes Kanban useful for durable, long-running multi-agent workflows.

And Kanban and delegation aren’t competing ideas.

A persistent Kanban worker could own a research task and then delegate parts of that research to several temporary sub-agents.

Think of it this way:

Delegation → temporary help

Bot Mode → persistent specialists

Kanban → coordination between specialists

🧠 7. Mixture of Agents: Multi-Model Reasoning

This is probably the easiest concept to confuse with multi-agent orchestration.

Mixture of Agents (MoA) operates at the model layer.

Imagine you’re solving a difficult software architecture problem.

Instead of asking one LLM to immediately decide what to do, you could ask several models to independently analyze the problem.

Reference Model A ──┐
                    │
Reference Model B ──┼──→ Aggregator Model → Hermes Agent Loop
                    │
Reference Model C ──┘

One model might notice a security problem.

Another might suggest a much simpler architecture.

Another might identify an assumption everyone else missed.

Their perspectives are then provided to an aggregator model, which becomes the model responsible for acting.

This is why Mixture of Agents is better understood as a model strategy rather than a traditional multi-agent workflow.

You’re not necessarily creating:

Research Agent
+
Coding Agent
+
Review Agent

Instead, you’re giving the acting model multiple model perspectives before it makes decisions.

The Hermes session, tools, and underlying agent loop can remain the same.

What changes is the model strategy powering that loop.

🗺️ The Mental Model

If you remember only one thing from this article, make it this:

Hermes Feature	Mental Model	Best For
🔁 Loop	Repetition	Repeated checks in the current session
🎯 Goal	Completion	Working toward a clearly defined outcome
⏰ Cron	Scheduling	Unattended scheduled workflows
🧩 Delegation	Temporary parallelism	Independent subtasks
🤖 Profiles / Bot Mode	Persistent identity	Long-lived specialist agents
📋 Kanban	Durable orchestration	Dependencies, handoffs, retries, and review
🧠 Mixture of Agents	Multi-model reasoning	Multiple LLM perspectives on difficult problems

The features become much easier to reason about once you stop grouping all of them under “autonomous AI agents.”

They’re different primitives for controlling different dimensions of autonomy.

🚀 Putting It All Together

The interesting part is that these concepts can be combined.

Imagine a software development workflow where:

Cron
  ↓
Starts scheduled workflow
  ↓
Kanban coordinates persistent specialist profiles
  ↓
Research Agent delegates work to temporary sub-agents
  ↓
Architecture Agent uses Mixture of Agents
  ↓
Goal keeps implementation focused until verification passes

Now you’re moving beyond a chatbot that happens to call tools.

You’re designing an agentic system where scheduling, persistence, parallelism, identity, orchestration, and model reasoning are separate architectural decisions.

And that distinction becomes increasingly important as AI agents become more autonomous.

Choosing an enterprise MCP gateway isn’t about the longest feature list. 👀 I compared Bifrost, Docker, Kong, Lunar MCPX & Microsoft MCP Gateway across identity, security, tool governance, deployment, and failure handling. Here’s what I found 👇

Vivek Shetye — Mon, 31 Aug 2026 15:05:34 +0000

Compares Bifrost to LiteLLM

Vivek Shetye

Aug 31

Best Enterprise MCP Gateway for Your AI Agents in 2026

#ai #mcp #agents #llm

10 min read

Best Enterprise MCP Gateway for Your AI Agents in 2026

Vivek Shetye — Mon, 31 Aug 2026 15:00:32 +0000

The best enterprise MCP gateway is not the product with the longest feature list. It is the one whose identity, policy, deployment, and failure model match your agents. After reviewing the current MCP specification and the leading gateway options, Bifrost is one of my strongest shortlist choices for application teams that want model routing and MCP tool access in the same self-hostable gateway, an embeddable Go SDK, and an explicit application-controlled tool-execution step. Its open-source codebase also gives platform teams an inspectable starting point.

That recommendation has boundaries. Bifrost Enterprise, not the open-source edition alone, is the relevant tier if you require high-availability clustering, enterprise identity federation, admin RBAC, and audit-grade logs. And if your main problem is container isolation, Kubernetes lifecycle management, or extending an existing API gateway, another product may fit better.

The short answer

Bifrost is a strong enterprise MCP gateway for teams that want one Go-based, self-hosted data plane for LLM provider traffic and MCP tools. It connects to multiple MCP servers, exposes their tools through one endpoint, supports shared and per-user upstream authentication, applies layered tool allow-lists, and keeps tool execution explicit by default.

It is not universally “the best.” Docker MCP Gateway is compelling for isolated local server runtimes; Kong is a natural extension of an existing Kong estate; Microsoft MCP Gateway targets Kubernetes-managed server lifecycle; and Lunar MCPX focuses on dedicated MCP aggregation and tool controls.

What is an enterprise MCP gateway?

An enterprise MCP gateway is an infrastructure layer between AI agents and Model Context Protocol servers. It gives agents one governed entry point for discovering and calling tools while centralizing identity, credential handling, authorization, routing, logging, and policy enforcement.

Without a gateway, every agent must connect to every server independently:

Direct connections

Agent A ─┬─ GitHub MCP server
         ├─ database MCP server
         └─ internal API MCP server
Agent B ─┬─ GitHub MCP server
         └─ database MCP server

With a gateway

Agents ── MCP gateway ─┬─ GitHub MCP server
                       ├─ database MCP server
                       └─ internal API MCP server
             │
             └─ identity, tool policy, credentials,
                approvals, traces, limits, audit

The direct model is fine for one developer and a few trusted tools. At enterprise scale, it duplicates configuration and secrets, scatters logs, and makes it difficult to answer a basic incident question: which user, through which agent, called which tool with what authority?

“MCP gateway” describes several different products

Flat comparison tables are misleading because the category contains at least four architectures:

Combined LLM and MCP gateways, such as Bifrost, govern model requests and MCP tool access in one gateway.
Dedicated MCP aggregation and control layers, such as Lunar MCPX, emphasize MCP federation, tool policy, and observability.
API gateways with MCP support, such as Kong, apply an established gateway and plugin ecosystem to MCP traffic and API-to-tool conversion.
Runtime and lifecycle gateways, such as Docker and Microsoft MCP Gateway, focus on where MCP servers run and how they are isolated or managed.

Before picking a vendor, decide which problem you actually have. A platform team replacing separate model and tool proxies has a different requirement from a security team placing policy in front of existing remote MCP servers.

Six tests for an enterprise MCP gateway

1. Whose identity reaches the tool?

Ask whether every caller collapses into one shared service account or whether the gateway preserves end-user identity. Shared credentials are simpler, but per-user OAuth lets the downstream system retain its own permission model and audit trail. Bifrost documents the available patterns in its MCP connection and authentication guide and lets operators inspect and revoke per-user MCP sessions.

The current MCP authorization specification requires each access token to be used only for its intended service. An MCP gateway must not reuse a client’s token to call another API; it should use a separate token with only the permissions that API requires.

2. Where is tool policy enforced?

Filtering tools/list reduces what the model sees, but discovery-time filtering alone is not authorization. Re-check access when tools/call executes. For destructive operations, the gateway or application should also support approval, argument validation, or a hardened read-only variant. Bifrost’s virtual-key MCP controls enforce an allow-list at inference and again at tool execution.

3. How are credentials stored and refreshed?

Verify shared OAuth, per-user OAuth, workload identities, static-key storage, token refresh, revocation, and secret redaction. Also check whether request headers are forwarded automatically. Bifrost’s connection documentation says incoming headers are not forwarded by default and describes per-client allow-lists. Safe defaults matter because a convenience feature can become a credential-exfiltration path.

4. Are logs actually audit evidence?

Operational logs, OpenTelemetry traces, and immutable administrative audit trails solve different problems. You normally need all three: traces for latency and failures, request logs for debugging, and retained audit events for security investigations and change accountability. Bifrost documents built-in observability and request logging, OpenTelemetry export, and separate Enterprise audit logs.

5. What fails, and how?

Test a dead upstream server, expired OAuth token, changed tool schema, slow tool, gateway-node loss, and duplicate call. “Retries supported” is not enough; retrying a read is different from retrying create_invoice after a timeout. Bifrost documents its MCP connection states, health checks, and retry behavior, but your proof of concept should still validate the failure semantics of each tool.

6. Which MCP specification does it implement?

The finalized MCP 2026-07-28 release made the HTTP protocol core stateless and added Mcp-Method and Mcp-Name routing headers. Many product pages still describe legacy SSE or session-affinity behavior. Require a version-compatibility matrix for your actual clients and servers rather than accepting “MCP compatible” as a complete answer.

How Bifrost’s MCP gateway works

Bifrost occupies a useful position because it is both an AI model gateway and an MCP gateway. According to its MCP architecture documentation, it acts as an MCP client to external tool servers and can act as an MCP server to clients such as Claude Desktop.

The verified request path looks like this:

Bifrost connects to upstream MCP servers and discovers their tools. Its connection guide documents STDIO, HTTP, and SSE connections.
Remote connections can use static headers, shared OAuth, or per-user OAuth. The connection and authentication documentation explains when each option applies.
Bifrost applies stacked tool filtering at client configuration, request, and virtual-key levels. Empty client tool lists deny access by default.
The model receives only the allowed tool definitions. Tool names are prefixed by client, avoiding collisions between servers.
By default, the model only proposes a tool call. Your application reviews it and invokes the separate tool-execution endpoint. Agent Mode can opt selected tools into automatic execution.
Requests and model operations can be exported through Bifrost’s OpenTelemetry integration.

That separation between proposal and execution is valuable. It creates a clean approval and validation point without claiming that a human is automatically in the loop. Your application still has to implement the approval policy.

Open source versus Enterprise

Bifrost’s open-source gateway is Apache 2.0 and includes the MCP connection/aggregation path, virtual-key governance, tool filtering, rate and budget controls, and observability plugins. The Enterprise overview describes a strict superset that adds high-availability clustering, identity-provider integrations, RBAC, audit-grade logging, and private deployment options. The clustering documentation covers peer discovery, state synchronization, and node failover.

Keep that boundary in your evaluation sheet. “Open source” does not mean every enterprise control is in the community edition. For deployment planning, Bifrost also provides an official Helm guide for OSS and Enterprise installations on Kubernetes.

Bifrost publishes impressive gateway-overhead figures, but I would treat them as vendor benchmarks. They measure Bifrost’s gateway path, not your end-to-end MCP tool latency, and there is no common independent benchmark here for ranking all six products.

Bifrost compared with enterprise MCP gateway alternatives

Gateway	Best fit	Verified differentiator	Watch before choosing
Bifrost	One self-hosted LLM + MCP gateway	Layered tool filtering, explicit execution, per-user upstream auth, Go gateway	Enterprise-only HA/identity/audit features; verify `2026-07-28` compatibility
Docker MCP Gateway	Docker-native development and server isolation	Runs MCP servers in restricted containers and manages lifecycle/credentials	Enterprise governance is a separate, invite-only offering in current docs
Kong AI Gateway	Existing Kong/API platform estates	MCP passthrough, API-to-MCP conversion, ACLs, rate limits, metrics	The AI MCP Proxy requires an AI Gateway Enterprise license
Lunar MCPX	Dedicated MCP aggregation and tool hardening	Tool groups, hardened tool variants, agent access control	Confirm which identity, audit, and secret capabilities require the Lunar platform beyond MCPX core
Microsoft MCP Gateway	Azure/Kubernetes server lifecycle	Reverse proxy plus adapter deployment, session-aware routing, Entra integration	A heavier, Kubernetes-oriented control plane; review the repository architecture

Why Bifrost belongs on the shortlist

Bifrost's case is strongest when the same application needs governed model routing and MCP tool access. Three documented design choices support that fit:

Bifrost returns proposed tool calls for review before the application invokes a separate execution API, creating an explicit point for approvals and argument validation.
Teams can deploy it as a gateway or integrate it directly through the Go SDK.
Its MCP Sessions UI supports inspection, re-authentication, editing, and revocation of per-user OAuth and header credentials.

Those capabilities do not make Bifrost the automatic winner for every deployment. They explain why it is a credible proof-of-concept candidate for the buyer profile in this article.

Kong’s configuration surface is an advantage if you already run Kong and overhead if you do not. Docker’s container isolation is excellent for local STDIO servers but does not automatically answer enterprise identity governance. Microsoft’s lifecycle manager is useful when the gateway should deploy servers, which Bifrost does not position as its primary job.

When I would choose Bifrost

I would put Bifrost on the proof-of-concept shortlist when:

the platform already needs multi-provider model routing as well as MCP;
the team prefers an inspectable, self-hosted Go service or wants to embed the gateway through its Go SDK;
explicit tool execution fits the approval design;
virtual-key and per-request tool allow-lists are sufficient for application-level access;
OpenTelemetry and private deployment are operational requirements; and
there is a clear path to Enterprise if clustering, SSO/SCIM, RBAC, and audit logs become mandatory.

I would look elsewhere first when the primary requirement is a curated connector catalog, container-per-server isolation, Kubernetes-managed MCP server deployment, or deep integration with an existing API gateway.

A proof-of-concept checklist

Do not evaluate an MCP security gateway from a demo dashboard alone. Run these tests with one read tool and one destructive tool:

two users with different upstream permissions;
discovery and execution denial for the restricted user;
OAuth expiry, refresh, and revocation;
attempted forwarding of an unapproved authorization header;
a tool description/schema change after approval;
gateway restart and upstream-server failure during a call;
trace correlation from agent through gateway to server;
audit export and actor attribution;
concurrent clients using the exact MCP protocol versions you deploy; and
edition/license mapping for every control in the acceptance criteria.

Final recommendation

Bifrost is one of the strongest enterprise MCP gateway options in 2026 for application teams that want unified LLM and MCP infrastructure, self-hosting or Go embedding, per-user MCP credential operations, layered tool governance, and an explicit execution boundary. That is a specific architectural fit, not a universal victory.

Choose the shape before the brand. Then require primary-source evidence and a failure-oriented proof of concept. For a Bifrost evaluation, compare the OSS and Enterprise editions explicitly and make 2026-07-28 protocol conformance part of acceptance testing.

Frequently asked questions

Do AI agents always need an MCP gateway?

No. A single trusted agent connected to a few local servers may be simpler without one. A gateway becomes valuable when multiple agents, users, credentials, policies, or audit requirements need one enforcement point.

What is the difference between an MCP gateway and an API gateway?

An API gateway primarily understands routes, services, and HTTP consumers. An MCP gateway understands MCP operations and entities such as tool discovery and execution. Products like Kong add MCP-aware capabilities to an API gateway; Bifrost combines MCP with model-provider routing.

Can Bifrost manage multiple MCP servers?

Yes. Bifrost connects to multiple upstream MCP servers, discovers their tools, filters them, and can expose the allowed aggregate through its /mcp gateway endpoint. It calls each upstream connection an MCP client.

Is Bifrost an open-source MCP gateway?

Yes. The Bifrost repository uses the Apache 2.0 license. Clustering, enterprise identity/RBAC, audit-grade logs, and some private deployment capabilities belong to Bifrost Enterprise, so evaluate the edition as well as the project.

How do you secure MCP servers behind a gateway?

Authenticate users and workloads, use audience-bound tokens, obtain separate upstream credentials, allow-list tools, enforce policy again at execution, require approval for risky calls, restrict header forwarding, trace every call, and retain security-relevant audit events. Also test protocol versions and tool-schema changes rather than treating the gateway as a complete security boundary by itself.

🤖 Hermes Agent Bot Mode or Kanban which should you use? I break down the difference, when each makes sense, and build a real workflow combining both for parallel AI agent research + automation. ⚡ Would love to hear how you’re using them! 👇

Vivek Shetye — Tue, 25 Aug 2026 15:23:17 +0000

Vivek Shetye

Aug 25

Hermes Agent Bot Mode vs Kanban: When to Use Each (and Why I Use Both)

#showdev #ai #hermes #agents

7 min read