DEV Community: Ted Kim

Harness Engineering — Building the Environment Your Coding Agent Works In

Ted Kim — Wed, 12 Aug 2026 16:50:11 +0000

Why I Wrote This

I actively use AI agents (Cursor, Claude Code, etc.) across multiple projects. At first, having an agent write code was impressive enough on its own. But as I integrated them more deeply into real projects, I kept running into recurring problems.

Every time I open a new session, the agent forgets the project conventions
It repeats the same mistakes today that we already solved yesterday
The quality of agent-generated code fluctuates wildly between sessions
When managing multiple projects, I have to repeat the same setup for each one

The root cause of these problems wasn't a lack of agent intelligence — it was that the environment surrounding the agent was not properly set up. As 2026 arrived, this concern spread across the industry and began to be systematized under the name "harness engineering."

After first applying harness engineering to a company project, I experienced its effectiveness firsthand and decided to apply the same structure to my personal projects. During this process, I felt the need for a "reference document that can quickly turn any project into a harness structure," which led me to write this guide.

This guide goes beyond explaining the concept of harness engineering — it covers what to apply based on your project type. From personal blogs to multi-agent automation systems, it's structured to help you design a harness that matches your project's scale and complexity.

1. What Is Harness Engineering?

Definition

Harness engineering is an infrastructure design discipline for operating AI agents safely and reliably. The term "harness" derives from horse tack that controls a horse's power, referring to a system that guides the powerful but unpredictable force of AI agents in the right direction.

"The model is a commodity. The harness is the moat."
— harness-engineering.ai

Analogy

Element	Analogy	Description
Horse	AI Model	Powerful and fast, but doesn't know where to go on its own
Harness	Infrastructure	Constraints, guardrails, and feedback loops that let the model's power be used productively
Rider	Human Engineer	Doesn't run directly — provides direction

Why Harness Engineering Now?

2025 was the year that proved AI agents can write code
2026 is the year we realized the key isn't the agent itself, but the harness

LangChain improved their Terminal Bench 2.0 score from 52.8% to 66.5% by changing only the harness, without modifying the model at all. The OpenAI Codex team built production applications exceeding 1 million lines without manually writing a single line of code.

Comparison with Related Concepts

Concept	Scope	Focus	Reliability Improvement
Prompt Engineering	Single interaction	Crafting effective prompts	5-15%
Context Engineering	Model context window	Optimizing information the model sees	15-30%
Harness Engineering	Entire agent system	Environment, constraints, feedback, lifecycle	50-80%

2. Core Components of Harness Engineering

2-1. Context Engineering

Ensuring the agent has the right information at the right time.

Static Context:

Repository-local documents (architecture specs, API specs, style guides)
CLAUDE.md files encoding project-specific rules (or AGENTS.md)
Interlinked design documents (linters automatically verify link validity and document freshness)

Dynamic Context:

Observational data accessible to the agent (logs, metrics, traces)
Directory structure mapping at agent startup
CI/CD pipeline status and test results

Core rule: From the agent's perspective, anything not accessible within the context doesn't exist. Knowledge in Google Docs, Slack threads, or people's heads is invisible to the system. The repository must be the single source of truth. — OpenAI

2-2. Architectural Constraints

Instead of telling the agent "write good code," mechanically enforce what good code looks like.

Dependency Layering:

Types → Config → Repo → Service → Runtime → UI

Each layer can only import from layers to its left, enforced through structural tests and CI verification.

Constraint Enforcement Tools:

Deterministic linters — custom rules that automatically flag violations
LLM-based auditors — agents that review other agents' code
Structural tests — architecture tests for AI-generated code
Pre-commit hooks — automatic checks before code is committed

Paradoxically, constraining the solution space improves agent productivity. When agents can generate anything, they waste tokens exploring dead ends. When the harness defines clear boundaries, agents reach correct solutions faster. — NxCode

2-3. Guardrails

Technically controlling both inputs and outputs of AI agents to preemptively block behavior outside the designed purpose scope.

Input stage: Detect / block prompt injection or confidential information leakage
Output stage: Automatically filter harmful content or hallucinations

2-4. Verification Loops

A structure that verifies the agent's work at each step before allowing progression. This is the component with the highest ROI in a harness.

# Verification loop pattern (pseudocode)
def run_agent_with_verification(task, tools, cost_ceiling):
    context = assemble_context(task)
    total_cost = 0

    while not task.is_complete():
        action = agent.plan(context, tools)
        result = execute_tool(action)

        verification = verify_output(result, action.expected_schema)
        if not verification.passed:
            if verification.retry_recommended:
                result = retry_with_backoff(action, max_retries=3)
            else:
                return TaskResult(status="failed", reason=verification.reason)

        total_cost += result.tokens_used
        if total_cost > cost_ceiling:
            return TaskResult(status="budget_exceeded", partial=context)

        context = update_context(context, result)

    return TaskResult(status="complete", output=context.final_output)

"Highest ROI" means the greatest effect relative to effort invested. Among all harness components, verification loops deliver the largest quality improvement with the least effort.

Implementation cost: 50-150ms additional latency per step, a few dozen lines of code
Effect: Task completion rate 83% → 96% (without changing model or prompts)

If you're building a harness for the first time and wondering "what should I build first?" — start with verification loops for the best cost-effectiveness.

2-5. Cost Envelope Management

Setting per-task budget ceilings that the harness enforces regardless of the agent's intent.

Cost envelopes are not just financial controls — they're reliability signals
A task hitting its budget ceiling means it's operating abnormally (bad upstream response, context drift, tool integration error, etc.)

2-6. Entropy Management

Regularly cleaning up entropy that accumulates in AI-generated codebases over time.

Document consistency agent — verifies documents match current code
Constraint violation scanner — re-scans code that passed previous checks
Pattern enforcement agent — identifies / fixes deviations from established patterns
Dependency auditor — tracks / resolves circular or unnecessary dependencies

2-7. Agent Memory

A structure for persistently accumulating knowledge, discovered patterns, and ongoing decisions that agents learn across sessions within the repository. Since agents are fundamentally stateless, all context disappears when a session ends. The harness compensates for this "amnesia."

Three time axes of context:

Time Axis	Nature	Examples	Change Frequency
Static Context	Pre-written documents	ARCHITECTURE.md, API specs	Rarely changes
Accumulated Context	Knowledge discovered / learned during work	Design decision records, failure pattern notes, ongoing interests	Grows per session
Dynamic Context	Data generated at runtime	Logs, metrics, CI status	Changes every time

Static context is written at project inception, and dynamic context is automatically generated at runtime. Accumulated context sits between them — a knowledge layer that grows incrementally as agent work accumulates.

Implementation pattern for accumulated context:

<project root>/
├── docs/
│   └── decisions/                    # Architecture Decision Records (ADR)
│       ├── 001-static-site-generator-choice.md
│       └── 002-deployment-strategy.md
├── .memory/                          # Agent memory
│   ├── learnings.md                  # Patterns, failure causes, know-how
│   ├── current-focus.md              # Current interests, priorities
│   └── session-notes/                # Per-session summaries (optional)
│       └── 2026-04-04.md

Native memory support by tool:

Tool	Memory Mechanism	Storage Location	Auto / Manual
Claude Code	Auto Memory	`~/.claude/projects/<project>/`	Auto-accumulate
Claude Code	CLAUDE.md	Project root	Manual
Cursor	`.cursor/rules/`	`.cursor/rules/*.mdc`	Manual
Cursor	Notepads	Cursor sidebar	Manual (inject via `@notepad`)

Claude Code has an Auto Memory feature that automatically saves patterns discovered during sessions, but Cursor requires manual knowledge management through .cursor/rules/ and Notepads. Using repository-based memory (the .memory/ pattern above) that doesn't depend on any specific tool makes it accessible from any tool.

Core principles:

Record failure causes and solutions so the agent "doesn't repeat the same mistakes"
Build a habit of committing "what I learned this session" to the repository before ending the session
Accumulated context is also subject to entropy management — regularly clean up outdated or invalidated notes

2-8. Observability & Evaluation

Observability: Structurally tracking what the agent did, why it did it, and what occurred at each step
Evaluation: An automated pipeline that continuously measures agent performance against defined criteria

3. Harness Engineering Application Levels

Harness engineering application depth varies based on project scale and complexity.

Level 1: Basic Harness (Individual Developer)

CLAUDE.md file containing project conventions (Cursor references via .cursor/rules/)
Pre-commit hooks for linting and formatting
Test suite the agent can run for self-verification
Clear directory structure with consistent naming conventions
Agent memory (.memory/) for cross-session knowledge accumulation

Suitable for personal blogs, documentation sites, personal tools, and side projects where a single agent is directly instructed by the user.

Level 2: Team Harness (Small Team)

Level 1 plus:

Architectural constraints enforced by CI
Shared prompt templates for common tasks
Documentation-as-code verified by linters
Code review checklists specifically for agent-generated PRs
Agent role boundaries and change scope limits

Suitable for environments where multiple developers use agents on the same repository. The focus is on preventing inter-agent conflicts and minimizing code quality variance.

Level 3: Production Harness (Engineering Organization)

Level 2 plus:

Custom middleware layers (loop detection, reasoning optimization)
Observability integration (agents read logs and metrics)
Entropy management agents running on schedule
Harness version control and A/B testing
Agent performance monitoring dashboards
Escalation policies for agent deadlocks

Suitable for systems where agents autonomously execute pipelines, integrate with external APIs at scale, and where agent judgment errors directly cause financial / operational losses.

4. Applying Harness Engineering to Your Project

4-1. Service Characteristics Analysis

Before designing a harness, first identify the target project's core characteristics through these questions:

Analysis Item	Key Question
Agent Structure	Single agent or multi-agent pipeline?
Data Processing	What data sources are involved? Is parallel processing needed?
Human Involvement	Is Human-in-the-Loop needed? At which stages is approval required?
Automation Level	Is there autonomous scheduling / automatic execution?
External Integration	What external APIs or services are integrated?
Risk Level	What impact does an agent's wrong judgment have? (financial, operational, security)

Based on these answers, classify your project into one of these types to determine which Phases to apply:

Project Type	Agents	Automation	External Integration	Risk	Recommended Phases	Examples
A. Personal / Static	Single (IDE-based)	None	None or minimal	Low	1~2	Blogs, doc sites, personal tools
B. Team / Web App	Single or few	CI/CD	Some APIs	Medium	1~3	Web apps, SaaS backends
C. Multi-Agent / Automation	Multiple, pipeline	Schedule / trigger	Many APIs	High	1~5	Ad automation, data pipelines
D. Production / Enterprise	Multiple, large-scale	Fully automated	Large-scale	Very high	1~7	Financial systems, infra automation

4-2. Risk Identification

Identify risks that can occur when operating without a harness based on project characteristics:

#	Risk Type	Description	Applicable Types
1	Financial malfunction	Agent's wrong judgment directly leads to monetary losses	C, D
2	Hallucination-based decisions	False analysis results lead to wrong follow-up actions	All types
3	Cost runaway	Repeated external API calls, LLM token accumulation cause cost control failure	B, C, D
4	Data contamination	Data from different contexts mixes during parallel execution	C, D
5	Agent doom loops	Same task repeatedly detected / processed in multi-agent pipelines	C, D
6	Duplicate execution	Periodic triggers cause the same task to execute redundantly	C, D
7	Quality drift	Agent-generated content / code quality fluctuates between sessions	All types
8	Context loss	Knowledge learned between sessions isn't carried over, repeating the same mistakes	All types

For Type A (personal projects), risks 2, 7, and 8 are most relevant, and Phase 1~2 harnesses (rule files, pre-commit hooks, agent memory) are sufficient. As you move toward Types C~D, risks 1, 3~6 become more critical, requiring Phase 3+ harnesses.

4-3. Harness Engineering Design

A. Context Engineering Application

Configure the project repository as the agent's single source of truth. Recommended directory structure:

<project root>/
├── CLAUDE.md                          # Entry point for agent behavior rules
├── ARCHITECTURE.md                    # Top-level system architecture map
├── .cursor/
│   └── rules/
│       └── general.mdc                # Connects Cursor to reference CLAUDE.md
├── docs/
│   ├── design-docs/                   # Feature design documents
│   ├── api-specs/                     # API endpoint specifications
│   ├── references/                    # SDK/framework references
│   └── quality/                       # Quality standards

Core principles:

Maintain CLAUDE.md as a table of contents, not an encyclopedia (about 100 lines)
Record all design decisions as documents within the repository (no Slack / Google Docs)
From the agent's perspective, information it can't search for doesn't exist

B. Agent Behavior Constraints

Applicable to: Project Type B and above (Team / Web App). For single-agent personal projects, glob-based rule files and pre-commit hooks are sufficient.

Clearly define agent behavior boundaries based on project characteristics.

1) Role Boundaries per Agent

Explicitly define what each agent can and cannot do:

<Agent Name> (<Role>)
  - Allowed: <list of tasks this agent can perform>
  - Forbidden: <list of tasks this agent must never perform>

Core principles for designing role boundaries:

Least privilege: Each agent has only the minimum permissions needed for its role
Separation: Separate data query / analysis agents from execution / modification agents
Approval required: High-risk actions (financial impact, data changes, external system calls) must go through an approval stage

2) Input / Output Guardrails

Input: External data source response validation, context isolation verification
Output: Change scope limits, hallucination filtering
Pipeline: Inter-stage data schema validation, identifier consistency checks

3) Task Cost Envelopes

Cost ceilings by task type (define per project):
  - Simple query/analysis: baseline × 1
  - Complex analysis/diagnosis: baseline × 2
  - Full pipeline execution: baseline × 10
  - Report generation: baseline × 5
  - Tasks involving external API calls: baseline × 3

  → On ceiling breach: halt task + alert + escalate

C. Verification Loop Application

Application level: Available for all project types, but implementation depth varies.

Type A (Personal): Pre-commit hooks and build validation serve as self-verification.

Type B (Team): Integrate automated tests, lints, and PR reviews into CI/CD.

Type C~D (Automation / Enterprise): Dual / triple structure of agent self-verification + Human-in-the-Loop is needed.

A dual structure is recommended for verification loops:

1) Agent Self-Verification

Pipeline step execution
  → Agent generates result
  → Self-verify result (data integrity, schema compliance)
  → Auto-retry on failure (max 3 attempts)
  → Pass to next step on success

2) Human Verification (Human-in-the-Loop)

High-risk actions request approval via operations tools
Operator approves / rejects
On rejection, feedback is passed to the agent to suggest alternatives

3) Output Verification Loop

Apply verification loops to agent-generated outputs as well:

Agent creates draft → verification (automated or human) → auto-rewrite on rejection

D. Multi-Agent Harness Middleware

Applicable to: Project Type C and above (Multi-Agent / Automation). Unnecessary for projects where a user manually directs a single agent.

For multi-agent systems, apply middleware to the pipeline:

Pipeline execution request
  → ContextIsolationMiddleware   (context isolation, prevent data contamination)
  → CostEnvelopeMiddleware       (cost envelope check, block on overrun)
  → LoopDetectionMiddleware      (prevent repeated task processing)
  → InputValidationMiddleware    (input data integrity validation)
  → [Agent Execution]
  → OutputValidationMiddleware   (result schema validation, change scope limits)
  → ApprovalGateMiddleware       (risk assessment, route to approval on high risk)
  → ExecutionAuditMiddleware     (execution history recording)

E. Observability Integration

Applicable to: Project Type C and above. For manual-use projects, agent memory and Git history provide sufficient observability.

Observable Data	Application
Agent execution history	Execution history dashboard, success / failure tracking
Pipeline stage status	Bottleneck identification, health monitoring
Approval / rejection ratio	Agent judgment accuracy evaluation
External API call volume	API quota management, call failure rate tracking
Cost tracking	LLM token, external service cost optimization

F. Entropy Management — Maintaining Agent Quality

Scope: Available for all project types, but scope varies.

Type A~B: Periodic cleanup of .memory/learnings.md, link verification is sufficient.

Type C~D: Dedicated agents perform regular quality audits.

[Periodic tasks — select/configure per project]
  - Prompt drift detector: Monitor consistency between system prompts and actual output
  - Data integrity checker: Verify referenced data/settings are still valid
  - Output quality audit: Verify auto-generated results are based on actual data
  - Tool function consistency check: Verify tool definitions are in sync with external service schemas

4-4. Harness Engineering Application Roadmap

You don't need to apply all Phases sequentially. Based on the project type classification, select and apply only the Phases you need.

Phase	Timing	Application Content	Expected Effect	Target Type
Phase 1	Immediate	Agent instruction files, directory structure documentation, pre-commit hooks	Development environment quality	All projects
Phase 2	Short-term	Agent memory structure (`.memory/`, decision records)	Cross-session continuity	All projects
Phase 3	Short-term	Agent role boundaries, I/O guardrails, context isolation	Pipeline stability	B+
Phase 4	Mid-term	Middleware chain, loop detection, cost envelope management	Parallel processing safety	C+
Phase 5	Mid-term	Automated task verification, duplicate execution prevention, API quota management	Automation stability	C+
Phase 6	Long-term	Observability dashboard, entropy management agent, A/B testing	Operations optimization	D
Phase 7	Long-term	Auto-approval threshold learning, agent benchmarks, harness version control	Autonomous operations	D

Application Examples by Project Type

Type A — Personal Blog / Documentation Site / Personal Tool

Phase 1: CLAUDE.md (or .cursor/rules/), ARCHITECTURE.md, Pre-commit hooks
Phase 2: .memory/learnings.md, current-focus.md, docs/decisions/
         → These two phases alone complete an environment where the agent
           consistently understands the project and accumulates knowledge
           across sessions.

Type B — Team Web Application / SaaS Backend

Phase 1~2: (Same as Type A)
Phase 3: Architecture lints enforced by CI, PR review checklists,
         change scope limits for agent-generated code
         → Reduces code quality variance between team members
           and prevents excessive agent changes.

Type C — Multi-Agent Automation System

Phase 1~3: (Apply through Type B)
Phase 4: Inter-agent data isolation middleware, cost envelope management,
         doom loop detection and auto-halt
Phase 5: Duplicate execution prevention for scheduled tasks (idempotency keys),
         external API quota management, retry policies on failure
         → Achieves "controlled autonomy."

Type D — Production Enterprise System

Phase 1~5: (Apply through Type C)
Phase 6: Real-time dashboard, automated entropy management agent,
         agent output A/B testing
Phase 7: Auto-approval threshold learning from repeated patterns,
         agent performance benchmark suite, harness config version control and rollback
         → Builds a sustainable operations framework.

5. Agent Instruction File Strategy

The pattern of "placing agent behavior instructions as markdown in the repository root" has become a de facto standard. However, each tool recognizes different filenames:

Tool	Default Instruction File	AGENTS.md Recognition	CLAUDE.md Recognition
OpenAI Codex	`AGENTS.md`	Official support	Not supported
Cursor	`.cursor/rules/`	Recognized (lowest priority)	Not supported
Claude Code	`CLAUDE.md`	Not supported	Official support

Covering all tools with a single file is currently impossible. (As of April 2026)

Recommended strategy: Main instruction file + tool-specific links

Choose the main instruction file based on your primary AI tool, and have other tools reference it:

Pattern A: Claude Code Main + Cursor Sub

CLAUDE.md                  ← Write actual rules here
.cursor/rules/general.mdc  ← "See CLAUDE.md"

Pattern B: Cursor Main + Claude Code Sub

.cursor/rules/<project>.mdc  ← Write actual rules here (alwaysApply: true)
CLAUDE.md                     ← "See .cursor/rules/"

Choose the AI tool you primarily use as the main one. Write rules in the main tool's instruction file and keep the sub tool's instruction file as a lightweight reference to the main. This way, you only manage rules in one place.

6. Agent Memory and Context Drift

Agent memory and context drift are two sides of the same coin.

Context drift is the phenomenon where the agent deviates from its original goal as conversations grow longer. Harness strategies for addressing this:

Session separation: Use multiple short, purpose-specific sessions rather than one long session. Claude Code's /compact command or starting a new Cursor chat serve this purpose.
Connect via memory files: Record key conclusions in memory files at session end, and reference them in the next session.
Leverage rule file priority: Rules recorded in .cursor/rules/ (alwaysApply) or CLAUDE.md are injected every turn regardless of session length, so core constraints must be in rule files.

[Context drift in long sessions]
  Session start → (rules + goals clear) → work progresses → ... → (context window saturated)
    → Initial rules' influence ↓ → drift occurs

[Memory-based short session strategy]
  Session 1: Work → record conclusions in memory → end session
  Session 2: Load memory → (rules + previous conclusions clear) → continue work

For developer-managed projects (blogs, personal projects, etc.), using markdown files within the repository (docs/decisions/, .memory/) is more practical than heavy automated memory systems. Simply instructing the agent to "add findings to .memory/learnings.md" is enough to establish cross-session knowledge continuity.

7. Pre-commit Hooks and Agents

Pre-commit hooks are scripts registered at .git/hooks/pre-commit that automatically run on git commit, blocking the commit itself on rule violations.

The core principle of harness engineering: "What can be mechanically enforced should be mechanically enforced."

Method	Use Case	Enforcement
Pre-commit hooks (scripts)	Formatting, linting, type checks — things with clear rules	100% — commit blocked on violation
Natural language instructions (CLAUDE.md, etc.)	Design judgments, naming conventions — things requiring judgment	Advisory level

Indirect communication with agents: Pre-commit hooks can't directly invoke agents. However, the structure where agents read and respond to hook error messages already works. The OpenAI Codex team leverages this by embedding fix instructions in linter error messages, so the agent reads the error, fixes it, and re-commits.

# Agent-friendly error message example
Error: line 42 - unused variable 'tempData'.
Fix: Remove the variable or use it in the fetchResult() call below.
Refer to docs/conventions/no-unused-vars.md for examples.

8. Key Takeaways

Harness engineering is the discipline of designing systems (constraints, feedback loops, documentation, lifecycle management) that make AI agents reliable.
Application scope varies by project characteristics. Personal projects only need Phase 1~2, multi-agent automation systems may need through Phase 5, and enterprise systems through Phase 7.
You don't need a perfect harness from the start — build incrementally by Phase, starting with basic context documents.
Design agent memory as a harness component. Since agents are fundamentally stateless, supplement cross-session learning and knowledge accumulation with repository-based memory.
Build flexible harnesses. As models improve, over-engineering becomes a burden — maintain a rippable structure that's easy to remove.
If you're building a harness for the first time and wondering what to start with — start with verification loops for the best ROI.

Real-World Application: Applying Harness to This Blog

While writing this guide, I simultaneously applied a harness to this blog project (Ted Factory). As a Type A (Personal / Static) project, I only applied Phases 1~2, and here's what I actually set up:

Phase 1 — Agent Instructions + Verification Automation

.cursor/rules/ted-blog-common-rules.mdc: Main Cursor rules (writing style, front matter, content structure)
CLAUDE.md: Lightweight reference file for Claude Code
ARCHITECTURE.md: Project structure documentation
scripts/pre-commit: Hugo build validation, front matter required field validation, ko / en symmetry validation

Phase 2 — Agent Memory

.memory/learnings.md: Patterns and know-how discovered during work
.memory/current-focus.md: Current interests and priorities
docs/decisions/: Architecture Decision Records (ADR)

Changes I noticed after applying these two Phases:

When opening a new session, the agent immediately understands project conventions (writing style, front matter structure, deployment method)
Pre-commit hooks catch front matter omissions and build failures before commit, eliminating "mistakes discovered after deployment"
Thanks to know-how accumulated in .memory/learnings.md, the agent doesn't repeat the same mistakes

Applying Phase 1~2 to a personal project takes about a day. The stability and consistency improvements you get in return are substantial. If you're actively using AI agents, I believe this is the first thing worth doing.

Try It Right Now

You can read through this guide and follow it step by step, but there's an even simpler way: give this article's URL to your AI agent and ask it to apply harness engineering to your project.

Agents like Cursor and Claude Code can read URLs, understand the content, and configure a harness for your project accordingly. When I applied a harness to this very blog, I took a similar approach — presenting a harness engineering design document to the agent and asking it to apply the structure.

In Cursor:

Open the project you want to harness in Cursor and type the following in the chat:

https://tedfactory.com/en/notes/essays/harness-engineering-guide/
Please apply harness engineering to this project based on the document above.

In Claude Code:

Run Claude Code from the root directory of the project you want to harness, and type:

https://tedfactory.com/en/notes/essays/harness-engineering-guide/
Please apply harness engineering to this project based on the document above.

The agent will read this guide, analyze your project's characteristics, and apply the appropriate Phases for your project type. Of course, you don't have to accept everything the agent suggests as-is — adjust it to fit your project's actual situation.

References

This post was originally published on Ted Factory. I write there about coding agents, harness engineering, and building things with AI.

Hermes Agent — Is It the Luxury Brand of AI Agents? A First Impression

Ted Kim — Tue, 04 Aug 2026 17:07:11 +0000

Introduction

Lately I've been hearing a lot about Hermes Agent. The most compelling example came directly from a teammate. They told me they had already connected Hermes to our company Slack and built an environment where the team could handle data lookups, task requests, and Q&A with a simple @Hermes message. That was enough to make me want to understand it properly, so I spent a single day doing all three of the following:

Researched what Hermes Agent is and what philosophy it was built on
Analyzed, at the code level, the Hermes deployment my teammate had built into our internal Pulumi project
Installed it on my own MacBook and connected it to Slack as a hands-on exercise

This essay is a record of the concepts and impressions I gathered along the way. To give the conclusion up front: Hermes is not just a coding assistant. It's something different in kind — "a self-improving autonomous agent you park on a server and command from anywhere via messaging." As someone who uses Cursor and Claude Code daily, I felt Hermes occupies a genuinely different niche.

What Is Hermes Agent?

Hermes Agent is an open-source autonomous AI agent built by Nous Research. Nous Research is a lab known for open-source LLMs (Large Language Models) like Hermes, Nomos, and Psyche, and the agent framework that shares the name is Hermes Agent. It's MIT-licensed and open-source on GitHub.

The three biggest differences from existing coding tools are:

1. Not Tied to a Local Development Session

Cursor is an IDE-centered coding copilot, while Claude Code started as a terminal-first coding agent and now spans IDEs, desktop, web, Slack, and more. So the real distinction is not simply whether a tool lives in an IDE. Hermes's defining trait is that it's a daemon-style agent you can put on a VPS, in Docker, or in a cloud sandbox (Modal, Daytona, Vercel Sandbox), then call from outside via messages. It runs on anything from a $5 VPS to a GPU cluster. A single gateway process bundles 20+ messaging platforms: Slack, Telegram, Discord, WhatsApp, Signal, Email, Matrix, Feishu, DingTalk, and more.

2. A Closed Learning Loop

It's not just a chatbot wrapping an LLM API. While you use it, it automatically accumulates the following:

Capability	Description
Memory	Persistent memory across sessions. FTS5 (SQLite Full-Text Search) based recall + LLM summarization
Skills	Builds procedural skills from work experience and self-improves them during use
User Modeling	Built-in user profile memory, with deeper user modeling available by connecting external memory provider plugins such as Honcho

In plain language: "the work you give it today gets done better tomorrow."

3. The Full Package

It also includes roughly 70 built-in tools (web search, browser automation, vision, voice, image generation), MCP (Model Context Protocol) server integration, a natural-language cron scheduler, isolated subagents for parallelization, and 7 terminal backends (local, Docker, SSH, Singularity, Modal, Daytona, Vercel Sandbox). For models, you can freely choose OpenRouter (200+ models), Anthropic, OpenAI, Nous Portal, or any OpenAI-compatible endpoint.

Official docs: https://hermes-agent.nousresearch.com/docs/

Hermes Running In My Team — A Code-Level Look

My teammate had deployed Hermes into our company's internal Pulumi project. Reading the code, several interesting design decisions stood out.

Everything on One AWS Lightsail Instance

The company runs most workloads on container-based services (EKS, Lambda, Fargate). But for Hermes, the choice was a single Lightsail instance ($40/month). Pulumi creates only 5 resources total.

Resource	Purpose
`aws.lightsail.Instance` (`hermes-server`)	Ubuntu 24.04, 2 vCPU, 8GB RAM
`aws.lightsail.KeyPair` (`hermes-key`)	SSH key pair
`aws.lightsail.StaticIp` (`hermes-ip`)	Stable IP across reboots
`aws.lightsail.InstancePublicPorts` (`hermes-ports`)	Only 22, 443 open
`aws.route53.Record` (`hermes-dns`)	Internal DNS record in the form of `hermes.example.com`

The reason for not going serverless is clear. The Hermes Slack gateway needs a persistent WebSocket connection (Socket Mode), which means it has to be always-on. That doesn't fit a Lambda or Fargate single-shot model. Flat fee, simple setup, and one box for the whole team — those three conditions converged on Lightsail.

Separation: Infra (Pulumi) + Bootstrap (Script) + Runtime Config (Templates)

What's interesting is the separation of responsibilities:

Pulumi: Owns only the 5 AWS resources
scripts/setup.sh: Installs Docker, Node.js, Hermes itself + registers a systemd service
hermes-config/config.yaml: Model, terminal backend, MCP servers, tool permissions
walnuts/: Company-wide context (a directory of domain knowledge for the LLM to read)
SOUL.md: The agent's persona and behavioral rules

Each layer can change independently. AWS resources change → pulumi up. Settings only change → scp + systemctl restart. Lean and tidy.

Docker Sandbox Protects the Host

When Hermes runs commands on the server, those commands go through a Docker container. Even with 25 teammates issuing commands freely from Slack, the risk of accidentally touching the Lightsail host's entire filesystem is greatly reduced. That said, mounted paths and forwarded tokens are still accessible inside the container, so the exposed surface should be kept intentionally small. In this setup, only the walnuts directory is volume-mounted into the container, and only one host environment variable (GITHUB_TOKEN) is forwarded.

Walnuts + Wiki — The Vessel for Company Knowledge

Hermes itself is the "agent engine," and company-specific knowledge lives in the Walnuts directory as files. The structure has three layers: _core/ (operational context, auto-refreshed by cron) + raw/ (immutable sources) + wiki/ (LLM-curated knowledge). It's clearly inspired by Karpathy's LLM Wiki idea. Hermes constantly reads the company's accumulated domain knowledge while simultaneously curating and extending it.

Installing on My MacBook — Step by Step

After dissecting the team's setup, I wanted to build a similar structure on my MacBook. I kept the requirements simple:

Docker sandboxing. But with the ability to mount specific local folders when needed
Slack integration. Not the team bot — a separate personal one

The whole flow took 6 steps. Capturing it end-to-end here so the same environment is reproducible later.

Step 1. Prerequisites — Docker + LLM API Key

You need a Docker daemon running. I prefer the lightweight OrbStack:

brew install --cask orbstack

You also need at least one LLM API key — either sk-ant-... from the Anthropic console or sk-or-... from OpenRouter (which gives you 200+ models in one place).

Pick the local folders you want to mount into the sandbox in advance. In my case, that's the blog folder and its parent project root.

Step 2. Install Hermes

The official installer is one line:

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

After install, open a fresh shell so PATH gets picked up. Add this to ~/.zshrc for permanence:

export PATH="$HOME/.local/bin:$PATH"

Then run the interactive initial setup:

hermes setup

It walks you through LLM provider, API key, default model, and messaging platform (choose Slack). At the end, ~/.hermes/config.yaml and ~/.hermes/.env are generated.

Step 3. Docker Sandbox + Folder Mount Configuration

Pre-pull the container image — same one the company's Pulumi deployment uses:

docker pull nikolaik/python-nodejs:python3.11-nodejs20

Open the terminal: section of ~/.hermes/config.yaml and switch to the Docker backend with explicit local folder mounts:

terminal:
  backend: docker
  docker_image: "nikolaik/python-nodejs:python3.11-nodejs20"
  container_cpu: 2
  container_memory: 4096
  container_persistent: true
  persistent_shell: true
  cwd: "/workspace"
  docker_volumes:
    - "/path/to/project-root:/workspace/project-root:ro"
  docker_forward_env:
    - GITHUB_TOKEN

The docker_volumes format is host_path:container_path[:ro], where :ro locks the folder as read-only. Anything you don't mount is invisible to the container. That's the heart of sandboxing — and the next section is dedicated to why this matters.

If you use Docker Desktop, check that Settings → Resources → File sharing includes /Users. OrbStack doesn't need this step.

Step 4. Create a New Slack App

Leave any existing Slack apps you use for other bots or MCP integrations untouched. Create a brand new app dedicated to Hermes. Reasons: permission separation, separate rate limits, avoiding conflicts between Socket Mode and existing event subscriptions, and bot name separation.

The official docs recommend pasting in a manifest generated by Hermes:

hermes slack manifest --write

If you configure it manually, the steps are:

Go to api.slack.com/apps → Create New App → From scratch → name it Hermes Agent (your identifier)
OAuth & Permissions → Bot Token Scopes box (NOT User Token Scopes!) — add 11 scopes: chat:write, app_mentions:read, channels:history, channels:read, groups:history, im:history, im:read, im:write, users:read, files:read, files:write
Settings → Socket Mode → Enable → generate an App-Level Token (name hermes-socket, scope connections:write) → copy the xapp-... token
Event Subscriptions → Enable Events → Subscribe to bot events: add app_mention, message.channels, message.groups, message.im → Save
App Home → Show Tabs: turn on Messages Tab + check "Allow users to send Slash commands and messages" (without this, DMs don't work)
Settings → Install App → Install to Workspace → copy the xoxb-... Bot User OAuth Token
Copy your own Slack Member ID (profile → more menu → Copy member ID)

Step 5. Inject Tokens + Run the Gateway

Add the two tokens and your Member ID to ~/.hermes/.env:

SLACK_BOT_TOKEN=<BOT_USER_OAUTH_TOKEN>
SLACK_APP_TOKEN=<APP_LEVEL_TOKEN>
SLACK_ALLOWED_USERS=<YOUR_MEMBER_ID>

With only your ID in SLACK_ALLOWED_USERS, the bot ignores anyone else. Lock the file permissions for safety:

chmod 600 ~/.hermes/.env

Now start the gateway:

hermes gateway

When the log shows Connected to Slack workspace ..., you're connected. Keep this terminal open as long as you want the bot alive. To background it, hermes gateway install registers it as a macOS LaunchAgent that auto-starts on login and auto-restarts on crash.

Step 6. Verify

In Slack, create a test channel and invite the bot:

/invite @Hermes Agent (your identifier)
@Hermes Agent (your identifier) hi, run pwd for me

If a thread reply comes back with a path like /workspace/ted-blog (the container path, not your host path), everything is working. In DMs, you don't need to mention the bot — just talk.

The Pitfalls I Hit

The setup itself is short, but a few small pitfalls are easy to trip on the first time. Collecting them here for anyone trying the same path.

Pitfall 1. The prompt_toolkit Error Right After `curl ... | bash`

The install script tries to launch the hermes TUI immediately after install, which produces this error:

File ".../prompt_toolkit/input/vt100.py", line 165, in _attached_input
    loop.add_reader(fd, callback_wrapper)
OSError: [Errno 22] Invalid argument

The cause is simple. A shell launched via curl ... | bash has its stdin tied to a pipe (curl's output), not a real TTY. The TUI requires stdin to be a real terminal to receive keyboard input, so it fails. Just open a fresh terminal and run hermes again — it works fine. The install itself was 100% successful.

Pitfall 2. The `app_mentions:read` Scope Doesn't Show Up

In the Slack app's OAuth & Permissions page, I went looking for app_mentions:read and it wasn't in the list. The reason is simple. That page has two boxes, and app_mentions:read is exclusive to the Bot Token Scopes box. It won't appear if you search inside User Token Scopes.

Box	Token Type	Purpose
Bot Token Scopes	`xoxb-` (Bot User OAuth Token)	The bot's own permissions
User Token Scopes	`xoxp-` (User OAuth Token)	The bot acts on behalf of the installing user

All scopes Hermes needs go into the Bot Token side. You don't need to touch User Token Scopes.

Pitfall 3. The Difference Between SLACK_BOT_TOKEN and SLACK_APP_TOKEN

Hermes's .env requires two Slack tokens. Confusing at first, but here's the breakdown:

Item	`SLACK_BOT_TOKEN`	`SLACK_APP_TOKEN`
Prefix	`xoxb-`	`xapp-`
Issued When	"Install to Workspace"	"Socket Mode" enable
Permission Basis	Bot Token Scopes	App-Level Scopes (`connections:write`)
Purpose	Web API calls (send message, upload file, etc.)	Open the Socket Mode WebSocket connection
Analogy	The bot's ID + action permissions	The key that opens a private phone line to Slack

Typical Slack MCP integrations use just one token (Bot or User). They follow a "call the API when needed" pattern. Hermes, in contrast, is a bot that receives the event stream in real time and reacts on its own, so Socket Mode is mandatory — and so both tokens are required.

The Real Value of the Docker Sandbox

I want to revisit the meaning of mounting only two folders via docker_volumes in Step 3 of the install. It matters more than it looks.

Coding agents that run tools locally generally operate with the permissions of the same OS user. Depending on configuration and approval settings, sensitive paths like ~/.ssh, ~/.aws, and local config files can fall within reach. The Docker setup above, by contrast, exposes only the two folders you explicitly named to the container. The rest of the host doesn't even exist from the container's perspective. On top of that, you can lock one folder as read-only (:ro).

For verification I had the bot query an arbitrary host path that had not been mounted. The response was "no such directory." Correct. Whatever you didn't mount does not show up. This is the real value of a sandbox: the user can precisely control the agent's blast radius.

A Luxury First Impression — and the Homework That Remains

Right after finishing install + Slack connection, I exchanged my first hello with the bot, and an unexpected problem was waiting. I sent "hi," waited 5 minutes, no response. A bit later I sent another message, and only then did the response to the first message land in Slack.

A search turned up several gateway issues with similar patterns. As of my May 2026 check, some of them had already been closed, but the symptom I saw looked like the same family of problems.

#4469 — Multiple rapid messages only last one processed: The gateway internally has two pending message queues (GatewayRunner._pending_messages, adapter._pending_messages) that aren't synchronized. Still open as of May 2026
#8221 — Telegram duplicate replies when interrupted session sends stale response before processing pending message: Interrupted sessions used to emit a stale response first. Fixed and closed as of May 2026
#5057 — Message handling behavior when multiple messages arrive during active agent execution: Covers queueing and interruption behavior when new messages arrive during active execution. Closed as of May 2026

At the time, issue #8221 looked closest to what I observed. But since it was already closed by May 2026, if the same problem reproduces, the first step should be checking the Hermes version and verbose logs. Whether the real cause is a message queue bug, a Docker container cold-start hang, or the Anthropic streaming response failing to emit a completion signal can only be confirmed from logs. I deferred that diagnosis to a later session.

There was one secondary issue worth noting. I use a light-background terminal, and Hermes's default skin uses yellow/white-heavy colors that were nearly unreadable. Two built-in skins solve this:

/skin daylight       # dark text + cool blue accents
/skin warm-lightmode # warm brown / gold tones

This same problem is even officially logged as an issue.

In Closing — Is It the Luxury Brand of AI Agents?

After installing it, wiring it into Slack, and tracing through a real team deployment, my first impression became fairly clear. Hermes feels less like "a tool with a lot of features" and more like a tool built around a strong idea of how agents should be operated. It is not mainly about opening a single local work session for a few minutes at a time. It is better understood as a longer-running system that ties messaging, sandboxing, memory, and skills into one workflow.

That is why the "luxury" comparison did not feel entirely exaggerated to me. Not because Hermes is flashy, but because its priorities are unusually clear. The Docker-first sandboxing model, the natural fit with messaging channels like Slack, and the idea that context and memory should accumulate over time all left a strong impression. Looking at the company's Pulumi deployment reinforced that feeling. The value of Hermes seems to lie less in any single feature than in the way it turns agent operation itself into a coherent system.

It is still too early for me to make a definitive judgment. At this point I have only spent a day installing and connecting it, and there are still practical issues to investigate, such as the delayed-response behavior I saw locally. Even so, it already feels clear that Hermes is aiming at a different place from the local development sessions where I usually use Cursor and Claude Code. If those tools are closer to "productivity right now," Hermes feels closer to "an environment you keep cultivating over time."

For now, my next step is to diagnose the response-delay issue on my MacBook. Once that is under control, I want to use my teammate's walnuts structure as a reference and slowly build a domain knowledge container of my own. After spending more real time with it, I would like to come back with a follow-up piece based not just on first impressions, but on actual operating experience.

References

This post was originally published on Ted Factory. I write there about coding agents, harness engineering, and building things with AI.

DEV Community: Ted Kim

Harness Engineering — Building the Environment Your Coding Agent Works In

Why I Wrote This

1. What Is Harness Engineering?

Definition

Analogy

Why Harness Engineering Now?

Comparison with Related Concepts

2. Core Components of Harness Engineering

2-1. Context Engineering

2-2. Architectural Constraints

2-3. Guardrails

2-4. Verification Loops

2-5. Cost Envelope Management

2-6. Entropy Management

2-7. Agent Memory

2-8. Observability & Evaluation

3. Harness Engineering Application Levels

Level 1: Basic Harness (Individual Developer)

Level 2: Team Harness (Small Team)

Level 3: Production Harness (Engineering Organization)

4. Applying Harness Engineering to Your Project

4-1. Service Characteristics Analysis

4-2. Risk Identification

4-3. Harness Engineering Design

A. Context Engineering Application

B. Agent Behavior Constraints

C. Verification Loop Application

D. Multi-Agent Harness Middleware

E. Observability Integration

F. Entropy Management — Maintaining Agent Quality

4-4. Harness Engineering Application Roadmap

Application Examples by Project Type

5. Agent Instruction File Strategy

6. Agent Memory and Context Drift

7. Pre-commit Hooks and Agents

8. Key Takeaways

Real-World Application: Applying Harness to This Blog

Try It Right Now

References

Hermes Agent — Is It the Luxury Brand of AI Agents? A First Impression

Introduction

What Is Hermes Agent?

1. Not Tied to a Local Development Session

2. A Closed Learning Loop

3. The Full Package

Hermes Running In My Team — A Code-Level Look

Everything on One AWS Lightsail Instance

Separation: Infra (Pulumi) + Bootstrap (Script) + Runtime Config (Templates)

Docker Sandbox Protects the Host

Walnuts + Wiki — The Vessel for Company Knowledge

Installing on My MacBook — Step by Step

Step 1. Prerequisites — Docker + LLM API Key

Step 2. Install Hermes

Step 3. Docker Sandbox + Folder Mount Configuration

Step 4. Create a New Slack App

Step 5. Inject Tokens + Run the Gateway

Step 6. Verify

The Pitfalls I Hit

Pitfall 1. The prompt_toolkit Error Right After curl ... | bash

Pitfall 2. The app_mentions:read Scope Doesn't Show Up

Pitfall 3. The Difference Between SLACK_BOT_TOKEN and SLACK_APP_TOKEN

The Real Value of the Docker Sandbox

A Luxury First Impression — and the Homework That Remains

In Closing — Is It the Luxury Brand of AI Agents?

References

Pitfall 1. The prompt_toolkit Error Right After `curl ... | bash`

Pitfall 2. The `app_mentions:read` Scope Doesn't Show Up