DEV Community: Rost

Idempotency in Distributed Systems That Actually Works

Rost — Mon, 11 May 2026 11:37:09 +0000

Idempotency in distributed systems is the property that saves you after the network lies, the queue retries, the client panics, and the operator hits replay. In production systems, duplicate delivery is normal. Duplicate side effects are the bug.

HTTP defines an idempotent method as one where multiple identical requests have the same intended effect on the server as one request. That is why PUT, DELETE, and safe methods are idempotent in protocol semantics and can be retried automatically after a communication failure.

That definition is useful, but it is not enough. In real architectures, idempotency is not an HTTP trivia answer. It is a business guarantee. If a customer hits "pay" once, you do not get to charge twice because a timeout happened between commit and response. If a worker updates inventory and crashes before acking the message, you do not get to decrement stock twice because the broker redelivered. That is the bar.

The mistake I see over and over is treating idempotency as a transport feature instead of a system property. Queue deduplication, HTTP verbs, and client retries help, but none of them rescue a design that lets the same business intent create a second side effect. If you want the broader framing for how these integration decisions fit service boundaries and persistence trade-offs, start with App Architecture in Production: Integration Patterns, Code Design, and Data Access.

Where duplicates come from in production

Duplicates do not appear because teams are careless. They appear because distributed systems retry, reorder, and replay.

A client can send a create request, the server can commit it, and the response can still disappear on the wire. That is exactly why HTTP distinguishes idempotent methods and why payment APIs such as Stripe and PayPal expose explicit idempotency mechanisms for unsafe methods like POST.

Message brokers make the problem even more obvious. At-least-once delivery means a consumer can be invoked repeatedly for the same message, and a handler can update the database successfully but fail before acknowledgment, causing the broker to deliver the same message again.

Webhooks are no different. GitHub says webhook deliveries can arrive out of order, failed deliveries are not automatically redelivered, and each delivery carries a unique X-GitHub-Delivery GUID that you should use when protecting against replay. For a practical architecture view of chat endpoints as interaction boundaries, see Chat Platforms as System Interfaces in Modern Systems.

Even systems that advertise stronger guarantees still leave you work to do. Kafka can prevent duplicate entries in Kafka logs with idempotent producers and can provide exactly-once delivery for read-process-write flows that stay inside Kafka with transactions and read_committed consumers. But Kafka's own design docs are clear that external systems still require coordination with offsets and outputs. Google Cloud Pub/Sub exactly-once delivery is limited to pull subscriptions, within a cloud region, and still requires clients to track processing progress until acknowledgment succeeds.

My opinionated summary is simple. Assume the transport will retry. Assume operators will replay. Assume webhooks will arrive late. Design the write path so a repeated intent cannot create a second business effect.

The API contract I actually trust

How do idempotency keys prevent duplicate API requests

The only API contract I trust for mutating operations is caller-supplied intent plus server-side persistence.

AWS recommends a caller-provided request identifier and warns that the service must atomically record the idempotency token together with the mutating work. Stripe stores the first status code and response body for a key, compares later parameters with the original request, and returns the same result for retries. PayPal uses PayPal-Request-Id on supported POST APIs and returns the latest status for the previous request with that same header.

That leads to a practical contract:

The client generates an idempotency key for a business operation.
The server scopes that key by tenant and operation name.
The server stores a request hash so the same key cannot be reused for a different payload.
The server records state such as pending, completed, or failed.
Retries with the same key either return the stored outcome or a stable pointer to it.
Retries with the same key and a different payload fail loudly.

There is an IETF Idempotency-Key header draft, but as of 2026-05-09 it is still listed in the IETF Datatracker as an expired Internet-Draft rather than a published RFC. In practice, the header name is still widely useful as a de facto convention, but you should document the contract in your own API instead of pretending the standard is finished.

What should the key represent? Intent. Not an HTTP attempt. Not a TCP connection. Not a retry counter. If the user means "create order 123 once", every retry for that same command must reuse the same key. If the user means "place a second order", that must use a different key.

A request ID is for tracing. An idempotency key is for correctness. If you mix those up, your dashboards look tidy while your money moves twice.

Why PUT is not enough

No, HTTP PUT is not enough to make an operation idempotent.

Yes, RFC 9110 gives PUT idempotent semantics. But if your PUT handler emits a new downstream event, sends an email on every retry, or charges an external provider again, then your implementation has violated the business contract even if your route name looks respectable.

Verb choice helps clients understand intent. It does not implement intent for you.

Use PUT when the resource model genuinely fits a full replacement or upsert style operation. Use POST when you are creating commands or actions. But for any mutation that might be retried across network boundaries, document an explicit idempotency contract. If your mutating actions are triggered from chat workflows, the same contract applies in Slack Integration Patterns for Alerts and Workflows and Discord Integration Pattern for Alerts and Control Loops. Hidden side effects are where architecture goes to die.

How long should an idempotency key be stored

Longer than your transport team wants.

Stripe says keys can be pruned after at least 24 hours. PayPal says retention is API specific and gives examples that can last up to 45 days. Amazon SQS FIFO deduplicates only within a 5-minute window. GitHub keeps recent deliveries for 3 days for manual redelivery. Those numbers are wildly different because the right retention period is a business decision, not a protocol default.

If you only keep keys for five minutes because your queue does, you are not designing idempotency. You are copying a transport limitation into your business layer.

Keep idempotency records for at least the maximum of these windows:

client retry horizon
queue redrive horizon
webhook replay horizon
operator replay horizon
settlement or compensation horizon for money-moving operations

For payments, bookings, and provisioning, that often means hours or days, not minutes.

AWS also calls out two anti-patterns I fully agree with. Do not use timestamps as the key, because clock skew and collisions make them unreliable. Do not blindly store entire request payloads as the dedup record for every request, because that harms performance and scalability. Store a normalized request hash plus the minimum response state you need to replay safely. If you must reproduce the first response byte for byte, store the canonical response body the way Stripe does.

The database patterns that make idempotency real

Idempotency becomes real when the persistence layer can win a race exactly once.

PostgreSQL gives you two critical primitives here. Unique constraints enforce uniqueness on one or more columns, and INSERT ... ON CONFLICT lets you define an alternative action instead of failing on a uniqueness violation. PostgreSQL also documents that ON CONFLICT DO UPDATE guarantees an atomic insert-or-update outcome under concurrency.

That means your idempotency layer should usually start with a table like this:

create table api_idempotency (
    tenant_id text not null,
    operation text not null,
    idempotency_key text not null,
    request_hash text not null,
    state text not null,
    status_code integer,
    response_body jsonb,
    resource_type text,
    resource_id text,
    created_at timestamptz not null default now(),
    expires_at timestamptz not null,
    primary key (tenant_id, operation, idempotency_key)
);

And the handling flow should look like this:

begin transaction

try insert (tenant_id, operation, idempotency_key, request_hash, state='pending')
on conflict do nothing

load row for (tenant_id, operation, idempotency_key) for update

if row.request_hash != incoming_request_hash
    fail with conflict or validation error

if row.state = 'completed'
    return stored response

if row.state = 'pending' and row was created by another live request
    either wait briefly, or fail fast with a retryable response

perform local business mutation

store stable result in idempotency row
set state = 'completed'

commit
return result

The important part is not the syntax. The important part is the atomicity. Recording the key and performing the mutation must succeed or fail together. AWS says this explicitly for API idempotency, and the same rule applies in SQL-backed services.

Do not do a naive check-then-act sequence like "select key; if missing then insert order". Under concurrency, two requests can pass the check and both create the side effect. A unique constraint is not optional. It is the mechanism that turns your architecture from optimistic folklore into something you can prove under load.

Here is the rule I use in reviews. If the dedup decision is not protected by the same transactional boundary as the mutation, you do not have idempotency. You have hope.

Messages, events, and webhooks need their own boundary

How do consumers handle duplicate events and messages

For message consumers, the classic pattern is still the right one. Record processed message IDs in the same database transaction as the business update. Chris Richardson describes the PROCESSED_MESSAGES table approach directly, using a primary key on subscriber and message ID so duplicates fail cleanly and can be ignored.

Many teams call that explicit processed_messages store an inbox table. The label matters less than the rule. The receiver must persist proof that it already handled the message before a retry can safely do nothing.

A minimal form looks like this:

create table processed_messages (
    subscriber_id text not null,
    message_id text not null,
    processed_at timestamptz not null default now(),
    primary key (subscriber_id, message_id)
);

And the consumer flow is just as strict as the HTTP flow:

begin transaction

insert into processed_messages (subscriber_id, message_id)
values (?, ?)
on conflict do nothing

if no row inserted
    rollback
    ack and ignore duplicate

apply business mutation

commit
ack message

That pattern is boring. Good. Idempotency should be boring.

It is also usually better than trying to lean on broker marketing terms. Kafka's exactly-once support is excellent when you stay inside Kafka's own transactional model, but Kafka's docs still warn that external destinations need cooperation. SQS FIFO reduces duplicate sends only within its 5-minute dedup window. Pub/Sub exactly-once still expects the subscriber to track progress and avoid duplicate work when acknowledgments fail.

Exactly-once is usually a local optimization. Idempotent side effects are the system guarantee.

Pair dedup with the outbox pattern

If your service updates local state and also publishes an event, idempotent consumption alone is not enough. You also need a safe way to get the event out after the local transaction commits.

That is why the transactional outbox pattern matters. Chris Richardson describes the basic idea as writing the event to an outbox table in the same transaction as the business update, and then publishing it asynchronously. Debezium says the outbox pattern avoids inconsistencies between a service's internal state and the events consumed by other services. NServiceBus goes further and shows how outbox processing deduplicates incoming messages and avoids zombie records and ghost messages.

This is the architecture I recommend for services that own data and publish integration events:

Validate and persist the command under an idempotency key.
Write business state and outbox event in one local transaction.
Let CDC or an outbox dispatcher publish the event.
Make downstream consumers idempotent too.

Outbox does not remove the need for idempotent consumers. It removes the need to pretend that a database commit and a broker publish can be one magical distributed transaction when they usually cannot.

Webhooks are just messages with better branding

Treat inbound webhooks exactly like messages from an untrusted network edge.

GitHub documents that deliveries can arrive out of order, recommends using X-Hub-Signature-256 to verify authenticity, and provides X-GitHub-Delivery as the unique delivery identifier. It also notes that redeliveries reuse the same delivery ID.

So the architecture is straightforward:

verify the signature first
use the delivery GUID as the dedup key
persist receipt before side effects
make handlers order-aware rather than assuming arrival order
enqueue the heavy work and return fast

If your webhook handler writes directly to business tables before it records receipt, it is not production-ready. It is just faster at making duplicate mistakes.

Sagas and workflow engines still need idempotency

Sagas and durable workflow engines do not delete the problem. They make it visible.

Temporal recommends writing Activities to be idempotent because Activities can be retried after failures or timeouts. Its docs even call out the edge case where a worker completes an external side effect successfully but crashes before reporting completion, which causes the Activity to run again. Temporal also suggests using a combination of Workflow Run ID and Activity ID as a stable idempotency key when calling downstream services. If you are applying this in service orchestration, Go Microservices for AI/ML Orchestration covers the broader workflow trade-offs.

That is exactly the right mental model. A workflow engine can preserve execution history and coordinate retries. It cannot retroactively uncharge a card or unsend an email unless your application gives it idempotent steps and idempotent compensations.

The same applies to sagas. Temporal's own saga guidance describes compensating actions that run when a step fails. Those compensations must be idempotent too. If "refund payment" runs twice, you may have solved the original bug by creating a new one.

My rule here is brutal and simple. Every Activity, every command handler, and every compensation that touches the outside world should either be naturally idempotent or carry a real idempotency key to the downstream system.

How to test idempotency before production

Most teams test happy paths and then act surprised when retries happen. That is not enough.

You should have automated tests for at least these cases:

the server commits the mutation but the response never reaches the client
two identical requests race with the same idempotency key
the same key is reused with a different payload
a consumer commits its database work and crashes before ack
a webhook is replayed with the same delivery ID
an outbox dispatcher publishes the same event more than once
a workflow Activity completes the external call and crashes before completion is reported
an idempotency record expires and a genuine late retry arrives

AWS explicitly recommends comprehensive test suites that include successful requests, failed requests, and duplicate requests. That advice is pedestrian and absolutely correct.

I would add one more failure drill. Verify that the replayed response is semantically equivalent to the first result. AWS discusses late-arriving retries and argues for responses that preserve the original meaning even after underlying state has changed. That is the difference between "no extra side effect happened" and "the caller still has a consistent contract."

Opinionated rules that save real systems

Here are the rules I would enforce in an architecture review.

First, idempotency keys belong to business intent, not transport attempts.

Second, scope every key by tenant and operation. Global key spaces are how unrelated requests collide.

Third, persist the dedup decision atomically with the mutation. If that is not true, the design is wrong.

Fourth, reject same-key different-payload retries. Stripe and AWS both do this for good reason.

Fifth, keep keys for the full replay horizon of the business process, not for the shortest queue window.

Sixth, pair producers with an outbox and consumers with message ID tracking. One side without the other is half a design.

Seventh, propagate the same operation identity downstream when the business action is the same. AWS explicitly recommends passing the idempotency token along the processing chain.

Eighth, never assume exactly-once marketing removes the need for idempotent side effects.

If that sounds strict, good. Idempotency is where optimistic architecture meets production reality. You do not need complexity everywhere. But wherever duplicate side effects would hurt money, state, or trust, idempotency should be a first-class part of the contract.

Useful Links

Hermes Voice Control from Your Phone

Rost — Sun, 10 May 2026 11:12:56 +0000

You already chat to Hermes Agent from your phone with text.
Now you want to talk to it directly and get spoken replies back.
That is usually the right move, especially if you already use Hermes as a persistent self-hosted assistant.
Typing long prompts on a small screen is slow and error-prone

Voice mode makes Hermes practical in the moments where it matters most, while walking, commuting, or doing admin work away from your desk.

The good news is that voice mode can run with zero paid APIs. A local faster-whisper model handles transcription, and Edge TTS handles spoken output for free. This guide covers setup, provider choices, platform differences, practical command patterns, and the failure modes that usually block first-time users.

How the Pipeline Works

Three stages, no magic:

Transcription STT — Your voice message becomes text.
Reasoning — Hermes processes that text exactly like a typed request.
Synthesis TTS — The response text is converted back to audio.

The important distinction from consumer assistants is execution depth. Hermes is not just answering trivia. It can call tools, inspect files, run code paths, and continue multi-step work from memory. In practice, that means voice can trigger real workflows such as incident triage, draft generation, and targeted debugging. If you want the broader architecture context, the AI Systems pillar explains how this voice layer fits into local agent infrastructure.

What Voice Control Is Great For

Use voice mode when keyboard precision is not required yet:

Operational checks while away from your laptop.
Idea capture for drafts, outlines, and rough specs.
Fast triage of alerts and errors before deeper desktop follow-up.
Hands-busy workflows where speaking is the only realistic input channel.

Voice Input: Pick an STT Provider

Provider	Cost	API Key	Notes
Local faster-whisper	Free	None	On-device, ~150 MB model, 90+ languages
Groq Whisper	Free tier	`GROQ_API_KEY`	Fast cloud inference
OpenAI Whisper	Paid	`VOICE_TOOLS_OPENAI_KEY`	Highest accuracy
Mistral Voxtral	Paid	`MISTRAL_API_KEY`	Alternative cloud option

Configuration in ~/.hermes/config.yaml:

stt:
  enabled: true
  provider: local
  local:
    model: base  # tiny, base, small, medium, large-v3

Start with local. It works immediately, handles multilingual speech, and adds no recurring cost. Move to Groq or OpenAI only if your local setup cannot meet your latency or accuracy requirements. For command-level setup and diagnostics while testing providers, keep the Hermes CLI cheat sheet nearby.

Faster Whisper Model Selection

Use a simple progression:

tiny for very low-power devices where speed matters most.
base as the default balance for laptops and small servers.
small when accents, noisy environments, or domain terms reduce accuracy.
medium or large-v3 when quality is critical and hardware budget is higher.

If your transcripts are consistently wrong, increase model size first before adding more prompt complexity.

Voice Output: TTS Providers

Provider	Quality	Cost	Best For
Edge TTS (default)	Good	Free	Quick start, 322 voices, 74 languages
ElevenLabs	Excellent	Paid	Premium quality, voice cloning
OpenAI TTS	Good	Paid	Natural voices, 6 options
MiniMax TTS	Excellent	Paid	Fine-grained speed/volume/pitch control
NeuTTS	Good	Free (local)	Fully offline, voice cloning

Configuration:

tts:
  provider: "edge"
  speed: 1.0

  edge:
    voice: "en-US-AriaNeural"

One critical detail is output format. Telegram voice bubbles are most reliable when audio is encoded as OGG with Opus. Hermes relies on ffmpeg for these conversions in common setups. If ffmpeg is missing, replies often show up as file attachments instead of inline voice bubbles.

Install ffmpeg early:

sudo apt install ffmpeg  # Ubuntu/Debian
brew install ffmpeg       # macOS

Platform Workflows and Practical Differences

Telegram is the easiest place to start. Voice messages are first-class on mobile, and the interaction loop is simple hold, speak, release, receive.

Setup:

# 1. Create a bot via @BotFather, get your token
# 2. Add to ~/.hermes/.env:
TELEGRAM_BOT_TOKEN=***
TELEGRAM_ALLOWED_USERS=your_user_id

# 3. Start the gateway
hermes gateway start

Then open the Hermes chat, tap the microphone, and speak. If STT and TTS are enabled, Hermes transcribes your request, executes it, and sends a voice reply.

Discord

Discord supports two useful modes. Voice messages in DMs or channels are close to Telegram behavior.

The more advanced option is live voice channels. In that flow, Hermes can participate continuously, transcribing speech and responding without explicit message bubbles.

Requirements:

Message Content Intent enabled in your bot settings
Server Members Intent enabled
Bot permissions: Connect and Speak

Signal

Signal works through the signal-cli daemon. Voice messages still use the same Hermes STT and TTS pipeline.

A useful pattern is running signal-cli as a linked device and using Signal Note to Self. You can leave yourself a voice note and get Hermes output in the same thread.

WhatsApp follows the same gateway model. Audio messages transcribe automatically once the connector is configured.

Mobile App Permissions

Both iOS and Android need microphone access for the messaging app you're using.

iOS: Settings → Telegram (or Discord) → Permissions → Microphone → Allow. Enable Background App Refresh for instant responses.

Android: Settings → Apps → Telegram → Permissions → Microphone → Allow. For Discord voice channels, enable overlay permission.

Pinning the Hermes bot chat to your home screen helps — one tap to start speaking.

Speaking Patterns That Work Reliably

Voice interaction has different ergonomics than typing. You cannot easily paste logs or quote long stack traces, so structure matters:

Be explicit. Say the action, scope, and output format in one sentence.
Keep one objective per message. Split multi-step jobs into short follow-ups.
Constrain output. Ask for numbered actions or a 3-point summary when mobile readability matters.
Stay short. Around 10 to 30 seconds per message usually transcribes better.
Use iterative turns. Correct and refine in the next voice message instead of overloading the first.

Example Prompts You Can Speak

"Check deployment logs for the last one hour and report only critical errors."
"Create a draft outline for a post about OpenTelemetry migration with five sections."
"Summarize this bug in three bullets and propose the most likely root cause."
"Review the config and tell me what to change for lower transcription latency."

Common Use Cases with Concrete Outcomes

Operations — "Check production health and list failed services." Outcome is a focused status update you can act on immediately.
Writing — "Turn these rough points into a publishable intro paragraph." Outcome is polished text from spoken notes.
Debug triage — "Investigate this TypeError and suggest the first fix to test." Outcome is a concrete next step before opening the IDE.
Research — "Find three recent sources on topic X and summarize differences." Outcome is a compressed briefing for later deep work.
Automation — "Run the home routine and confirm device states." Outcome is direct action plus confirmation.

Troubleshooting

Voice messages not transcribing: Confirm stt.enabled: true in config.yaml. Verify local dependencies are installed. Then restart with hermes gateway restart.

TTS not responding: Confirm tts.provider is set. If using a paid provider, verify the API key in .env. Validate current voice settings from the Hermes CLI status commands.

Poor transcription quality: Increase stt.local.model from base to small or medium. Reduce noise and speak in shorter segments. If needed, switch to cloud STT for better accuracy.

Voice bubbles showing as files on Telegram: Install ffmpeg and restart the gateway. This is the most common issue.

The Free Stack

For cost-conscious setups, this baseline is strong:

STT: Local faster-whisper with no API key
TTS: Edge TTS with wide language coverage
Total cost: $0

This is a meaningful advantage over many closed assistants where voice quality and automation quickly become paid-only features.

If quality requirements increase, upgrade one layer at a time. Usually STT upgrades produce the biggest immediate gain, then TTS quality can be improved later if needed.

FAQ Topics in Practice

The four most common user questions are predictable. They also overlap with memory and profile design concerns covered in Hermes Agent Memory System and Hermes production setup patterns.

Whether voice commands get the same tool access as text.
Whether a free stack is viable for daily use.
Why Telegram sometimes shows attachments instead of voice bubbles.
Which local Whisper model should be used first.

This guide addresses each of these directly in setup, tuning, and troubleshooting sections so you can move from first run to stable daily usage quickly.

Quick Start Recap

# 1. Install voice extras
pip install "hermes-agent[all]"

# 2. Set up Telegram gateway
hermes gateway setup

# 3. Install ffmpeg (required for Telegram voice bubbles)
sudo apt install ffmpeg

# 4. Send a voice message from your phone
# Hermes transcribes, processes, and responds

From there, iterate based on your real bottleneck. If latency is the issue, tune model size or cloud STT. If audio quality is the issue, tune TTS provider and voice preset. Start free, measure, then upgrade only where it actually improves your workflow.

Kanban in Hermes Agent for Self Hosted LLM Workflows

Rost — Fri, 08 May 2026 09:48:56 +0000

Hermes Agent ships with a Kanban-style board and the Hermes Gateway that can saturate your self-hosted LLM if too many tasks are dispatched at once.

I can say you can easily ddos your own LLM this way.

Hermes Kanban is a durable multi-profile board backed by ~/.hermes/kanban.db.

Each lane represents a phase of work, and each card is a task that can be claimed by a specific Hermes profile.

Out of the box, the dispatcher can promote many ready tasks in one pass. That is fine for elastic cloud APIs, but it can overload a small self-hosted GPU cluster.

If you are new to this stack, start with the broader Hermes setup and operations guide and the AI Systems pillar for surrounding architecture.

This post shows how to:

Understand how Hermes Kanban dispatch interacts with your LLM gateway.
Control parallelism safely for heavy tasks.
Batch promotions with cron so background jobs do not collide with interactive use.
Monitor and tune the system so GPUs stay busy without overload.

How Hermes Kanban and the dispatcher work

At a high level, the system has three layers:

Board - durable SQLite state for tasks, columns, relations, and history.
Workers - Hermes profiles started in isolated workspaces to process a task.
Dispatcher - a long-lived process that scans for dispatchable cards and starts runs.

Tasks created from CLI or dashboard usually start in backlog or ready.

The dispatcher scans for eligible cards, claims one atomically, and starts the assigned profile with its tools and memory.

Each worker then calls your LLM gateway or local runtime (for example, OpenAI-compatible endpoints backed by Ollama, vLLM, or llama.cpp). For deployment choices across these runtimes, use the LLM Hosting in 2026 Local Self-Hosted and Cloud Infrastructure Compared. If you are tuning request fan-out on Ollama itself, this pairs well with How Ollama Handles Parallel Requests.

If you add many heavy tasks and do not cap promotions, your gateway can get flooded with concurrent requests.

On a single-GPU or CPU-bound host, that often means queueing, thrashing, and timeouts instead of better throughput.

The practical limitation today

In current Hermes builds many teams run, dispatcher config exposes only two Kanban dispatch keys and does not apply a global active-task cap from config:

kanban:
  dispatch_in_gateway: false
  dispatch_interval_seconds: 10

For active-task control, rely on explicit dispatch cadence (hermes kanban dispatch --max ...) plus dependency modeling.

Known gotchas:

Do not run gateway-embedded dispatch and hermes kanban daemon --force against the same board, or you can get claim races.
If the gateway is down, ready tasks do not dispatch and can burst later when service returns.
Longer dispatch intervals feel uneven because claiming happens in ticks.
Behavior can vary across versions because run-state and reclaim edge cases were patched over time.

Quick verification when behavior looks wrong:

# 1) confirm exactly one dispatcher path is active
pgrep -af "hermes gateway start|hermes kanban daemon"

# 2) check the wired Kanban dispatcher keys
rg "dispatch_in_gateway|dispatch_interval_seconds" ~/.hermes/config.yaml

# 3) inspect queue shape
hermes kanban list --status ready
hermes kanban list --status running

Key ideas:

Dispatcher config wires dispatch_in_gateway and dispatch_interval_seconds.
dispatch --max limits new spawns in that pass, not total running tasks.
For small self-hosted clusters, start conservative and increase only after latency stays stable.

When first deploying Hermes near your LLM gateway:

Keep only supported Kanban dispatcher keys in config.
Observe GPU and CPU utilization under real queue pressure.
Use Strategy 1 or Strategy 2 for deterministic pacing.

Investigation findings and root cause

hermes kanban dispatch does not read config.yaml for max_active_tasks.

In hermes_cli/kanban.py, the dispatch command exposes --max as a CLI cap (default None) and passes only args.max into kb.dispatch_once(...). There is no max_active_tasks config lookup in this path. See hermes_cli/kanban.py raw.

Then in kanban_db.dispatch_once, the only cap is max_spawn, with logic equivalent to:

if max_spawn is not None and spawned >= max_spawn:
    break

There is no check of already running tasks and no max_active_tasks reference in that dispatch path. See hermes_cli/kanban_db.py raw.

Effective behavior:

hermes kanban dispatch

unbounded for that pass (limited by ready queue size).

hermes kanban dispatch --max 2

caps only new spawns in that pass, not total running tasks.

The wired config knobs around gateway dispatch are kanban.dispatch_in_gateway and kanban.dispatch_interval_seconds.

So max_active_tasks is ignored in this dispatch path because it is not implemented there.

Strategy 1 - Encode dependencies for strictly sequential flows

Some workflows should run strictly one after another — for example:

multi step data pipelines with shared intermediate artefacts
migrations or infrastructure changes
batch jobs that write to the same object store or database

Hermes Kanban supports parent child dependencies between tasks so that a child card becomes dispatchable only when its parent is done.

You can model this with a small helper script around the Hermes CLI:

#!/usr/bin/env bash

set -euo pipefail

parent_id="$(hermes kanban add \
  --title 'Ingest customer logs for April' \
  --profile 'etl-worker' \
  --column backlog)"

hermes kanban add \
  --title 'Generate April anomaly report' \
  --profile 'analytics-worker' \
  --column backlog \
  --parent "${parent_id}"

hermes kanban add \
  --title 'Publish April summary to dashboard' \
  --profile 'reporting-worker' \
  --column backlog \
  --parent "${parent_id}"

With an appropriate board policy and low dispatcher limits only the parent task runs first.

Once it finishes the child tasks gradually become ready, and the dispatcher pulls them one by one without ever exceeding your concurrency caps.

Strategy 2 - Use Linux cron with a running-aware dispatch cap

If you want deterministic pacing, use host cron plus a small wrapper script.

Instead of always calling dispatch --max 2, first count currently running tasks, then dispatch only the remaining slots.

Create hermes-kanban-dispatch-capped.sh:

#!/usr/bin/env bash
set -euo pipefail

MAX_PARALLEL="${MAX_PARALLEL:-2}"
BOARD="${BOARD:-}"

board_args=()
if [[ -n "$BOARD" ]]; then
  board_args=(--board "$BOARD")
fi

# or where your hermes is installed
export PATH="/home/abc/.local/bin:$PATH"

running_out="$(hermes kanban "${board_args[@]}" list --status running)"

if [[ "$running_out" == *"(no matching tasks)"* ]]; then
  running_count=0
else
  running_count="$(printf '%s\n' "$running_out" | wc -l)"
fi

slots=$(( MAX_PARALLEL - running_count ))

if (( slots <= 0 )); then
  echo "Already at limit running=$running_count max=$MAX_PARALLEL dispatch skipped"
  exit 0
fi

echo "running=$running_count max=$MAX_PARALLEL slots=$slots dispatching up to $slots"

hermes kanban "${board_args[@]}" dispatch --max "$slots"

Make it executable:

chmod +x ./hermes-kanban-dispatch-capped.sh

Run it with:

MAX_PARALLEL=2 ./hermes-kanban-dispatch-capped.sh

For a specific board:

BOARD=my-board MAX_PARALLEL=2 ./hermes-kanban-dispatch-capped.sh

Schedule it once per minute with cron:

* * * * * /opt/hermes/scripts/hermes-kanban-dispatch-capped.sh >> /var/log/hermes/kanban-cron.log 2>&1

Operational notes:

Cron often has a minimal PATH, so if hermes is not found, use its full path inside the script (for example /usr/local/bin/hermes).
If you log to /var/log/hermes/..., create that directory first and ensure the cron user has write access.

Example:

sudo mkdir -p /var/log/hermes
sudo chown "$USER":"$USER" /var/log/hermes

Create or edit cron entries with:

crontab -e

Then verify with:

crontab -l

Sub-minute cadence with one cron entry

Cron ticks once per minute, but you can still dispatch more frequently by running a short loop inside the script.

Example hermes-kanban-dispatch-subminute.sh:

#!/usr/bin/env bash
set -euo pipefail

LOCK_FILE="/tmp/hermes-kanban-dispatch.lock"
RUNS_PER_MINUTE="${RUNS_PER_MINUTE:-4}"    # 4 runs => every 15 seconds
CAP_SCRIPT="${CAP_SCRIPT:-/opt/hermes/scripts/hermes-kanban-dispatch-capped.sh}"

exec 9>"$LOCK_FILE"
flock -n 9 || exit 0

sleep_seconds=$(( 60 / RUNS_PER_MINUTE ))

for ((i=1; i<=RUNS_PER_MINUTE; i++)); do
  "$CAP_SCRIPT"

  if (( i < RUNS_PER_MINUTE )); then
    sleep "$sleep_seconds"
  fi
done

Make it executable:

chmod +x ./hermes-kanban-dispatch-subminute.sh

Schedule it once per minute:

* * * * * /opt/hermes/scripts/hermes-kanban-dispatch-subminute.sh >> /var/log/hermes/kanban-subminute.log 2>&1

This gives an effective sub-minute cadence while flock prevents overlapping runs.

Why this works:

list --status running gives current running load.
dispatch --max N caps only new spawns for that pass.
Computing N as remaining slots keeps total running tasks near your target limit.

Important caveat: this cap works only for dispatches made through this script.

Disable gateway embedded dispatch, otherwise it can still promote tasks independently:

kanban:
  dispatch_in_gateway: false

The official docs describe both command capabilities and note gateway dispatch defaults in the Kanban feature guide: Hermes Kanban docs.

Internal Hermes Cron

Do not use it.
Do you really want your llm to process regular prompts like Execute in terminal the command /path/hermes-kanban-dispatch-capped.sh, especially when it's busy doing some useful work?

Hermes Kanban Monitoring and Tuning

Whichever strategy you choose you should monitor:

LLM gateway metrics — request rate, latency, error rate, token throughput.
Node health — GPU utilisation, VRAM usage, CPU load and RAM.
Hermes metrics — how many tasks are in backlog, ready, active and done.

For production metric baselines and dashboards, see Monitor LLM Inference in Production with Prometheus and Grafana and the broader LLM Performance hub.

Start with low concurrency, then gradually raise limits while watching for:

rising latency at constant throughput
increasing timeout or rate limit errors
long tails where some tasks stay active for a very long time

As soon as you see these symptoms roll back to the previous stable configuration and keep that as your default.

When Kanban is the right tool

Hermes Kanban shines when you have:

long lived research or engineering backlogs
multi agent collaboration with named profiles
workflows that must survive restarts and host reboots
humans who want a dashboard to triage work

If you only need a single run to create a few temporary helpers, the built in delegate task tools are usually simpler.

Once you need history, dashboards and strict control over how your agents hit self hosted LLMs the Kanban board plus dispatcher is the right foundation.

With a few configuration tweaks and optional cron based batching you can keep Hermes Kanban responsive while protecting your gateway and hardware.

Hermes Agent Skill Authoring — SKILL.md Structure and Best Practices

Rost — Wed, 06 May 2026 08:10:04 +0000

Hermes Agent treats skills as the default way to teach repeatable workflows. Official documentation describes them as on-demand knowledge documents aligned with the open agentskills.io shape, loaded through progressive disclosure so the model sees a small index first and only pulls full instructions when a task actually needs them.

Authoring is less about clever wording than about packaging—you are telling the runtime when to load a procedure, what order of steps counts as “done,” and how to tell success from a silent failure. This article stays focused on SKILL.md structure, supporting folders, visibility rules, and the split between secrets and non-secret settings—the details that decide whether a skill shows up in /slash commands, survives a hub install, or quietly disappears on CI.

Hermes sits inside the broader AI Systems: Self-Hosted Assistants, RAG, and Local Infrastructure cluster, where assistants are treated as systems built from inference, retrieval, memory, and tooling rather than as a single chat surface. Install paths, provider wiring, gateway behavior, and the layout of ~/.hermes are all spelled out in the Hermes AI Assistant - Install, Setup, Workflow, and Troubleshooting guide; day-to-day shell ergonomics—hermes skills, profiles, gateway, memory—are easier to scan in the Hermes Agent CLI cheat sheet — commands, flags, and slash shortcuts. In real deployments, skills inherit isolation from profiles (separate config, secrets, memories, and skill trees). Hermes AI Assistant Skills for Real Production Setups argues for treating those profiles—not individual markdown files—as the unit of ownership; keep that in mind when you name skills and decide what belongs in shared external_dirs versus a single profile.

Skill or tool?

Official guidance is blunt. Use a skill when the capability is mostly prose instructions plus shell commands and tools Hermes already exposes—wrapping a CLI, driving git, calling curl, or using web_extract for structured fetches. Use a tool when you need tight integration for API keys and auth flows, deterministic binary handling, streaming, or Python that must execute the same way every time.

That boundary matters in practice because skills ship without changing agent code, while tools carry review and release overhead. Most teams benefit from starting with a skill, then promoting only the brittle core to a tool once the failure modes are obvious (auth refresh loops, binary parsers, strict idempotency).

Procedures versus curated memory

Skills answer how to run a workflow; Hermes’ bounded core memory answers what has already been agreed about the user and the project. A skill loads when the task matches its description; MEMORY.md and USER.md stay in the prompt as a small, curated fact layer. The two mechanisms stack rather than compete, and the full picture of snapshots, limits, and external providers is laid out in Hermes Agent Memory System: How Persistent AI Memory Actually Works.

Anatomy of a skill directory

On disk, every skill is a folder under ~/.hermes/skills/, often nested under a category such as devops/ or research/. Hermes expects SKILL.md at the leaf; everything else is optional structure you add when the instructions would otherwise sprawl. The usual pattern is references/ for long tables or vendor docs, templates/ for output skeletons, scripts/ for deterministic helpers, and assets/ for static files the agent should not re-fetch.

That layout mirrors how progressive disclosure works in practice: the agent can stay at the main file until it truly needs a deep appendix. Keeping “happy path” prose in SKILL.md and pushing rarely used detail into references/ is one of the cheapest ways to protect token budgets.

Hermes can also merge in external skill directories via skills.external_dirs in config.yaml. Those paths are scanned for discovery, but the agent still writes through skill_manage into the primary ~/.hermes/skills/ tree. Local names shadow external ones, so if you “fix” a shared skill in your home directory, teammates pulling the same external repo will not see your edit until they remove or rename the local copy—a common source of “it works on my machine” confusion.

SKILL.md frontmatter that survives review

The body of SKILL.md is Markdown; the opening block must be valid YAML between --- delimiters. Real skills accumulate long fenced examples, so the small habits from Markdown Code Blocks: Complete Guide with Syntax, Languages & Examples—consistent language tags, readable excerpts, tight fences—keep large files maintainable for humans and slightly easier for the model to scan.

Required fields are name and description. The name becomes the slash route and index key; it stays lowercase with hyphens and must respect the documented length cap. The description is the only prose many sessions ever pay for at level zero, so it should read like a search result or router string (“when backups look stale, verify latest archive and checksum”), not the first paragraph of a blog post.

Optional top-level keys such as version, author, and license help hub packaging and audits. The platforms list (macos, linux, windows) is sharper than it looks—when set, Hermes omits the skill entirely on non-matching hosts, which is why a skill that “works on my Mac” can vanish in Linux CI with no error message beyond a shorter skill list.

Hermes-specific knobs live under metadata.hermes: tags, related_skills, and the conditional visibility fields in the next section. required_environment_variables declares secrets that should land in .env and pass into sandboxes; required_credential_files covers OAuth token files and other on-disk credentials that must mount into Docker or Modal; metadata.hermes.config declares non-secret preferences stored under skills.config in config.yaml.

Official docs stress size discipline for a reason. Trim the description to its budget, front-load the procedure, and push historical notes or giant option matrices into references/ so a partial skill_view still gives the agent something actionable.

Below is a minimal SKILL.md you can drop into ~/.hermes/skills/devops/backup-check/SKILL.md (or any category folder) and iterate from there.

---
name: backup-check
description: Verify nightly backup archives exist, are non-empty, and pass a quick checksum spot-check on the latest file.
version: 1.0.0
metadata:
  hermes:
    tags: [devops, backups, shell]
    requires_toolsets: [terminal]
    config:
      - key: backup_check.archive_dir
        description: Absolute path to the directory that holds backup archives
        default: "/var/backups"
        prompt: Backup archive directory (absolute path)
---

# Backup archive spot-check

## When to use

Use when the user asks to confirm backups ran, to audit the latest archive on disk, or to catch empty or stale backup files before a restore drill.

## Quick reference

- Latest archive directory is configured under `skills.config.backup_check.archive_dir` (set via `hermes config migrate` if declared in metadata).
- Default check uses `ls` by mtime and `test -s` for non-empty files.

## Procedure

1. Resolve the archive directory from skill config or ask the user once if unset.
2. List the most recently modified file matching the expected pattern (for example `*.tar.zst`).
3. Confirm the file exists, is non-empty, and record its path and size for the reply.
4. If a checksum file exists beside the archive, verify it with the documented tool (for example `sha256sum -c`).

## Pitfalls

- Empty files can still have a recent mtime if a failed job touched the path; always check size.
- Relative paths break when the terminal cwd is not the backup host; use absolute paths in config.

## Verification

The user should see the latest archive path, byte size, and either a checksum OK line or an explicit note that no `.sha256` sidecar was found.

Progressive disclosure in practice

Progressive disclosure is the difference between a skill library that feels snappy and one that burns thousands of tokens before the first user message. Hermes walks three conceptual steps: a compact catalog (names and short descriptions), the full SKILL.md when the task matches, and—only if needed—a slice of a reference file via skill_view paths. Assume level zero is all the model will read until it explicitly commits; every sentence in the description and the first screen of body text should help routing, not storytelling.

A practical outline that survives partial loads is When to use (triggers in plain language), Quick reference (commands, env vars, file paths), Procedure (ordered steps the agent should not improvise away), Pitfalls (known failure modes), and Verification (what “green” looks like). Narrative history, vendor changelog dumps, and twenty-row option tables belong in references/ with stable headings so the agent can pull a single section.

When a skill activates, Hermes can rewrite ${HERMES_SKILL_DIR} and ${HERMES_SESSION_ID} in the body so shell lines point at the installed folder without hand-built paths. Optional inline shell snippets (!cmd`) can inject fresh context (current branch, disk free space), but they execute on the host and stay disabled unlessskills.inline_shell` is on—treat that flag as a trust boundary for the whole skill source, not a convenience toggle.

Conditional activation and prompt hygiene

Skills can show or hide based on which toolsets or tools exist in the current session. requires_toolsets / requires_tools gate a skill behind capabilities that must be present; fallback_for_toolsets / fallback_for_tools surface a cheaper or local path when a premium integration is absent—the DuckDuckGo fallback when a paid web search API is not configured is the canonical example.

These predicates directly shape prompt noise. An overly strict requires_* rule hides a skill from newcomers who have not finished hermes tools setup yet; an overly loose fallback_for_* rule duplicates half your library whenever someone omits an API key. The useful middle ground is to name real prerequisites, test with hermes chat --toolsets skills, and toggle keys or toolsets on purpose while watching whether the skill list breathes the way you expect.

Secrets, config, and credential files

Secrets should be declared in required_environment_variables. Hermes can prompt when a skill loads in the local CLI, persist values in .env, and pass them into terminal and execute_code sandboxes without streaming the raw secret back into the model transcript. Remote chat surfaces refuse to collect keys inline and instead point people at hermes setup or manual .env edits—author your skill text so it matches that behavior (tell users that a key is required, not *to paste it into Telegram).

Non-secret preferences—default paths, org names, feature toggles—belong in metadata.hermes.config. Values resolve into skills.config inside config.yaml, show up in hermes config show, and arrive in the skill message as resolved facts so the model does not need to open your config file mid-task.

File-shaped credentials (OAuth token JSON, service account keys) map to required_credential_files. When those files exist, Hermes can bind-mount them into Docker or sync them into Modal jobs; declaring them upfront avoids the classic “script works locally, dies in sandbox” gap.

Supporting scripts and dependencies

The upstream guide pushes authors toward boring dependencies: stdlib Python, curl, and Hermes’ own tools (web_extract, read_file, terminal). That is less about purity than about reproducibility—every extra pip install is another silent failure when the agent runs in a clean container.

When JSON or XML parsing is fiddly, a short script under scripts/ plus a ${HERMES_SKILL_DIR} path beats asking the model to re-derive parsers each run. If you truly need a package, state the install command in Procedure, repeat the failure symptom in Pitfalls, and give a Verification command that fails loudly when the dependency is missing.

Publishing, hub installs, and trust

Community skills move through the Skills Hub and the other discovery paths the user guide lists—official optional skills, GitHub slugs, skills.sh entries, .well-known indexes, and raw SKILL.md URLs. Installs are scanned for obvious exfiltration, injection, and destructive patterns; trust tiers run from builtin through community, and some findings only clear with --force while the worst cases stay blocked entirely.

The SKILL.md file shape is not Hermes-specific; IDE-centric assistants use the same progressive-loading idea with different discovery and triggers. Claude Skills and SKILL.md for Developers: VS Code, JetBrains, Cursor is a useful contrast read—frontmatter discipline and “load only when relevant” carry over, even when the installer and slash-command wiring differ.

Org-wide rollouts usually pair a private tap or shared Git repo with external_dirs for read-only sharing, while keeping the agent-writable copy under each profile when skill_manage is allowed to mutate skills in place.

Troubleshooting and optimization

When a skill misbehaves, walk this checklist before rewriting prose.

Visibility — Confirm platforms, requires_*, and fallback_for_* predicates. A skill that “works on my Mac” but not in Linux CI is often a platform guard.
Name collisions — Duplicate names across local and external directories follow local precedence. Rename or namespace aggressively.
Discovery layout — A misplaced SKILL.md or wrong category folder can drop the skill from indexing entirely.
Token load — If sessions feel slow, shorten level-zero descriptions, move depth into references/, and deduplicate giant tables.
Agent edits — Hermes can create, patch, or delete skills via skill_manage. Treat valuable skills like code: review diffs, export snapshots, and reset bundled skills deliberately when upgrades drift.

A tight regression loop beats rereading the whole file: hermes chat --toolsets skills -q "Use the <skill> workflow to <concrete task>" should show the agent pulling the right disclosure level before it freestyles. If it never invokes skill_view, your When to use text or description probably does not match how people phrase requests.

Official references stay authoritative for behavior changes—the Skills System user guide for runtime semantics, Creating Skills for author-facing rules, the Bundled Skills Catalog for copy-paste examples, and the agentskills.io specification for the shared file format Hermes aligns with.

Hermes Agent CLI cheat sheet — commands, flags, and slash shortcuts

Rost — Mon, 04 May 2026 10:57:09 +0000

Hermes Agent from Nous Research is a model-agnostic, tool-using assistant you run locally or on a VPS.

Hermes does not lock you into one surface. You can use

the classic hermes / hermes chat CLI,
the full-screen hermes --tui session,
a long-running hermes gateway for Telegram, Discord, Slack, and other messaging platforms,
hermes dashboard for a local browser UI when the web extra is installed.

Those paths share the same config and data under ~/.hermes; this page lists shell commands that matter across those modes.

Below is a dense command reference grouped by task.

Install Hermes Agent and first-run CLI commands

For install and troubleshooting, start with Hermes AI Assistant — Install, Setup, Workflow, and Troubleshooting.

The installer pulls the repo, sets up a Python environment, and wires the hermes executable. After source ~/.bashrc or ~/.zshrc, your default entry point for interactive chat is simply hermes (same family as hermes chat).

Command	Description
`curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \	bash`
`hermes` / `hermes chat`	Start interactive chat after install (default daily entry).
`hermes --version` / `hermes version`	Print version information.
`hermes completion bash` \	`zsh` \
`hermes update [--check] [--backup] [--restart-gateway]`	Pull latest code, reinstall deps, optional pre-update home snapshot or gateway restart.
`hermes uninstall [--full] [--yes]`	Remove Hermes; optional full data deletion.

Native Windows is not supported; use WSL2. Android installs via Termux follow a dedicated path in the upstream docs.

Global flags for every `hermes` invocation

These flags apply before subcommands and change which profile, which session, or how much personal config loads.

Flag	Description
`--profile`, `-p`	Select Hermes profile for this run (overrides sticky default from `hermes profile use`).
`--resume`, `-r`	Resume a session by ID or title.
`--continue [name]`, `-c`	Continue the latest session, or latest matching a title.
`--worktree`, `-w`	Start in an isolated Git worktree for parallel agents.
`--yolo`	Bypass dangerous-command approval prompts (use with care).
`--pass-session-id`	Include session ID in the system prompt.
`--ignore-user-config`	Skip `~/.hermes/config.yaml` (defaults only); `.env` still loads.
`--ignore-rules`	Skip auto-injection of AGENTS.md, SOUL.md, `.cursorrules`, memory, preloaded skills.
`--tui`	Launch the TUI (`HERMES_TUI=1` equivalent).
`--dev`	With `--tui`, run TS sources via `tsx` for TUI development.

Isolated automation often pairs hermes chat --ignore-user-config --ignore-rules with hermes -z for reproducible one-shots.

`hermes chat`, one-shot prompts, and `hermes -z`

Command / pattern	Description
`hermes chat`	Interactive or scripted chat; main surface for `-q`, `-m`, `--provider`, toolsets, resume, worktree, checkpoints.
`hermes chat -q "..."`	One-shot prompt (non-interactive); keeps richer output than `-z` when tools run.
`hermes -z "..."`	Scripted one-shot — final answer only on stdout, no banner or session noise. Same agent and tools; best for pipes and scripts.
`hermes chat --quiet`, `-Q`	Quieter programmatic mode (banner and tool previews suppressed).
`-m` / `--model`, `--provider`	Per-run model and provider overrides; env `HERMES_INFERENCE_MODEL` / `HERMES_INFERENCE_PROVIDER` mirror behavior.
`-t` / `--toolsets`	Enable comma-separated toolsets for the run.
`-s` / `--skills`	Preload skills (repeat or comma-separated).
`--image path`	Attach a local image to a single query.
`--checkpoints`	Enable filesystem checkpoints before destructive edits.
`--max-turns N`	Cap tool-calling iterations per turn (default from config).
`--source`	Session source tag (`cli` vs `tool` for integrations).

Hermes model outside the session vs /model inside it — Running hermes model from the shell is where you add providers, keys, and OAuth. Slash /model only switches among already configured providers. If you only see OpenRouter in /model, exit the session and complete hermes model.

Model picker, credential pools, and fallback providers

Command	Description
`hermes model`	Interactive provider and model picker; keys, OAuth, custom endpoints.
`hermes auth`	Credential pools — `add`, `list`, `remove`, `reset` for rotation-friendly keys and OAuth.
`hermes fallback [list \	add \
{% raw %}`hermes setup [model \	tts \

Deprecated {% raw %}hermes login / hermes logout — use hermes auth and hermes model instead.

Picking local OpenAI-compatible endpoints versus hosted APIs for hermes model sits on the same trade-offs as general LLM hosting (latency, cost, ops).

Config files and `hermes config` commands

Configuration resolves as CLI overrides → config.yaml → .env → defaults. API keys belong in .env; structured settings in config.yaml.

Command	Description
`hermes config show`	Display effective configuration.
`hermes config edit`	Open `config.yaml` in `$EDITOR`.
`hermes config set key value`	Set values (secrets routed to `.env`, non-secrets to YAML).
`hermes config path` / `hermes config env-path`	Print paths to config and env files.
`hermes config check`	Detect missing or stale settings.
`hermes config migrate`	Apply newly introduced options interactively.

Where files live — Everything sits under HERMES_HOME (default ~/.hermes) for config, secrets, memories, skills, sessions, gateway state, and logs.

Session management and `hermes profile`

Command	Description
`hermes sessions list`	List recent sessions.
`hermes sessions browse`	Interactive picker with search and resume.
`hermes sessions export`	Export sessions (e.g. JSONL).
`hermes sessions delete`, `prune`, `rename`, `stats`	Delete one session, prune old ones, rename titles, show store stats.
`hermes profile list` \	`use` \
`hermes profile export` / `import`	Archive or restore a profile tarball.
`hermes profile alias`	Short wrapper scripts for fast profile switching.

Use hermes -p work chat -q "..." for ad hoc runs without changing the sticky default profile.

Skills hub, toolsets, shell hooks, and plugins

For profile-first configuration and skills tuned to real production workflows by role, see Hermes AI Assistant Skills for Real Production Setups.

Command	Description
`hermes tools`	Interactive per-platform tool enablement; `--summary` prints current choices.
`hermes skills browse`, `search`, `inspect`, `install`, `list`, `check`, `update`, `audit`, `uninstall`, `publish`, `snapshot`, `tap`, `config`	Skills hub workflows including registries and URL installs.
`hermes curator status`, `run`, `pause`, `pin`, `rollback`, …	Background skill maintenance and safe rollback.
`hermes hooks list`, `test`, `revoke`, `doctor`	Declared shell hooks and allowlists in config.
`hermes plugins`	Composite UI or subcommands to install, enable, disable, remove plugins.

Built-in memory and `hermes memory` providers

Built-in MEMORY.md / USER.md stay active; external providers add optional recall layers. For how that architecture behaves in practice, read Hermes Agent Memory System — How Persistent AI Memory Actually Works. To compare external backends and activation trade-offs, see Agent Memory Providers Compared — Honcho, Mem0, Hindsight, and Five More.

Command	Description
`hermes memory setup`	Interactive external memory provider configuration.
`hermes memory status`	Show active provider settings.
`hermes memory off`	Disable external provider; built-in files remain.

When a provider is active it may register extra provider-specific top-level subcommands — run hermes --help to see what is wired today.

Messaging gateway, DM pairing, and platforms

Command	Description
`hermes gateway setup`	Interactive messaging platform setup.
`hermes gateway run`	Foreground gateway (recommended on WSL, Docker, Termux).
`hermes gateway start` \	`stop` \
`hermes gateway install` \	`uninstall`
`hermes pairing list` \	`approve` \
`hermes whatsapp`	WhatsApp bridge pairing flow.
`hermes slack manifest`	Generate Slack app manifest with gateway slash parity.

On WSL, hermes gateway run inside tmux is the resilient pattern when gateway start misbehaves.

Cron scheduler, webhooks, and Kanban

Command	Description
`hermes cron …`	Create, edit, pause, resume, run, remove scheduled prompts (`tick` for manual scheduler pass).
`hermes webhook subscribe`, `list`, `remove`, `test`	Dynamic webhook routes for event-driven runs.
`hermes kanban …`	Multi-profile task board backed by SQLite; `dispatch` drives workers.

`hermes doctor`, logs, backup, and usage insights

Command	Description
`hermes doctor [--fix]`	Interactive diagnostics and optional auto-repair.
`hermes status [--all] [--deep]`	Concise status; deeper checks when needed.
`hermes dump [--show-keys]`	Paste-friendly setup summary for Discord or GitHub issues.
`hermes debug share`	Upload redacted debug bundle to a paste service (or `--local`).
`hermes logs [agent \	errors \
{% raw %}`hermes backup`, `hermes import`	Zip snapshots of home data and restore paths.
`hermes insights [--days N] [--source …]`	Token, cost, and activity analytics.

When something breaks after an upgrade, hermes doctor, hermes status, and hermes logs errors -f form the fastest triage loop.

MCP, ACP, web dashboard, and OpenClaw migration

Command	Description
`hermes mcp serve`	Run Hermes as an MCP server.
`hermes mcp add`, `remove`, `list`, `test`, `configure`	Manage MCP client connections from Hermes.
`hermes acp`	Agent Client Protocol stdio server for editors (extra install may apply).
`hermes dashboard [--port …] [--host …]`	Local web dashboard (`pip install hermes-agent[web]`).
`hermes claw migrate …`	Migrate OpenClaw-style configs into Hermes (`--dry-run`, presets, optional secrets).

OpenClaw migration — hermes claw migrate reads legacy OpenClaw home directories; for what that stack looked like before moving, see the OpenClaw case study.

Slash commands in the Hermes CLI session

Type / for autocomplete. Commands are case-insensitive; skills register extra /skill-name routes. The tables below are a curated subset; for the full registry see Official Hermes Agent documentation at the end of this article.

Session flow, background tasks, and goals

Command	Description
`/new`, `/reset`	New session ID and history.
`/resume [name]`	Resume a named session.
`/compress [focus]`	Manual context compression with optional focus topic.
`/retry`, `/undo`	Retry last turn or drop last exchange.
`/title …`	Name the session for later `/resume`.
`/background …`, `/queue …`, `/steer …`	Parallel background run, queued next prompt, mid-loop nudge after next tool.
`/goal …`	Persistent multi-turn objective with judge loop (`status`, `pause`, `resume`, `clear`).
`/branch`, `/fork`	Branch the conversation for alternate exploration.

Models, tool toggles, skills, and reload

Command	Description
`/model … [--global]`	Switch models among configured providers; `--global` persists default.
`/tools …`, `/toolsets`	Session tool toggles and toolset listing.
`/skills …`	Search, install, and manage skills from chat.
`/cron …`	Scheduled tasks UI from the CLI session.
`/reload-mcp`	Reload MCP servers from config.
`/reload`	Reload `.env` into the running session without restart.

Usage, help, and quitting

Command	Description
`/usage`, `/insights`	Token and cost visibility; analytics snapshot.
`/help`, `/quit`	Help or exit the CLI.

Messaging apps (Telegram, Discord, Slack, and others) expose an overlapping slash set plus /approve, /restart, /commands, and related gateway-only helpers — platform differences are documented in the slash command reference under Official Hermes Agent documentation below.

Official Hermes Agent documentation

Upstream documentation on hermes-agent.nousresearch.com:

Tip. Keep hermes dump and hermes doctor --fix in muscle memory — they turn vague "something broke" reports into actionable diffs against a known-good setup.

MinIO CE in 2026: Retired Upstream, Source-Only, and What to Use

Rost — Mon, 04 May 2026 10:56:52 +0000

MinIO Community Edition is no longer a safe default for new production systems.

As of 2026, the public project status and distribution model changed enough that many teams now treat MinIO CE as end of life for serious workloads.

If you are deciding whether to keep MinIO CE, fork it, or migrate, this guide gives you:

a factual timeline of what changed
the practical risk for operators
a technical comparison of SeaweedFS, Garage, RustFS, and Ceph RGW
a migration plan you can execute in phases

For broader context around storage, databases, and search in production AI stacks, see the Data Infrastructure for AI Systems pillar.

What happened to MinIO CE

The community concern is not one single event. It is the sequence.

Date	What changed	Why it matters
May 2025	Key management features moved out of CE path	Reduced CE parity for auth and admin workflows
Oct 2025	Community Docker images and public binaries stopped	Operators must build and verify from source
Dec 2025	Public maintenance mode messaging became explicit	Fewer expectations for active OSS iteration
Feb 2026	Repository archived for the first time	Read only state blocks normal OSS collaboration
Apr 2026	Repository archived again and stayed locked	Confirms long term frozen upstream posture

The core operational impact is simple

you inherit more supply chain, patching, and maintenance responsibility than most teams expect from a mainstream S3 compatible store.

Is MinIO still open source in 2026

A common question is whether MinIO is still open source at all.

The server code in the public repository is still under AGPLv3.

However, the practical community path changed from normal binary-first consumption to source-first self build.

For many teams, that feels less like a living OSS ecosystem and more like unsupported source availability.

So the accurate answer is nuanced

license status remains open source, but operationally the community experience is no longer what most platform teams need for low risk production adoption.

Is MinIO CE safe for new production deployments

For greenfield deployments, usually no, especially when compared with documented options in this MinIO vs Garage vs AWS S3 comparison.

Why the risk profile changed

Patch cadence risk no stable, trusted community binary channel means every CVE cycle becomes your build and release cycle
Verification burden your team must own provenance, repeatability, and rollback strategy
Ecosystem drift risk tooling that assumed public images may lag or break
People risk senior SRE and security time is consumed by platform plumbing instead of product work

If you already run MinIO CE internally, this does not mean panic shutdown.

It means treat the platform as controlled technical debt and put a migration runway on your roadmap.

Community verdict and market response

Across operator communities in 2025 to 2026, the pattern is consistent:

fewer teams choose MinIO CE for net new deployments
more teams evaluate Garage and SeaweedFS first
enterprise teams with strict S3 semantics often move toward Ceph RGW
RustFS gets attention as a direct successor style option, but with alpha caution

This trend matters because platform safety is partly social

healthy ecosystems reduce integration risk, improve troubleshooting velocity, and widen hiring pools.

Best alternatives to MinIO CE

SeaweedFS

SeaweedFS is a strong option when you care about huge object counts, small file behavior, and practical efficiency in commodity environments.

Choose SeaweedFS when

you need high small-object density
you prefer Apache 2.0 governance and licensing clarity
you want production readiness without the heavy footprint of Ceph

Garage

Garage is attractive for lightweight self-hosted clusters, edge nodes, and geo distributed deployments on modest hardware.

If you want a concrete setup path, use this Garage S3 quickstart to validate replication and operations before migration.

Choose Garage when

resource efficiency matters more than full S3 feature parity
you run mixed ARM or small node environments
you want simple operations over maximal feature surface

RustFS

RustFS is frequently discussed as the closest successor narrative to MinIO style deployment and UX.

Choose RustFS when

you accept alpha-stage software risk
you can test deeply before production
you want to track a fast moving project with potential upside

For regulated or high uptime systems, keep RustFS in pilot until maturity is proven in your own reliability tests.

Ceph RGW

Ceph RGW remains the enterprise heavyweight with broad capability and scale.

Choose Ceph RGW when

you need mature enterprise S3 behavior
your team already has Ceph operational expertise
you can support higher infrastructure and on-call complexity

Which object store is best for your use case

Use this pragmatic filter:

Small team and low ops budget start with Garage or SeaweedFS
Large enterprise and strict compatibility needs prefer Ceph RGW
Experimental migration from MinIO style workflows pilot RustFS, but keep rollback options

No option is universally best.

The correct target depends on required S3 features, RPO and RTO goals, team maturity, and how much platform ownership you want.

If your team still needs legacy MinIO background before deciding, this MinIO vs AWS S3 overview and this MinIO command cheatsheet help with current-state audits.

Migration plan from MinIO CE

If you are currently on MinIO CE, this phased approach avoids risky big-bang moves.

Phase 1 inventory and risk scoring

list buckets, object counts, and growth rates
classify workloads by criticality and recovery objectives
identify hard S3 dependencies such as versioning, object lock, or policy behavior

Phase 2 proof of compatibility

stand up one or two candidate platforms
replay representative read and write workloads
verify auth, lifecycle rules, retention behavior, and SDK edge cases

Plan to instrument your pilot from day one with metrics and alerts from the Observability pillar so migration regressions are measurable rather than anecdotal.

Phase 3 pilot cutover

migrate one low blast radius workload first
run dual read validation where possible
measure latency, error rates, and operational overhead

Phase 4 production migration

migrate high priority internet facing workloads first
keep rollback artifacts and retention windows
document final runbooks before decommissioning MinIO CE paths

Bottom line

MinIO CE may still run, but it is no longer the low-friction default for new production object storage.

Treat current clusters as transition infrastructure, not a long horizon foundation.

For most teams in 2026, safer direction is:

SeaweedFS or Garage for pragmatic self hosted deployments
Ceph RGW for enterprise scale and mature S3 requirements
RustFS for monitored pilot environments only

Make the migration decision early while you can still choose your timeline instead of reacting to the next forced change.

NemoClaw practical guide for secure OpenClaw operations in 2026

Rost — Fri, 01 May 2026 11:02:25 +0000

Most AI agent stacks still treat security as a post-demo fix.
NemoClaw starts from the opposite assumption and makes isolation, policy, and routing day-zero defaults.

OpenClaw stays the assistant while OpenShell stays the enforcement layer, and NemoClaw acts as the opinionated glue between them. That glue matters because it makes the safer path easier to install, easier to observe, and much harder to skip when you are rushing.

That is exactly why NemoClaw matters in 2026. It is not just another wrapper around an LLM agent, because it is designed as a reference stack for running always-on OpenClaw assistants inside sandboxed OpenShell containers, with routed inference, policy-based egress control, and lifecycle tooling built in from day one. If you want broader context for where this fits, start with the AI Systems hub and the baseline OpenClaw system overview.

There is one hard truth you should not bury for the sake of hype. NVIDIA marks NemoClaw as alpha software in early preview, beginning March 16, 2026, and explicitly warns that interfaces and behavior may still shift between releases. Treat it like a serious lab tool, not finished production furniture.

What NemoClaw is and when to use it

NemoClaw exists for a specific operational job rather than for experimentation theater. It gives you a practical way to run an always-on OpenClaw assistant with guardrails around network access, filesystem access, process privileges, and model routing. If you have ever looked at an autonomous agent and thought that it should not have casual host access, NemoClaw is a strong answer to that discomfort.

The stack is easier to understand when you separate the layers:

OpenClaw is the assistant runtime, tools, memory, and behaviour inside the container.
OpenShell is the execution environment that provides sandbox lifecycle, credential-storing gateway, inference proxying, and policy enforcement.
NemoClaw is the opinionated reference stack that onboards, configures, and operates OpenClaw correctly inside OpenShell.

That distinction matters because it explains the product's purpose. NemoClaw is not trying to replace OpenClaw. It is trying to make OpenClaw survivable in real environments.

Typical use cases are obvious and sensible:

running an always-on assistant with controlled egress
testing agent behaviour before granting broader access
pushing a sandboxed assistant onto a remote GPU host for persistent operation

My blunt take is this. If you only want a throwaway demo, raw OpenClaw is simpler and faster to get moving, and the OpenClaw quickstart is the fastest route. If you want something that behaves like it belongs on an actual machine, NemoClaw is the more serious choice because its defaults are built for operators instead of screenshots.

NemoClaw security and operations features that matter

A long feature list is cheap. The right feature list is not. These are the capabilities that actually change how you operate the system.

Feature	Why it matters
Guided onboarding	`nemoclaw onboard` validates prerequisites, credentials, providers, and policy before the sandbox is created.
Hardened blueprint	NemoClaw builds on a versioned blueprint and a security-first image rather than a pile of one-off shell steps.
Routed inference	The agent talks to `inference.local`, while provider credentials stay on the host.
Layered protection	Network, filesystem, process, and inference controls are enforced together instead of as optional extras.
Policy tiers and presets	You can start restricted and selectively add access for package registries, search, messaging, or other services.
State management	Snapshots and rebuild flows exist so upgrades do not have to mean memory loss.
Channel messaging	Telegram, Discord, Slack, and similar bridges can be wired in through controlled host-side operations.
Skill installation	You can push skills into a running sandbox without turning the whole environment into mutable sludge.

NemoClaw supports several inference paths, including NVIDIA Endpoints, OpenAI, Anthropic, Google Gemini, compatible OpenAI style and Anthropic style endpoints, and local Ollama. For compatible endpoints, onboarding validates the endpoint with a real inference request because many services copy the OpenAI shape but fail on real runtime behavior. If you are choosing an inference runtime strategy first, the broader LLM hosting guide for 2026 is a useful companion. Experimental local NIM and local vLLM paths also exist, but they are gated behind an environment flag for a reason, so use them for evaluation instead of unattended long-running workloads.

The security model is the real headline. NemoClaw starts with deny-by-default egress, keeps provider credentials on the host, uses a read-only OpenClaw config inside the sandbox, and lets operators review unknown network requests in the OpenShell TUI. This is not flashy, but that is exactly the point, because flashy agent stacks are common while boring control surfaces are the scarce resource in production.

The defaults you should not casually relax

NemoClaw does have escape hatches. It also tells you, very politely, when you are about to do something foolish.

The biggest foot-guns are these:

--dangerously-skip-permissions trades the default sandbox posture for a permissive one
adding permanent baseline policy entries for one-off requests makes privilege creep feel normal
writing directly into /sandbox/.openclaw is the wrong mental model because that config is meant to stay locked down
using openclaw agent --local as if it were your standard operating mode is a bad habit for anything security-sensitive

That last point deserves emphasis. Local mode is convenient for smoke testing and one-off checks, but it is not the posture you should normalize for an always-on assistant that has any real permissions.

NemoClaw quickstart for your first sandbox

Prerequisites

Here is the practical baseline before you waste an afternoon pretending a tiny laptop is enough. The official prerequisites page currently lists Node.js 22.16 or later and npm 10 or later in addition to Docker runtime requirements.

Resource	Minimum	Recommended
CPU	4 vCPU	4 plus vCPU
RAM	8 GB	16 GB
Disk	20 GB free	40 GB free

The tested runtime matrix is also straightforward:

Platform	Runtime	Notes
Linux	Docker	Primary tested path
macOS on Apple Silicon	Colima or Docker Desktop	Works with limitations
DGX Spark	Docker	Tested
Windows	WSL2 with Docker Desktop backend	Works with limitations

If you are on macOS, install Xcode Command Line Tools first. If you are on Linux, make sure Docker is actually running and that your user can talk to it without permission drama.

There is also a resource detail that catches many first-time users. The sandbox image is around 2.4 GB compressed, and the export pipeline can temporarily consume enough memory to trigger OOM on weak machines. If you cannot add RAM, adding at least 8 GB swap is an official workaround, though it slows onboarding. For small dedicated AI boxes, the NVIDIA DGX Spark overview gives a concrete reference point for local always-on deployments.

Install and onboard

The official install path is intentionally simple:

curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

Then confirm the CLI is present:

nemoclaw --help
nemoclaw --version

From there, the real work is onboarding. nemoclaw onboard drives sandbox creation, provider setup, and policy application in one guided flow, which is why it should be your default lifecycle entry point.

nemoclaw onboard

During onboarding you will choose an inference provider, a sandbox name, and a policy tier. The tier choice matters more than most people expect:

restricted keeps the base sandbox only
balanced is the default and adds development tooling plus web search related access
open adds broad third-party access, including messaging and productivity services

My recommendation is not subtle. For an always-on assistant, start with the smallest posture that can possibly work. If that means restricted, good. Add only what the agent proves it needs.

If you want a scripted run, the non-interactive flow looks like this:

NEMOCLAW_POLICY_TIER=restricted \
nemoclaw onboard --non-interactive --yes-i-accept-third-party-software

Use a sane sandbox name. NemoClaw expects lowercase alphanumeric characters and hyphens. If you keep trying to be clever with names, the validator will win.

First connection and first prompt

Once onboarding completes, connect to the sandbox:

nemoclaw my-assistant connect

Inside the sandbox, open the terminal UI:

openclaw tui

If you only want a one-message smoke test, you can do this:

openclaw agent --agent main --local -m "hello" --session-id test

That said, do not confuse a smoke test with an operating model. For real day-two usage, I would rather stay honest about the system and use the TUI plus host-side monitoring instead of normalising --local.

NemoClaw operations that matter on day two

Once the sandbox exists, NemoClaw becomes an operations tool, not just an installer. These are the commands that pull their weight.

Task	Command	Why it matters
List sandboxes	`nemoclaw list`	Shows provider, model, and applied presets
Check health	`nemoclaw my-assistant status`	Shows sandbox health and inference state
Stream logs	`nemoclaw my-assistant logs --follow`	Your first stop for failed blueprint runs and runtime errors
Watch blocked egress	`openshell term`	Lets you review and approve unknown network requests
Add a preset	`nemoclaw my-assistant policy-add pypi --yes`	Permanent access for a known integration
Remove a preset	`nemoclaw my-assistant policy-remove pypi --yes`	Roll back access when you no longer need it
Pause a channel	`nemoclaw my-assistant channels stop telegram`	Keeps credentials but disables the bridge
Re-enable a channel	`nemoclaw my-assistant channels start telegram`	Brings a paused bridge back without re-entering tokens
Install a skill	`nemoclaw my-assistant skill install ./my-skill/`	Pushes a skill into the running sandbox
Create a snapshot	`nemoclaw my-assistant snapshot create --name before-upgrade`	Fast insurance before risky changes
Restore a snapshot	`nemoclaw my-assistant snapshot restore before-upgrade`	Rewind state cleanly
Rebuild safely	`nemoclaw my-assistant rebuild --yes`	Upgrade while preserving workspace state

Changing network access after onboarding

This is where NemoClaw is noticeably better than ad hoc agent setups. Instead of loosening all controls after the first block, you can keep a constrained baseline and then approve or persist only what is necessary.

For one-off blocked destinations, use the TUI:

openshell term

That lets you review host, port, binary, and method or path information when available. Approved requests stay available for the current session, but they do not become permanent baseline policy. That is a feature, not a bug.

For durable changes, add or remove presets:

nemoclaw my-assistant policy-add github --yes
nemoclaw my-assistant policy-remove github --yes

If you need something more specific than a stock preset, define a custom policy entry and keep protocol: rest with method and path restrictions for HTTP APIs whenever possible. L4-only rules are a compromise. Pretending otherwise just makes bad policy look neat.

Switching models without rebuilding your whole life

If you are staying within the same provider family, model changes are simple:

openshell inference set --provider openai-api --model <model>

Then verify the result:

nemoclaw my-assistant status

If you are switching across provider families, the story gets more opinionated. You are not just flipping a runtime pointer. You are changing the route and some baked image configuration. In practice, that means you should treat the change like a real reconfiguration and rerun onboarding or recreate the sandbox with the appropriate overrides.

Cost is another practical reason to keep this workflow clean. Public pricing pages in April 2026 show large spreads between model tiers, such as GPT-5.4 mini at low single-digit dollars per million output tokens versus premium frontier tiers that cost an order of magnitude more. Anthropic pricing similarly ranges from Haiku class to Opus class, and the wider pricing shift is covered in Claude, OpenClaw, and the End of Flat Pricing for Agents. If you need a practical playbook for spending less under these conditions, see token optimization strategies for LLM cost control, because being able to switch models without policy chaos is an operational advantage, not just a convenience.

Understanding what persists and what does not

The useful state lives in the workspace under /sandbox/.openclaw/workspace/. That includes files such as AGENTS.md, IDENTITY.md, MEMORY.md, SOUL.md, USER.md, and the memory/ directory of daily notes. If you are designing longer-lived assistants, the AI Systems Memory hub and agent memory provider comparison are useful next reads.

The good news is that sandbox restarts preserve this state. The bad news is that nemoclaw destroy does not care about your feelings. Destroying the sandbox deletes its persistent volume and your workspace goes with it.

That is why the rebuild and snapshot flows matter. NemoClaw is usable precisely because it does not force you to choose between upgrading and losing the assistant's memory.

The rule everyone learns late

Here is the rule that saves the most time once you internalize it. A surprising amount of NemoClaw configuration is build-time or image-time configuration, not live mutable state.

That explains several behaviours that confuse new users:

messaging channels are baked into the image and host-side commands rebuild the sandbox when channels change
the OpenClaw config path inside the sandbox is read-only
some auth, proxy, and port settings require re-onboarding or sandbox recreation
editing the right host-side state is usually the correct move, while editing from inside the sandbox is usually the wrong one

Once you accept that model, the platform stops feeling random and starts feeling deliberate.

NemoClaw troubleshooting that actually saves time

Install and platform problems

Problem	What is really happening	What to do
`nemoclaw` not found after install	Your shell has not refreshed its PATH	Run `source ~/.bashrc` or `source ~/.zshrc`, or open a new terminal
Docker permission denied on Linux	Your user is not in the `docker` group	Run `sudo usermod -aG docker $USER` then `newgrp docker`
Docker is not running	The installer or onboarding cannot reach the runtime	Start Docker and rerun `nemoclaw onboard`
Colima socket not detected on macOS	Colima is not running or the socket path is missing	Run `colima status` and start Colima if needed
Unsupported platform error	You are outside the tested matrix	Move to a tested Docker-based runtime before wasting more time

Runtime and policy problems

If the agent cannot reach an external host, the first answer is usually not that the provider is broken. The first answer is usually that the destination is not allowed by policy yet, especially on new sandboxes.

Open the TUI:

openshell term

If the request is legitimate, approve it for the session or add the correct preset or custom policy entry permanently.

If onboarding fails because port 18789 is taken, find and kill the conflict:

sudo lsof -i :18789
kill <PID>

If an older release left an orphaned SSH forward behind after a destroy, current NemoClaw versions can clean that up automatically during re-onboard. Older ones may need the manual kill.

If the dashboard does not load after setting NEMOCLAW_DASHBOARD_PORT, rerun onboarding on a current release with the desired port. Older builds had a bug where the host respected the custom port but the sandbox still listened on the default one.

Memory, rebuilds, and channels

If sandbox creation dies with exit code 137, you probably hit an out-of-memory condition during the image push path. Add swap or use a machine with more RAM. The cheap machine was not actually cheap if it cost you a day.

Before risky changes, snapshot first:

nemoclaw my-assistant snapshot create --name before-upgrade

If you need to upgrade the sandbox but keep the assistant state, rebuild instead of destroy:

nemoclaw my-assistant rebuild --yes

If you rotate Telegram, Discord, or Slack tokens, rerun onboarding so NemoClaw can detect the credential change and recreate the sandbox correctly.

And if you try to fix channels from inside the sandbox with openclaw channels commands, stop. Channel config is baked into the image and the config path is read-only. Use the host-side commands instead:

nemoclaw my-assistant channels add telegram
nemoclaw my-assistant channels remove telegram
nemoclaw my-assistant channels stop telegram
nemoclaw my-assistant channels start telegram

Inference and local model pain

Compatible endpoints are the classic source of false confidence. Just because a server exposes an OpenAI-looking API does not mean it supports the streaming behaviour OpenClaw expects.

If onboarding succeeded but runtime calls fail on a compatible endpoint, rerun onboarding and let NemoClaw re-probe the endpoint. Do not assume a config override alone is enough.

For local backends, keep an eye on health and binding issues:

nemoclaw my-assistant status

If local inference health checks look wrong on older releases, IPv6 versus IPv4 resolution may be the culprit. If Ollama behaves badly in WSL, make sure Docker Desktop integration is working and consider increasing OLLAMA_CONTEXT_LENGTH before restarting ollama serve.

If all else fails, collect diagnostics instead of guessing:

nemoclaw debug --sandbox my-assistant --output ./nemoclaw-debug.tar.gz

That is a much better bug report than a screenshot of a half-visible terminal.

Should you use NemoClaw in 2026

NemoClaw is opinionated in the right places. It assumes that an always-on agent should begin inside a cage, that inference credentials should stay on the host, and that network access should be earned rather than assumed. For this class of tooling, that philosophy is still the right default.

It is also still alpha. That means rough edges are real, the runtime model takes time to learn, and the issue you hit may genuinely be a product issue rather than operator error. If you are honest about that constraint, the stack is usable today for serious evaluation and controlled internal workloads.

My recommendation is simple. Use NemoClaw if you care about secure-by-default agent operations, want a clearer separation between assistant and enforcement layer, and are willing to operate within a deliberate lifecycle. If you only want the fastest possible demo, there are simpler toys, but if you want a safer long-running stack, NemoClaw is one of the most convincing options available right now. Once you are stable, the practical follow-on is OpenClaw production setup patterns with plugins and skills, which maps day-to-day operating models. At that stage, add formal monitoring with LLM inference observability using Prometheus and Grafana so operations do not depend on terminal intuition alone.

Agent Memory Providers Compared — Honcho, Mem0, Hindsight, and Five More

Rost — Thu, 30 Apr 2026 10:15:44 +0000

Modern assistants still forget everything when you close the tab unless something persists beyond the context window. Agent memory providers are services or libraries that hold facts and summaries across sessions — often wired in as plugins so the framework stays thin while memory scales.

This guide compares eight backends that ship as Hermes Agent external memory plugins — Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, and Supermemory — and explains how they fit into broader AI systems stacks. The same vendors appear in OpenClaw and other agent tooling via community or official integrations. The AI Systems Memory hub lists this article alongside Cognee and related guides.

For Hermes-specific bounded core memory (MEMORY.md and USER.md), freezing behaviour, and triggers, see Hermes Agent Memory System.

Hermes Agent lists eight external memory provider plugins for persistent, cross-session knowledge. Only one external provider can be active at a time. Built-in MEMORY.md and USER.md stay loaded alongside it — additive, not replacement.

External dependencies. Every external provider except Holographic requires at least one external service call — an LLM for memory extraction, an embedding model for semantic search, or a database like PostgreSQL for storage. These dependencies have direct implications for privacy, cost, and whether your memory stack can run fully self-hosted. Hindsight and ByteRover bundle or eliminate the most dependencies; Honcho, Mem0, and Supermemory require the most moving parts. Where a provider supports Ollama or any OpenAI-compatible endpoint, you can route LLM and embedding calls to a local model and keep data off third-party servers entirely.

Activation with Hermes Agent

hermes memory setup   # Interactive picker + configuration
hermes memory status  # Check what's active
hermes memory off     # Disable external provider

Or manually in ~/.hermes/config.yaml:

memory:
  provider: openviking  # or honcho, mem0, hindsight, holographic, retaindb, byterover, supermemory

Provider Comparison

Provider	Storage	Cost	External Dependencies	Self-hostable	Unique Feature
Honcho	Cloud/Self-hosted	Paid/Free	LLM + Embedding model + PostgreSQL/pgvector + Redis	Yes — Docker / K3s / Fly.io	Dialectic user modeling + session-scoped context
OpenViking	Self-hosted	Free	LLM (VLM) + Embedding model	Yes — local server; Ollama-native init wizard	Filesystem hierarchy + tiered loading
Mem0	Cloud/Self-hosted	Paid/Free OSS	LLM + Embedding model + Vector store (Qdrant or pgvector)	Yes — Docker Compose OSS; fully local possible	Server-side LLM extraction
Hindsight	Cloud/Local	Free/Paid	LLM + bundled PostgreSQL + built-in embedder + built-in reranker	Yes — Docker or embedded Python; fully local with Ollama	Knowledge graph + `reflect` synthesis
Holographic	Local	Free	None	Native — no infra required	HRR algebra + trust scoring
RetainDB	Cloud	$20/mo	Cloud-managed (LLM + retrieval on RetainDB servers)	No	Delta compression
ByteRover	Local/Cloud	Free/Paid	LLM only — no embedding model, no DB	Yes — local-first by default; Ollama supported	File-based context tree; no embedding pipeline
Supermemory	Cloud	Paid	LLM + PostgreSQL/pgvector (enterprise Cloudflare deploy)	Enterprise plan only	Context fencing + session graph ingest

Detailed Breakdown

Honcho

Best for: multi-agent systems, cross-session context, user-agent alignment.

Honcho runs alongside existing memory — USER.md stays as-is, and Honcho adds an additional layer of context. It models conversations as peers exchanging messages — one user peer plus one AI peer per Hermes profile, all sharing a workspace.

External dependencies: Honcho requires an LLM for session summarisation, user-representation derivation, and dialectic reasoning; an embedding model for semantic search across observations; PostgreSQL with the pgvector extension for vector storage; and Redis for caching. The managed cloud at api.honcho.dev handles all of this for you. For self-hosted deployments (Docker, K3s, or Fly.io), you supply your own credentials. The LLM slot accepts any OpenAI-compatible endpoint, including Ollama and vLLM, so inference can stay on-premises. The embedding slot defaults to openai/text-embedding-3-small but supports configurable providers via LLM_EMBEDDING_API_KEY and LLM_EMBEDDING_BASE_URL — any OpenAI-compatible embedding server works, including local options like vLLM with a BGE model.

Tools: honcho_profile (read/update peer card), honcho_search (semantic search), honcho_context (session context — summary, representation, card, messages), honcho_reasoning (LLM-synthesized), honcho_conclude (create/delete conclusions).

Key config knobs:

contextCadence (default 1): Minimum turns between base layer refresh
dialecticCadence (default 2): Minimum turns between peer.chat() LLM calls (1-5 recommended)
dialecticDepth (default 1): .chat() passes per invocation (clamped 1-3)
recallMode (default 'hybrid'): hybrid (auto+tools), context (inject only), tools (tools only)
writeFrequency (default 'async'): Flush timing: async, turn, session, or integer N
observationMode (default 'directional'): directional (all on) or unified (shared pool)

Architecture: Two-layer context injection — base layer (session summary + representation + peer card) + dialectic supplement (LLM reasoning). Automatically selects cold-start vs warm prompts.

Multi-peer mapping: Workspace is a shared environment across profiles. User peer (peerName) is a global human identity. AI peer (aiPeer) is one per Hermes profile (hermes default, hermes.<profile> for others).

Setup:

hermes memory setup  # select "honcho"
# or legacy: hermes honcho setup

Config: $HERMES_HOME/honcho.json (profile-local) or ~/.honcho/config.json (global).

Profile management:

hermes profile create coder --clone  # Creates hermes.coder with shared workspace
hermes honcho sync                   # Backfills AI peers for existing profiles

OpenViking

Best for: self-hosted knowledge management with structured browsing.

OpenViking provides a filesystem hierarchy with tiered loading. It's free, self-hosted, and gives you full control over your memory storage.

External dependencies: OpenViking requires a VLM (vision-language model) for semantic processing and memory extraction, and an embedding model for vector search — both are mandatory. Supported VLM providers include OpenAI, Anthropic, DeepSeek, Gemini, Moonshot, and vLLM (for local deployment). For embeddings, supported providers include OpenAI, Volcengine (Doubao), Jina, Voyage, and — via Ollama — any locally served embedding model. The openviking-server init interactive wizard can detect available RAM and recommend suitable Ollama models (e.g. Qwen3-Embedding 8B for embeddings, Gemma 4 27B for VLM) and configure everything automatically for a fully local, zero-API-key setup. No external database is required; OpenViking stores memory in the filesystem.

Tools: viking_search, viking_read (tiered), viking_browse, viking_remember, viking_add_resource.

Setup:

pip install openviking
openviking-server init   # interactive wizard (recommends Ollama models for local setup)
openviking-server
hermes memory setup  # select "openviking"
echo "OPENVIKING_ENDPOINT=http://localhost:1933" >> ~/.hermes/.env

Mem0

Best for: hands-off memory management with auto extraction.

Mem0 handles memory extraction server-side via an LLM call on every add operation — it reads the conversation, extracts discrete facts, deduplicates, and stores them. The managed cloud API handles all infrastructure. The open-source library and self-hosted server give you full control.

External dependencies: Mem0 requires an LLM for memory extraction (default: OpenAI gpt-4.1-nano; 20 providers supported, including Ollama, vLLM, and LM Studio for local models) and an embedding model for retrieval (default: OpenAI text-embedding-3-small; 10 providers supported, including Ollama and HuggingFace for local models). Storage uses Qdrant at /tmp/qdrant in library mode, or PostgreSQL with pgvector in self-hosted server mode — both can run locally. A fully local, zero-cloud Mem0 stack is achievable: Ollama for LLM, Ollama for embeddings, and a local Qdrant instance, all configured via Memory.from_config.

Tools: mem0_profile, mem0_search, mem0_conclude.

Setup:

pip install mem0ai
hermes memory setup  # select "mem0"
echo "MEM0_API_KEY=your-key" >> ~/.hermes/.env

Config: $HERMES_HOME/mem0.json (user_id: hermes-user, agent_id: hermes).

Hindsight

Best for: knowledge graph-based recall with entity relationships.

Hindsight builds a knowledge graph of your memory, extracting entities and relationships. Its unique reflect tool performs cross-memory synthesis — combining multiple memories into new insights. Recall runs four retrieval strategies in parallel (semantic, keyword/BM25, graph traversal, temporal), then merges and re-orders results using reciprocal rank fusion.

External dependencies: Hindsight requires an LLM for fact and entity extraction on retain calls, and for synthesis on reflect calls (default: OpenAI; supported providers include Anthropic, Gemini, Groq, Ollama, LM Studio, and any OpenAI-compatible endpoint). The embedding model and cross-encoder reranking model are bundled inside Hindsight itself — they run locally within the hindsight-all package and require no external API. PostgreSQL is also bundled with the embedded Python installation via a managed pg0 data directory; you can alternatively point Hindsight at an external PostgreSQL instance. For a fully local, zero-cloud setup, set HINDSIGHT_API_LLM_PROVIDER=ollama and point it at a local Ollama model — retain and recall work fully; reflect requires a tool-calling-capable model (e.g. qwen3:8b).

Tools: hindsight_retain, hindsight_recall, hindsight_reflect (unique cross-memory synthesis).

Setup:

hermes memory setup  # select "hindsight"
echo "HINDSIGHT_API_KEY=your-key" >> ~/.hermes/.env

Auto-installs hindsight-client (cloud) or hindsight-all (local). Requires >= 0.4.22.

Config: $HERMES_HOME/hindsight/config.json

mode: cloud or local
recall_budget: low / mid / high
memory_mode: hybrid / context / tools
auto_retain / auto_recall: true (default)

Local UI: hindsight-embed -p hermes ui start

Holographic

Best for: privacy-focused setups with local-only storage.

Holographic uses HRR (Holographic Reduced Representation) algebra for memory encoding, with trust scoring for memory reliability. No cloud dependency — everything runs locally on your own hardware.

External dependencies: None. Holographic requires no LLM, no embedding model, no database, and no network connection. Memory encoding is done entirely through HRR algebra running in-process. This makes it unique among all eight providers — it is the only one that operates with zero external calls. The trade-off is that retrieval quality is lower than embedding-based semantic search, and there is no cross-memory synthesis like Hindsight's reflect. For users where privacy and zero-dependency operation are non-negotiable, Holographic is the only option that delivers that unconditionally.

Tools: 2 tools for memory operations via HRR algebra.

Setup:

hermes memory setup  # select "holographic"

RetainDB

Best for: high-frequency updates with delta compression.

RetainDB uses delta compression to efficiently store memory updates and hybrid retrieval (vector + BM25 + reranking) to surface relevant context. It's cloud-based with a $20/month cost, with all memory processing handled server-side.

External dependencies: RetainDB's LLM calls, embedding pipeline, and reranking all run on RetainDB's own cloud infrastructure — you supply only a RETAINDB_KEY. Memory extraction uses Claude Sonnet server-side. There is no self-hosting option and no local mode. All conversation data is sent to RetainDB servers for processing and storage. If data sovereignty or offline operation matters for your use case, this provider is not suitable.

Tools: retaindb_profile (user profile), retaindb_search (semantic search), retaindb_context (task-relevant context), retaindb_remember (store with type + importance), retaindb_forget (delete memories).

Setup:

hermes memory setup  # select "retaindb"

ByteRover

Best for: local-first memory with human-readable, auditable storage.

ByteRover stores memory as a structured markdown context tree — a hierarchy of domain, topic, and subtopic files — rather than embedding vectors or a database. An LLM reads source content, reasons about it, and places extracted knowledge into the right location in the hierarchy. Retrieval is MiniSearch full-text search with tiered fallback to LLM-powered search, with no vector database required.

External dependencies: ByteRover requires an LLM for memory curation and search (18 providers supported, including Anthropic, OpenAI, Google, Ollama, and any OpenAI-compatible endpoint via the openai-compatible provider slot). It requires no embedding model and no database — the context tree is a local directory of plain markdown files. Cloud sync is optional and used only for team collaboration; everything works fully offline by default. For a fully self-contained local setup, connect Ollama as the provider (brv providers connect openai-compatible --base-url http://localhost:11434/v1) and no data leaves your machine.

Tools: 3 tools for memory operations.

Setup:

hermes memory setup  # select "byterover"

Supermemory

Best for: enterprise workflows with context fencing and session graph ingest.

Supermemory provides context fencing (isolating memory by context) and session graph ingest (importing entire conversation histories). It automatically extracts memories, builds user profiles, and runs hybrid retrieval combining semantic and keyword search. The managed cloud API is the primary deployment target.

External dependencies: Supermemory's cloud service handles all LLM inference and embedding server-side — you supply only a Supermemory API key. Self-hosting is available exclusively as an enterprise plan add-on and deploys to Cloudflare Workers; it requires you to provide PostgreSQL with the pgvector extension (for vector storage) and an OpenAI API key (mandatory, with Anthropic and Gemini as optional additions). There is no Docker-based or local self-hosting path — the architecture is tightly coupled to Cloudflare Workers edge compute. For users who need full data sovereignty without an enterprise contract, this provider is not the right choice.

Tools: 4 tools for memory operations.

Setup:

hermes memory setup  # select "supermemory"

How to Choose

Need multi-agent support? Honcho
Want self-hosted and free? OpenViking or Holographic
Want zero-config? Mem0
Want knowledge graphs? Hindsight
Want delta compression? RetainDB
Want bandwidth efficiency? ByteRover
Want enterprise features? Supermemory
Want privacy (local only)? Holographic
Want fully local with zero external services? Holographic (no dependencies at all) or Hindsight/Mem0/ByteRover with Ollama
Want human-readable, auditable memory with no embedding pipeline? ByteRover

For full profile-by-profile provider configurations and real-world workflow patterns, see Hermes Agent production setup.

Related guides

AI Systems Memory hub — scope of this subcluster and links to Cognee guides
Hermes Agent Memory System — core two-file memory before plugins
Hermes Agent production setup — profile wiring for providers in practice

Hermes Agent Memory System: How Persistent AI Memory Actually Works

Rost — Thu, 30 Apr 2026 10:15:40 +0000

You know the drill. You open a chat with an AI agent, explain your project, share your preferences, get some work done, and close the tab. Come back the following week and it's like talking to a stranger — all context gone, every preference forgotten, the project re-explained from scratch.

This isn't a bug. It's how Large Language Models work by design. They're stateless: each request is independent, each response generated from whatever prompt you send right now, with no memory, no history, and no continuity beyond the tokens in the current context window.

For single-turn interactions, that's fine. Ask a question, get an answer, move on. But for agents — systems that are supposed to do things across sessions, learn from mistakes, and evolve with you — statelessness is a hard architectural limit. It's one of the central unsolved problems in self-hosted AI systems.

The industry has tried to solve this. LangChain added memory modules. OpenAI introduced assistants with threads. Frameworks like Letta, Zep, and Cognee built entire architectures around persistent memory. Databricks published on "memory scaling" — the idea that agent performance improves with accumulated experience. Dedicated benchmark papers, episodic memory surveys, and a rapidly growing ecosystem of tools have all emerged since 2024 to address what is increasingly recognised as one of the central unsolved problems in agentic AI.

Most of these approaches share a common problem: they treat memory as an afterthought — a database you query, a context window you stuff, a retrieval system that adds latency and noise rather than clarity.

Hermes Agent takes a fundamentally different approach. Memory isn't something the agent retrieves when needed. It's something the agent is at all times — built into the system prompt, curated, bounded, and always active. It's small enough to be fast, structured enough to be useful, and disciplined enough to know what to forget.

This article explains exactly how that works.

Part 1: The AI Agent Memory Problem

Why "Just Add Context" Doesn't Scale for Agents

The obvious solution to stateless AI is to add context. Attach the previous conversation. Include the project documentation. Send the entire history.

For a while, that works. You've got a 128K context window. You can fit a lot of text in there.

But context isn't memory — there's a real and important difference between them. Context is everything you're shown right now; memory is what you actively keep and carry forward.

Context has no curation. It's a dump: as it grows, the model has to process thousands of tokens of irrelevant history to find the one fact it needs. That costs tokens and money, compounds latency, and eventually hits the ceiling.

Memory is curated. It's the distillation of experience into something compact and actionable. It doesn't grow indefinitely — it consolidates, updates, and forgets.

Human memory works the same way. You don't remember every conversation you've ever had. You remember the parts that matter: who you're talking to, what they care about, what you've agreed on, what you've learned. The rest is either forgotten or searchable when you need it.

The Research Landscape

The AI agent memory space has exploded since 2024, with dedicated benchmark suites, a growing research literature, and a measurable performance gap between different architectural approaches. Here's where things stand.

Letta (formerly MemGPT) was one of the earliest frameworks to treat persistent memory as a first-class concern, reaching 21.7K GitHub stars. It uses an OS-inspired three-tier model: core memory (small, always in context), recall memory (searchable conversation history), and archival memory (long-term cold storage). The insight that not all memory is equal was correct. The implementation, however, requires agents to run entirely inside the Letta runtime — adopting it means adopting the whole platform, not just a memory layer.

Zep / Graphiti focuses on conversational memory with temporal entity tracking — facts carry validity windows so the graph knows when something was true. It's strong for chatbots that need relationship graphs, less suited for autonomous agents tracking environment facts and project conventions.

Cognee is built for knowledge extraction from documents and structured data, with 30+ ingestion connectors and a knowledge graph backend. It excels at institutional knowledge and RAG pipelines but is less focused on personal agent memory. See self-hosting Cognee with local LLMs for a practical setup guide.

Hindsight does knowledge graph-based recall with entity relationships and a unique reflect synthesis tool that performs cross-memory synthesis — combining multiple memories into new insights. It's among the top performers on agent memory benchmarks and is available as a memory provider for Hermes Agent.

Mem0 handles memory extraction server-side via LLM analysis, requiring minimal configuration. The Mem0 research paper, published at ECAI 2025 (arXiv:2504.19413), benchmarked ten distinct approaches to AI memory and validated the selective extraction approach — storing discrete facts, deduplicating, and retrieving only what's relevant. Mem0 has grown to approximately 48K GitHub stars and supports 21 framework integrations. The trade-off is cloud dependency and cost.

Databricks' memory scaling research introduced the concept that agent performance improves with accumulated experience. Their architecture holds system prompts, enterprise assets, and episodic/semantic memories scoped at organization and user level, validating the idea that memory quality matters as much as model capability.

The common thread across most frameworks is that they treat memory as a retrieval problem: store it somewhere, query it when needed, inject it into context. Hermes does the opposite — memory isn't retrieved on demand, it's injected at session start and always present. Always active, always available, curated enough to stay useful.

Part 2: Architecture — Two Files, One Brain

Hermes Agent's built-in memory system lives in two files.

~/.hermes/memories/MEMORY.md — Agent's personal notes (2,200 chars, ~800 tokens)
~/.hermes/memories/USER.md — User profile (1,375 chars, ~500 tokens)

That's the entire persistent memory surface: two files, under 3,600 characters total, fewer than 1,300 tokens. It looks deliberately small because it is — and that's exactly the design intent.

MEMORY.md: The Agent's Notes

This is where the agent stores everything it learns about its environment, the project, tools, conventions, and lessons learned. Here's what it looks like:

User's project is a Go microservice at ~/code/gateway using gRPC + PostgreSQL
This machine runs Ubuntu 22.04, has Docker and kubectl installed
User prefers snake_case for variable names and avoids camelCase

These aren't logs. They're facts. Dense, declarative, information-packed. No timestamps, no fluff, no "on January 5th the user asked me to..."

USER.md: The User Profile

This is where the agent stores everything it knows about you.

User is a full-stack developer comfortable with TypeScript, Go, and Python.
User prefers snake_case for variable names and avoids camelCase.
User primarily uses Linux Ubuntu 22.04.
User deploys to AWS using Terraform.

Identity, role, preferences, technical skills, communication style, pet peeves. The stuff that makes the agent respond differently to you than to anyone else.

The Frozen Snapshot Pattern

At session start, both files are loaded from disk and injected as a frozen block into the system prompt. Here's what it looks like:

══════════════════════════════════════════════
MEMORY (your personal notes) [7% — 166/2,200 chars]
══════════════════════════════════════════════
User's project is a Go microservice at ~/code/gateway using gRPC + PostgreSQL
§
This machine runs Ubuntu 22.04, has Docker and kubectl installed
§
User prefers snake_case for variable names and avoids camelCase
§

══════════════════════════════════════════════
USER PROFILE (who the user is) [8% — 110/1,375 chars]
══════════════════════════════════════════════
User is a full-stack developer comfortable with TypeScript, Go, and Python.
§
User prefers snake_case for variable names and avoids camelCase.
§

The format uses headers, usage percentages, character counts, and § (section sign) delimiters. Entries can be multiline. It's designed to be parseable by the model while remaining human-readable.

Why frozen? Prefix caching. The system prompt is the same across every turn in a session. By keeping memory static after session start, the model can cache the prefix computation and only process the variable parts — the conversation. This is a significant performance optimization. You're not re-computing attention over the same memory tokens on every turn.

Changes made during a session persist to disk immediately, but they only appear in the system prompt at the next session start. Tool responses always show the live state, but the model's "mind" doesn't change mid-session. This prevents the model from chasing its own tail — updating memory and then reacting to its own update in the same conversation.

Character Limits as a Feature

2,200 characters. 1,375 characters. These aren't arbitrary limits. They're design constraints that force curation.

Unlimited memory is a liability. It encourages dumping everything in, never consolidating, and eventually becoming noise. Bounded memory forces the agent to be selective. What's actually important? What will I need again? What can be compressed without losing meaning?

When memory is full, the agent doesn't just fail silently. It gets an error with current entries and usage, then follows a workflow:

Read current entries from error response
Identify removable or consolidatable entries
Use replace to merge related entries into shorter versions
Add the new entry

This is how memory stays useful. It's not a database. It's a curated collection of facts that matter.

Security: Prompt Injection Scanning

Every memory entry is scanned before acceptance. The system blocks prompt injection attempts, credential exfiltration, SSH backdoors, and invisible Unicode characters.

Memory is also deduplicated. Exact duplicate entries are rejected automatically. This prevents adversaries from trying to inject malicious content through repeated submissions.

Part 3: When Memory Fires — Triggers & Decisions

The most common question about Hermes Agent's memory is when it actually saves something.

The answer is: constantly, but selectively. The agent manages its own memory via the memory tool, and the decision to save is driven by a combination of explicit signals and implicit patterns.

Writing Triggers: When Does the Agent Decide to Save?

The agent saves memory proactively. It doesn't wait for you to ask. Here's what triggers it.

User corrections. When you correct the agent, that's a signal to remember. "Don't do that again." "Use this instead." "Remember this." These are explicit instructions to update memory.

Example: you ask the agent to configure a Python environment. It suggests pip. You say "I use poetry for everything." The agent saves: User prefers using the 'poetry' package manager for all Python projects.

Discovered preferences. The agent observes patterns and infers preferences. If you consistently use a certain tool, framework, or workflow, it gets saved.

Example: after seeing you use poetry multiple times across different projects, the agent saves it as a preference.

Environment facts. Things about the machine, the project, the tools installed. These are discovered through exploration and saved as facts.

Example: the agent checks what's installed and saves: This machine runs Ubuntu 22.04, has Docker and kubectl installed.

Project conventions. How the project is structured, what tools it uses, what patterns it follows. These are discovered through code inspection and saved.

Example: User's project is a Go microservice at ~/code/gateway using gRPC + PostgreSQL.

Completed complex workflows. After completing a task that took 5+ tool calls, the agent considers saving the approach as a skill or at least noting what worked.

Tool quirks and workarounds. When the agent discovers something non-obvious about a tool, API, or system — a limitation, a workaround, a convention — it saves it.

What gets skipped:

Trivial or obvious information
Things easily re-discovered
Raw data dumps
Session-specific ephemera
Information already in context files (SOUL.md, AGENTS.md)

Reading Triggers: When Does the Agent Recall?

Memory isn't retrieved — it's always there. But there are different levels of access.

Session start (automatic). MEMORY.md and USER.md are injected into the system prompt. The agent has them from the first token. No query needed, no latency, no tool call. This is the core memory — always active.

session_search (on-demand). When the agent needs to find something from past conversations that isn't in core memory, it uses the session_search tool. This queries SQLite (~/.hermes/state.db) with FTS5 full-text search and Gemini Flash summarization.

Example: you ask "Did we discuss Docker networking last week?" The agent searches session history and returns a summary of the relevant conversation.

External provider tools (when configured). When an external memory provider is active, the agent has additional tools available: honcho_search, hindsight_recall, mem0_search, etc. These are used when the agent determines that external context is needed.

The Decision Tree

Here's how the agent weighs "is this worth remembering?":

Is this a correction or explicit instruction?
  YES → Save to memory
  NO → Is this a preference or pattern?
    YES → Save to user profile
    NO → Is this an environment fact or convention?
      YES → Save to memory
      NO → Is this easily re-discovered?
        YES → Skip
        NO → Is this session-specific?
          YES → Skip
          NO → Save to memory

The agent doesn't overthink this. It saves proactively, consolidates when full, and trusts the character limits to keep things tight.

Part 4: Internal Memory vs. External Knowledge Bases

This is where confusion often happens. Hermes Agent has internal memory (MEMORY.md, USER.md, external providers) and external knowledge bases (LLM Wiki, Obsidian, Notion, ArXiv, filesystem), and they serve completely different roles. This is similar to the distinction between retrieval-augmented generation pipelines and agent working memory — external retrieval is good for deep knowledge lookups, not for carrying identity and preferences. Internal memory is the agent's brain — always active, curated, carried into every session. External knowledge bases are its library — vast reference resources consulted on demand.

The Distinction

Internal Memory (the brain):

Small, persistent, injected into system prompt
Contains: user preferences, agent conventions, immediate lessons
Always "in mind" during conversation
Curated, bounded, actively managed
Examples: MEMORY.md, USER.md, Honcho, Hindsight, Mem0

External Knowledge Bases (the library):

Vast, reference-only, accessed on-demand
Contains: documents, papers, code, notes, databases
Accessed via tools when needed
Not "remembered" — looked up
Examples: LLM Wiki, Obsidian, Notion, ArXiv, filesystem, GitHub

How They Relate

The agent accesses external bases via tools when needed. It doesn't "remember" them — it looks them up.

LLM Wiki (llm-wiki): Karpathy's interlinked Markdown knowledge base for building and querying domain knowledge. The agent uses the llm-wiki skill to read, search, and query it. It's a reference resource, not memory.

Obsidian: Personal note vaults with bidirectional links. The agent uses the obsidian skill to read, search, and create notes. Obsidian is part of the broader personal knowledge management ecosystem that Hermes can tap into as a library resource.

Notion/Airtable: Structured databases and wikis accessed via API. The agent queries them when needed.

ArXiv: Academic paper repositories. The agent searches and extracts papers when researching a topic.

Filesystem: Project code, documentation, configurations. The agent reads files when working on a project.

The Distillation Pattern

Here's the key insight: critical insights from external bases can be distilled into internal memory.

Example: the agent reads a paper from ArXiv about memory scaling for AI agents. It doesn't save the entire paper to memory. It saves the key takeaway: Memory scaling: agent performance improves with accumulated experience through user interaction and business context stored in memory.

The external resource is vast. The internal memory is the distillation.

When to Use Which

Internal memory for:

"Who am I helping?"
"What do they prefer?"
"What did we just learn?"
"What's the project setup?"
"What tools are available?"

External knowledge bases for:

"What's the latest research on X?"
"What's in my project's documentation?"
"What did we discuss last month?"
"What's the API for this service?"
"What's the code structure?"

The agent understands the difference and uses each appropriately — it doesn't conflate looking up a document with recalling something it has learned about you and your environment.

Part 5: How It Actually Works

Let's look at the mechanics.

The `memory` Tool

The agent manages memory through a single tool with three actions: add, replace, remove.

There is no read action — memory content is auto-injected into the system prompt. The agent doesn't need to read it because it's always there.

add — Adds a new entry.

memory(action="add", target="memory",
       content="User runs macOS 14 Sonoma, uses Homebrew, has Docker Desktop installed.")

replace — Replaces an existing entry using substring matching.

memory(action="replace", target="memory",
       old_text="dark mode",
       content="User prefers light mode in VS Code, dark mode in terminal")

remove — Removes an entry using substring matching.

memory(action="remove", target="memory",
       old_text="temporary project fact")

Substring Matching

replace and remove use short unique substrings via old_text. You don't need the full entry text. This makes surgical edits possible without knowing the exact content.

If a substring matches multiple entries, an error is returned requesting a more specific match. The agent then refines its query.

Target Stores: `memory` vs `user`

The target parameter determines which file gets updated.

memory — Agent's personal notes. Environment facts, project conventions, tool quirks, lessons learned.
user — User profile. Identity, role, timezone, communication preferences, pet peeves, workflow habits.

Capacity Management

When memory is >80% full, the agent consolidates. It merges related entries, removes outdated facts, and compresses information.

Good memory entries are compact and information-dense:

User runs macOS 14 Sonoma, uses Homebrew, has Docker Desktop installed. Shell: zsh with oh-my-zsh. Editor: Neovim with Telescope plugin.

Bad memory entries are vague or verbose:

User has a project.
On January 5th, 2026, the user asked me to look at their project which is located at ~/code/gateway and it uses Go with gRPC and PostgreSQL for the database layer.

The first is dense and useful. The second is either too vague or too verbose.

Session Search vs Persistent Memory

session_search and persistent memory serve different purposes.

Feature	Persistent Memory	Session Search
Capacity	~1,300 tokens total	Unlimited (all sessions)
Speed	Instant (in system prompt)	Requires search + LLM summarization
Use Case	Key facts always available	Finding specific past conversations
Management	Manually curated by agent	Automatic — all sessions stored
Token Cost	Fixed per session (~1,300 tokens)	On-demand (searched when needed)

Rule of thumb: use memory for critical facts that should always be in context. Use session search for historical lookups.

Part 6: External Memory Providers

Beyond built-in MEMORY.md and USER.md, Hermes Agent can attach one external memory plugin at a time — Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, or Supermemory — for persistent, cross-session knowledge. Only one external provider is active at once; the two core files remain loaded alongside it (additive, not replacement).

Activate and inspect providers with hermes memory setup, hermes memory status, and hermes memory off, or set memory.provider in ~/.hermes/config.yaml.

For the full comparison table, LLM and embedding dependency notes, per-provider breakdowns, and how these backends relate to OpenClaw and other stacks, see Agent memory providers compared.

For profile-specific wiring and production workflows, see Hermes Agent production setup. The AI Systems Memory hub lists this guide plus related Cognee and knowledge-layer articles.

Part 7: The Philosophy

Why Bounded Memory Beats Unlimited Memory

The instinct is to make memory as large as possible. Store everything. Retrieve what you need.

Bounded memory works better. Here's why.

Curation forces quality. When you have limited space, you only save what matters. You compress, consolidate, and prioritize. Unlimited memory encourages dumping everything in and never cleaning up.

Speed matters. 1,300 tokens in the system prompt is fast. 100,000 tokens retrieved from a database is slow. Memory should be instant, not a query.

Noise degrades performance. More memory isn't better memory. It's noisier memory. The model has to distinguish signal from noise, and that takes attention — attention that should be spent on the actual task.

Forgetting is a feature. Human memory forgets. That's not a bug — it's how we prioritize. Agents should forget too. Not everything deserves to be remembered.

The "Forgetting" Problem

Agents need to unlearn. Not just forget, but actively remove outdated information.

Here's how Hermes Agent handles it:

remove action: Delete entries that are no longer relevant.
replace action: Update entries with new information.
Capacity pressure: When memory is full, the agent consolidates and removes old entries.
Security scanning: Blocks malicious or corrupted entries.

Forgetting isn't failure — it's maintenance. An agent that can't unlearn will eventually carry as much noise as signal.

Memory Scaling

Databricks introduced the concept of "memory scaling": does an agent with thousands of users perform better than one with a single user?

Their research suggests yes, but with caveats. Memory scaling requires:

Quality extraction: Not all interactions are worth remembering. The agent must extract insights, not logs.
Effective retrieval: Retrieved memories must be relevant. Noise degrades performance.
Generalization: Memories should be patterns, not specifics. "User prefers Python" scales. "User ran command X at timestamp Y" does not.

Hermes Agent's bounded memory naturally supports memory scaling. By forcing curation, it ensures that memories are generalizable, compact, and useful.

What This Means for the Future

Memory is becoming the competitive moat in agentic AI — not the model itself, but what the model carries between sessions. Two agents with identical underlying models can perform very differently: one remembers your preferences, your environment, and your past mistakes; the other starts cold every time.

The question is no longer whether agents should have persistent memory. It's settled: they must. The open question is how to design that memory well — what to keep, what to discard, how to make it instant, and how to prevent it from becoming noise.

Hermes Agent's answer is to keep memory small, curated, and always active — not a database you query, but a working model of the user that the agent carries with it into every conversation.

Conclusion

Hermes Agent's memory system is deliberately simple: two files, firm character limits, no retrieval pipeline, no vector database, and no per-query latency. What sounds like a constraint is the whole point.

It works because it treats memory the way a brain works rather than the way a database does — small, curated, and always active. The agent doesn't retrieve memory when it needs it; the memory is simply always there, woven into the system prompt from the first token of every session.

External memory providers extend this system for users who need more: knowledge graphs, multi-agent support, self-hosted storage, enterprise features. But the core remains the same: bounded, curated, always available.

And external knowledge bases — LLM Wiki, Obsidian, Notion, ArXiv — serve a different role. They're the library, not the brain. The agent looks them up, doesn't remember them. Critical insights get distilled into internal memory; the rest stays in the library.

This is how an AI agent remembers you. Not by storing everything, but by remembering what matters.

Hermes Agent was released by Nous Research in February 2026 and reached over 64,000 GitHub stars by April 2026 (v0.9.0), with 242+ contributors. It is open-source and available at github.com/NousResearch/hermes-agent. For install, configuration, and workflow guides, see the Hermes Agent overview.

OpenClaw Rise and Fall — Timeline and Real Reasons Behind the Collapse

Rost — Tue, 28 Apr 2026 12:50:44 +0000

OpenClaw did not fail as a product. It lost its fuel.

What looks like a dramatic boom and collapse is actually something more mechanical and more interesting. OpenClaw was a thin layer on top of a temporary economic advantage in the AI ecosystem. Once that advantage disappeared, so did the attention.

This article breaks down the exact timeline, the real drivers behind the spike, and why the drop was inevitable.

The illusion of product-driven growth

Most people assume OpenClaw grew because it was a great AI agent — and that is only partially true.

OpenClaw was genuinely useful. It supported more than 50 integrations, worked across Claude, GPT-4o, Gemini, and DeepSeek, and attracted enterprise adoption — Tencent built a platform directly on top of it. But capability alone did not set it apart from comparable alternatives:

Cline
LangChain-based setups
Other agent wrappers

The real driver was access rather than capability — a distinction that explains the entire arc of OpenClaw's rise and collapse.

OpenClaw made powerful models cheap to use at scale.

Phase 1. Quiet emergence (November 2025)

The story begins in November 2025, when Peter Steinberger built the first prototype in roughly one hour. He was annoyed that the tool did not exist yet, so he built it, calling it Clawdbot — a nod to Anthropic's Claude, complete with a lobster mascot.

The first version was practical rather than flashy: an AI agent that could manage calendars, check email, book appointments, and automate computer tasks on the user's behalf. Steinberger shared it in developer communities and early adopters recognized something promising, though growth at this stage remained slow and organic with no visibility outside technical circles.

Phase 2. The viral ignition (January–February 2026)

The spike began when several forces aligned in quick succession.

1. Naming drama and forced rebrands

In late January 2026, Anthropic sent Steinberger a trademark notice over "Clawdbot," citing phonetic similarity to "Claude." By his account, Anthropic handled it professionally — but the notice forced a rename. The project became Moltbot for three days, then OpenClaw, and the forced rebranding generated exactly the kind of attention that marketing budgets cannot buy.

2. The agent hype wave

The market was already primed for an agent breakthrough:

autonomous agents were trending across social media and the tech press
"AI that can act" had become the dominant narrative
developers were actively searching for tools that could automate complex workflows

OpenClaw arrived at exactly the right moment, when demand for this kind of tool was at its highest and the story of autonomous AI agents was capturing mainstream attention.

3. The cheap compute loophole

The most decisive factor was a compute pricing loophole that no amount of good engineering could have manufactured deliberately.

Users discovered that OpenClaw could connect to Claude by grabbing the OAuth token from a Claude Pro or Max subscription and spoofing the authentication headers of Anthropic's own Claude Code client. Instead of paying per token through the API, they effectively got:

near unlimited agent execution for a fixed monthly cost

The numbers made this explosive. A Claude Max subscription cost $200 per month, while running equivalent workloads through the API would cost far more — industry analysts estimated a price gap of more than five times, meaning Anthropic was quietly subsidising each heavy OpenClaw user by hundreds of dollars a month.

This changed behavior instantly:

developers ran heavy experiments they would never have attempted at API prices
viral demos flooded social media
large-scale automation became accessible to solo developers

Nothing in the software changed — the economics did, and that shift alone was enough to ignite a viral adoption curve. By March 2, 2026, the OpenClaw repository had accumulated 247,000 GitHub stars and 47,700 forks, reaching 100,000 stars in under 48 hours — a pace widely described as the fastest-growing GitHub project in history.

Phase 3. Peak usage and inflated expectations

At peak interest, developers pushed agents to extremes, social media amplified the results, and expectations exploded around what personal AI automation could achieve. An estimated 135,000 OpenClaw instances were running simultaneously when Anthropic made its announcement, and one founder described publicly how she had deployed nine separate AI agents to manage her administrative work and personal household logistics.

Why do AI tools suddenly become popular and then fade

Because the initial spike is driven by novelty and perceived leverage. Once users test the limits, reality sets in — the tool proves harder to use reliably, and the economic conditions that made it attractive often turn out to be temporary. In OpenClaw's case, the perceived leverage was real but built on borrowed economics that Anthropic had not priced for agentic workloads.

The creator leaves for OpenAI (February 2026)

Before the collapse arrived, OpenClaw lost its original architect.

On February 14–15, 2026, Steinberger announced he was leaving the project to join OpenAI. Sam Altman posted that Steinberger would "drive the next generation of personal agents" at the company, and Steinberger wrote that "teaming up with OpenAI is the fastest way to bring this to everyone." OpenClaw was transferred to an independent open-source foundation with OpenAI's continued support.

The timing was striking. Anthropic had declined to hire or partner with Steinberger, despite the fact that his tool had become arguably their best free marketing in years — a project built explicitly to showcase how good Claude was. Instead, he went directly to their biggest competitor, taking with him both the project's momentum and its community relationships.

Phase 4. The correction begins

Two things started happening at the same time.

1. Reality of agent limitations

Users who had deployed OpenClaw at scale began encountering its real constraints:

agents are brittle and fail unpredictably on multi-step tasks
reliability is inconsistent across different workflows and environments
setup and maintenance is non-trivial for most users outside technical circles

These limitations alone would have caused a gradual decline, but OpenClaw did not taper off gradually — it dropped sharply, because a second and more decisive force hit at exactly the same time.

2. The economic layer breaks

Anthropic had already run this playbook once. In January 2026, just weeks before OpenClaw peaked, they blocked OpenCode — another popular third-party coding client — from using Claude subscription tokens in what was framed as a terms of service violation, not a capacity issue. OpenClaw users had every reason to expect the same treatment, and that moment arrived in April.

Anthropic then introduced restrictions that closed the loophole entirely:

third-party tools were blocked from using subscription OAuth tokens
usage shifted to pay-as-you-go extra billing or full API keys

This removed the key advantage:

cheap large-scale execution

Now users faced a very different cost structure:

Metric	Before cutoff	After cutoff
Monthly plan cost	$20–$200 (flat)	$20–$200 + usage
Cost per task	Effectively $0	$0.50–$2.00
API rate (Sonnet 4.6 input)	Covered by sub	$3 per million tokens
API rate (Sonnet 4.6 output)	Covered by sub	$15 per million tokens
Increase for heavy users	—	10× to 50×

What caused the sudden drop in interest in AI agent tools

The answer is straightforward: not a lack of innovation, but the loss of affordable compute. Once the pricing floor disappeared, the incentive to experiment and share disappeared with it, and search interest followed almost immediately.

April 4, 2026 — The hard cutoff

On April 4, 2026, at 12 PM Pacific Time, the subscription access ended for all third-party tools.

Boris Cherny, Head of Claude Code at Anthropic, posted on X that Claude Pro and Max subscriptions would no longer cover usage from third-party tools, effective immediately. An Anthropic spokesperson confirmed that using subscriptions with third-party tools was always against the terms of service, and that those tools were placing "an outsized strain on our systems." Additional context made the timing feel urgent: on April 1, the full source code of Claude Code — 512,000 lines of TypeScript — had leaked through an npm package, exposing exactly how Anthropic's first-party tools authenticated with the backend and making it more pressing to lock down third-party tools that were spoofing those same patterns.

Anthropic offered a one-time credit equal to one month's subscription fee and a 30% discount on pre-purchased usage bundles to ease the transition. For light users, the credit covered the adjustment period, but for power users running multiple instances the new numbers simply did not work. The effect on activity was immediate:

experimentation stopped
viral sharing disappeared
search interest collapsed

This matches the sharp drop in Google Trends almost perfectly. The full policy mechanics and migration options after the cutoff are covered in Claude, OpenClaw, and the End of Flat Pricing for Agents.

OpenAI moves in the opposite direction

On the same day as the Anthropic ban, OpenAI publicly confirmed that ChatGPT Plus, Pro, and Team subscribers were entirely free to use their subscriptions to power OpenClaw through OAuth — including with models like GPT-5.3 Codex for complex coding tasks.

This was not accidental timing. By hiring Steinberger and explicitly opening their subscription gates, OpenAI positioned themselves as the developer-friendly alternative at the exact moment Anthropic cut off its most active community, securing the loyalty of the developers who were building the next generation of AI tools.

Phase 5. Where OpenClaw users actually went

Users did not disappear after the ban — they redistributed across a spectrum of alternatives depending on their technical depth and budget.

Direct usage of chat assistants

Many users moved back to direct chat interfaces, trading agent automation for the simplicity and reliability they had given up:

ChatGPT
Claude UI
Gemini

Are AI agents replacing traditional chat assistants

No — for most users, agents add complexity without enough reliability gains. The chat interface remains the default for daily use because it is faster to start, easier to debug when something goes wrong, and requires no infrastructure setup. Agents serve a committed minority of power users, not the general population. The AI developer tools ecosystem has evolved to fill this gap with tools that sit between raw agents and simple chat, giving developers structured assistance without full agentic overhead.

Cheaper model ecosystems

Power users with the technical ability to self-host migrated toward lower-cost alternatives:

Qwen
DeepSeek
other low-cost models accessible through Ollama for fully local setups

Which models are popular for low-cost AI experimentation

Models that offer lower pricing, fewer usage restrictions, and flexible deployment including local self-hosting absorbed the bulk of displaced OpenClaw power users. These ecosystems grew quietly rather than generating public hype, which is why the migration was largely invisible in trend data even as it represented a significant redistribution of compute demand.

Alternative agent frameworks

Developers who still needed agent capabilities switched to leaner approaches:

custom scripts tailored to specific workflows
lightweight frameworks with fewer dependencies
self-hosted solutions combining local models with minimal tooling

The key difference from OpenClaw is that these users optimized for cost and control rather than convenience, and built for sustainability rather than maximum automation at minimum price. This is the pattern common across the self-hosted AI systems ecosystem — provider independence treated as a design requirement, not an afterthought.

The overlooked factor — why cost is the real product

The most important insight from OpenClaw's trajectory is that cost functions as the real product in AI adoption.

Why is cost important in AI adoption

Because usage scales non-linearly with compute costs. When compute is cheap, experimentation explodes, innovation accelerates, and attention grows because viral sharing becomes economically rational. When compute becomes expensive, usage contracts to serious workflows only, casual users leave, and hype disappears almost overnight — which is precisely why token optimization and cost reduction strategies become critical skills once compute stops being subsidized.

OpenClaw demonstrated this rule in an unusually clear form: between February and April 2026, the software did not change, but the economics of running it did — and that single shift was enough to collapse the community in a matter of days.

OpenClaw was never the core story

OpenClaw functioned as a surface layer on top of more fundamental forces.

The real story involved three factors operating simultaneously:

access to Claude models at subscription prices rather than API rates
a five-to-one pricing mismatch between what users paid and what usage actually cost Anthropic
a policy correction that had to happen eventually given the scale of that mismatch

Once those underlying conditions changed, any tool that depended on them would show the same pattern — which is exactly why similar tools spiked and declined in lockstep, regardless of their individual quality or feature sets. Anthropic's decision also revealed something strategic: by blocking third-party clients while protecting Claude Code, the company chose to concentrate developer engagement inside its own first-party tooling at a moment when independent communities were iterating faster than any centralized lab.

The pattern repeats across AI

OpenClaw's trajectory is not unique — the same cycle has played out repeatedly across the AI ecosystem.

The same pattern appears in AutoGPT, BabyAGI, and other early agent frameworks that attracted massive attention and then faded as compute costs, reliability limits, or platform restrictions were enforced. The cycle is consistent:

New capability appears
Cheap or free usage emerges
Viral experimentation begins
Costs or limits are enforced
Attention collapses

Each cycle leaves behind a smaller, more committed user base and a clearer understanding of what actually works at scale — which is how progress compounds even through the boom-and-bust pattern.

OpenClaw vs Hermes Agent — what the trend data shows

The chart above compares worldwide Google Trends search interest for OpenClaw AI (blue) and Hermes Agent (red) over the past three months. OpenClaw peaked at an index of 100 in mid-March 2026 and collapsed sharply in April after the subscription cutoff. Hermes Agent barely registered during OpenClaw's peak, then gradually picked up interest as OpenClaw faded — reaching an index of around 40 in bursts through April, compared to OpenClaw's average of 49 and Hermes's average of 8.

Hermes Agent is an open-source framework built by Nous Research and released in February 2026. Unlike OpenClaw, which is optimized for broad reactive tool use across many integrations, Hermes is built around a learning loop: it generates reusable skills from successful task completions, refines them through continued use, and maintains a persistent model of the user across sessions. The result is an agent that improves the more it is used on the same task types, rather than approaching each job from the same baseline. It reached 95,600 GitHub stars in its first seven weeks.

The gap in the chart is significant. OpenClaw's hype surplus did not transfer to Hermes — it evaporated. Casual experimenters who had been running agents cheaply on Claude subscriptions simply left the space rather than migrating to an alternative. The users who did move to Hermes were the committed technical minority who needed persistent, self-hosted automation and were willing to set it up properly — which is exactly the kind of smaller, more sustainable user base that remains after every AI hype cycle collapses. For those users, Hermes production setup patterns are worth exploring.

Final takeaway — follow the economics, not the interface

OpenClaw did not rise because it was revolutionary — it rose because it unlocked something temporarily underpriced, and it fell not because it failed as a product but because that pricing advantage was removed by the platform it depended on.

This was not a product lifecycle. It was a pricing event.

Understanding this distinction is critical for predicting the next spike in AI tooling. The same pattern will repeat whenever a new compute subsidy appears, whether through a subscription loophole, a generous free tier, or a new open-weight model that undercuts established pricing. Track where compute is temporarily cheap and you will find the next wave of viral AI tools before the hype arrives.

Llama-Server Router Mode - Dynamic Model Switching Without Restarts

Rost — Mon, 27 Apr 2026 11:42:46 +0000

For a long time, llama.cpp had a glaring limitation:

you could only serve one model per process, and switching meant a restart.

That era is over.

Recent updates introduced router mode in llama-server, bringing something much closer to what people expect from modern local LLM runtimes:

dynamic model loading
unloading on demand
switching per request
no process restart

In other words: Ollama-like behavior, but without the training wheels.

If you are still deciding between local runtimes, cloud APIs, and self-hosted infrastructure, the
LLM hosting overview is a good starting point.

Prerequisites

Router mode requires a recent llama-server build — roughly post mid-2024. Older builds do not have the --models flag.

For install options (package manager, pre-built binaries, or full source build with CUDA), see the
llama.cpp quickstart.

Once you have llama-server, confirm your build supports router mode:

llama-server --help | grep -i models

If the --models flag appears, you are good. If it is absent, update to a newer build.

My current output of models-related help:

-cl,   --cache-list                     show list of models in cache
                                        Prefix/Suffix/Middle) as some models prefer this. (default: disabled)
                                        models with dynamic resolution (default: read from model)
                                        models with dynamic resolution (default: read from model)
                                        embedding models (default: disabled)
--models-dir PATH                       directory containing models for the router server (default: disabled)
                                        (env: LLAMA_ARG_MODELS_DIR)
--models-preset PATH                    path to INI file containing model presets for the router server
                                        (env: LLAMA_ARG_MODELS_PRESET)
--models-max N                          for router server, maximum number of models to load simultaneously
                                        (env: LLAMA_ARG_MODELS_MAX)
--models-autoload, --no-models-autoload
                                        for router server, whether to automatically load models (default:
                                        (env: LLAMA_ARG_MODELS_AUTOLOAD)

What router mode actually does

Router mode turns llama-server into a model dispatcher.

Instead of binding to a single model via -m, the server:

starts with no model loaded
receives a request that names a model
loads that model if it is not already in memory
runs inference
optionally unloads the model after the response, or keeps it warm for the next request

The key idea

You are no longer running:

./llama-server -m model.gguf

You are running:

./llama-server --models models.ini --port 8080

And letting the server decide what to load and when, based on what the client actually requests.

This matters because it means one persistent process can serve an entire fleet of models, with clients selecting the right one per task — a coding model, a chat model, a summarisation model — without any coordination overhead on your side.

Configuration: defining your models

This is where things are still a bit raw.

There is no fully stable official format yet, but current builds support INI-style model definitions via a config file.

Example models.ini

[llama3]
model = /opt/models/llama-3-8b-instruct.Q5_K_M.gguf
ctx-size = 8192
ngl = 35
threads = 8

[mistral]
model = /opt/models/mistral-7b-instruct-v0.3.Q4_K_M.gguf
ctx-size = 4096
ngl = 20
threads = 8

[qwen]
model = /opt/models/qwen2.5-coder-7b-instruct.Q5_K_M.gguf
ctx-size = 16384
ngl = 35
threads = 8

Each section name becomes the model identifier that clients use in the "model" field of their API requests.

Key config parameters

Parameter	What it controls
`model`	Absolute path to the GGUF file
`ctx-size`	Context window size in tokens. Larger values use more VRAM.
`ngl`	Number of GPU layers offloaded. Set to `0` for CPU-only; increase until you hit VRAM limits.
`threads`	CPU threads for the layers that remain on CPU.

Choosing the right ngl value depends on your GPU's available VRAM — for GPU selection and hardware economics, the compute hardware guide is a useful reference. To watch live VRAM consumption while dialing it in, see the GPU monitoring tools for Linux.

Starting the server with config

./llama-server --models /opt/llama.cpp/models.ini --port 8080

Confirm the server started correctly:

curl http://localhost:8080/v1/models | jq '.data[].id'

You should see each section name from your models.ini listed as a model ID.

A note on stability

The INI config interface is still evolving:

flags may change between commits
some parameters are only recognised by specific build configurations
documentation lags behind implementation

Pin to a specific llama.cpp commit if you need reproducibility across restarts.

API usage: switching models on request

Once the server is running, model switching happens through the standard OpenAI-compatible API. You simply set the "model" field.

List registered models

curl http://localhost:8080/v1/models

Completion request — first model

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [
      {"role": "user", "content": "Explain router mode in one paragraph"}
    ]
  }'

Switch to a different model — same endpoint, same port

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "messages": [
      {"role": "user", "content": "Write a Python function that reads a CSV file"}
    ]
  }'

The server handles the unload/load cycle transparently. Your client code does not change — only the model field.

Python example

If you are using openai Python client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

# Use the coding model
response = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "Write a Go HTTP handler"}],
)
print(response.choices[0].message.content)

# Switch to the chat model — same client, different model name
response = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "What is the capital of Australia?"}],
)
print(response.choices[0].message.content)

What happens internally

When a request arrives for qwen and llama3 is currently loaded:

llama3 is unloaded from VRAM
qwen weights are read from disk and loaded into VRAM
inference runs
the next request determines whether to keep qwen loaded or swap again

This directly answers the common question:

How can a local LLM server switch models without restarting

By dynamically loading models per request, not binding at startup.

Systemd service: production-ready setup

Create a dedicated user and directories

sudo useradd --system --shell /usr/sbin/nologin --home-dir /opt/llama.cpp llm
sudo mkdir -p /opt/llama.cpp/models
sudo chown -R llm:llm /opt/llama.cpp

Copy your binary and model config into place:

sudo cp build/bin/llama-server /opt/llama.cpp/
sudo cp models.ini /opt/llama.cpp/

/etc/systemd/system/llama-server.service

[Unit]
Description=Llama.cpp Router Server
After=network.target

[Service]
Type=simple
User=llm
WorkingDirectory=/opt/llama.cpp
ExecStart=/opt/llama.cpp/llama-server --models /opt/llama.cpp/models.ini --port 8080
Restart=always
RestartSec=5

Environment=LLAMA_LOG_LEVEL=info

[Install]
WantedBy=multi-user.target

Enable and start

sudo systemctl daemon-reload
sudo systemctl enable llama-server
sudo systemctl start llama-server

Verify and inspect logs

sudo systemctl status llama-server

journalctl -u llama-server -f

On a successful start you will see lines indicating the server is listening and the model registry has been loaded. A quick sanity check:

curl -s http://localhost:8080/v1/models | jq '.data[].id'

Now you have a persistent service with auto-restart and centralised model switching — no manual process management required. If you want to apply the same pattern to other binaries, hosting any executable as a Linux service walks through the general approach.

The llama-server --metrics flag exposes a Prometheus-compatible endpoint. For llama.cpp-specific dashboards, PromQL queries, and alerting rules, see the LLM inference monitoring guide. For the broader observability setup, the observability guide covers the full stack.

Limitations you need to understand

Router mode is genuinely useful, but it comes with tradeoffs you should be clear about before relying on it in production.

Only one model in memory at a time

Even though multiple models are defined in models.ini, only one is resident in VRAM per worker at any given moment. Switching means a full unload-and-reload cycle.

switching means reload
latency spike is unavoidable
on a typical 7B model at Q5, a reload can take 3–10 seconds depending on disk speed and VRAM bandwidth

This answers another key question:

Does llama.cpp support serving multiple models at once

Not really. It supports multiple definitions, not simultaneous residency. If you need two models genuinely loaded in parallel, you need two processes on two separate GPUs.

For measured VRAM consumption and tokens-per-second across model sizes, the LLM performance benchmarks cover the full picture. For numbers specific to llama.cpp on a 16 GB GPU — dense and MoE models at multiple context sizes — see the 16 GB VRAM llama.cpp benchmarks.

No smart caching

Unlike Ollama, which maintains a warm pool and evicts models based on recency:

there is no automatic model eviction strategy
no background pre-warming
no priority queue for frequently used models

If you send alternating requests for llama3 and mistral, every single request triggers a reload. This is the fundamental cost of being closer to the metal.

Latency is unpredictable for mixed workloads

A well-behaved workload that uses one model consistently will be fast. A workload that interleaves multiple models will be slow. Plan your client routing logic accordingly — group requests by model where possible.

Config is not stable

The INI support exists and works in most recent builds, but it is not fully standardised. Flags and parameter names have changed across versions. If you upgrade llama-server, test your models.ini against the new build before deploying.

Llama.cpp vs Ollama: honest comparison

Feature	llama.cpp router	Ollama
Dynamic loading	Yes	Yes
Model switching	Yes	Yes
Built-in registry	Partial (INI)	Yes (pull-based)
Memory management	Basic	Advanced
Model eviction	None	TTL-based
UX polish	Low	High
OpenAI API compat	Yes	Yes
Control	Maximum	Opinionated
Config stability	Experimental	Stable

Opinionated take

Choose llama.cpp router mode when you want:

maximum control over runtime parameters per model
minimal process overhead
direct access to llama.cpp flags without abstraction layers
a hackable base for custom tooling

Choose Ollama when you want:

a stable, polished experience
automatic model downloading and versioning
smart keep-alive and eviction without configuration
batteries included from day one

Neither is wrong. The choice depends on how much you want to manage yourself.

If you go with Ollama, the Ollama CLI cheatsheet covers day-to-day commands. For a broader comparison that also includes vLLM, LM Studio, and LocalAI, see how different local runtimes compare in 2026.

Llama.cpp vs llama-swap

llama-swap is an external orchestrator that sits in front of one or more llama-server instances:

it intercepts requests and inspects the model field
it starts the appropriate llama-server process for that model
it shuts down idle instances after a configurable timeout
it proxies the request through once the model is ready

For a hands-on setup, see the llama-swap quickstart.

Key difference

Aspect	router mode	llama-swap
Built-in	Yes	No (separate binary)
Maturity	Experimental	More stable
Flexibility	Limited	High
Control layer	Internal	External proxy
Per-model config	INI file	YAML file
Process model	Single process	One process per model

When to use llama-swap

llama-swap gives you process-level isolation per model, which means a crash in one model instance does not affect others. It also lets each model run with completely independent llama-server flags.

Use it if you need:

better lifecycle control and isolation
smarter switching logic with configurable idle timeouts
more predictable latency (each model has a warm process after first load)
production stability today, not eventually

When native router mode is enough

Use the built-in router if you want:

zero external dependencies
a single process to manage
simpler deployment (one binary, one config file)
minimal stack for dev or single-user setups

Final thoughts

Router mode is a meaningful step forward for llama-server.

It answers the long-standing demand:

What is router mode in llama.cpp server

It is the missing layer that turns a static binary into a dynamic inference service — one process that can field requests for a whole catalogue of models.

But it is not finished.

Today it is:

powerful enough for real workloads
promising as a foundation for more sophisticated routing
slightly rough around the config and stability edges

If your workload is predictable and you can group requests by model, router mode works well today. If you need production-grade reliability and per-model isolation, reach for llama-swap while the native implementation matures.

Either way, you get Ollama-like behavior, without hiding the machinery.

Claude Skills and SKILL.md for Developers: VS Code, JetBrains, Cursor

Rost — Sat, 25 Apr 2026 08:10:03 +0000

Most teams misuse Claude Skills in one of two ways. They either turn SKILL.md into a dumping ground, or they never graduate from giant copy-pasted prompts.

Both approaches are sloppy. If you want Skills to work in a real dev workflow, you need to treat them like code and operations logic, not like prompt poetry.

Claude Skills are directories anchored by SKILL.md, with optional scripts, references, and assets. They work because of progressive disclosure. The agent starts by loading only compact metadata such as the skill name and description, then reads the full instructions only when the task matches. That lets an agent keep many skills available without bloating every session from the start.

Anthropic's own guidance makes the intended division of labour pretty clear. CLAUDE.md is for durable, always-on project context. Skills are for reusable knowledge, playbooks, and invocable workflows that should load on demand. Claude Code even folded old custom commands into the same mechanism, so legacy .claude/commands/*.md files still work, but Skills are now the better long-term shape — and the most reusable building block in any AI-powered development workflow.

When to Use Claude Skills: CLAUDE.md vs Skills vs Hooks

A Claude Skill is worth creating when you keep pasting the same checklist, the same deployment playbook, the same code review rubric, or the same internal API gotchas into chat. Anthropic explicitly recommends creating a skill when you keep reusing the same procedure, or when a section of CLAUDE.md has grown into a process rather than a fact. That is the practical answer to the FAQ question "What is a Claude Skill and when should you use one". Use a Skill for repeatable procedure, not for general taste or broad repo rules.

The real win is control over context cost and behaviour. A good Skill is loaded only when relevant, while a bloated CLAUDE.md is loaded every session. Anthropic recommends keeping CLAUDE.md short and moving domain knowledge or procedures into Skills precisely because on-demand loading keeps the agent focused on the task in front of it.

My opinionated rule is simple. If the instruction should apply every single session, it belongs in CLAUDE.md. If the instruction is a reusable method, checklist, or workflow that matters only sometimes, it belongs in a Skill. If the action must happen automatically on every matching event, it probably belongs in a hook, not a Skill. Anthropic's feature overview frames those tools in almost exactly that layering model.

Layer	Tool	When to use
`CLAUDE.md`	Always loaded	Project facts, durable conventions, repo-wide rules
Skill	Loaded on demand	Repeatable procedures, playbooks, domain checklists
Hook	Event-triggered	Automatic side effects on file save, commit, or session start

A practical smell for each: if you find yourself pasting the same instructions into every chat, that is a Skill. If a CLAUDE.md section has grown into a step-by-step process, extract it into a Skill. If you want something to fire silently every time a file is saved, write a hook instead.

Claude Skills IDE Support: VS Code, JetBrains, Cursor, and Codex

Claude Code runs across CLI, Desktop, VS Code, JetBrains, web, and mobile-related remote control flows. Anthropic describes the CLI as the most complete local surface, while the IDE integrations trade some CLI-only capabilities for editor-native review, file context, and tighter workflow ergonomics. Configuration, project memory, and MCP servers are shared across the local surfaces, so your .claude setup follows you rather than being trapped in one editor.

For VS Code, Anthropic says the extension is the recommended interface inside the editor. It provides plan review, inline diffs, file mention support, and integrated access to the CLI. The same install flow also exposes a direct path for Cursor. For JetBrains, the current supported list includes IntelliJ IDEA, PyCharm, Android Studio, WebStorm, PhpStorm, and GoLand, with diff viewing, selection sharing, file-reference shortcuts, and diagnostic sharing built into the plugin.

JetBrains support is better than many developers realise. If you run claude from the IDE's integrated terminal, the integration features are active automatically. If you start from an external terminal, Anthropic documents the /ide command to connect Claude Code back to the JetBrains session, and it explicitly recommends launching from the same project root so Claude sees the same files your IDE sees. If you use auto-edit modes in JetBrains, Anthropic also warns that IDE configuration files can become part of the editable surface, so manual approvals are the safer default in that environment.

Now the bigger point. Claude Skills are not only a Claude Code thing. Agent Skills is an open standard. The official Agent Skills quickstart says the same skill can work in VS Code with GitHub Copilot, Claude Code, and OpenAI Codex, and OpenAI's own Codex docs say Skills are available in the Codex CLI, IDE extension, and app. The Agent Skills implementation guide adds an important portability detail: .agents/skills has emerged as the cross-client convention, while some clients also scan .claude/skills for pragmatic compatibility.

So here is the practical compatibility rule I recommend. If you are building for Claude Code first and only, author in .claude/skills. If you genuinely want cross-client portability, target the open Agent Skills shape and use .agents/skills as the canonical path. Do not pretend those two goals are identical. They are related, not identical.

Quick compatibility reference:

Client	Skills path	Notes
Claude Code CLI	`.claude/skills/` or `~/.claude/skills/`	Most complete surface; full `allowed-tools` support
VS Code + Claude extension	`.claude/skills/`	Inline diffs, plan review, file mention
Cursor	`.claude/skills/`	Same install path as VS Code
JetBrains (IDEA, PyCharm, etc.)	`.claude/skills/`	Run `claude` from IDE terminal or use `/ide` to reconnect
GitHub Copilot, OpenAI Codex	`.agents/skills/`	Open Agent Skills standard; cross-client portability
Claude.ai web	Upload via UI	Dir name must match `name` field; 200-char description cap

SKILL.md File Structure, Folder Layout, and Storage Locations

A proper Skill is a folder, not a random markdown file sitting at repo root. The core specification requires a directory with a SKILL.md file and allows optional scripts/, references/, and assets/ directories. SKILL.md must contain YAML frontmatter followed by markdown instructions. In the spec, name and description are required, name is limited to 64 characters using lowercase letters, numbers, and hyphens, compatibility is only for real environment requirements, and allowed-tools is explicitly experimental across implementations.

Claude Code is a bit looser than the portable spec because it can derive a name from the directory and fall back to the first paragraph when description is missing. You should not rely on that if you care about portability or predictability. Claude.ai requires the directory name to match the name field, and its custom-skill upload path caps descriptions at 200 characters even though the broader spec allows much more. The portable choice is to set an explicit name, keep the directory identical, and write a precise description that fits in tight limits. That answers the FAQ topic "What should a SKILL.md file contain" without hand-waving.

Start from a structure this boring:

repo/
  .claude/
    skills/
      review-pr/
        SKILL.md
        scripts/
          review.sh
        references/
          checklist.md
        assets/
          comment-template.md

If portability across Skills-compatible clients matters more than Claude Code convenience, keep the same internal shape and swap .claude/skills/ for .agents/skills/. The folder structure is the same idea either way.

For Claude Code, the storage locations are straightforward. Project skills live at .claude/skills/<skill-name>/SKILL.md. Personal skills live at ~/.claude/skills/<skill-name>/SKILL.md. Plugin-distributed skills live under <plugin>/skills/<skill-name>/SKILL.md. Anthropic documents precedence across the built-in scopes as enterprise over personal over project, while plugin skills avoid collisions by using a namespaced form such as plugin-name:skill-name. On Windows, ~/.claude resolves to %USERPROFILE%\.claude, and CLAUDE_CONFIG_DIR can relocate the whole base directory.

The choice between project and personal scope is straightforward. Use .claude/skills/ inside the repo when the Skill is tightly coupled to that codebase — for example, a deploy playbook that knows your specific cluster names or a review rubric tuned to your team's conventions. Use ~/.claude/skills/ for Skills that travel with you across projects: personal checklists, generic changelog generators, preferred debugging workflows. Anything you would put in a dotfiles repo belongs in personal scope.

A few sharp edges are worth memorising. SKILL.md must be named exactly with that casing. Anthropic's PDF guide recommends kebab-case folder names and explicitly says not to place a README.md inside the skill folder, because the operative documentation should live in SKILL.md or references/. That same guide also stresses that SKILL.md naming is case-sensitive. These are boring constraints, but boring constraints are what make tooling reliable.

Claude Code also does the right thing for monorepos. It automatically discovers nested .claude/skills/ directories when you work inside subdirectories, which is ideal for package-level or service-level skills. It also watches existing skill directories for live changes during the current session. The one restart trap is creating a top-level skills directory that did not exist when the session started. Anthropic documents that as the case where you do need to restart so the new directory can be watched.

Claude Skills Best Practices: Descriptions, Scripts, and Scope

The fastest way to create a useless Skill is to ask an LLM to invent one from generic training knowledge. Anthropic's best-practices guide warns against exactly that. The valuable bits are the domain-specific corrections, edge cases, tool choices, and conventions the model would not reliably invent on its own. The right workflow is to solve the task once with the agent, correct it until it works, then extract the method into a Skill.

Scope the Skill like a good function, not like a wiki. Anthropic says Skills should encapsulate a coherent unit of work. Too narrow, and you force multiple skills to stack for one task. Too broad, and the agent cannot activate them precisely. The best-practices guide is blunt that overly comprehensive skills can hurt more than they help because the model chases irrelevant instructions and loses the signal.

Description quality is not a cosmetic concern. It is the routing layer. Both Anthropic and the Agent Skills docs say the description field is the primary mechanism the model uses to decide whether to load a Skill at all. Good descriptions say what the Skill does, when to use it, and the trigger phrases or file types a user would actually mention. Bad descriptions are vague, overly technical, or broad enough to match nonsense. That is the real answer to the FAQ question "Why is a Claude Skill not triggering". Usually the router is bad, not the model.

The contrast is clear side by side:

Bad descriptions — too vague to route reliably:

Helps with code review — matches everything, disambiguates nothing
Useful for development tasks — broader than a search query
Assists with writing — not a router, just a category label

Good descriptions — specific trigger language:

Review pull requests for security issues, migration risk, and missing tests. Use when reviewing a PR, git diff, or release critical change.
Generate a changelog from git log output. Use when preparing a release, writing release notes, or summarising commits since last tag.
Scaffold a new Go HTTP handler with request validation and error middleware. Use when adding a new endpoint or route to a Go service.

The pattern is the same each time: state what the Skill does, name the exact user phrases that should activate it, and optionally name file types or tools that are relevant. If your description would match a generic Google query, it is not specific enough.

If a workflow has side effects, make it manual. Claude Code exposes that directly. disable-model-invocation: true makes a Skill user-invoked only, which Anthropic recommends for actions like deploys, commits, or outbound messages. user-invocable: false goes the other way and hides the Skill from the slash menu while still letting Claude use it as background knowledge. That answers the FAQ topic "When should a skill be manual instead of automatic" in one sentence: manual for risk, automatic for safe repeatable guidance.

Keep SKILL.md small enough to stay intelligible. Anthropic recommends keeping it under 500 lines and around 5,000 tokens, then moving detailed material into references/ or similar files with explicit loading instructions. "Read references/api-errors.md if the API returns a non-200" is a good pattern. "See references/" is lazy. Claude Code also injects the rendered Skill into the conversation as a message and does not keep re-reading the file on later turns. After context compaction, only recent Skill content is carried forward within token budgets. Huge Skills are therefore not merely ugly. They are brittle over long sessions.

A good SKILL.md can stay very plain:

---
name: review-pr
description: Review pull requests for security issues, migration risk, and missing tests. Use when reviewing a PR, git diff, or release critical change.
compatibility: Designed for Claude Code. Requires git and gh.
disable-model-invocation: true
allowed-tools: Bash(git diff *) Bash(gh pr diff *) Read Grep Glob
---
# Review PR

Read references/checklist.md before running any commands.

1. Collect the diff and changed files.
2. Flag correctness, security, and test coverage issues.
3. Return findings grouped by severity with file references.
4. Suggest the smallest safe fix first.

Use scripts when determinism matters more than eloquence. The Skills scripts guide is excellent here. It says agent-facing scripts must avoid interactive prompts, document usage through --help, emit helpful error messages, prefer structured output such as JSON or CSV on stdout, send diagnostics to stderr, and support retry-safe use. It also recommends pinning one-off tool versions and describing runtime requirements explicitly in SKILL.md or the compatibility field rather than assuming the environment has the right packages.

A minimal but correct agent-facing script looks like this:

#!/usr/bin/env bash
# scripts/collect-diff.sh — called by review-pr skill
# Usage: collect-diff.sh <base-ref> [<head-ref>]
set -euo pipefail

BASE="${1:?Usage: collect-diff.sh <base-ref> [<head-ref>]}"
HEAD="${2:-HEAD}"

# Structured output to stdout so the agent can parse it
git diff "${BASE}...${HEAD}" --stat --name-only \
  | jq -Rs '{
      "changed_files": split("\n") | map(select(length > 0))
    }' \
  || { printf '{"error":"git diff failed"}\n' >&2; exit 1; }

Three things make this agent-safe. set -euo pipefail ensures the script exits loudly on any failure rather than silently proceeding. JSON on stdout gives the agent a format it can parse without guessing. Diagnostics go to stderr so the agent's stdout stream stays clean. None of this is clever. All of it is necessary.

One subtle trap is allowed-tools. In the spec it is experimental and support varies. In Claude Code it pre-approves specific tools while the Skill is active, but it does not restrict the universe of callable tools, and deny rules still belong in Claude Code permissions. In the Claude Agent SDK, Anthropic explicitly says the allowed-tools frontmatter in SKILL.md does not apply, so SDK apps must enforce tool access in the main allowed_tools or allowedTools configuration instead. If you ignore that difference, your Skill will behave differently in the CLI and in SDK-powered automation.

One more advanced pattern is worth stealing. When a workflow would flood your main thread with logs, file searches, or long research output, Claude Code lets a Skill run in a forked subagent using context: fork and an agent such as Explore. Anthropic shows this for research workflows, where the heavy lifting happens in isolated context and the main conversation gets the summary. For deep codebase exploration, that is a much better design than a giant inline Skill that pollutes the main session.

A forked Skill looks like this in frontmatter:

---
name: explore-codebase
description: Deep exploration of an unfamiliar codebase. Use when onboarding to a new repo, auditing architecture, or mapping module dependencies.
context: fork
agent: Explore
compatibility: Requires Claude Code CLI.
---
# Explore Codebase

1. Walk the directory tree and summarise the top-level modules.
2. Identify the main entry points and their responsibilities.
3. Map the dependency graph between packages.
4. Return a structured summary to the main session — not the raw file list.

The key line is context: fork. Without it, the exploration output lands inline in your conversation. With it, the subagent runs in its own context window and hands back a summary. The difference matters on large repos where exploration alone can consume thousands of tokens.

Testing Claude Skills: Triggers, Correctness, and Baseline Comparisons

A Skill is not tested because one happy-path demo worked once. Anthropic's guide breaks testing into three layers: manual testing in Claude.ai, scripted testing in Claude Code, and programmatic testing via the Skills API. The recommended evaluation areas are triggering, functional correctness, and performance against a baseline without the Skill. That is also the best answer to the FAQ question "How do you test whether a skill is reliable". You test route selection, output quality, and efficiency, not just whether the model sounded confident.

The official eval guidance gives a clean structure for test cases. Each case should include a realistic user prompt, a human-readable description of the expected output, and optional input files. The docs store those in evals/evals.json inside the Skill directory, which is a sensible convention even if you roll your own harness.

Use a fixture file and a no-nonsense eval layout like this:

{
  "skill_name": "review-pr",
  "evals": [
    {
      "id": 1,
      "prompt": "Review this PR for security issues and missing tests",
      "expected_output": "Findings grouped by severity with file references and at least one test recommendation.",
      "files": ["evals/files/pr-diff.patch"]
    },
    {
      "id": 2,
      "prompt": "Summarise last week's commits",
      "expected_output": "The skill should not activate.",
      "files": []
    }
  ]
}

My own testing rule is harsher than most teams use, but it lines up with the official guidance. Every serious Skill should have should-trigger queries, should-not-trigger queries, at least one edge-case test, and a baseline comparison without the Skill. Anthropic's examples compare tool calls, failed API calls, clarification loops, and token use with and without the Skill because "works" is not the same as "improves the workflow".

If you test through the Claude Agent SDK, remember the plumbing. Skills are filesystem artefacts there, not programmatic registrations. Anthropic says you must enable the "Skill" tool and load the relevant filesystem settings through settingSources or setting_sources. If you omit user or project, or point cwd at the wrong place, the SDK will not discover the Skill. Anthropic even recommends asking "What Skills are available?" as a direct discovery check.

Also test on the model and client you actually intend to ship. The open Agent Skills quickstart explicitly warns that tool-use reliability varies across models, and some models may answer directly instead of executing the command the Skill intends. That is not always a Skill design problem. Sometimes it is a model-selection problem, and your test matrix should expose it.

Claude Skills Troubleshooting: Common Failures and Fixes

When a Skill misbehaves, assume packaging before intelligence. The most common failures are still the boring ones.

If the Skill is not found at all, verify the file is named exactly SKILL.md, with the right case, inside the correct directory. Anthropic's troubleshooting guide calls out filename case explicitly, and its Claude Code and SDK docs point you straight at .claude/skills/*/SKILL.md and ~/.claude/skills/*/SKILL.md as the first checks.
If frontmatter is invalid, check the YAML delimiters and quotes first. Anthropic's examples show the classic mistakes: missing ---, unclosed quotes, or invalid names with spaces and capitals. Skill names should be lowercase and hyphenated.
If the Skill exists but does not trigger, the description is usually too vague. Claude Code's own troubleshooting says to include keywords users would naturally say, verify the Skill appears when you ask "What skills are available?", and try rephrasing closer to the description. Anthropic's PDF guide adds a great diagnostic trick: ask Claude when it would use the Skill and listen to how it paraphrases the description back to you.
If the Skill triggers too often, narrow the scope. Anthropic recommends making the description more specific, adding negative triggers, and using disable-model-invocation: true for workflows you want only by explicit command. Over-triggering is usually just under-specified routing language.
If the Skill seems to lose influence in long sessions, remember that descriptions can be shortened in the Claude Code catalogue when many skills are present, and invoked Skills are later carried within token budgets after compaction. Anthropic recommends front-loading keywords in the description, trimming excess text, and, for Claude Code specifically, adjusting SLASH_COMMAND_TOOL_CHAR_BUDGET if description listings are being squeezed too aggressively.
If a bundled script hangs or behaves erratically, check whether it expects interactive input. The scripts guide says agents run in non-interactive shells, so TTY prompts, password dialogs, and confirmation menus are design bugs. Accept input through flags, environment variables, or stdin and make failures explicit.
If the SDK does not see your Skill, confirm that allowed_tools includes "Skill", that settingSources or setting_sources contains user and or project, and that cwd points at the directory that actually contains .claude/skills/. Without that setup, the Skill system is not enabled no matter how correct your markdown looks.
If an MCP-backed Skill loads but the tool calls fail, Anthropic's troubleshooting checklist is sensible: verify the MCP server is connected, confirm authentication and scopes, test the MCP tool directly without the Skill, then check the exact tool names because they are case-sensitive.

The boring truth is that good Claude Skills look like good operational engineering. Clear names. Small files. Explicit triggers. Deterministic scripts where needed. Real tests. If your Skill reads like a crisp runbook, the agent has a fighting chance. If it reads like a brainstorm, you have simply hidden chaos in a folder.

DEV Community: Rost

Idempotency in Distributed Systems That Actually Works

Where duplicates come from in production

The API contract I actually trust

How do idempotency keys prevent duplicate API requests

Why PUT is not enough

How long should an idempotency key be stored

The database patterns that make idempotency real

Messages, events, and webhooks need their own boundary

How do consumers handle duplicate events and messages

Pair dedup with the outbox pattern

Webhooks are just messages with better branding

Sagas and workflow engines still need idempotency

How to test idempotency before production

Opinionated rules that save real systems

Useful Links

Hermes Voice Control from Your Phone

How the Pipeline Works

What Voice Control Is Great For

Voice Input: Pick an STT Provider

Faster Whisper Model Selection

Voice Output: TTS Providers

Platform Workflows and Practical Differences

Telegram

Discord

Signal

WhatsApp

Mobile App Permissions

Speaking Patterns That Work Reliably

Example Prompts You Can Speak

Common Use Cases with Concrete Outcomes

Troubleshooting

The Free Stack

FAQ Topics in Practice

Quick Start Recap

Kanban in Hermes Agent for Self Hosted LLM Workflows

How Hermes Kanban and the dispatcher work

The practical limitation today

Investigation findings and root cause

Strategy 1 - Encode dependencies for strictly sequential flows

Strategy 2 - Use Linux cron with a running-aware dispatch cap

Sub-minute cadence with one cron entry

Internal Hermes Cron

Hermes Kanban Monitoring and Tuning

When Kanban is the right tool

Hermes Agent Skill Authoring — SKILL.md Structure and Best Practices

Skill or tool?

Procedures versus curated memory

Anatomy of a skill directory

SKILL.md frontmatter that survives review

Progressive disclosure in practice

Conditional activation and prompt hygiene

Secrets, config, and credential files

Supporting scripts and dependencies

Publishing, hub installs, and trust

Troubleshooting and optimization

Hermes Agent CLI cheat sheet — commands, flags, and slash shortcuts

Install Hermes Agent and first-run CLI commands

Global flags for every hermes invocation

hermes chat, one-shot prompts, and hermes -z

Model picker, credential pools, and fallback providers

Config files and hermes config commands

Session management and hermes profile

Skills hub, toolsets, shell hooks, and plugins

Built-in memory and hermes memory providers

Messaging gateway, DM pairing, and platforms

Cron scheduler, webhooks, and Kanban

hermes doctor, logs, backup, and usage insights

MCP, ACP, web dashboard, and OpenClaw migration

Slash commands in the Hermes CLI session

Session flow, background tasks, and goals

Models, tool toggles, skills, and reload

Usage, help, and quitting

More useful reading

Official Hermes Agent documentation

MinIO CE in 2026: Retired Upstream, Source-Only, and What to Use

What happened to MinIO CE

Is MinIO still open source in 2026

Is MinIO CE safe for new production deployments

Why the risk profile changed

Global flags for every `hermes` invocation

`hermes chat`, one-shot prompts, and `hermes -z`

Config files and `hermes config` commands

Session management and `hermes profile`

Built-in memory and `hermes memory` providers

`hermes doctor`, logs, backup, and usage insights