DEV Community: Fred the Fox 🦊

Can Your Hermes Agent Recover From Failure?

Fred the Fox 🦊 — Mon, 28 Sep 2026 11:17:42 +0000

For the drill, I keep the old VPS powered off. I begin on a replacement host with an empty shell and bring across only the backup ZIP. I treat persistent memory as recovered only after that archive rebuilds the required state.

I run the fresh-VPS test with the original host unavailable. The restored agent has to recover its memory, skills, credentials, schedules, and tool access. If you are evaluating Hermes on a virtual server, treat the server as replaceable from day one. Hermes remains self-managed software, which puts the recovery plan in the operator's hands.

Define Recovery Before Touching the VPS

The recovery target includes several kinds of state:

Configuration and provider settings
Memory, sessions, skills, and local agent data
Authentication material and tool credentials
Cron jobs and their recent execution state
Workspace files and any application data stored elsewhere

Before starting, I write down two limits. Recovery point objective, or RPO, is the amount of recent state the business can afford to lose. Recovery time objective, or RTO, is how long the agent may remain unavailable. “We have a backup” offers no testable target for either one.

Run the Fresh-VPS Drill

1. Record the Known-Good Environment

In the runbook, I record the Hermes release, operating system, Python environment, enabled skills, model provider, and external services. Hermes publishes versioned code and notes in its GitHub releases. Keeping the release in the runbook separates restoration problems from upgrade problems.

2. Create a Full Backup

For the full archive, I run:

hermes backup

The official CLI reference says a full backup packages configuration, skills, sessions, and data. The archive includes credentials. I encrypt it, restrict access, and store it outside the server it protects.

Hermes also offers a quick backup for configuration and core state. I use the full form for this drill. The documented exclusions include prior backups and the checkpoint directory, so workspace recovery needs a separate inventory.

3. Provision a Clean Host

On a new VPS with no copied home directory or hidden state, I install Hermes using the official installation instructions:

curl -fsSL <https://hermes-agent.nousresearch.com/install.sh> | bash

Through restoration and validation, I keep the old gateway stopped. This reduces the chance of scheduled jobs or external actions running twice.

4. Import the Archive

Move the encrypted archive through your approved secure channel, decrypt it on the new host, and run:

hermes import /path/to/backup.zip

The CLI documentation advises stopping the gateway before import. I start it after the restore completes and file permissions look correct.

5. Test Recovered Behavior

First, I run hermes doctor to check the installation and configuration. For scheduled work, I use [hermes cron doctor](https://hermes-agent.nousresearch.com/docs/user-guide/features/cron), then inspect recent execution records with hermes cron runs [job-id] --limit 20.

My canary task exercises memory retrieval, one representative skill, and a low-risk tool call. I also test denied actions. A restore that broadens permissions is a security regression. Expected secrets should work, provider-revoked secrets should fail, and a log scan should find no secret values.

6. Measure the Gaps

When the canary passes, I stop the clock. Process launch time is an intermediate marker. I compare the elapsed time with the RTO and measure missing state against the RPO. I add each manual discovery to the runbook or turn it into an automated check.

Keep Checkpoints in Their Lane

Hermes checkpoints and rollback can protect workspace changes during agent activity. They are opt-in, and I keep the off-host backup as a separate control. Restoring local files leaves an email, API mutation, or transaction in another system untouched.

Reliable agent operations need backups for local state, idempotency or approval controls for side effects, and separate recovery plans for external data.

I repeat the fresh-VPS test after material configuration changes and before I need it. A timed restore with a passing canary turns the backup plan into evidence.

You Can't Outsource Everything

Fred the Fox 🦊 — Thu, 03 Sep 2026 00:23:02 +0000

Building next-level tech requires strategic decisions. Whether it's AI or whatnot. What do you own and what do you delegate? Some things make sense when they are handled from a place of convenient capacity. Infrastructurally, decentralization has always been an advantage.

OriginTrail - $TRAC is what happens when someone decides the data layer is too important to leave in someone else's hands. A decentralized knowledge graph that organizes verified information for AI systems: supply chain records, scientific data and asset provenance owned by the network, rather than by a platform that can change its API pricing on a Tuesday afternoon.

Fluence - $FLT sits in the middle of this conversation in an interesting way. It's not asking anyone to abandon the cloud entirely. It's asking what happens when the compute layer is coordinated by a protocol instead of a corporation. Guess. It's the same output but different relationships. The key advantage here is ownership.

Nillion - $NIL takes it further. Blind computation. This means processing sensitive data without ever exposing it, even to the infrastructure running the calculation. For AI systems handling medical records, financial data, or anything that can't leave a jurisdiction, this isn't a nice-to-have. It's the only way the application can legally exist.

Maybe the tickers were unnecessary but a lot of great products now thrive on tokenization. This remains though: there are some things you can rent and the foundation isn't one of them.

Holding or Selling in 2026?

Fred the Fox 🦊 — Mon, 31 Aug 2026 03:18:48 +0000

In 2026, I have $300K worth of crypto.
I need $100K.

Why would my first thought be:
“Which $100K do I sell?”

How about:
“What can I borrow against?”

Can I keep my exposure while accessing liquidity?

Here are some markets already available on Tangent Finance in 2026

👉 $USDC / $fxUSD - HEC market with a TVL of $812.86K
👉 $fxUSD / $reUSD - HEC market with a TVL of $286.69K

I'm highlighting these two because I really trust FX protocol products

The catch is borrowing $USG against accepted collateral. Borrowing can make that much sense, yes.

Stop thinking about your portfolio as something you either hold or sell. Start thinking about it as productive collateral.

AI Agents Age Through Their State

Fred the Fox 🦊 — Fri, 28 Aug 2026 11:52:23 +0000

A chatbot usually gets old in the obvious way: a stronger model ships, and yesterday’s answers start looking weak by comparison.

Persistent agents have a less visible aging problem. They can degrade while the model weights stay frozen.

The cause is the state around the model.

A normal chatbot starts with a relatively clean context. The user sends a prompt, the model answers, and most of the session disappears afterward. Persistent agents carry more forward. They save memories, update project facts, write reusable skills, run scheduled jobs, and reuse earlier decisions in later work.

The model may be unchanged. Its operating context keeps moving.

That distinction matters because agent evaluation still tends to center the model. For a long-lived agent, the model is one part of a system that keeps editing the information it will rely on next time.

That is where aging enters.

The agent includes its accumulated state

A recent paper on agent lifespan engineering studies this directly. Across roughly 400 experiments and agent lifespans ranging from 8 to 200 sessions, the researchers measured how performance changed as the agents accumulated history and updates.

Its most useful contribution is a taxonomy of lifespan failures. The paper names several ways an agent can degrade without any change to the underlying model:

Compression aging: important details disappear when long histories are summarized or compressed.
Interference aging: new memories compete with, distort, or override older information.
Revision aging: the agent fails to replace an outdated fact cleanly with a newer one.
Maintenance aging: changes to tools, prompts, or memory systems produce unexpected behavior.

Those categories sound like infrastructure work because they are. Agent memory turns ordinary state-management problems into model behavior.

The awkward part is how quiet the failures can be. An agent may still answer fluently, call tools, and follow the surface shape of the task while its memory becomes less precise. A broken API call is easy to spot. Memory drift can look like competence until the wrong fact changes an action.

Day-one benchmarks miss the aging path

Most agent benchmarks still test a fresh system. The agent receives a task, runs through a controlled environment, and gets a score.

That is useful for measuring planning, browsing, tool use, or task completion under clean conditions. It says less about the same agent after months of conversations, summaries, corrections, saved conventions, outdated project facts, skills, and maintenance changes.

For persistent systems such as Hermes AI agents, the difference shows up in the product itself. Value comes partly from continuity: the agent stays available, remembers project context, learns recurring tasks, and can run work on a schedule.

Each saved memory also becomes another state object the system has to retrieve, rank, interpret, and sometimes revise.

A strong short-run benchmark score only proves that the fresh version of the system performed well under the test. It cannot prove that the agent will make the same decision after its memory layer has been rewritten 100 times.

For persistent agents, lifespan is part of the product surface.

Memory needs production observability

Teams already monitor model calls: prompts, responses, latency, token use, tool errors. Long-lived agents need the same seriousness around memory.

Useful observability has to answer more than “what did the model say?” Operators need to know:

What information did the agent save?
Which memory influenced a specific action?
Has an older fact been superseded?
Did compression remove a constraint the agent still needed?
Can a damaged or misleading memory be rolled back?
Did a maintenance change alter behavior on the same task?

Prompt logs miss much of that path. The memory pipeline has its own lifecycle: write, summarize, embed, retrieve, rank, inject, use, revise, prune, and delete.

Failures can enter at any step.

A memory can be accurate when written and wrong later. Summaries can drop the exception that made a rule safe. Retrieval can pull a stale convention because it is semantically close to the current task. A repair can add the new fact while leaving the old one active. Maintenance can improve one retrieval path and quietly damage another.

This is closer to operating a production data system than keeping a notebook.

The risk rises once an agent can act. If it can call APIs, execute commands, edit files, or run scheduled workflows, memory quality becomes an operational control. A bad answer wastes time. Automated action with bad context can create cost, delete work, or push the wrong change into a live process.

Persistence needs maintenance

Persistent memory is still one of the more useful ideas in agentic AI. An assistant that remembers project conventions, previous decisions, recurring tasks, and user preferences should beat one that starts from zero every morning.

Accumulation alone can leave the system worse.

The same agent can collect useful knowledge, stale assumptions, contradictory instructions, compressed summaries, and obsolete project state. Without lifecycle controls, “learning from every interaction” turns into carrying old mistakes forward with more confidence.

The answer is to operate persistent agents as long-lived systems.

That means testing across weeks or months, measuring memory quality separately from task completion, and building repair paths before the memory layer is visibly wrong.

Memory pruning, provenance tracking, versioning, rollback, and maintenance regression tests are likely to become normal parts of the agent stack. They are less exciting than a new model release, but they target the part of the system that changes every day.

Agent reliability has to be measured after launch

A team deploying persistent agents needs to know whether the system still behaves correctly after context compression, memory edits, tool updates, prompt changes, and scheduled-task history have accumulated.

That creates a different testing loop. Re-run old tasks after new memories are added. Check which memories were retrieved. Make superseded facts visible. Test maintenance changes against known workflows before assuming the agent’s behavior stayed stable.

From the outside, the failure mode often looks competent. The agent may still produce a confident answer and call the expected tool, even when its internal state has been damaged, compressed, or made stale.

That is the practical difference between model aging and agent aging.

A model gets old when the frontier moves past it. Persistent agents can age because their own history becomes harder to manage.

Keeping agents online is only the first reliability problem. The harder one is keeping their accumulated state dependable as they grow older.