DEV Community: Rose

I gave an AI agent SSH access for a month. The agent was fine. These five things weren't.

Rose — Sun, 15 Nov 2026 05:37:32 +0000

I gave an agent SSH access to a real server and let it work unattended for a month. The agent part was fine. Everything I had to build around it is what this post is about.

If you are wiring an agent into anything that can send mail, spend money, or touch a production box, these are the five things that bit me — in the order they bit.

1. A hold that is a file is not a hold

My first safety mechanism was a HALT file. If it exists, the sender refuses. Clean, obvious, easy to reason about.

Then I moved the mailer to a new machine and the file didn't come with it. The new host reported halt: null — meaning no hold — and would have sent. Nothing was broken. Nothing errored. The control had simply not travelled with the code.

The lesson is not "copy the file". It's that a control you have only ever observed in its passing state is a control you have not tested. The check that catches this is not ls HALT. It is: arm a real send and confirm it refuses, on that host, today. A refusal is evidence. A file listing is a hope.

I now run four steps on any new host before trusting it:

The halt exists as text and names the right incident.
The status command reads it back — not null.
An armed send actually refuses.
The refusal appears in that host's own ledger.

Step 4 matters more than it looks, and it leads to the next problem.

2. "It's in the ledger" does not mean "this host did it"

A refusal row proves a refusal happened. It does not prove which machine made it. If two hosts can write to one ledger path, every row stays real and no row is attributable — and you will not notice, because the file looks correct.

The check is boring and it is the one I'd keep: before anything writes, resolve every configured path and assert it lives inside that host's own tree. Not "the config looks right" — resolve it and compare.

3. Two watchers on one mailbox is worse than one watcher

I ran a mail watcher on the old box while bringing up the new one. Same mailbox, two independent UID cursors. My assumption was that the worst case was a duplicate notification.

It was worse. A watcher-written row carried the send's id, not the finding's, and no host field. Two polls on two hosts would write two rows with the same id, same outcome, same subject, differing by seconds. A verifier passes both. Any human reading it concludes "the provider reported this twice" — which is completely ordinary — rather than "we polled twice".

The record would have asserted that something happened twice when it happened once, in a way indistinguishable from a genuine duplicate. That is the worst shape a bad record takes: not wrong in a way anyone would check.

The fix is one string — put the host in the via field. The reason it took a month to find is that the failure is silent and the file stays valid.

4. A quiet poll writes nothing, so "no new rows" proves nothing

I tried to prove a watcher had stopped by watching for a quiet period with no new ledger rows. That test passes identically for a stopped job and a running job with nothing to report.

What actually moves on every run, quiet or not, is the state file — last_poll, the cursor, an mtime. Watch that instead. It is a positive observation rather than an absence, and it also catches the case a disable can't: something else shelling the script.

Absence of evidence is the most comfortable kind of wrong.

5. The bill is the same prompt, sent again

Separate post, but it belongs in this list because it is the thing that surprised me most about running agents continuously rather than in bursts.

Every turn resends the system prompt, the tool definitions and the conversation. On a long task that repeated prefix is most of the cost. Prompt caching discounts it — but only on an exact prefix match. A timestamp at the top of the system prompt, a task counter that gets rewritten, tool definitions serialised from a hash map in a different order: each one silently turns a cache read into a cache write, every single turn.

The discipline is: nothing that changes goes at the top. Volatile state goes at the end, or in a tool result, or nowhere.

What I'd tell someone starting today

Build the refusal before the feature. Prove every control by trying to violate it, on the machine it will run on, and keep the proof. And put the host name in every row you write, because the day you have two of anything is the day your records stop being readable.

I run a crew of agents on AgentParley — hosted, nothing to install. If you're coming from a self-hosted setup, the comparison is here.

Your agent's token bill is mostly the same prompt, sent again

Rose — Sun, 15 Nov 2026 05:33:03 +0000

Most of what you pay for when an agent works is the same text, sent again.

Every time an agent thinks, the whole prompt goes back to the model: its instructions, its tool definitions, the conversation so far. On a long task that repeated part is the majority of the bill. Providers know this and sell a discount on it called prompt caching — but the discount only applies if the start of the prompt is byte-for-byte identical to last time. Change one character near the top and you pay full price for everything after it.

That sounds like a small implementation detail. It is the single biggest lever on what an agent costs to run, and most agent frameworks lose it by accident.

What breaks the cache

The cache is a prefix match. Anything that varies at the top of the prompt invalidates everything below it. The usual culprits:

A current time: 14:32:07 line in the system prompt.
A to-do list or task counter that gets rewritten each turn.
Tool definitions serialised from a hash map, so their order changes between runs.
Re-stamping old messages with a new timestamp when you replay history.
Trimming one old message per turn once the context gets long, so the prefix shifts every single turn.

None of these look like a cost decision when you write them. All of them are.

The heartbeat case

The clearest example I know is in OpenClaw's own issue tracker: an open report that heartbeats put the current time into the cached start of the prompt, so each heartbeat pays to write the cache again instead of reading it. A heartbeat is supposed to be the cheapest thing an agent does — wake up, check nothing needs attention, go back to sleep. If the clock sits in the cached prefix, every one of those wake-ups is a full-price prompt write.

The fix is not clever. It is discipline about what goes at the top.

Four rules that keep the prefix stable

1. The system prompt is byte-identical every turn. Persona, instructions, skills — no clock, no counters, no "you have 3 tasks remaining". If the agent needs to know the time, it calls a tool and the answer lands in the conversation, below the cached part.

2. History only grows at the end. Old messages are replayed exactly as they were, stamped with the time they arrived, never re-stamped. Appending is free; rewriting is not.

3. Summarise once, don't trim every turn. When a conversation gets long, compact it once and then let it grow at the end again. Dropping one old message per turn means the prefix changes on every call, which means you never get a cache hit at all.

4. Waiting costs nothing. An agent blocked on a colleague, a timer or a build should make zero model calls until something arrives. This is the one people miss, because a polling loop feels harmless — it is a full prompt every few seconds.

What it's worth

Take an agent whose instructions, tools and history add up to 30,000 tokens, making 20 model calls on a task.

Sent uncached, that is 600,000 input tokens at full price.

At typical cached-input pricing, where a cache read costs roughly a tenth of normal input, the first call writes the cache and the other 19 read it. That is about 85% less on the repeated part — and the repeated part is most of the bill.

The same arithmetic applies to specialists. A sub-agent that starts with its own short brief instead of inheriting the parent's whole conversation is cheaper on every call it makes, and it can run on a smaller model without losing anything, because it only has one job.

The uncomfortable part

Caching is also why the subscription question keeps coming up. In April 2026 Anthropic stopped letting Claude Pro and Max subscriptions cover third-party agent tools, citing its terms of service and load on its systems; in May it let them back in on a separate monthly credit. If you are running an agent on a flat-rate subscription, your cost model depends on a policy someone else can change, and the caching behaviour underneath it is not yours to control.

If you are building this yourself, the four rules above are most of the work, and they are worth more than any model upgrade.

I work on AgentParley, a hosted platform for crews of AI agents. We build every prompt so the repeated part stays byte-identical, which is where the numbers above come from. If you want the longer version, including how it compares to self-hosting, there is a full comparison here.