I gave an agent SSH access to a real server and let it work unattended for a month. The agent part was fine. Everything I had to build around it is what this post is about.
If you are wiring an agent into anything that can send mail, spend money, or touch a production box, these are the five things that bit me — in the order they bit.
1. A hold that is a file is not a hold
My first safety mechanism was a HALT file. If it exists, the sender refuses. Clean, obvious, easy to reason about.
Then I moved the mailer to a new machine and the file didn't come with it. The new host reported halt: null — meaning no hold — and would have sent. Nothing was broken. Nothing errored. The control had simply not travelled with the code.
The lesson is not "copy the file". It's that a control you have only ever observed in its passing state is a control you have not tested. The check that catches this is not ls HALT. It is: arm a real send and confirm it refuses, on that host, today. A refusal is evidence. A file listing is a hope.
I now run four steps on any new host before trusting it:
- The halt exists as text and names the right incident.
- The status command reads it back — not
null. - An armed send actually refuses.
- The refusal appears in that host's own ledger.
Step 4 matters more than it looks, and it leads to the next problem.
2. "It's in the ledger" does not mean "this host did it"
A refusal row proves a refusal happened. It does not prove which machine made it. If two hosts can write to one ledger path, every row stays real and no row is attributable — and you will not notice, because the file looks correct.
The check is boring and it is the one I'd keep: before anything writes, resolve every configured path and assert it lives inside that host's own tree. Not "the config looks right" — resolve it and compare.
3. Two watchers on one mailbox is worse than one watcher
I ran a mail watcher on the old box while bringing up the new one. Same mailbox, two independent UID cursors. My assumption was that the worst case was a duplicate notification.
It was worse. A watcher-written row carried the send's id, not the finding's, and no host field. Two polls on two hosts would write two rows with the same id, same outcome, same subject, differing by seconds. A verifier passes both. Any human reading it concludes "the provider reported this twice" — which is completely ordinary — rather than "we polled twice".
The record would have asserted that something happened twice when it happened once, in a way indistinguishable from a genuine duplicate. That is the worst shape a bad record takes: not wrong in a way anyone would check.
The fix is one string — put the host in the via field. The reason it took a month to find is that the failure is silent and the file stays valid.
4. A quiet poll writes nothing, so "no new rows" proves nothing
I tried to prove a watcher had stopped by watching for a quiet period with no new ledger rows. That test passes identically for a stopped job and a running job with nothing to report.
What actually moves on every run, quiet or not, is the state file — last_poll, the cursor, an mtime. Watch that instead. It is a positive observation rather than an absence, and it also catches the case a disable can't: something else shelling the script.
Absence of evidence is the most comfortable kind of wrong.
5. The bill is the same prompt, sent again
Separate post, but it belongs in this list because it is the thing that surprised me most about running agents continuously rather than in bursts.
Every turn resends the system prompt, the tool definitions and the conversation. On a long task that repeated prefix is most of the cost. Prompt caching discounts it — but only on an exact prefix match. A timestamp at the top of the system prompt, a task counter that gets rewritten, tool definitions serialised from a hash map in a different order: each one silently turns a cache read into a cache write, every single turn.
The discipline is: nothing that changes goes at the top. Volatile state goes at the end, or in a tool result, or nowhere.
What I'd tell someone starting today
Build the refusal before the feature. Prove every control by trying to violate it, on the machine it will run on, and keep the proof. And put the host name in every row you write, because the day you have two of anything is the day your records stop being readable.
I run a crew of agents on AgentParley — hosted, nothing to install. If you're coming from a self-hosted setup, the comparison is here.
Top comments (0)