Your Agent Finished the Job. Did the Email Actually Leave?

An agent run can be complete while its outbound side effect is still unknown.

That distinction matters for email, GitHub comments, payments, ticket updates, browser actions, and every other operation where a remote system accepts a request asynchronously. A process exit of 0 proves only that your worker reached its local completion path. It does not prove delivery.

This article presents a small delivery ledger and a failure-injection test you can add before trusting an always-on agent.

The two facts you need to store

For each externally visible action, record two independent facts:

1. Execution state: did the agent decide, validate, and dispatch the action?
2. Delivery state: did the provider confirm acceptance, or can you reconcile the result later?

A minimal state machine is:

action_created -> dispatch_started -> delivery_confirmed
                              \
                               -> delivery_unknown -> reconciled

Do not collapse dispatch_started and delivery_confirmed into a single done flag. A timeout after the HTTP request leaves the result ambiguous: the provider may have rejected it, or it may have accepted it before the connection failed.

A practical ledger schema

Use a stable idempotency key that is generated before the first attempt and survives retries:

CREATE TABLE outbound_actions (
  action_id TEXT PRIMARY KEY,
  provider TEXT NOT NULL,
  operation TEXT NOT NULL,
  idempotency_key TEXT NOT NULL UNIQUE,
  execution_state TEXT NOT NULL,
  delivery_state TEXT NOT NULL,
  provider_id TEXT,
  attempt_count INTEGER NOT NULL DEFAULT 0,
  last_error TEXT,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL
);

The important constraint is not the exact database. It is that a restart cannot create a new identity for the same logical action.

Before dispatch, write the row and commit it. Then:

• increment attempt_count in a transaction;
• send the provider request with the same idempotency key;
• store the provider message or request ID when returned;
• mark delivery confirmed only after the provider acceptance response is parsed;
• treat timeouts, broken connections, and process termination as delivery unknown;
• reconcile unknown rows using the provider lookup API or an operator review queue.

A retry should not mean “send again.” It should mean “ask whether this logical action already exists, then retry only when the provider says it does not.”

Why a worker health check is insufficient

A supervisor can prove that the process is alive. It cannot prove that an outbound action reached its destination. Even a successful TCP write is weaker than provider acceptance, and provider acceptance can still be weaker than final delivery.

Expose these counters separately:

• actions created
• dispatch attempts
• provider acceptances
• delivery-unknown records
• reconciled successes
• reconciled failures
• actions awaiting operator review

If those numbers are merged into one successful runs metric, an outage can look healthy while the delivery backlog grows.

A reproducible failure-injection test

Run the same test against a fake provider and, where possible, a sandbox provider:

1. Create one action with key deploy-42-notify.
2. Commit action_created.
3. Let the provider accept the request, but drop the client connection before the response is read.
4. Kill and restart the worker.
5. Confirm that the ledger contains exactly one logical action.
6. Reconcile by idempotency key or provider lookup.
7. Verify that the recovery path produces either one confirmed delivery or one explicit failure, never an unbounded duplicate loop.

Repeat with the connection dropped before the provider receives the request. Your recovery code must be able to distinguish the two cases through reconciliation, not guess from the local exception.

A useful assertion set is:

same logical action => same idempotency key

unknown delivery    => no blind duplicate send

restart             => ledger survives

reconciliation      => terminal state is explicit

operator review     => includes provider and request evidence

Hosting changes the recovery surface

If the worker is expected to run continuously, the host is part of this protocol. You need durable storage for the ledger, a restart policy, observable queue pressure, and a rebuild procedure that restores state without restoring stale credentials into an unintended environment. A managed option such as always-on agent hosting on Ampere can be evaluated against those requirements, but hosting does not remove the need for idempotency or reconciliation.

Before choosing a host, test a clean rebuild: restore the ledger, start one worker, replay unknown actions, and verify that no second worker can claim the same row without a lease or compare-and-swap transition.

Checklist

• Is execution completion separate from outbound delivery confirmation?
• Is the idempotency key created before dispatch and persisted across restarts?
• Can a timeout enter an explicit unknown state?
• Is there a provider lookup or human reconciliation path?
• Do metrics distinguish accepted, delivered, unknown, and failed?
• Does a clean-host rebuild preserve the ledger without widening credential access?
• Have you injected a response-loss failure after provider acceptance?

An agent is not reliable because its final log line says done. It is reliable when the system can explain what happened after the process, network, or provider stopped cooperating.