DEV Community

Zira
Zira

Posted on

Your AI Agent Finished the Job, But the Message Never Sent

#mcp

A restarted agent can be healthy and still lose the most important part of a workflow: telling the outside world what happened.

That is because execution recovery and delivery recovery are different failure domains.

  • Execution recovery asks: did the task run, and can we reconstruct its result?
  • Delivery recovery asks: did the notification, webhook, email, or side effect leave the system?

Treating both as one boolean such as status = done creates an awkward failure: the agent completed the work, crashed before sending the message, then refused to retry because the task already looked complete.

This article shows a small pattern you can adapt to an agent worker, MCP tool runner, or scheduled automation.

1. Record execution and delivery separately

Use a durable record instead of process memory. SQLite is enough for a single-worker prototype:

CREATE TABLE IF NOT EXISTS jobs (
  id TEXT PRIMARY KEY,
  input_json TEXT NOT NULL,
  result_json TEXT,
  execution_state TEXT NOT NULL,
  delivery_state TEXT NOT NULL,
  attempt INTEGER NOT NULL DEFAULT 0,
  updated_at TEXT NOT NULL
);
Enter fullscreen mode Exit fullscreen mode

A successful execution should set execution_state = succeeded while leaving delivery_state = pending until the external system acknowledges the message.

2. Make the external operation idempotent

Recovery means retrying. Retrying without an idempotency key can duplicate an email, ticket, deployment, or payment request. Derive the key from the durable job ID, not from a random value generated on each retry:

def deliver(job_id, payload):
    key = f"agent-job:{job_id}"
    response = requests.post(
        "https://example.invalid/webhook",
        json=payload,
        headers={"Idempotency-Key": key},
        timeout=10,
    )
    response.raise_for_status()
Enter fullscreen mode Exit fullscreen mode

The receiver must also persist the key, or otherwise document that it deduplicates it. A client-generated header alone is not a guarantee.

3. Recover each state independently

On worker startup, do not only look for running jobs. Also find completed executions whose delivery is still pending:

SELECT id, result_json
FROM jobs
WHERE execution_state = succeeded
  AND delivery_state IN (pending, failed)
ORDER BY updated_at
LIMIT 20;
Enter fullscreen mode Exit fullscreen mode

For each row: resend the stored result with the same idempotency key; mark delivery as sent only after a confirmed success response; keep failed rows with an error and next retry time; alert on age, not merely count.

4. Test the crash window deliberately

Force termination after the result is committed, before delivery, after delivery but before marking sent, and during a delivery timeout. The third case is why idempotency matters: the receiver may have accepted the request even though the worker never recorded the response.

5. Use this acceptance checklist

  • [ ] Result is durable before execution success.
  • [ ] Delivery has its own state and retry schedule.
  • [ ] Retries reuse a stable idempotency key.
  • [ ] Receiver deduplicates or exposes a delivery ID.
  • [ ] Restart recovery scans pending delivery.
  • [ ] Metrics distinguish execution failures from delivery failures.
  • [ ] Logs include job ID and attempt, but never credentials.

Hosting is part of recovery

If an agent must stay available for scheduled work or browser automation, the host needs persistent state, restart behavior, and a way to inspect failed deliveries. A managed option such as always-on OpenClaw hosting on Ampere can be relevant when those operational requirements matter more than running a process on a laptop. It does not remove the need for durable state or idempotent receivers.

The practical lesson is simple: “the agent is running” is a liveness signal, not proof that work was recovered or delivered. Model those boundaries explicitly, then test the crash between them.

Top comments (0)