DEV Community

Zira
Zira

Posted on

Your AI Agent Finished the Job. Did the Notification Actually Send?

#testing

An agent run can be correct while the user-facing result is missing, duplicated, or impossible to reconcile.

That happens when execution state and outbound delivery state share one vague status such as done. A worker may commit the result, crash before sending the webhook, retry after a timeout, and send the same notification twice. Or the webhook may be accepted by the provider while the response is lost, leaving the sender unsure whether to retry.

This post shows a small delivery ledger and a failure-injection test you can adapt to webhooks, email, Slack, or browser automation.

Separate execution from delivery

Keep two state machines. The execution state answers: “Did the agent complete the work?” The delivery state answers: “What do we know about the outbound side effect?”

A minimal record can look like this:

CREATE TABLE agent_runs (
  run_id TEXT PRIMARY KEY,
  result_hash TEXT NOT NULL,
  execution_state TEXT NOT NULL, -- RUNNING, SUCCEEDED, FAILED, UNKNOWN
  created_at TEXT NOT NULL
);

CREATE TABLE deliveries (
  delivery_id TEXT PRIMARY KEY,
  run_id TEXT NOT NULL,
  channel TEXT NOT NULL,
  idempotency_key TEXT NOT NULL UNIQUE,
  state TEXT NOT NULL, -- PENDING, SENT, UNKNOWN, CONFIRMED, FAILED
  provider_message_id TEXT,
  attempts INTEGER NOT NULL DEFAULT 0,
  last_error TEXT,
  FOREIGN KEY (run_id) REFERENCES agent_runs(run_id)
);
Enter fullscreen mode Exit fullscreen mode

The important detail is the stable idempotency_key. Derive it from the logical event, not from the retry attempt:

sha256(run_id + ":" + event_type + ":" + channel + ":" + recipient)
Enter fullscreen mode Exit fullscreen mode

If the same logical delivery is retried after a timeout, it must address the same key. A new key per retry turns a network ambiguity into a duplicate notification.

Use an explicit delivery protocol

A practical protocol is:

  1. Commit the agent result and insert a PENDING delivery in one database transaction.
  2. A delivery worker claims the row with a lease and increments attempts.
  3. Send using the stable idempotency key, if the provider supports one.
  4. Store the provider message ID and mark CONFIRMED only when the provider gives an authoritative acknowledgement.
  5. If the worker times out after sending, mark UNKNOWN, not FAILED.
  6. Reconcile UNKNOWN deliveries with the provider lookup API before retrying.
  7. Retry only when the provider proves that the key was not accepted, or when a channel-specific deduplication rule makes retry safe.

Do not let a successful HTTP response from your own worker stand in for provider confirmation. They are different facts.

Make the ambiguity visible

Expose at least these counters and queries:

  • pending deliveries older than their SLO
  • unknown deliveries awaiting reconciliation
  • confirmed deliveries with no linked execution result
  • duplicate-key conflicts
  • attempts by channel and error class

An incident dashboard should be able to answer three questions for one run: did execution finish, did delivery leave the system, and can the receiver identify the exact logical event?

Failure-injection test

Run this matrix against a staging provider or a local fake:

Injection point Expected state Safe recovery
Crash before delivery insert no outbound work replay transaction or mark run incomplete
Crash after insert, before claim PENDING worker claims after lease expiry
Timeout before provider response UNKNOWN provider lookup by idempotency key
Provider rejects request FAILED or PENDING classify error before retry
Crash after provider accepts UNKNOWN lookup, then record provider ID
Duplicate worker claim one accepted logical event row lease plus provider idempotency
Receiver returns 500 after acceptance provider-dependent lookup before retry

The test is not complete until you verify the final receiver-visible outcome, not merely the worker log.

Hosting implication

If this worker must stay available while the main agent is idle, a managed runtime such as always-on OpenClaw hosting on Ampere can be one option to evaluate. Hosting does not solve delivery semantics: you still need the ledger, leases, provider reconciliation, and idempotency tests.

That distinction is the point. A live process reduces one failure mode; it does not prove that an outbound side effect was delivered exactly once.

Checklist

Before calling an agent run “complete,” verify:

  • execution and delivery have separate states
  • each logical event has a stable idempotency key
  • timeouts become UNKNOWN until reconciled
  • provider message IDs are stored
  • retries are classified by provider evidence
  • leases prevent concurrent workers from racing
  • dashboards show pending, unknown, and duplicate-key states
  • a clean rebuild can recover the delivery ledger

If you build agents that send messages, trigger deployments, or drive browsers, test delivery as its own subsystem. “The agent finished” is not the same as “the user received one correct notification.”

Follow for practical AI-agent reliability patterns that are small enough to test and specific enough to debug.

Top comments (0)