Long-running agents rarely fail because a model returned invalid JSON. They fail at the boundary between queued, leased, running, and completed work.
A worker can restart while holding a lease. A queue can accept more work than the runtime can safely execute. A retry can begin before the first attempt has actually stopped. If these states are not explicit, reliable automation can run the same side effect twice or lose work without an obvious error.
This article shows a small control-plane model you can implement with a relational database and a worker loop. The goal is to make pressure and ambiguity visible before putting an agent on an always-on host.
Start with four separate states
Do not represent work with only pending and done. At minimum, distinguish:
- queued: accepted, but not assigned to a worker
- leased: assigned to a worker until a deadline
- running: the worker has started execution
- succeeded, failed, or unknown: the outcome is recorded
unknown is important. It means the control plane cannot prove whether the attempt finished. It is not the same as failure, and it must not be retried blindly when the tool had an external side effect.
CREATE TABLE agent_jobs (
id TEXT PRIMARY KEY,
status TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
lease_owner TEXT,
lease_until INTEGER,
idempotency_key TEXT NOT NULL UNIQUE,
outcome_ref TEXT,
last_error TEXT
);
The unique idempotency key protects the enqueue path. It does not, by itself, make an email, browser action, payment, or deployment idempotent. The external operation needs its own stable request key or a reconciliation step.
Make admission control explicit
A queue is not a plan for infinite concurrency. Define a limit for each expensive resource:
- maximum queued jobs per tenant
- maximum running jobs per worker pool
- maximum tool calls per job
- maximum wall-clock age
- maximum retry count
When a threshold is reached, return a retryable reason such as tenant_queue_limit, not a generic 500. That gives the caller a choice: wait, reduce scope, or use another pool.
Do not hide backpressure by accepting everything into memory. If the process dies, those accepted jobs disappear. Persist the queue before acknowledging it.
Lease work, then re-check ownership
A worker should claim one job atomically and receive a short lease. PostgreSQL workers commonly use SELECT FOR UPDATE SKIP LOCKED, ordered by creation time, then update the row with a lease owner and deadline.
The lease is not permission to run forever. Before every expensive or externally visible step, the worker should verify that it still owns the lease and that the job is not cancelled. A heartbeat proves recent liveness; it does not prove that a previous worker stopped.
If a lease expires while the old worker is still alive, two workers can execute the same job. That is an expected failure mode, not an edge case.
Separate execution from side effects
For an agent that edits files, opens a browser, sends a notification, or calls a deployment API, record an intent before the effect:
- Create a stable operation key from job_id and step number.
- Persist INTENT_RECORDED.
- Call the provider with that key when supported.
- Persist the provider reference and OUTCOME_CONFIRMED.
- If the process disappears, mark the operation UNKNOWN.
On recovery, reconcile UNKNOWN operations using the provider lookup API or an application audit trail. Only issue a new request when the original operation is proven not to have happened.
This is why a retry counter is not enough. Attempt 2 tells you how many times the worker tried. It does not tell you whether attempt 1 sent the email, pushed the commit, or clicked the button before it died.
A failure-injection test
Use a fake tool that sleeps for two seconds and writes a record keyed by operation_key. Run two workers with a lease shorter than the tool duration. Kill worker A after it starts, then let worker B reclaim the lease.
Your assertions should be explicit:
- the job has one stable idempotency key
- the effect table has at most one row for that key
- the job is not marked succeeded without an outcome reference
- an ambiguous result becomes unknown, not automatically failed
- reconciliation can move unknown to succeeded without running the effect again
Repeat with cancellation, a database restart, a blocked provider lookup, and a worker that pauses during its heartbeat. Capture the state transition log, not just the final row.
What to monitor
A green process health check says little about queue safety. Track queue age, oldest leased job age, lease expirations, running jobs at each concurrency limit, unknown operations and reconciliation age, retries by reason, jobs accepted but never started, and per-tenant pressure.
Alert on a growing unknown set and on lease expirations that coincide with active workers. Those signals indicate ambiguity or overload even when the process is technically healthy.
Hosting is part of the experiment
When moving this worker to an always-on host, validate the recovery contract there. You need persistent database state, preserved credentials with narrow scope, observable restarts, and a rebuild path.
For teams that do not want to assemble those pieces from scratch, managed always-on hosting for agent workloads on Ampere is one option to evaluate. It does not remove the need for leases, idempotency, or reconciliation. The application still owns those semantics.
Practical checklist
- What exactly does accepted mean?
- Can a worker be alive after its lease expires?
- How does a second worker detect an ambiguous first attempt?
- Which side effects accept an idempotency key?
- How are unknown outcomes reconciled?
- What happens when one tenant saturates the pool?
- Can a clean host rebuild the queue and credentials from documented inputs?
- Can you show the state transitions for a killed worker?
The model can be brilliant and the queue can still be unsafe. Reliability comes from making pressure, ownership, and ambiguity first-class states around the model.
Top comments (0)