AI Agent Durable Queue: Stop Long-Running Work From Dying Mid-Task

An AI agent does not need to be wrong to disappoint users. Sometimes it is halfway through a useful job, a worker restarts, a model call times out, a tool hangs, and the whole task disappears like it never existed.

That failure feels small in logs and huge to the customer. They asked the agent to analyze accounts, draft a migration plan, process documents, update records, or prepare a report. The agent looked busy. Then nothing happened.

If you are building production AI workflows, the fix is not a longer prompt. You need a durable queue: a system that stores agent work as recoverable state, not as a fragile loop living inside one process.

This guide walks through a practical architecture for solo developers, Micro SaaS builders, and AI product teams who want long-running agents to finish jobs with evidence.

Why agent work dies in production

The first agent prototype is often a while loop: call the model, append tool results, repeat until the model returns a final answer. That is fine for a demo. It is risky for a real customer task because everything important lives in memory:

• the conversation state
• the current tool call
• the retry count
• the model choice
• the user and tenant context
• the approval status
• the partial output
• the reason a step failed

If the process restarts, the stack is gone. If the tool succeeds but the response write fails, you may repeat the action. If the model call times out, you may not know whether to retry, pause, or mark the job as failed.

Long-running AI work has the same reliability needs as payments, imports, billing jobs, and webhooks: durable state, leases, idempotency, retries, and audit logs.

What is an AI agent durable queue?

An AI agent durable queue is a persistence-backed execution layer for agent tasks.

Instead of running a whole agent job inside one request or one worker stack, you break the job into durable records:

• a run record for the overall user request
• step records for each model call, tool call, approval, or verification
• event records for every important state transition
• artifact records for files, summaries, reports, or generated outputs
• receipt records that explain what happened and why

A normal task queue says, "Run this background job." A durable agent queue says, "Remember every step, resume safely, avoid duplicate side effects, prove what happened, and pause for human review when needed."

That difference matters because a single user request can involve retrieval, planning, tool calls, file generation, database updates, approvals, and final verification.

The durable queue mental model

Think of the agent as a state machine, not a chat loop.

A run moves through states:

queued -> planning -> waiting_for_tool -> waiting_for_approval
       -> verifying -> completed
       -> failed
       -> paused
       -> cancelled

Each transition is written before the next risky action. That gives you a recovery point.

If a worker crashes while calling a model, another worker can inspect the last committed state and continue. If a tool call already created a record, the system can detect the idempotency key and avoid doing it twice. If a step needs approval, the run pauses instead of trying to be clever.

The goal is not to make failures impossible. The goal is to make failures visible, recoverable, and safe.

A simple database schema

You can build this with Postgres before reaching for complex infrastructure. Start small.

create table agent_runs (
  id uuid primary key,
  tenant_id uuid not null,
  user_id uuid not null,
  status text not null,
  task text not null,
  current_step_id uuid,
  priority int not null default 100,
  attempts int not null default 0,
  max_attempts int not null default 3,
  leased_until timestamptz,
  leased_by text,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  completed_at timestamptz
);

create table agent_steps (
  id uuid primary key,
  run_id uuid not null references agent_runs(id),
  type text not null, -- model_call, tool_call, approval, verify, final_answer
  status text not null,
  input_json jsonb not null default '{}',
  output_json jsonb,
  error_json jsonb,
  idempotency_key text,
  attempt int not null default 0,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now()
);

create unique index agent_steps_idempotency_key_idx
on agent_steps(run_id, idempotency_key)
where idempotency_key is not null;

create table agent_events (
  id bigserial primary key,
  run_id uuid not null,
  step_id uuid,
  event_type text not null,
  event_json jsonb not null default '{}',
  created_at timestamptz not null default now()
);

This schema is not fancy, but it gives you the foundation:

• runs are queryable
• steps are replayable
• duplicate tool calls can be blocked
• failures have structured context
• support can inspect what happened
• metrics can be calculated later

For small teams, boring tables beat magical agent memory.

Claim work with leases, not hope

A common mistake is letting many workers grab the same job. Another is letting one dead worker hold a job forever.

Use leases. A worker claims a run for a short period. If the worker keeps making progress, it extends the lease. If it dies, the lease expires and another worker can resume.

update agent_runs
set leased_by = $1,
    leased_until = now() + interval '60 seconds',
    status = 'running',
    updated_at = now()
where id = (
  select id
  from agent_runs
  where status in ('queued', 'running')
    and (leased_until is null or leased_until < now())
  order by priority asc, created_at asc
  for update skip locked
  limit 1
)
returning *;

skip locked is useful because multiple workers can safely look for work without blocking each other.

Keep leases short enough to recover quickly, but long enough to avoid noisy takeovers. For many agent workflows, 30 to 120 seconds is a reasonable starting point.

Break the agent loop into recoverable steps

Do not treat the whole agent run as one opaque job. Store each meaningful operation before and after it happens.

type StepType = "model_call" | "tool_call" | "approval" | "verify";

async function executeNextStep(runId: string) {
  const run = await db.getRun(runId);
  const context = await buildContextPacket(run);

  const modelStep = await db.createStep({
    runId,
    type: "model_call",
    status: "started",
    input_json: { contextVersion: context.version }
  });

  try {
    const response = await llm.chat(context.messages);

    await db.completeStep(modelStep.id, {
      output_json: {
        model: response.model,
        usage: response.usage,
        tool_calls: response.tool_calls ?? [],
        content_preview: response.content?.slice(0, 500)
      }
    });

    if (response.tool_calls?.length) {
      await enqueueToolSteps(runId, response.tool_calls);
      return;
    }

    await enqueueVerificationStep(runId, response.content);
  } catch (err) {
    await db.failStep(modelStep.id, normalizeError(err));
    await scheduleRetryOrPause(runId, err);
  }
}

Notice the pattern:

1. Write intent.
2. Perform the risky operation.
3. Write the result.
4. Decide the next step.

That sequence makes the workflow inspectable. It also gives you a clean place to add budgets, policies, approvals, and tests.

Make every tool call idempotent

Agents call tools. Tools create side effects. Side effects are where reliability bugs become customer pain.

If an agent sends the same invoice twice, deletes the same file twice, or posts the same message twice, the user will not care that the model was smart.

Each tool call needs an idempotency key. The key should represent the intended effect, not just a random retry attempt.

function toolIdempotencyKey(runId: string, toolName: string, args: unknown) {
  return hashJson({
    runId,
    toolName,
    args,
    purpose: "agent-tool-effect-v1"
  });
}

Before executing a tool, check whether the same key already completed.

async function runToolStep(step: AgentStep) {
  const previous = await db.findCompletedStepByKey(
    step.run_id,
    step.idempotency_key
  );

  if (previous) {
    await db.completeStep(step.id, {
      output_json: previous.output_json,
      reused_result: true
    });
    return;
  }

  const result = await tools.execute(step.input_json.name, step.input_json.args);

  await db.completeStep(step.id, {
    output_json: result
  });
}

For external APIs, also pass the idempotency key when the API supports it. For internal writes, store the key on the created row.

Idempotency is not only for payments. It is for any agent action you may retry.

Use retry policy by failure type

A durable queue should not retry everything blindly.

Use different policies for different failures:

Failure	Retry?	Action
Model timeout	Yes	Retry with backoff, then fallback model
Rate limit	Yes	Respect retry-after, lower priority if needed
Invalid JSON	Maybe	Repair once, then ask model with stricter schema
Permission denied	No	Pause and show user/admin action needed
Tool validation error	No	Ask model to revise arguments or fail safely
Approval rejected	No	Mark cancelled or request a new plan
Suspected prompt injection	No	Quarantine step and require review

Here is a simple retry decision function:

function retryDecision(error: AgentError, attempt: number) {
  if (error.kind === "rate_limit") {
    return { retry: true, delayMs: error.retryAfterMs ?? 60_000 };
  }

  if (error.kind === "timeout" && attempt < 3) {
    return { retry: true, delayMs: Math.pow(2, attempt) * 1000 };
  }

  if (error.kind === "invalid_tool_args") {
    return { retry: false, next: "revise_plan" };
  }

  if (error.kind === "permission_denied") {
    return { retry: false, next: "pause_for_admin" };
  }

  return { retry: false, next: "fail_with_receipt" };
}

The point is to avoid turning retries into a hidden cost leak. Every retry should have a reason, a limit, and a trace.

Add approval pauses as first-class queue states

Long-running agents often reach moments where they should not continue alone:

• sending an email
• changing billing data
• deleting records
• publishing content
• updating customer-facing settings
• running a high-cost operation
• using a sensitive integration

Do not handle this with a prompt that says, "Ask for approval when needed." Store approval as a durable step.

{
  "type": "approval",
  "status": "waiting",
  "input_json": {
    "risk": "high",
    "action": "send_customer_email",
    "summary": "Send a renewal reminder to 84 trial users",
    "sample": "Hi {{first_name}}, your trial ends...",
    "required_role": "workspace_admin"
  }
}

The run stays paused until the right human approves, rejects, or edits the action.

This gives you clean product behavior:

• workers do not spin while waiting
• users can resume later
• reviewers see the exact proposed action
• audit logs show who approved what
• the agent cannot silently skip the gate

Approval gates are not friction when they protect trust. They are part of the workflow contract.

Store artifacts separately from chat history

A long-running agent produces more than messages. It may create files, reports, diffs, SQL queries, charts, summaries, or extracted data.

Do not bury those artifacts inside a huge prompt transcript. Store them as separate records, then pass only the relevant artifact summaries back into the model. This keeps context smaller and makes the final output easier to verify.

For example, a data-cleaning agent might store:

• uploaded file snapshot
• schema summary
• cleaning plan
• generated Python script
• validation report
• final CSV
• user-facing summary

The model does not need the entire file in every call. It needs the right summary, the right links, and the right constraints.

Verify before completing the run

A durable queue should not mark work as complete just because the model wrote a confident final answer.

Add a verification step.

For deterministic tasks, verification can be code:

async function verifyReport(runId: string) {
  const artifacts = await db.listArtifacts(runId);
  const report = artifacts.find(a => a.artifact_type === "final_report");
  const sources = artifacts.filter(a => a.artifact_type === "source_snapshot");

  return {
    hasReport: Boolean(report),
    hasSources: sources.length > 0,
    hasEmptySections: report?.content_text?.includes("TODO") ?? false
  };
}

For fuzzy tasks, use a mixed approach:

• schema validation
• required section checks
• citation checks
• policy checks
• cost checks
• LLM-as-judge for narrow rubrics
• human review for risky outputs

A useful completion rule: the agent is done only when the queue can explain what changed, what evidence supports it, and what remains unresolved.

Metrics to track from day one

You do not need a massive observability stack to start. Track metrics that reveal whether the queue is making agent work more reliable.

Start with:

• run completion rate
• median and p95 run duration
• retries per run
• tool failures by tool name
• model failures by provider
• approval wait time
• duplicate tool calls prevented
• runs recovered after expired lease
• cost per completed run
• user-visible failure rate

The most important metric is cost per successful outcome. A cheap failed run is still waste.

A practical build order

If you are building this from scratch, do not boil the ocean. Start with runs, steps, events, leases, idempotent tool calls, typed retries, approval pauses, artifacts, verification, and customer-visible status.

For a solo builder, Postgres plus one worker process is enough to begin. Larger teams can later move dispatching to dedicated queues or workflow engines while keeping durable run state and receipts in the product database.

Conclusion

A long-running AI agent is not just a smarter prompt. It is a distributed workflow with unreliable models, flaky APIs, human approvals, variable cost, and real side effects.

The core pattern is simple: persist the run, persist each step, use leases, make tools idempotent, retry carefully, pause for approval, verify completion, and leave a receipt.

FAQ

What is an AI agent durable queue?

An AI agent durable queue is a persistence-backed execution system that stores agent runs, steps, events, artifacts, retries, approvals, and results so long-running AI work can resume after crashes or timeouts.

Is a normal background job queue enough for AI agents?

A normal queue is a good start, but AI agents usually need more state. You need step history, tool idempotency, approval pauses, model usage, artifacts, and verification receipts. A basic queue can run the worker, but durable agent state should live in your application database or workflow store.

How do I stop an agent from repeating the same tool action?

Use idempotency keys for every side-effecting tool call. Store the key with the tool step and, when possible, pass it to the external API. On retry, check whether the same key already completed before executing the action again.

What should happen when an agent worker crashes?

The worker lease should expire. Another worker should claim the run, inspect the last completed step, and continue from the next safe state. If the previous step may have caused a side effect, the new worker should use idempotency records before retrying.

When should an AI agent ask for human approval?

Require approval for actions that send messages, spend money, delete or modify important data, publish externally, access sensitive integrations, or exceed cost and risk thresholds. Approval should be stored as a durable step, not left only to prompt instructions.

Comments

[Reid Marlow]

This is the pattern that starts looking boring only after the first retry bug. I would add a hard boundary between resumable computation and side effects. The run can replay planning steps, but external writes need idempotency keys, receipts, and a reconciliation state when the worker dies after the write but before the ack.