What Happens When an AI Agent Runs Longer Than Your HTTP Request?

A user clicks “Run agent”, your backend receives a normal HTTP request, and the agent starts doing what agents do: calling tools, reading documents, querying APIs, waiting for a human approval, retrying a flaky search, and generating a long report.

Two minutes later, your load balancer returns 504 Gateway Timeout.

The user sees an error.

The agent, depending on where it is running, may still be alive. It may still be spending tokens. It may have already sent an email, created a ticket, or updated a record. Then the user clicks retry. Now you may have two agents doing overlapping work, with no clean way to know which one is authoritative.

That is the core problem:

An HTTP request is a short-lived delivery mechanism.

An AI agent run is often a long-lived state machine.

When the agent outlives the request, the request/response model stops being the right abstraction. You need run IDs, durable state, queues, idempotency, cancellation, progress events, and a way to resume after failure.

This article is about what breaks when an AI agent runs longer than the HTTP request that started it — and how to design systems that survive that reality.

TL;DR

• HTTP requests are bounded by clients, proxies, gateways, and platform timeouts.
• AI agents often run for seconds, minutes, hours, or even days.
• If an agent can outlive the request, do not treat the request as the job.
• Create an agent run resource, return 202 Accepted, and execute the work asynchronously.
• Use idempotency keys to prevent duplicate runs.
• Store agent state durably, not just in process memory.
• Use SSE/WebSockets for progress, but keep the run state as the source of truth.
• Treat tool calls as side effects that need idempotency, retries, and sometimes compensation.
• Cancellation must be explicit and cooperative.
• For long human pauses or multi-step workflows, prefer durable execution patterns.

📋 Table of Contents

• The real mismatch
• 1. The Gateway Times Out, but the Agent Keeps Spending Tokens
• 2. The Client Retries, and Now You Have Two Agents
• 3. In-Memory State Dies on Deploy, Restart, or Scale-In
• 4. Streaming Is a Delivery Channel, Not a Source of Truth
• 5. Tool Calls Turn Agent Runs Into Distributed Transactions
• 6. Cancellation Has to Be a Protocol, Not a Hope
• 7. Human Pauses Make “Long-Running” Look Cute
• 8. Auth Tokens Expire While the Agent Is Still Thinking
• 9. Choosing the Right Execution Model for Agent Workloads
• 10. The Production Shape I’d Use
• A practical checklist before shipping long-running agents

The real mismatch

A typical HTTP request has a simple lifecycle:

client connects
client sends request
server processes
server responds
connection closes

An AI agent run has a much messier lifecycle:

queued
started
planning
waiting for model
waiting for tool
waiting for approval
retrying
streaming progress
completed / failed / cancelled

Those two lifecycles do not line up.

HTTP assumes the server can finish quickly enough for the client to stay interested. Agent work often cannot make that promise. The agent may be waiting on:

• a slow model response
• a tool call to an external API
• a browser automation step
• a document parser
• a code execution sandbox
• a human approval
• a scheduled follow-up
• a rate limit cooldown
• a multi-step plan that only becomes clear halfway through

The first mistake is pretending the HTTP request is the agent. It is not. The request is only the trigger.

Once you accept that, the architecture changes.

1. The Gateway Times Out, but the Agent Keeps Spending Tokens

Scenario:

Your frontend calls POST /agent-runs and waits for the final answer. The agent takes longer than your gateway timeout. The client receives a 504, but the backend worker continues executing.

Why it matters:

Now you have split-brain behavior. The user thinks the operation failed. The system may still be doing work, consuming tokens, calling APIs, and mutating state.

This is especially dangerous when the agent has side effects. A timed-out read-only query is annoying. A timed-out agent that can send messages, create orders, or delete records is an incident waiting to happen.

Solution:

Do not run long-lived agents synchronously inside the HTTP request. Create a run record, return 202 Accepted, and execute the agent asynchronously.

import express from "express";
import crypto from "node:crypto";

const app = express();
app.use(express.json());

app.post("/agent-runs", async (req, res) => {
  const idempotencyKey = req.get("Idempotency-Key") ?? crypto.randomUUID();

  const run = await startAgentRunOnce({
    idempotencyKey,
    input: req.body,
  });

  res
    .status(202)
    .setHeader("Location", `/agent-runs/${run.id}`)
    .json({
      runId: run.id,
      status: run.status,
      statusUrl: `/agent-runs/${run.id}`,
      eventsUrl: `/agent-runs/${run.id}/events`,
    });
});

The client receives a run ID immediately. The actual agent work happens in a worker, queue, or workflow engine.

Why this works:

The HTTP request is no longer responsible for completing the agent run. It is only responsible for starting it safely.

The run becomes a first-class resource:

POST /agent-runs        → start
GET  /agent-runs/:id    → status
GET  /agent-runs/:id/events → progress stream
POST /agent-runs/:id/cancel → cancel

💡 Practical note:

202 Accepted is the honest response for work that has been received but not completed. It is not a hack. It is the correct HTTP semantics.

2. The Client Retries, and Now You Have Two Agents

Scenario:

The user does not see a result, so they click “Run” again. Or the mobile app automatically retries after a network drop. Now two agent runs are executing for the same intent.

Why it matters:

Agents are not pure functions. If the agent can call tools, retrying from scratch can produce different decisions and duplicate side effects.

A normal API retry problem becomes much worse with agents because the retry may:

• generate a different plan
• call different tools
• interpret the task differently
• perform side effects twice
• leave conflicting artifacts

Solution:

Require an idempotency key for agent creation.

A simple Postgres pattern:

CREATE TABLE agent_runs (
  id uuid PRIMARY KEY,
  idempotency_key text NOT NULL UNIQUE,
  input jsonb NOT NULL,
  status text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

Then insert safely:

INSERT INTO agent_runs (id, idempotency_key, input, status)
VALUES ($1, $2, $3, 'queued')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;

If no row is returned, fetch the existing run:

SELECT id, status
FROM agent_runs
WHERE idempotency_key = $1;

In application code:

async function startAgentRunOnce(params: {
  idempotencyKey: string;
  input: unknown;
}) {
  const runId = crypto.randomUUID();

  const inserted = await db.query(
    `
      INSERT INTO agent_runs (id, idempotency_key, input, status)
      VALUES ($1, $2, $3, 'queued')
      ON CONFLICT (idempotency_key) DO NOTHING
      RETURNING id, status
    `,
    [runId, params.idempotencyKey, JSON.stringify(params.input)]
  );

  if (inserted.rows.length > 0) {
    return inserted.rows[0];
  }

  const existing = await db.query(
    `
      SELECT id, status
      FROM agent_runs
      WHERE idempotency_key = $1
    `,
    [params.idempotencyKey]
  );

  return existing.rows[0];
}

Why this works:

The idempotency key becomes the client’s declaration of intent. If the same intent is submitted again, the system returns the existing run instead of creating a new one.

⚠️ Gotcha:

Idempotency keys need a meaningful scope. A key like run-agent is useless. Use something tied to the user, tenant, action, and input hash, or let the client generate a UUID per logical attempt.

3. In-Memory State Dies on Deploy, Restart, or Scale-In

Scenario:

Your agent keeps its conversation history, current plan, and tool-call progress in process memory. Then you deploy a new version, a container scales in, or the machine restarts.

The run disappears.

Why it matters:

Long-running agents need durable state. If the process can die before the run finishes, the state must live somewhere else.

This is not just about crashes. In production, processes restart all the time:

• deployments
• autoscaling
• node draining
• memory pressure
• platform maintenance
• worker restarts after errors

If your agent state is only in memory, you do not have a long-running agent. You have a fragile process with amnesia.

Solution:

Persist the agent run state in a durable store. The exact store can be Postgres, Redis, DynamoDB, or a workflow engine, but the state model should be explicit.

A useful run state shape:

type AgentRunStatus =
  | "queued"
  | "running"
  | "waiting_for_approval"
  | "completed"
  | "failed"
  | "cancelled";

interface AgentRunState {
  runId: string;
  status: AgentRunStatus;
  input: unknown;
  cursor?: string;
  messages: AgentMessage[];
  pendingToolCall?: ToolCall;
  artifacts: Artifact[];
  error?: string;
  updatedAt: string;
}

More important than the exact fields is the discipline:

• every meaningful step updates durable state
• the worker can reload the run after restart
• the API can answer status questions without touching the worker
• the frontend can reconnect without losing context

Why this works:

The worker becomes replaceable. The run state survives independently of any one process.

This also makes debugging much easier. When someone asks, “What is the agent doing right now?” you can answer from stored state instead of guessing from logs.

🚨 Production warning:

If you cannot answer “What step is this agent on?” without attaching a debugger, your agent is not production-ready.

4. Streaming Is a Delivery Channel, Not a Source of Truth

Scenario:

You use Server-Sent Events or WebSockets to stream agent progress. The connection drops. The user reloads the page. Now the frontend has missed events and has no reliable way to recover.

Why it matters:

Streaming is excellent for user experience. It is terrible as the only record of what happened.

A stream is ephemeral. It tells you what is happening now, or what happened recently, but it does not by itself answer:

• what is the current run status?
• what was the last completed step?
• which tool calls succeeded?
• did the run finish while the client was disconnected?
• can the client safely resume?

Solution:

Keep durable run state as the source of truth. Use streaming as a notification layer on top of that state.

A resilient SSE endpoint should support reconnection:

app.get("/agent-runs/:id/events", async (req, res) => {
  const runId = req.params.id;

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  const lastEventId = Number(req.headers["last-event-id"] ?? "0");

  const missedEvents = await eventStore.eventsSince(runId, lastEventId);

  for (const event of missedEvents) {
    sendEvent(res, event);
  }

  const unsubscribe = eventStore.subscribe(runId, (event) => {
    sendEvent(res, event);
  });

  const heartbeat = setInterval(() => {
    res.write(": ping\n\n");
  }, 15000);

  req.on("close", () => {
    clearInterval(heartbeat);
    unsubscribe();
  });
});

function sendEvent(
  res: express.Response,
  event: { id: number; type: string; payload: unknown }
) {
  res.write(`id: ${event.id}\n`);
  res.write(`event: ${event.type}\n`);
  res.write(`data: ${JSON.stringify(event.payload)}\n\n`);
}

The important part is Last-Event-ID. When the client reconnects, the server can replay missed events instead of pretending the connection never dropped.

Why this works:

The client can lose the stream without losing the run. The UI becomes a projection of durable state, not the only place where progress exists.

💡 Practical note:

Heartbeats are not optional. Many proxies and load balancers close idle connections quietly. A periodic comment line like : ping helps keep the stream alive.

5. Tool Calls Turn Agent Runs Into Distributed Transactions

Scenario:

Your agent calls three tools: search, create ticket, send email. The email tool succeeds, then the run crashes before the state is saved. The worker restarts and retries. Now the email may be sent again.

Why it matters:

Agents with tools are not just inference loops. They are distributed systems that take actions.

Once an agent can affect external systems, you have the usual distributed-workflow problems:

• retries
• duplicates
• partial failure
• timeouts
• compensation
• auditability

But agents make these problems harder because the sequence of actions may not be deterministic.

Solution:

Make tool execution idempotent wherever possible. Give each meaningful tool call a stable idempotency key derived from the run and the logical operation.

async function sendInvoiceEmail(run: AgentRunState, invoiceId: string) {
  const toolCallKey = `${run.runId}:send_invoice_email:${invoiceId}`;

  const existing = await externalCallStore.find(toolCallKey);

  if (existing) {
    return existing.result;
  }

  const result = await emailClient.send({
    idempotencyKey: toolCallKey,
    to: run.input.customerEmail,
    subject: `Invoice ${invoiceId}`,
    template: "invoice",
  });

  await externalCallStore.record(toolCallKey, result);

  return result;
}

If the downstream API does not support idempotency keys, you still need local deduplication:

const alreadyPerformed = await sideEffectLog.exists(toolCallKey);

if (alreadyPerformed) {
  return sideEffectLog.resultFor(toolCallKey);
}

For destructive or irreversible operations, consider requiring explicit approval or using a two-phase pattern:

propose action
store proposal
wait for approval
execute once
record result

Why this works:

You are separating “the agent decided to do something” from “the system actually did it.” That separation gives you a place to enforce safety, retries, and auditing.

🧠 The important part:

If a tool call can happen twice, the agent run is not safe to retry unless that tool call is idempotent or guarded.

6. Cancellation Has to Be a Protocol, Not a Hope

Scenario:

The user clicks “Cancel” while the agent is running. Your API updates a database row. The agent, currently waiting on a model call or external tool, has no idea.

Why it matters:

Cancellation is easy as a UI concept and hard as an execution concept. If the agent does not check for cancellation, it keeps doing work. If it checks too late, it may perform side effects after the user asked it to stop.

Solution:

Treat cancellation as a cooperative protocol.

First, expose a cancellation endpoint:

app.post("/agent-runs/:id/cancel", async (req, res) => {
  const runId = req.params.id;

  await agentRunStore.requestCancellation(runId);

  res.status(202).json({
    runId,
    status: "cancel_requested",
  });
});

Then make the worker check cancellation at safe boundaries:

async function executeAgentRun(runId: string, signal: AbortSignal) {
  while (!signal.aborted) {
    const run = await agentRunStore.get(runId);

    if (!run) return;

    if (run.status === "cancel_requested") {
      await agentRunStore.markCancelled(runId, {
        reason: "user_requested",
      });
      return;
    }

    const nextStep = await planNextStep(run);

    if (!nextStep) {
      await agentRunStore.markCompleted(runId);
      return;
    }

    await executeStep(runId, nextStep, signal);
  }
}

The critical detail is where you check.

Good cancellation points:

• before starting a new model call
• before executing a tool call
• after a long external wait
• before committing a side effect
• before moving to a new planning stage

Bad cancellation strategy:

• hoping the process gets killed
• relying only on the client disconnecting
• checking cancellation only at the very end

Why this works:

Cancellation becomes part of the run lifecycle instead of an afterthought.

⚠️ Gotcha:

Cancellation does not automatically undo side effects. If the agent already sent the email, cancellation may only mean “stop doing more work.” Your system needs to know the difference.

7. Human Pauses Make “Long-Running” Look Cute

Scenario:

Your agent needs approval before sending a high-risk email. The approval may come in ten seconds, ten hours, or three days.

Now your agent is not merely long-running. It is suspended.

Why it matters:

Many agent systems are designed for “slow API calls,” not for “pause until a human responds.” These are different problems.

A slow API call can be handled with timeouts and retries. A human pause requires:

• durable suspension
• expiration rules
• reminders
• escalation
• resume semantics
• audit trail
• possibly re-authentication

If you keep the agent process alive while waiting for a human, you are wasting resources. If you do not persist the pause, you lose the run.

Solution:

Model waiting states explicitly.

await agentRunStore.update(runId, {
  status: "waiting_for_approval",
  pendingAction: {
    type: "send_customer_email",
    payload: emailDraft,
    requestedAt: new Date().toISOString(),
    expiresAt: new Date(Date.now() + 48 * 60 * 60 * 1000).toISOString(),
  },
});

Then resume when the approval arrives:

app.post("/agent-runs/:id/approvals", async (req, res) => {
  const runId = req.params.id;
  const { approved, approver } = req.body;

  await agentRunStore.resolveApproval(runId, {
    approved,
    approver,
    resolvedAt: new Date().toISOString(),
  });

  await queue.enqueue("agent.resume", { runId });

  res.status(202).json({ runId, status: "resuming" });
});

This is where durable execution engines start to make sense. They are designed for workflows that can wait for long periods without keeping a process alive.

Why this works:

The agent run becomes a workflow with explicit pause and resume points. That is far more honest than pretending a human approval is just another fast function call.

💡 Practical note:

If your agent can wait for humans, design expiration early. A pending action that lives forever becomes a security and compliance problem.

8. Auth Tokens Expire While the Agent Is Still Thinking

Scenario:

The agent starts with the user’s access token. Twenty minutes later, it needs to call another API. The token has expired.

Why it matters:

User sessions and agent lifetimes do not naturally align.

A user may:

• close the browser
• log out
• revoke consent
• switch tenants
• have a token expire
• require step-up authentication for a sensitive action

Meanwhile, the agent may still be running.

If you casually pass the original request token into a long-lived background worker, you create both security and reliability problems.

Solution:

Decide explicitly what identity the agent runs under.

Common patterns:

Short-lived user token, narrow task

Useful when the run is brief and the token lifetime is sufficient.

Risk: the token expires mid-run.

Refresh token stored securely

Useful when the agent acts on behalf of the user for longer periods.

Requirements:

• secure token storage
• narrow scopes
• revocation handling
• audit logging

Service principal with delegated permissions

Useful when the agent performs system-level work.

Requirements:

• least privilege
• per-tenant policy checks
• explicit authorization mapping

Step-up reauthorization for sensitive steps

Useful when the agent needs approval for high-risk actions.

Example:

agent requests permission to refund payment
system pauses run
user re-authenticates
system resumes run with fresh approval

A practical authorization check before a sensitive tool call:

async function authorizeToolCall(run: AgentRunState, tool: ToolDefinition) {
  const policy = await policyStore.forRun(run.runId);

  if (!policy.allows(tool.name)) {
    throw new Error(`Tool ${tool.name} is not allowed for this run`);
  }

  if (tool.requiresRecentUserApproval) {
    const approval = await approvalStore.latest(run.runId, tool.name);

    if (!approval || approval.approvedAt < minutesAgo(5)) {
      await agentRunStore.update(run.runId, {
        status: "waiting_for_approval",
        pendingAction: {
          type: tool.name,
          payload: tool.input,
        },
      });

      throw new ToolCallPaused("Recent approval required");
    }
  }
}

Why this works:

You stop assuming that the original HTTP request’s auth context is valid forever.

🚨 Production warning:

Do not solve token expiration by giving agents broad admin tokens “just to keep things moving.” That turns every long-running agent into a privilege-escalation risk.

9. Choosing the Right Execution Model for Agent Workloads

Not every agent needs the same architecture. The right choice depends on how long the agent can run, whether it has side effects, and whether humans are involved.

Approach	Best for	Complexity	Weakness
Synchronous HTTP	Very fast, read-only agent calls	Low	Breaks as soon as work is slow or unreliable
Async run + polling	Simple background agent jobs	Medium	Polling can be inefficient; needs run store
SSE/WebSocket progress	Interactive UX with live updates	Medium	Connection loss handling is required
Queue workers	Scalable background execution	Medium	Retry and state discipline needed
Durable workflow engine	Multi-step, long-lived, human-in-loop agents	High	More operational complexity

When synchronous HTTP is fine

Use synchronous request/response when:

• the agent usually finishes in a few seconds
• tool calls are read-only or low-risk
• there is no human approval
• losing a run is acceptable
• retries are safe

Example:

POST /summarize-text

When async run + polling is better

Use async runs when:

• the agent may take tens of seconds or more
• the client may disconnect
• you need a stable run ID
• retries need to be deduplicated
• you want auditability

Example:

POST /agent-runs
GET /agent-runs/:id

When streaming matters

Use SSE or WebSockets when:

• users need live progress
• tokens or characters are being generated
• tool activity should be visible
• you want a responsive UI

But keep durable state underneath.

When durable execution earns its complexity

Use a durable workflow engine when:

• runs can last hours or days
• human approvals are common
• steps need guaranteed retries
• timers and schedules matter
• compensation or rollback is needed
• you need strong execution history

This is the category where “agent” starts looking less like a chat endpoint and more like business process automation.

10. The Production Shape I’d Use

If I were shipping an AI agent feature that could run longer than a normal HTTP request, I would use this shape:

Client
  ↓
API layer
  ↓
Run store
  ↓
Queue / workflow engine
  ↓
Agent worker
  ↓
Tool execution layer
  ↓
Event store / notifications

API layer

Responsible for:

• creating runs
• enforcing idempotency
• returning run URLs
• serving status
• exposing event streams
• accepting cancellation and approvals

Run store

Stores:

• run ID
• tenant/user
• input
• status
• state cursor
• pending action
• error
• timestamps
• idempotency key

Queue or workflow engine

Responsible for:

• scheduling workers
• retrying failed executions
• handling long waits
• enforcing timeouts
• preserving execution order where needed

Agent worker

Responsible for:

• loading run state
• calling the model
• choosing next steps
• executing tools
• checkpointing progress
• checking cancellation
• emitting events

Tool execution layer

Responsible for:

• validating tool input
• enforcing authorization
• applying idempotency keys
• logging side effects
• rate limiting
• wrapping external APIs

Event store / notifications

Responsible for:

• progress events
• reconnection support
• webhooks
• audit trail
• UI updates

The most important design decision is this:

The HTTP request starts and observes the run.

It does not own the run.

That one mental shift prevents a large class of production pain.

A practical checklist before shipping long-running agents

Before letting an agent run longer than the HTTP request that started it, I’d want these boxes checked.

Lifecycle

• [ ] Does the run have a unique ID?
• [ ] Is creation idempotent?
• [ ] Is the initial response 202 Accepted?
• [ ] Can the client check status without restarting work?
• [ ] Can the client reconnect after losing the stream?

State

• [ ] Is run state stored durably?
• [ ] Can a worker resume after restart?
• [ ] Are checkpoints saved after meaningful steps?
• [ ] Is there a clear status model?

Side effects

• [ ] Are tool calls idempotent where possible?
• [ ] Are destructive actions gated?
• [ ] Are external calls logged?
• [ ] Can partial failure be understood and repaired?

Cancellation

• [ ] Can the user request cancellation?
• [ ] Does the worker check cancellation at safe boundaries?
• [ ] Are side effects prevented after cancellation where possible?
• [ ] Is the final cancelled state recorded?

Human interaction

• [ ] Are approvals modeled as explicit run states?
• [ ] Do pending actions expire?
• [ ] Can the run resume after a long pause?
• [ ] Is the approver recorded?

Auth and policy

• [ ] Is the agent’s identity explicit?
• [ ] Are tokens scoped narrowly?
• [ ] Are sensitive actions reauthorized?
• [ ] Are permissions checked per tool call, not just at run start?

Observability

• [ ] Can you see the current step?
• [ ] Can you trace a run from request to final result?
• [ ] Are token usage, tool calls, and errors logged?
• [ ] Can you distinguish user retries from internal retries?

The deeper truth is that long-running AI agents are not just “slower APIs.” They are workflows with non-deterministic planning, external side effects, and user expectations.

HTTP can trigger them. HTTP can report on them. HTTP can stream progress from them.

But once the agent can outlive the request, HTTP should not be the container for the entire execution.

The request is the doorway.

The agent run is the process.

Design them separately, and the system becomes far easier to operate. Design them as the same thing, and every timeout, retry, deploy, and disconnected browser becomes a potential corruption of the run.