A Timeout Is Not a Failed AI Job: Designing Safe Retries for Paid Generation APIs

A customer clicks Generate. Your server sends the request, then the connection times out. The tempting fix is to send another request. The expensive possibility is that the first job is already running.

This is a distributed-systems problem with a visible price tag. The client knows that it did not receive an answer; it does not know whether the server accepted the work.

We build SpicyAPI, so the API examples below use our media task protocol. The application design applies to other asynchronous services too, but each service has its own replay guarantees. Check those guarantees before sharing a retry helper across providers.

Separate a user's intent from an HTTP attempt

Give each intentional generation an operation record in your own database. An operation might be “make take 2 for shot 7.” Several HTTP attempts can belong to that one operation.

An intentional third take is a different operation, even if the prompt happens to be identical. A network retry of take 2 is still take 2.

A useful record contains:

• Your operation ID and the owning customer or project.
• A stable idempotency key generated once for that operation.
• The exact request body and a reference to the API credential that will send it.
• The creation time and an application recovery deadline.
• The remote task ID once you obtain one.
• A state that can represent uncertainty.

Store sensitive input in a protected application record; do not copy the full body or credential into ordinary logs. Create the record transactionally before making the network call. Generating a UUID in memory is insufficient if a worker can crash before saving it.

Add an “unknown” state

A small application state machine can be enough:

prepared -> submission_unknown -> accepted -> terminal
          \-> rejected

“Submission unknown” means that the next action is recovery. It should not be presented as proof that the model failed.

An operation can enter that state when a socket closes, a client deadline expires, or a response cannot be parsed. Even a successful server response can be lost before your application commits the task ID.

Once the remote task ID is durable, stop resubmitting. Follow that task through callbacks or status reads. These are two separate problems: recovering acceptance and waiting for completion.

Freeze the submission while recovering it

For SpicyAPI media tasks, reuse the same Idempotency-Key, API key and request semantics to recover an accepted submission. The replay window lasts 24 hours from acceptance; it is not a permanent deduplication service. Model, input or callback changes make it a different request. See the idempotency contract.

That has practical consequences:

• Do not regenerate a key inside a retry loop.
• Do not reconstruct a prompt from mutable project settings during recovery.
• Do not switch credentials and assume the old task will be visible.
• Do not keep an automatic recovery queue running beyond the service's replay window.

Use an application deadline much shorter than that window. If recovery is still unresolved, flag the operation for inspection instead of quietly submitting a fresh job.

Keep one HTTP attempt small

Here is a deliberately limited Node.js helper. It makes one attempt and reports what the caller actually knows. A durable worker, rather than this function, decides when to try the saved operation again.

operation.requestBody is a saved JSON string. Build it from the current model schema and your confirmed request before calling this function. The example does not select a model or start a generation by itself.

export async function sendSavedOperation(operation, apiKey, send = fetch) {
  let response;
  let envelope;

  try {
    response = await send(
      'https://api.spicyapi.ai/api/v1/jobs/createTask',
      {
        method: 'POST',
        headers: {
          Authorization: `Bearer ${apiKey}`,
          'Content-Type': 'application/json',
          'Idempotency-Key': operation.idempotencyKey,
        },
        body: operation.requestBody,
        signal: AbortSignal.timeout(20_000),
      },
    );
    envelope = await response.json();
  } catch {
    return { kind: 'unknown', operationId: operation.id };
  }

  const taskId = envelope?.data?.taskId;
  if ([200, 202].includes(response.status)
      && envelope?.code === 200 && taskId) {
    return { kind: 'accepted', taskId, requestId: envelope.request_id };
  }

  // This is a response to classify, not permission to create another job.
  return {
    kind: 'response_to_review',
    httpStatus: response.status,
    businessCode: envelope?.code,
    requestId: envelope?.request_id,
  };
}

The helper intentionally does not label every non-success response “safe to retry.” Invalid input, authentication failures, spend limits and conflicting keys need different handling. Rate limits require bounded backoff; follow the service's retry guidance. HTTP status alone is often insufficient when an API also has business error codes.

After an accepted result, commit the task ID before acknowledging your own queue message. After unknown, retain the same saved operation for recovery. Do not catch every exception and create a new operation.

For multiple workers, claim or lease the operation atomically. Idempotency at the remote API does not prevent two local workers from sending duplicate emails or writing two output records.

Do not confuse acceptance, completion and payment

In this protocol, an HTTP 202 response acknowledges a submitted task. It does not say that generation succeeded. The task's state describes the generation outcome, while settled tells you whether its reported cost is final. A local timeout does not cancel accepted work. The production integration guide describes those boundaries and cost confirmation.

Your interface can reflect this without exposing protocol jargon:

• “Checking whether your generation started…” during uncertain submission.
• “Generating…” after acceptance.
• A completed or failed result only after reading a terminal task state.

An explicit “Generate another take” action should create another operation with a new key. A “Check progress” action should keep following the existing one.

Test the lost response, not only a failed connection

A mock that always throws before accepting anything misses the difficult case. Make your fake server remember the operation, create one task, and then drop the first response. Send the saved operation again.

The useful assertions are:

1. Two HTTP attempts carry the same key and body.
2. The fake server creates exactly one remote task.
3. The recovered task ID is committed locally.
4. A process restart reads the old operation instead of generating another key.
5. Two callbacks for that task produce one application-side effect.
6. An operation beyond its recovery deadline is escalated rather than resubmitted automatically.

Also test a changed body with the old key. A conflict should be visible to your operator, not “fixed” by silently inventing a new key and spending money.

The key question for each failure path is simple: what do we know, and which durable record lets the next worker continue from there?

Written by the SpicyAPI Team. Examples describe our media task API; streaming and chat endpoints have different replay behavior.

Comments

[Qingfeng]

The prepared -> submission_unknown -> accepted split is the part retry helpers usually skip, and it is where the money goes. We hit the same wall from the other side: an agent running locally on the user machine that wants to re-issue a tool call after a 30s timeout (we build one, agent-home.ai). Giving unknown its own state is what makes fail-closed possible - the run parks and asks instead of guessing "probably failed" and doing it twice. Two things I would add to your state machine: a repair budget so unknowns cannot pile up silently, and a receipt a human can actually read, because "the agent did something we are not sure about" is not a status anyone can act on. Do you cap recovery attempts per operation or per time window?