AI API call hangs in production, then 504: timeouts and budgets

We read 1,662 public posts from builders whose apps broke at or after launch and verified 215 recent cases. AI model and API failures make up 4% of them. Six of the verified cases share one pattern: an outbound or AI call with no timeout, no budget and no fallback. In most of the six, nobody noticed for a while, because the app didn't log model calls or their outcomes.

From the builder's side it shows up as one of these:

• A spinner that runs for minutes, then a 504 from Vercel or a 524 from Cloudflare. On localhost it was fine.
• Users read polite fallback text while the spend dashboard looks normal, because failed calls cost almost nothing.
• An agent or background job burns a week's budget in a day. Then the provider's spend limit trips and every user is cut off at once.

All three have the same cause. The call has no limits of its own, so the only limits that apply are set by other layers.

Reproduce it

Here is a Next.js App Router route using the OpenAI Node SDK with its defaults. Most generated code ships in this shape.

// app/api/summary/route.ts
import OpenAI from "openai";

const openai = new OpenAI(); // defaults: 10 min per attempt, 2 retries

export async function POST(req: Request) {
  const { text } = await req.json();
  try {
    const r = await openai.chat.completions.create({
      model: "gpt-5-mini",
      messages: [{ role: "user", content: `Summarise this:\n${text}` }],
    });
    return Response.json({ summary: r.choices[0].message.content });
  } catch {
    // the user sees this with a 200, and nothing is logged
    return Response.json({ summary: "Summary unavailable right now." });
  }
}

You don't need a provider outage to see it fail. The SDK reads OPENAI_BASE_URL, so you can point it at a server that accepts connections and never replies:

# terminal 1: a provider that never answers
node -e "require('http').createServer(() => {}).listen(4010)"

# terminal 2
OPENAI_BASE_URL=http://127.0.0.1:4010/v1 OPENAI_API_KEY=test npm run dev

# terminal 3
curl -s -X POST localhost:3000/api/summary \
  -H 'content-type: application/json' -d '{"text":"hello"}' -w '\n%{time_total}s\n'

curl waits for at least 15 minutes. On Vercel, the same request is killed at 300 seconds with a 504 FUNCTION_INVOCATION_TIMEOUT before the catch runs, so nothing is logged. Behind Cloudflare the user gets a 524 even sooner.

Now make the stub answer 500 instead: node -e "require('http').createServer((q, s) => { s.statusCode = 500; s.end('{}') }).listen(4010)". After a few seconds of retries the route returns 200 with the fallback sentence. Every monitor you have counts that as a success.

Why it happens

Every layer between the browser and the model has its own clock. In the default setup the clock in your own code is the longest, so an outer layer always runs out first.

Layer	Default limit	What happens
Cloudflare proxy	125 s with no response from the origin	524 to the user
Vercel function (fluid compute)	maxDuration 300 s	function terminated, 504
Node's fetch (undici)	300 s waiting for response headers	connection error
OpenAI and Anthropic Node SDKs	10 minutes per attempt	APIConnectionTimeoutError

Here is what the official docs say about each part:

• The SDK timeout is per attempt, and retries multiply it. By default, both official Node SDKs retry connection errors, 408, 409, 429 and 5xx twice with a short backoff, and they retry timed-out requests too. That is where the 15 minutes comes from: three attempts, each waiting at least 300 seconds. For non-streaming requests with a large max_tokens, Anthropic's SDK raises its default timeout above 10 minutes, up to 60.
• A non-streamed response sends nothing until the model finishes. Every proxy in front of it sees an idle connection. Anthropic's SDK docs recommend streaming for long requests for this reason. The SDK also throws an error for a non-streaming request expected to run past roughly 10 minutes, unless you set a timeout.
• The platform ends the process, not the promise. Vercel terminates a function that runs past its maxDuration. Your catch never runs, and neither does the logging inside it.
• The fallback hides the errors you did catch. A catch that returns a 200 turns every provider error into a success in your metrics. Failed calls use few tokens, so the cost chart stays flat.
• Nothing caps spend per user or per run. Provider spend limits are monthly and cover a whole project or workspace. When OpenAI's hard limit is reached, requests return 429 with the code project_spend_limit_exceeded. Because the SDKs retry every 429, each user's request retries an error that won't clear until the limit resets. Anthropic returns 400 invalid_request_error when a workspace spend limit is reached, and retry logic that only looks for 429 won't recognise it.

Catch it before your users do

This suite starts a fake provider on localhost with three behaviours: it never answers, it plays an agent that never finishes, or it returns a spend-limit 429. It imports the wrapper from the next section, and its assertions define what that wrapper has to do. It needs no API key and costs nothing to run.

// lib/ai-guards.test.ts
import http from "node:http";
import type { AddressInfo } from "node:net";
import { afterAll, beforeAll, expect, test } from "vitest";
import { callModel, makeClient, newRun, runAgent } from "./ai";

process.env.OPENAI_API_KEY ??= "test";
let hits = 0;

// One fake provider; the URL prefix picks its behaviour.
const server = http.createServer((req, res) => {
  hits++;
  if (req.url?.startsWith("/stall/")) return; // accept, never answer
  res.setHeader("content-type", "application/json");
  if (req.url?.startsWith("/spent/")) {
    res.statusCode = 429;
    return res.end(JSON.stringify({ error: { code: "project_spend_limit_exceeded", message: "limit reached" } }));
  }
  res.end(JSON.stringify({ // "/loop/": a model that never says it is done
    id: "c1", object: "chat.completion", created: 0, model: "stub",
    choices: [{ index: 0, finish_reason: "stop", message: { role: "assistant", content: "keep going" } }],
    usage: { prompt_tokens: 500, completion_tokens: 500, total_tokens: 1000 },
  }));
});
const client = (mode: string) =>
  makeClient(`http://127.0.0.1:${(server.address() as AddressInfo).port}/${mode}/v1`);
const hi = [{ role: "user" as const, content: "hi" }];

beforeAll(() => new Promise<void>((done) => { server.listen(0, "127.0.0.1", done); }));
afterAll(() => { server.closeAllConnections(); server.close(); });

test("a provider that never answers fails inside the deadline", async () => {
  const t0 = Date.now();
  await expect(callModel(client("stall"), newRun(), hi, 2_000)).rejects.toThrow();
  expect(Date.now() - t0).toBeLessThan(3_000);
});

test("an agent that never finishes stops at its step cap", async () => {
  const run = newRun(5, 1);
  await expect(runAgent(client("loop"), run, "plan my week")).rejects.toThrow("run_budget_exceeded");
  expect(run.steps).toBe(5);
});

test("a spend-limit 429 surfaces its code after at most one retry", async () => {
  hits = 0;
  await expect(callModel(client("spent"), newRun(), hi)).rejects.toMatchObject({
    status: 429, code: "project_spend_limit_exceeded",
  });
  expect(hits).toBeLessThanOrEqual(2);
});

npm i -D vitest
npx vitest run lib/ai-guards

Each test takes a few seconds. Run the suite in CI next to your other tests.

The fix

Give every call its own limits, shorter than every layer outside it, and make failures visible.

// lib/ai.ts
import OpenAI from "openai";

export type Run = { steps: number; maxSteps: number; spentUsd: number; capUsd: number };
export const newRun = (maxSteps = 1, capUsd = 0.05): Run => ({ steps: 0, maxSteps, spentUsd: 0, capUsd });

// 20 s per attempt, one retry. The deadline in callModel caps the total.
export const makeClient = (baseURL?: string) => new OpenAI({ baseURL, timeout: 20_000, maxRetries: 1 });

// Set these from your model's price page; at 0 the spend cap never trips.
const USD_PER_TOKEN = { input: 0, output: 0 };

export async function callModel(
  client: OpenAI,
  run: Run,
  messages: OpenAI.Chat.ChatCompletionMessageParam[],
  deadlineMs = 45_000,
) {
  if (run.steps >= run.maxSteps || run.spentUsd >= run.capUsd) throw new Error("run_budget_exceeded");
  run.steps++;
  const res = await client.chat.completions.create(
    { model: "gpt-5-mini", messages, max_completion_tokens: 800 },
    { signal: AbortSignal.timeout(deadlineMs) }, // one clock for the whole call, retries included
  );
  const u = res.usage;
  run.spentUsd += (u?.prompt_tokens ?? 0) * USD_PER_TOKEN.input + (u?.completion_tokens ?? 0) * USD_PER_TOKEN.output;
  return res;
}

export async function runAgent(client: OpenAI, run: Run, task: string) {
  const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [{ role: "user", content: task }];
  for (;;) {
    const res = await callModel(client, run, messages);
    const text = res.choices[0].message.content ?? "";
    if (text.includes("[done]")) return text;
    messages.push({ role: "assistant", content: text }, { role: "user", content: "Continue." });
  }
}

The route now fails loudly and leaves a record:

// app/api/summary/route.ts
import { callModel, makeClient, newRun } from "@/lib/ai";

const client = makeClient();

export async function POST(req: Request) {
  const { text } = await req.json();
  const t0 = Date.now();
  try {
    const r = await callModel(client, newRun(), [{ role: "user", content: `Summarise this:\n${text}` }], 25_000);
    console.info(JSON.stringify({ tool: "summary", status: "ok", ms: Date.now() - t0, usage: r.usage }));
    return Response.json({ summary: r.choices[0].message.content });
  } catch (err: any) {
    console.error(JSON.stringify({ tool: "summary", status: "error", code: err?.code ?? err?.name, http: err?.status, ms: Date.now() - t0 }));
    return Response.json({ error: "summary_failed" }, { status: 503 });
  }
}

Pick deadlineMs below both your proxy's timeout and your function's maxDuration, with time left over to log and respond. If an answer really needs longer, stream it so bytes reach the proxy early, or return a job ID and have the client poll. With streaming, a 200 only means the stream started. Log the call as failed if the stream ends without a finish_reason.

For a per-user daily cap, add up the user's spend for today from the same log rows before creating the run, and pass what's left as capUsd. Once those rows are in a table, this query shows the failures you already have. It sorts by failure rate rather than spend, because failed calls barely move a cost chart:

select tool, count(*) as calls,
       avg((status <> 'ok')::int) as failure_rate
from ai_calls
where created_at > now() - interval '7 days'
group by tool
order by failure_rate desc;

Checklist

• [ ] Every provider client sets timeout and maxRetries explicitly.
• [ ] Every call has one overall deadline (AbortSignal.timeout), shorter than maxDuration and your proxy's timeout.
• [ ] Long answers stream, or run as a background job that the client polls.
• [ ] Every agent run has a step cap and a spend cap. Every user has a daily cap, checked before the call.
• [ ] A failed call returns a non-2xx status and writes one log line with tool, status, error code and duration.
• [ ] OpenAI's 429 project_spend_limit_exceeded and Anthropic's spend-limit 400 show the user a clear paused state instead of starting a retry loop.
• [ ] A provider hard spend limit is set as a backstop, not used as the budget.
• [ ] The three tests above run in CI.

Gemmein (in beta) handles the per-user half of this for you: its AI tools spend or reserve a person's credits before the provider is called, and every call is recorded with who, tool, tokens, credits and outcome.

Which layer in your stack has the shortest clock today, and did you set it yourself?

Comments

[Seohyun Lee]

I can sense the frustration of dealing with AI API calls hanging in production, leading to timeouts and budget issues. It's like hitting a roadblock when you're trying to launch or scale your app. I've seen similar issues in my own work with AI services, where a small glitch can cause a ripple effect and impact the entire workflow. Have you explored any strategies for optimizing API call performance or implementing fail-safes to prevent these kinds of disruptions?

[Vlad Zoff]

The retry multiplication is the nasty part. A 5-minute outer timeout sounds safe until the SDK can spend that budget several times on the same request. I usually want one explicit deadline propagated through the whole path, rather than separate timeouts that don't know about each other.

[Ahmed]

Agreed, that's the trap. Each layer's timeout looks reasonable on its own, and the retries quietly multiply it. One deadline set at the edge and passed down (so every retry checks the time left, not a fresh timeout) is the cleanest fix I've found. Thanks for putting it so clearly.

[Doyoon Kim]

The 504s you observed often stem from unbounded token generation combined with a fixed request timeout—once the model exceeds the budget, the API stalls until the client aborts. In production I've mitigated this by adding a token‑limit guard on the LLM side and coupling it with a circuit‑breaker that falls back to a cached response or a lighter‑weight model. Have you tried instrumenting per‑call token usage metrics to dynamically adjust those limits?

[Mudassir Khan]

the silent failure mode is the one that compounds. the catch block returning 200 with "unavailable right now" means you have no signal that anything broke — spent budget, no log, customer never complains because they got a response.

the part worth adding to that code sample: log the exception before you return the fallback. doesn't fix the timeout but at least the spend dashboard and the error count move together when something breaks.

what did you find for per request timeouts on streaming completions? the AbortController pattern behaves differently there.

[Ahmed]

Good addition. Logging before the fallback is the minimum, otherwise the 200 hides everything. On streaming: a single total timeout fits badly, because a long answer that's still producing tokens looks the same as a hung one. What's worked better is two limits: a time-to-first-token deadline, then an idle timeout reset on every chunk, both firing the same AbortController. When it aborts, close the stream and log which limit tripped, so a slow start and a stall mid-answer show up as different errors.