Next.js API Proxy Times Out After Long ML Inference (502) — Navigating Undici's Timeout Quagmire

📝 Originally published (in Japanese) at forge.workstyle.tech.

Relaying API Requests from Next.js Frontend to a Separate Inference Service (FastAPI) via API Routes — A Common BFF Pattern

When using Next.js to relay requests from the frontend to an inference service (FastAPI) running in a separate process via API routes, it's a common BFF (Backend for Frontend) architecture. However, when running audio generation with a 44.1kHz model on the CPU, processing can take several minutes — or even hours. And then, at some point, this happens:

The frontend receives a 502 error even though the generation hasn't finished.

Checking the inference service logs shows that processing is still running smoothly. The issue isn't with the inference itself — the bottleneck is the proxy in between. This article documents how we identified that the culprit was Next.js's global fetch (powered by undici) and its default timeout, and how we rewrote the proxy using Node's standard http/https modules to bypass it.

---

Symptoms: Only Long-Running Generation Fails with 502 — and Always at the Same Time

The key observations during debugging were:

• Short generations (a few seconds to tens of seconds) work fine
• Only long-running generations fail with 502
• And they always fail after "about the same amount of time"

The inference service continues running, but the proxy layer gives up first. The fact that it fails "after a fixed time" is a strong hint: there’s a hardcoded timeout somewhere.

---

Root Cause: Default Timeout in Global fetch (undici)

When you use fetch() directly in a Next.js API route, under the hood it uses undici, Node.js’s built-in HTTP client. Undici has a default timeout for receiving headers (around 300 seconds), and if no response comes back within that window, it forcibly closes the connection. When inference takes more than 5 minutes, it gets cut off right there — resulting in a 502.

"Just disable undici’s timeout then!"

We tried adjusting headersTimeout/bodyTimeout via an Agent, but ran into another wall: it’s hard to directly import and inject undici in this setup. Modifying the internal implementation of global fetch isn’t clean, and the override may not work across environments.

Temporarily increasing the timeout just kicks the can down the road — eventually, another long-running generation will hit the same limit. The real fix wasn’t to tweak the timeout — it was to rewrite the proxy layer using a different timeout model entirely.

---

Solution: Rewrite the Proxy Using Node’s Standard http/https

Instead of fighting with undici, we rewrote the proxy using Node’s built-in http/https modules. With standard modules, we gain full control over timeout behavior.

The key design principle: distinguish between connection setup and response waiting.

• We want to fail fast if the target isn’t reachable (TCP connection) → connection timeout: 30 seconds
• Once connected, we’re willing to wait however long it takes → no timeout on response

This separation is critical.

// Disable response timeout entirely (allow long-running generation). Only enforce connection timeout.
preq.setTimeout(0);
preq.on('socket', (s) => {
  s.setTimeout(30_000, () => {
    if (!s.destroyed && (s.connecting || !s.writable)) s.destroy(new Error('connect timeout'));
  });
  s.once('connect', () => s.setTimeout(0));
});

Breaking it down:

• preq.setTimeout(0) — disables the overall request timeout, allowing the response to be awaited indefinitely
• On the socket, we set a 30_000ms timeout that triggers only if the socket is still connecting or not writable — i.e., only when the target is unreachable
• Once connect fires, we immediately disable the socket timeout (setTimeout(0)), letting the response stream in however long it takes

This way:

• If the service is down (can’t connect), we fail fast with 502
• If the service is up but the inference takes 20 minutes, we wait patiently

We also added proper error handling:

preq.on('error', (e) => {
  if (!res.headersSent) res.status(502).json({ error: `backend API unreachable: ${String(e)}` });
  resolve();
});

---

Another Trap: Next.js Response Monitoring Warnings

Even after removing the timeout, Next.js API routes have a built-in mechanism that warns when a handler takes too long to respond. To suppress this warning for long-running proxies, we declare:

export const config = {
  api: {
    bodyParser: { sizeLimit: '25mb' },
    responseLimit: '25mb',
    externalResolver: true, // Tell Next.js: "This route resolves externally; don't monitor it"
  },
};

We also increased the body and response size limits since we’re dealing with audio data — the defaults would reject large recordings.

---

Pitfalls & Lessons Learned

• fetch uses undici under the hood. If you casually use fetch in API routes without realizing it has default timeouts, you’ll end up with a hard-to-debug scenario: "The service is running, but the proxy returns 502."
• Don’t disable timeouts globally — split them by purpose. By separating "connection setup" from "response waiting," we can fail fast on unreachable services while waiting indefinitely for long-running tasks. Disabling all timeouts risks hanging forever on a dead service.
• Going back to standard modules gives you control. Instead of wrestling with undici’s internal settings, dropping down to http/https and managing the socket directly is more reliable and readable for this kind of requirement.
• Don’t forget to set externalResolver: true for long-running API routes.

---

Summary

• When proxying long-running ML inference via Next.js API routes, the global fetch (undici) default timeout (~300s) causes 502 errors
• In setups where injecting undici directly is difficult, rewriting the proxy with Node’s http/https is the most reliable fix
• Split timeouts logically: 30s connection timeout, no timeout on response
• Switch to setTimeout(0) on connect to fail fast only when unreachable
• Use externalResolver: true to suppress Next.js monitoring warnings, and increase body limits for audio data