This was an extraction endpoint for support tickets. A request comes in, the model turns the message into { category, priority, summary }, and the API returns the validated object to the dashboard.
It worked until the provider got slow.
The dashboard waited behind a model call with no upper bound. A slow request held the HTTP connection open, the gateway eventually gave up, and the model call was still running somewhere behind it. The user saw an error, but the provider still finished generating an answer nobody would use.
My first timeout only timed out my code
The obvious patch was Promise.race():
const result = await Promise.race([
generate(model, TicketSchema, message),
new Promise((_, reject) =>
setTimeout(() => reject(new Error("LLM timeout")), 5_000)
),
]);
That made the API return after five seconds. It did not cancel generate(). The losing promise kept its reference to the provider request, and the provider kept doing work. On a busy endpoint, a timeout stopped being a user-experience problem and became a concurrency problem: requests the application had abandoned were still consuming connections and model capacity.
I tried adding a retry around the race. That made the numbers worse. A slow first call kept running while the retry started a second call, so one user request could turn into two active generations. Retrying a timeout also felt wrong: the model had not produced a bad shape; the backend had simply not finished within the budget.
The timeout belongs inside the generation call
I was already using shapecraft for the structured extraction, so I checked whether its options could carry cancellation into the model layer:
import { generate, TimeoutError } from "@aviasole/shapecraft";
try {
const result = await generate(model, TicketSchema, message, {
timeoutMs: 5_000,
maxRetries: 3,
});
return result.data;
} catch (error) {
if (error instanceof TimeoutError) {
return { status: "deferred", reason: "model took too long" };
}
throw error;
}
The important detail is that timeoutMs is enforced by the core, not left entirely to the backend. The core races the model call against its own timeout guard, so generate() stops waiting after the configured number of milliseconds even if a custom model implementation ignores the signal.
For a backend that does honor AbortSignal, the same signal is passed through to the underlying request too. That lets an HTTP-based provider or SDK actually cancel the request rather than merely letting shapecraft stop awaiting it.
Timeout errors are deliberately not retries
I initially assumed maxRetries: 3 meant the call could take roughly three times the timeout. It does not in this case, and that is the behavior I wanted once I understood it.
Shapecraft retries SchemaViolationError, because a model that returned the wrong shape may succeed when asked again. TimeoutError is a different kind of failure. Asking the same slow backend again immediately does not repair a timeout, so it propagates on the first timed-out attempt:
const result = await generate(model, TicketSchema, message, {
timeoutMs: 5_000,
maxRetries: 3, // applies to schema failures, not a timeout
});
That distinction gave me a predictable boundary: malformed output can use the retry budget, but a slow call exits through the timeout path without multiplying the work.
Cancellation is a different signal from a deadline
A timeout is an application policy. An AbortSignal is useful when the caller has already decided the work no longer matters, such as when a browser tab closes or an HTTP client disconnects:
const controller = new AbortController();
const deadline = setTimeout(() => controller.abort(), 5_000);
try {
const result = await generate(model, TicketSchema, message, {
signal: controller.signal,
});
return result.data;
} finally {
clearTimeout(deadline);
}
The caller can also abort with its own reason:
controller.abort(new Error("client disconnected"));
That reason is propagated to the caller instead of being turned into a TimeoutError. An already-aborted signal fails before the backend is called at all, which matters when a request has already gone away before the generation starts.
The boundary I needed to document
Cancellation still depends on the backend underneath. Built-in cloud backends forward the signal to their SDK or fetch call. llamaCpp() can still be bounded by shapecraft's core, but local inference does not get an underlying network request to cancel. The result is still returned to my application within the core timeout, but the backend's own cancellation behavior is a separate capability.
That was the useful mental model: a timeout always bounds how long my application waits; a backend that honors the signal can also stop the work itself.
Where it landed
The endpoint now has a five-second upper bound, timeout failures do not create duplicate generations, and client disconnects can abort work that no longer has a consumer. The structured-output retry loop still handles malformed answers, but it no longer treats a slow provider as if it had made a formatting mistake.
If you are wrapping an LLM call in Promise.race(), check whether the losing operation is still alive. Passing timeoutMs or an AbortSignal into the generation layer gave me both the response deadline and a clean cancellation path:
await generate(model, TicketSchema, message, { timeoutMs: 5_000 });
The API stopped waiting at five seconds. That was the part I could guarantee. Whether the provider stopped doing the work depended on whether it honored the signal.
Top comments (1)
Solid overview of a problem that bites way more people than admit it. One thing I'd add from running this exact pattern in production: Promise.race() also leaks the loser's timers, so if you build the race manually, keep a handle to the setTimeout and clearTimeout it in a finally block — otherwise every abandoned generation holds a timer reference until it fires and keeps the Node event loop alive during graceful shutdown.
The retry distinction you describe is worth committing to a team convention: our rule became "retry only on structured/parseable failure, fail fast on time". One extra gotcha — when you do pass an AbortSignal down to fetch, remember that aborting mid-stream doesn't give you a partial completion event unless you handle response.body readers explicitly. We log whether the provider actually honored the cancel (connection closed vs request completed server-side) so we can tell "bounded latency" from "bounded billing". They are not the same number.