- Book: AI That Ships
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
A CRUD endpoint answers in tens of milliseconds. An agent that makes
six model calls and hits three tools takes tens of seconds, sometimes
minutes.
Every layer between your user and your process was configured for the
first case. Load balancer idle timeouts, platform execution limits,
client fetch timeouts, and your own graceful-shutdown grace period —
all sized for requests that finish quickly.
None of them error usefully when an agent run exceeds them. You get a
504 with no logs, or a truncated stream, or a run that vanished
mid-execution.
Know your actual ceiling
Before designing around limits, find out what they are for your
platform and configuration. They vary widely between hosts, between
plan tiers, and between a streaming and a buffered response — and
they change.
The three that bind, in order:
Platform execution limit. Serverless functions have a hard
maximum. Exceed it and the process is killed mid-run with no chance
to clean up.
Proxy idle timeout. Load balancers close connections with no
bytes for N seconds. A model thinking for forty seconds before its
first token looks identical to a dead connection.
Client timeout. Browser fetch has no default timeout, but every
HTTP client wrapper adds one, and mobile networks add their own.
Check the current numbers for your platform rather than assuming.
Then design so the common case fits and the long case has somewhere
to go.
Streaming resets the clock
The change that buys the most: send bytes early and keep sending
them.
app.post("/agent", async (req, res) => {
res.writeHead(200, {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache, no-transform",
"X-Accel-Buffering": "no",
Connection: "keep-alive",
});
res.flushHeaders();
const beat = setInterval(() => {
if (!res.writableEnded) res.write(": ping\n\n");
}, 15_000);
try {
for await (const ev of runAgentStream(req.body.task, ctx)) {
res.write(`data: ${JSON.stringify(ev)}\n\n`);
}
res.write("data: [DONE]\n\n");
} finally {
clearInterval(beat);
res.end();
}
});
flushHeaders() sends the response head before any content, which
tells the proxy a response has begun. The heartbeat keeps the
connection non-idle during a long tool call.
no-transform and X-Accel-Buffering: no matter because a proxy
that buffers the whole response defeats all of this — the user waits
for everything, then receives it at once, and the bug report says
"streaming does not work" with nothing in your logs.
Streaming does not extend a platform's hard execution limit. It
solves idle timeouts, not wall-clock caps.
Past the ceiling, use a queue
When runs can exceed the hard limit, the request must stop being the
unit of work.
app.post("/agent", async (req, res) => {
const runId = crypto.randomUUID();
await queue.add("agent-run", { runId, task: req.body.task,
userId: req.user.id });
res.status(202).json({ runId, status: "queued" });
});
app.get("/agent/:runId", async (req, res) => {
const run = await store.get(req.params.runId);
if (!run) return res.status(404).end();
res.json(run);
});
202 Accepted with a run id. The worker has whatever runtime you
give it, the API stays fast, and a client that disconnects loses
nothing.
The worker writes progress as it goes, so the polling endpoint has
something real to return:
worker.process("agent-run", async (job) => {
const { runId, task } = job.data;
await store.set(runId, { status: "running", startedAt: Date.now() });
for await (const ev of runAgentStream(task, ctx)) {
if (ev.type === "turn") {
await store.update(runId, { turns: ev.turn, costUsd: ev.costUsd });
}
}
await store.update(runId, { status: "complete", result });
});
Combining both is usually right: stream for the runs that finish in
seconds, fall back to the queue for the ones that will not. Decide by
task type at the boundary rather than making users poll for
everything.
Graceful shutdown, sized for agents
Deploys are the most common way to lose an in-flight run, and the
default grace period is far too short.
let shuttingDown = false;
const inFlight = new Set<Promise<unknown>>();
process.on("SIGTERM", async () => {
shuttingDown = true;
server.close();
const deadline = Date.now() + 120_000;
while (inFlight.size && Date.now() < deadline) {
await Promise.race([
Promise.allSettled([...inFlight]),
new Promise((r) => setTimeout(r, 1000)),
]);
}
process.exit(0);
});
app.use((req, res, next) => {
if (shuttingDown) {
res.set("Connection", "close");
return res.status(503).json({ error: "shutting_down" });
}
next();
});
Stop accepting, finish what is running, then exit. Two configuration
details make or break it: your orchestrator's termination grace period
must be longer than this deadline, or it sends SIGKILL and the whole
handler is pointless. And the readiness probe must fail as soon as
shuttingDown flips, so no new traffic arrives.
For queued work, the same idea applies to the worker: stop taking new
jobs, let current ones checkpoint, then exit.
Cold starts
Cold starts matter less for agents than for normal endpoints — a
second of initialisation against a thirty-second run is noise. Two
things still help.
Create clients at module scope so they are reused across warm
invocations:
// module scope, not per request
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const pool = new Pool({ max: 5 });
And be careful with connection pools on serverless. Each instance
gets its own pool, so a max of 20 across 50 concurrent instances is
a thousand connections against a database configured for a hundred.
Size pools per instance with the fan-out in mind, or put a pooler in
front.
Abort on disconnect
If you stream, a client that leaves should stop the work.
const ac = new AbortController();
res.on("close", () => {
if (!res.writableEnded) ac.abort();
});
for await (const ev of runAgentStream(task, { ...ctx, signal: ac.signal })) {
// ...
}
The signal has to reach the model call and the tool calls, not just
the outer loop. An abort that stops your writes while the provider
request continues has saved nothing.
The deploy checklist
Before an agent goes to production:
Platform execution limit known, and runs that can exceed it routed to
a queue. Proxy idle timeout known, and heartbeats shorter than it.
Response headers set so nothing buffers. Graceful shutdown longer
than a typical run, with the orchestrator's grace period longer
still. Readiness failing on shutdown. Client disconnect aborting the
work. Pool sizes computed per instance.
None of these are about AI. They are the ordinary operational
concerns of long-running requests, which most Node services have
never had to have.
If this was useful
AI That Ships covers
deployment properly — streaming versus queued execution, timeouts at
every layer, graceful shutdown, and the observability that tells you
which limit you hit.
The full series is at
xgabriel.com/ai-in-typescript.
Top comments (0)