Parallel Tool Calls in Node: Promise.all, Rate Limits, Partial Failure

#typescript #agents #ai #node

Book: AI That Acts
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Models emit several tool calls in a single turn when the calls are
independent. Ask about four orders and you get four tool_use
blocks at once, which is the model being efficient.

The obvious way to run them is the wrong one.

const results = await Promise.all(
  toolUses.map((b) => execute(b, ctx)),
);

Three succeed, one throws, and Promise.all rejects. You have
discarded three completed calls — three round trips you paid for —
and the turn fails. The model never learns that three quarters of
what it asked for worked.

Settle, do not race to the first failure

const settled = await Promise.allSettled(
  toolUses.map((b) => execute(b, ctx)),
);

const results: ToolResultBlockParam[] = settled.map((s, i) =>
  s.status === "fulfilled"
    ? s.value
    : {
        type: "tool_result",
        tool_use_id: toolUses[i].id,
        content: `Tool failed: ${describe(s.reason)}`,
        is_error: true,
      },
);

Every call produces a result block. Successes carry data, failures
carry a message. The model receives an accurate picture of what
happened and can decide — retry the one that failed, answer with
three quarters of the data, or tell the user what is missing.

This is the shape the protocol expects. Every tool_use in an
assistant turn needs a matching tool_result in the next user turn;
a missing one is a malformed request, not a degraded one.

Match by id, never by position

The mapping above uses toolUses[i], which is correct because
allSettled preserves order. It is also fragile: the moment someone
filters, sorts, or groups the array, the correspondence breaks
silently and results attach to the wrong calls.

Carry the id through instead.

type Outcome = { id: string; result: ToolResultBlockParam };

const settled = await Promise.allSettled(
  toolUses.map(async (b): Promise<Outcome> => ({
    id: b.id,
    result: await execute(b, ctx),
  })),
);

Now a reordering cannot misattribute anything. This matters more than
it looks: attaching order data to the wrong tool_use_id produces an
agent that confidently reports the wrong customer's information, and
nothing anywhere throws.

Unbounded parallelism is its own failure

Promise.allSettled over whatever the model emitted means the model
decides your concurrency. Usually that is two to five calls. Give an
agent a list of thirty ids and it may fan out to thirty.

Thirty simultaneous requests to an internal API sized for normal
traffic is a self-inflicted load spike, and the rate limiter will
answer with 429s that arrive as tool failures.

import pLimit from "p-limit";

const limit = pLimit(4);

const settled = await Promise.allSettled(
  toolUses.map((b) => limit(() => execute(b, ctx))),
);

Better: per-tool limits, because a database read and a payment API
have nothing in common.

const LIMITS = {
  search_docs: pLimit(8),
  get_order: pLimit(4),
  charge_card: pLimit(1),
} as const;

const runner = (b: ToolUseBlock) =>
  (LIMITS[b.name as keyof typeof LIMITS] ?? pLimit(2))(
    () => execute(b, ctx),
  );

charge_card at one is deliberate. Side-effecting tools should not
run concurrently with themselves, whatever the model asked for.

The ordering assumption models make

Parallel execution means no ordering guarantee, and the model may
have assumed one.

tool_use: create_invoice { customerId }
tool_use: send_invoice_email { invoiceId: "?" }

That second call cannot be right — the invoice does not exist yet.
Usually a model will sequence dependent calls across turns. When it
does not, running both in parallel produces a confusing failure.

Enforce the constraint in your executor rather than hoping:

const SEQUENTIAL = new Set(["create_invoice", "send_invoice_email"]);

const [seq, par] = partition(toolUses, (b) => SEQUENTIAL.has(b.name));

const parOut = await Promise.allSettled(par.map(runner));
const seqOut: PromiseSettledResult<Outcome>[] = [];
for (const b of seq) {
  seqOut.push(await settle(() => runner(b)));
}

Writes in order, reads in parallel, results merged. The set of
sequential tools is small and you will know which ones belong in it.

Timeouts belong per call

One slow tool holds the whole turn. Promise.allSettled waits for
everything.

async function execute(b: ToolUseBlock, ctx: Ctx) {
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), TIMEOUTS[b.name] ?? 10_000);
  try {
    return await runTool(b, { ...ctx, signal: ac.signal });
  } finally {
    clearTimeout(timer);
  }
}

The finally matters in Node. An uncleaned timer keeps the event
loop alive, and a process that will not exit after its work is done
is a genuinely annoying thing to debug later.

Pass the signal down to fetch so the abort actually cancels the
request rather than just abandoning the promise.

Budget the combined result size

Four parallel calls return four payloads into one message. Each may
be individually reasonable and collectively enormous.

function fitResults(
  results: ToolResultBlockParam[],
  budget = 40_000,
): ToolResultBlockParam[] {
  const share = Math.floor(budget / results.length);
  return results.map((r) => {
    const s = String(r.content);
    if (s.length <= share) return r;
    return {
      ...r,
      content: s.slice(0, share) +
        `\n[truncated ${s.length - share} chars — narrow the query]`,
    };
  });
}

An equal share is crude and predictable, which is what you want.
Saying the truncation happened, and suggesting what to do, keeps the
model from treating a cut list as a complete one.

Assembling the turn

export async function runToolTurn(res: Message, ctx: Ctx) {
  const uses = res.content.filter(
    (b): b is ToolUseBlock => b.type === "tool_use",
  );
  if (!uses.length) return null;

  const outcomes = await executeAll(uses, ctx);

  const byId = new Map(outcomes.map((o) => [o.id, o.result]));
  const results = uses.map((u) => {
    const r = byId.get(u.id);
    if (!r) {
      return {
        type: "tool_result" as const,
        tool_use_id: u.id,
        content: "Tool did not run.",
        is_error: true,
      };
    }
    return r;
  });

  return { role: "user" as const, content: fitResults(results) };
}

The final uses.map rebuilds in the model's original order and
guarantees one result per call, even if something upstream dropped an
outcome. That last guard has never fired in code I would ship — and
it costs four lines to make the malformed-request failure impossible.

The rule

Independent calls run in parallel, with a concurrency cap you choose.
Failures come back as results rather than exceptions. Everything is
matched by tool_use_id. Writes are sequential. The combined payload
has a budget.

Promise.all gets exactly one of those right.

If this was useful

AI That Acts covers the
execution layer properly — parallel calls, partial failure,
concurrency limits, timeouts, and keeping tool results inside a
context budget.