DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Prompt parallelization: fan one input out to N prompts at once, then aggregate — buy latency or reliability for N the tokens

#ai #promptengineering #llm #beginners

Most prompt pipelines grind through everything in one long serial pass, or cram a whole job into a single bloated prompt. Both waste something. Parallelization fixes both by fanning the same input out to several prompts that run at the same time, then combining their outputs with an aggregator. It's map/reduce for prompts, and once you see it that way it's really just a Promise.all and a reduce. I built an interactive demo that fires the calls and shows the wall-clock and reliability wins in real time — here's the shape.

The slow serial version you're replacing

The thing that almost works: run each piece one after another with await in a loop. It's correct, but every call blocks the next even though none of them depend on each other, so the wall clock is the sum of all the calls.

// ❌ each independent call blocks the next -> wall clock = SUM
async function serial(subtasks, input) {
  const parts = [];
  for (const t of subtasks) parts.push(await llm(t.prompt + input));
  return merge(parts);
}
// 4 calls of ~1.5s each -> ~6s, even though none depend on another.
Enter fullscreen mode Exit fullscreen mode

Flavor one — sectioning (a latency win)

Break the task into pieces that don't need each other's output and give each its own small, focused prompt. A launch brief becomes: headline, features, FAQ, tweet — four narrow prompts instead of one that's thin and rushed at every piece. Because they're independent, they run at once, and the wall clock drops from the sum to the slowest single call. That's the whole trick — start them all without awaiting, then await Promise.all once.

// ✅ parallel fan-out: start them all, then await once. Clock = MAX, not SUM.
async function fanOut(subtasks, input) {
  const calls = subtasks.map(t => llm(t.prompt + input)); // start ALL now
  return await Promise.all(calls);                        // wait for the slowest
}
// 4×1.5s serially = ~6s;  fanned out = ~1.5s. Same tokens, 4× less waiting.
Enter fullscreen mode Exit fullscreen mode

Flavor two — voting (a reliability win)

Same fan-out, but every call is the identical prompt, run warm so samples vary. A single sample can be a coin-flip; run it N times and take the majority, and the random noise cancels — a shaky one-shot becomes a confident, self-consistent answer. The best part: the winning vote share is a confidence score for free. In the demo's bat-and-ball question, two of five samples fall for the classic $0.10 trap; the majority still lands on the correct $0.05.

async function vote(prompt, n = 5) {
  const runs = Array.from({ length: n }, () => llm(prompt)); // identical prompt
  const votes = await Promise.all(runs);                     // all at once
  return majority(votes.map(parse));   // { answer, confidence } — noise cancels
}
Enter fullscreen mode Exit fullscreen mode

The aggregator is the reduce

Fan-out is only half of it; the results have to come back together. For sectioning that's a merge — stitch the sections into one document, sometimes with a final synthesis pass. For voting it's a tally — count the answers, take the majority, and read the confidence off the winning share.

const merge = (parts) => parts.join("\n\n");        // sectioning: stitch pieces

function majority(votes) {                          // voting: tally + confidence
  const c = {}; votes.forEach(v => c[v] = (c[v]||0) + 1);
  const winner = Object.keys(c).sort((a,b)=>c[b]-c[a])[0];
  return { answer: winner, confidence: c[winner] / votes.length };
}
Enter fullscreen mode Exit fullscreen mode

Both flavors, one shape — and the limits

Sectioning and voting are the same two moves: fanOut → aggregate. The demo lets you flip between them and toggle sequential vs parallel, and a little Gantt chart makes the point visceral — the parallel row is just the length of the slowest call plus one aggregator step, while the sequential row stacks end to end.

async function parallelize(input, mode) {
  if (mode === "section") return merge(await fanOut(SUBTASKS, input)); // latency
  if (mode === "vote")    return vote(input, 5);                       // reliability
}
// only for INDEPENDENT work; costs N× tokens; cap concurrency vs rate limits.
Enter fullscreen mode Exit fullscreen mode

The trade-off is flat and worth stating plainly: N calls means roughly N× the tokens. So reach for parallelization when the pieces are genuinely independent (sectioning) or when a correct answer is worth several tries (voting) — not to speed up steps that must feed each other in order. That's chaining, a different pattern entirely, and provider rate limits will throttle a big fan-out, so cap your concurrency. This is a core agent building-block: the next step up is an orchestrator that dynamically decides the subtasks and hands them to worker models, where here the split is fixed.

Pick a flavor and watch the calls fire, with the wall-clock and token cost side by side:
https://dev48v.infy.uk/prompt/day37-parallelization.html

Top comments (0)