Introduction: The Serial Agent Bottleneck
As AI engineers build increasingly sophisticated autonomous systems, we frequently hit a frustrating wall: latency.
In conventional multi-step reasoning workflows, agents operate serially. An orchestrator receives a complex prompt, breaks it down, asks an LLM to evaluate the architecture, waits 2.5 seconds, then asks it to evaluate security implications, waits another 3 seconds, evaluates cost implications, waits another 2.5 seconds, and finally writes a synthesis.
Mathematically, serial execution compounds your wall-clock wait time:
$$\text{Wall-Clock Time}{\text{Sequential}} = T{\text{decompose}} + \sum_{i=1}^{N} T_{\text{worker}i} + T{\text{synthesis}}$$
If you have 4 analytical angles and each inference call takes 2.5 seconds, your user is staring at a loading spinner for over 12 seconds. In interactive web interfaces, 12 seconds is an eternity that degrades user retention.
To solve this, we built the Parallel Fan-Out Orchestrator—a lightweight, client-side application in TypeScript, React 19, and Tailwind CSS that proves how the Agentic Fan-Out / Fan-In pattern slashes total latency by 2.5× to 6× while delivering richer, multi-perspective outputs.
Here is the engineering breakdown of how we architected and implemented it.
The Core Concept: Fan-Out / Fan-In Parallelization
The Fan-Out / Fan-In pattern is borrowed from distributed computing and adapted for LLMs. Instead of a single model attempting to reason through multiple orthogonal dimensions sequentially, the pipeline is divided into three distinct stages:
- Decompose (Fan-Out Trigger): An Orchestrator Agent analyzes the original task and decomposes it into $N$ mutually exclusive, independently answerable subtask angles.
-
Execute (Parallel Fan-Out): All $N$ subtasks are dispatched concurrently across separate asynchronous HTTP request streams using
Promise.all. - Synthesize (Fan-In / Merge): A Merger Agent receives the collective findings, resolves contradictions, removes redundant boilerplate, and weaves them into a unified, high-density executive report.
Under parallel execution, wall-clock time is governed by the slowest worker, not the sum:
$$\text{Wall-Clock Time}{\text{Parallel}} \approx T{\text{decompose}} + \max(T_1, T_2, \dots, T_N) + T_{\text{synthesis}}$$
System Architecture
Here is the complete end-to-end data flow:
[ User Task Prompt ]
│
▼
┌───────────────────────────────┐
│ STAGE 1: DECOMPOSE │
│ (Orchestrator Agent) │
│ Breaks task into N subtasks │
│ via JSON Schema / Fallback │
└───────────────┬───────────────┘
│
┌─────────────────────────┼─────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────────────┐ ┌───────────────────────┐ ┌───────────────────────┐
│ WORKER 1 │ │ WORKER 2 │ │ WORKER N │
│ (Subtask Angle 1) │ │ (Subtask Angle 2) │ │ (Subtask Angle N) │
│ HTTP Stream Req 1 │ │ HTTP Stream Req 2 │ │ HTTP Stream Req N │
│ [performance.now()] │ │ [performance.now()] │ │ [performance.now()] │
└───────────┬───────────┘ └───────────┬───────────┘ └───────────┬───────────┘
│ │ │
└─────────────────────────┼─────────────────────────┘
│
(Promise.all)
│
▼
┌───────────────────────────────┐
│ STAGE 3: SYNTHESIZE │
│ (Merger Agent) │
│ Combines findings, resolves │
│ contradictions & deduplicates│
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Final Synthesized Answer │
│ + Wall-Clock Comparison │
└───────────────────────────────┘
Engineering Deep Dive: The Code
Let's look at how this is implemented cleanly in TypeScript without backend middleware.
1. Robust 3-Tier Task Decomposition
The Orchestrator must reliably split the user's prompt into exactly $N$ distinct subtasks. The biggest failure mode with LLMs here is invalid JSON output or extra conversational filler ("Sure, here are your tasks: ...").
To guarantee reliability across both commercial and open-source models, we implemented a 3-tier resilient parser:
export async function decomposeTask(
config: ApiConfig,
task: string,
n: number,
signal?: AbortSignal
): Promise<{ subtasks: string[]; latencyMs: number }> {
const systemPrompt = `You are a task planner. Break the user's task into exactly ${n} distinct, non-overlapping subtasks, each from a different useful angle. Output ONLY a JSON object of the form {"subtasks": ["...", "..."]} with exactly ${n} strings. Each subtask must be self-contained and independently answerable.`;
const jsonSchemaFormat = {
type: 'json_schema',
json_schema: {
name: 'subtasks',
schema: {
type: 'object',
properties: {
subtasks: {
type: 'array',
items: { type: 'string' },
minItems: n,
maxItems: n,
},
},
required: ['subtasks'],
additionalProperties: false,
},
},
};
const t0 = performance.now();
// Tier 1: Try strict response_format json_schema
try {
const rawContent = await callOpenAiCompatible(
config,
[
{ role: 'system', content: systemPrompt },
{ role: 'user', content: task },
],
{ responseFormat: jsonSchemaFormat, temperature: 0.3, signal }
);
const parsed = extractSubtasksFromJson(rawContent);
if (parsed && parsed.length > 0) {
return { subtasks: parsed.slice(0, n), latencyMs: Math.round(performance.now() - t0) };
}
} catch (err) {
// Graceful fallback if provider doesn't support json_schema (e.g., older local models)
}
// Tier 2: Fallback to prompt-only JSON with markdown fence stripping & regex matching
// Tier 3: One-shot LLM repair retry if output contains malformed brackets
}
2. Concurrent Fan-Out with High-Resolution Benchmarking
The magic happens in Stage 2. We use standard browser Promise.all combined with performance.now() microsecond timers:
// STAGE 2: Fan Out N in Parallel via Promise.all
setStage('parallel_fanout');
const parallelStartTime = performance.now();
// Launch all worker promises concurrently
const workerPromises = decomposeResult.subtasks.map(async (subtask, idx) => {
const workerId = idx + 1;
const result = await executeWorker(
config,
workerId,
workersCount,
subtask,
controller.signal
);
// Optimistic UI: Update individual card immediately when it finishes
setWorkers((prev) =>
prev.map((w) => (w.id === workerId ? result : w))
);
return result;
});
// Await all parallel workers to complete
const completedWorkerResults = await Promise.all(workerPromises);
const parallelEndTime = performance.now();
// Calculate exact wall-clock vs sequential sum
const calculatedParallelWallClock = Math.round(parallelEndTime - parallelStartTime);
const calculatedSequentialSum = completedWorkerResults.reduce(
(sum, w) => sum + (w.latencyMs || 0),
0
);
const speedupMultiplier = (calculatedSequentialSum / calculatedParallelWallClock).toFixed(1);
Notice the UX touch: as each worker finishes, we update that specific card's status in real time. The user sees Worker 1 and Worker 3 finish in 1.8 seconds, while Worker 2 finishes in 2.4 seconds, providing visceral visual confirmation of concurrent execution.
3. Synthesis & Context Merging
Once all workers return, their findings are formatted and fed to the Merger Agent:
export async function mergeOutputs(
config: ApiConfig,
originalTask: string,
workers: WorkerResult[],
signal?: AbortSignal
): Promise<{ finalAnswer: string; latencyMs: number }> {
const systemPrompt = `You are a synthesis editor. Combine the ${workers.length} worker answers below into one cohesive final answer to the original task. Integrate the angles, remove repetition, and structure the output with clear sections. Do not invent facts not present in the worker outputs.`;
const formattedWorkers = workers
.map(
(w) =>
`### Worker ${w.id} (Subtask: ${w.subtask})\n${w.output.trim()}`
)
.join('\n\n---\n\n');
const userContent = `Original Task:\n${originalTask}\n\n====================\nWorker Subtask Findings:\n====================\n\n${formattedWorkers}`;
const t0 = performance.now();
const finalAnswer = await callOpenAiCompatible(
config,
[
{ role: 'system', content: systemPrompt },
{ role: 'user', content: userContent },
],
{ maxTokens: 2048, temperature: 0.5, signal }
);
return { finalAnswer, latencyMs: Math.round(performance.now() - t0) };
}
Real-World Benchmarks: The Latency Results
Running a 4-worker decomposition on a complex prompt ("Design an enterprise migration strategy from monolithic Postgres to distributed CockroachDB"):
| Execution Model | Worker 1 | Worker 2 | Worker 3 | Worker 4 | Total Worker Wall-Clock | Speedup |
|---|---|---|---|---|---|---|
| Sequential (Serial) | 2,410 ms | 2,850 ms | 3,120 ms | 2,290 ms | 10,670 ms (~10.7s) | 1.0× (Baseline) |
| Parallel Fan-Out | 2,410 ms | 2,850 ms | 3,120 ms | 2,290 ms | 3,120 ms (~3.1s) | 3.42× Faster |
Even after accounting for the initial decomposition (~850ms) and final synthesis (~1,200ms), the overall pipeline finishes in ~5.1 seconds compared to ~12.7 seconds sequentially.
More importantly: the final report synthesized from 4 dedicated, focused perspectives is demonstrably more comprehensive than a single-prompt generation trying to balance all 4 angles in one breath.
Key Takeaways for AI Engineers
- Orthogonal subtasks should never run sequentially. If subtasks don't depend on each other's outputs, running them sequentially is throwing away user attention.
- Specialized sub-prompts beat single mega-prompts. A worker focused exclusively on "Identify security and compliance risks" generates deeper domain insights than a generic prompt asking the model to do everything at once.
- Plan for schema failures. When building multi-agent pipelines, never trust an LLM to return valid JSON 100% of the time. Always implement structured schemas with regex-based fallback extraction.
-
Client-side zero-overhead architecture works. By using native
fetchagainst OpenAI-compatible endpoints directly from the browser, we avoid proxy server latency and eliminate backend credential storage vulnerabilities.
Code & more: https://www.dailybuild.xyz/project/242-parallel-fan-out-orchestrator
Top comments (0)