Building Distributed Systems with LLM: Best Practices and Architectures

We're building a scatter-gather research cluster that parallelizes complex technical investigations across multiple LLM workers, then synthesizes the results into a unified report. This architecture solves the single-context bottleneck and lets you combine heterogeneous reasoning styles without managing tokens across a long prompt. I use this exact pattern to pre-screen open-source libraries before adding them to production services.

What you'll need

• Python 3.10 or newer
• pip install openai tenacity
• An Oxlo.ai API key from https://portal.oxlo.ai
• The standard concurrent.futures module (no extra install)

Step 1: Configure the worker pool

Each worker is a function bound to a specific Oxlo.ai model. We keep the client initialization minimal because Oxlo.ai is fully OpenAI SDK compatible. I run three specialists in parallel: a reasoning node, a code-analysis node, and a market-context node.

from openai import OpenAI
from concurrent.futures import ThreadPoolExecutor, as_completed
from tenacity import retry, stop_after_attempt, wait_exponential

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

WORKERS = {
    "deep_researcher": {
        "model": "deepseek-v3.2",
        "prompt": "You are a methodical research analyst. Given a technical topic, identify core concepts, trade-offs, and failure modes. Be concise and factual."
    },
    "code_analyst": {
        "model": "kimi-k2.6",
        "prompt": "You are a senior staff engineer. Evaluate the topic from a code quality, maintenance burden, and integration risk perspective. Focus on concrete engineering concerns."
    },
    "market_analyst": {
        "model": "llama-3.3-70b",
        "prompt": "You are a product strategist. Assess community adoption, vendor risk, and long-term viability. Reference specific ecosystem trends if relevant."
    }
}

Step 2: Define the coordinator prompt

The coordinator is the only stateful agent in the system. Its job is to interpret the user's request, broadcast subtasks, and reconcile conflicting worker outputs into one coherent directive. Here is the system prompt I use for the synthesis node.

SUPERVISOR_PROMPT = """You are the coordinator of a distributed research cluster. You will receive a user query and three specialist reports produced in parallel by heterogeneous workers. Your job is to synthesize the reports into a single, actionable technical brief. Highlight disagreements between workers and explain which perspective should carry more weight and why. Output valid Markdown. Do not omit dissenting opinions; reconcile them."""

Step 3: Implement scatter with retries

Real distributed systems tolerate partial failure. I wrap each Oxlo.ai call with tenacity so that transient errors on one worker do not crash the entire job. Because Oxlo.ai has no cold starts on popular models, the retry loop is fast.

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def call_worker(worker_id: str, topic: str) -> dict:
    config = WORKERS[worker_id]
    response = client.chat.completions.create(
        model=config["model"],
        messages=[
            {"role": "system", "content": config["prompt"]},
            {"role": "user", "content": f"Topic: {topic}\nProduce your specialist report now."},
        ],
        temperature=0.2,
        max_tokens=2048,
    )
    return {
        "worker": worker_id,
        "model": config["model"],
        "content": response.choices[0].message.content,
    }

Step 4: Implement gather and fault isolation

I use ThreadPoolExecutor to fan out requests concurrently. If one node throws an exception, we capture it and continue with the survivors. This is the same fault-isolation pattern I use when fanning out to microservices.

def scatter(topic: str) -> list[dict]:
    with ThreadPoolExecutor(max_workers=len(WORKERS)) as executor:
        future_to_id = {
            executor.submit(call_worker, wid, topic): wid
            for wid in WORKERS
        }
        results = []
        for future in as_completed(future_to_id):
            wid = future_to_id[future]
            try:
                results.append(future.result())
            except Exception as exc:
                results.append({"worker": wid, "error": str(exc), "content": ""})
        return results

Step 5: Synthesize results

After gathering partial reports, we feed them into a dedicated synthesis model. I use Qwen 3 32B here because it handles multilingual reasoning and agent workflows well, which makes it ideal for reconciling conflicting technical opinions.

def synthesize(topic: str, reports: list[dict]) -> str:
    context = "\n\n---\n\n".join(
        f"Worker: {r['worker']} (model: {r['model']})\n{r['content']}"
        for r in reports if not r.get("error")
    )
    
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": SUPERVISOR_PROMPT},
            {"role": "user", "content": f"User query: {topic}\n\nSpecialist reports:\n{context}"},
        ],
        temperature=0.3,
        max_tokens=4096,
    )
    return response.choices[0].message.content

Step 6: Wire the pipeline

The main entrypoint orchestrates the full flow: scatter, then synthesize. This keeps the topology explicit and makes it easy to add a persistence layer or caching later.

def run_cluster(topic: str):
    print(f"Scattering topic: {topic}")
    raw_reports = scatter(topic)
    
    for r in raw_reports:
        if r.get("error"):
            print(f"Worker {r['worker']} failed: {r['error']}")
    
    print("Synthesizing final report...")
    return synthesize(topic, raw_reports)

if __name__ == "__main__":
    topic = "Evaluate using Oxlo.ai for a long-context document processing pipeline instead of a token-based provider."
    report = run_cluster(topic)
    print("\n=== FINAL REPORT ===\n")
    print(report)

Run it

Save the complete script as distributed_research.py, export your key, and execute:

export OXLO_API_KEY="sk-oxlo.ai-..."
python distributed_research.py

When I ran this against the topic above, the deep-researcher node highlighted request-based pricing advantages for long-context workloads, the code-analyst node noted the OpenAI SDK compatibility reduces migration risk, and the market-analyst node commented on the breadth of the 45+ model catalog. The Qwen 3 32B coordinator correctly surfaced the tension between flat-request pricing predictability and token-based granularity, then recommended a hybrid approach for variable workloads. Total wall time was under eight seconds because Oxlo.ai serves the models without cold starts.

Next steps

Add a Redis cache layer to store worker outputs keyed by topic hash so you do not pay for identical subtasks twice. You can also introduce a circuit-breaker around each worker so repeated Oxlo.ai timeouts for a specific model route traffic to a standby worker without human intervention.