Building Distributed Systems with LLM: Best Practices and Considerations

#engineering #oxlo #ai

We are going to build a distributed log diagnostician, a small Python agent that collects logs from three simulated microservices and uses an LLM to pinpoint cross-service failures. It is the kind of tool you reach for when tailing logs across ten containers stops scaling.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Simulate the microservices cluster

I need something to monitor, so I created three fake services. The payment-service is configured to fail so we have a realistic distributed failure to detect.

import random
from datetime import datetime, timedelta

class ServiceNode:
    def __init__(self, name, failure_mode=False):
        self.name = name
        self.healthy = True
        self.logs = []
        self.failure_mode = failure_mode

    def generate_logs(self, count=5):
        self.logs = []
        for i in range(count):
            ts = (datetime.utcnow() - timedelta(seconds=i * 15)).isoformat()
            if self.failure_mode:
                msg = f"{ts} ERROR {self.name} connection reset by peer, upstream timeout"
                self.healthy = False
            else:
                msg = f"{ts} INFO {self.name} health=OK req_id={random.randint(1000, 9999)}"
            self.logs.append(msg)
        return "\n".join(self.logs)

nodes = [
    ServiceNode("user-service"),
    ServiceNode("payment-service", failure_mode=True),
    ServiceNode("notification-service"),
]

Step 2: Collect logs from all nodes

The coordinator gathers the latest logs from every node into a single context block so the model sees the whole cluster state.

def collect_cluster_state(nodes):
    blocks = []
    for node in nodes:
        logs = node.generate_logs(count=6)
        blocks.append(f"--- {node.name} ---\n{logs}")
    return "\n\n".join(blocks)

cluster_state = collect_cluster_state(nodes)
print(cluster_state)

Step 3: Define the system prompt

I want structured reasoning, not prose. This prompt tells the model to act as an SRE and respond in a fixed format we can parse.

SYSTEM_PROMPT = """You are a distributed systems SRE. You are given logs from multiple microservices.
Analyze the logs to identify the root cause. Respond in this exact format:

Root Cause: 
Affected Services: 
Recommended Action: 
Confidence: 

Be concise. If no issue is found, state "No issue detected"."""

Step 4: Diagnose with Oxlo.ai

I send the full log bundle to Llama 3.3 70B through Oxlo.ai. Because Oxlo.ai uses flat per-request pricing, pulling in six log lines from three services does not make the call more expensive. That matters when you move from three services to thirty.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def diagnose(state: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": state},
        ],
    )
    return response.choices[0].message.content

diagnosis = diagnose(cluster_state)
print(diagnosis)

Step 5: Act on the diagnosis

Finally, I parse the structured response and execute the recommended action against the simulated cluster. In production, this hook would call kubectl or your container orchestrator.

def act(diagnosis: str, nodes):
    affected = []
    action = "none"
    for line in diagnosis.splitlines():
        if line.startswith("Affected Services:"):
            raw = line.split(":", 1)[1]
            affected = [s.strip() for s in raw.split(",")]
        if line.startswith("Recommended Action:"):
            action = line.split(":", 1)[1].strip().lower()

    for node in nodes:
        if node.name in affected and action == "restart":
            node.healthy = True
            node.failure_mode = False
            node.logs = []
            print(f"Executed restart on {node.name}")

act(diagnosis, nodes)

print("\nPost-recovery state:")
print(collect_cluster_state(nodes))

Run it

Save everything into diagnostician.py, replace YOUR_OXLO_API_KEY, and run:

python diagnostician.py

Example output:

--- user-service ---
2024-05-20T14:12:00 INFO user-service health=OK req_id=4821
2024-05-20T14:11:45 INFO user-service health=OK req_id=3912
...

--- payment-service ---
2024-05-20T14:12:00 ERROR payment-service connection reset by peer, upstream timeout
2024-05-20T14:11:45 ERROR payment-service connection reset by peer, upstream timeout
...

--- notification-service ---
2024-05-20T14:12:00 INFO notification-service health=OK req_id=9912
...

Root Cause: payment-service is failing due to upstream timeouts likely caused by connection resets.
Affected Services: payment-service
Recommended Action: restart
Confidence: high

Executed restart on payment-service

Post-recovery state:
--- payment-service ---
2024-05-20T14:12:45 INFO payment-service health=OK req_id=5543
...

Wrap-up

Next, wire the ServiceNode class to real HTTP health endpoints so the agent monitors actual containers. If you want to go further, send your logs through Oxlo.ai's embedding endpoint and store them in a vector database so the agent can retrieve similar past incidents before it diagnoses.

DEV Community