I recently shipped a small internal agent that chews through our Oxlo.ai request logs and tells us exactly why a particular call was slow. It saves us from guessing whether the bottleneck is prompt bloat, model size, or greedy generation parameters. In this walkthrough, I will show you how to build the same thing in under fifty lines of Python.
What you'll need
Before we start, grab the following:
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Capture a slow baseline
We need something to debug. I will fire a deliberately inefficient request at Oxlo.ai and wrap it in a timer so we have ground truth.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
start = time.perf_counter()
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in exhaustive detail, covering every major algorithm and physical implementation."},
],
max_tokens=4000,
temperature=0.9,
)
elapsed = time.perf_counter() - start
log_entry = {
"model": "llama-3.3-70b",
"request_messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in exhaustive detail, covering every major algorithm and physical implementation."},
],
"max_tokens": 4000,
"temperature": 0.9,
"total_duration_sec": round(elapsed, 2),
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
}
print(f"duration={log_entry['total_duration_sec']}s prompt_tokens={log_entry['prompt_tokens']} completion_tokens={log_entry['completion_tokens']}")
Step 2: Serialize the log
The debugger needs a compact JSON view. I will format the log, truncating long strings so we do not flood the context window.
import json
def serialize_log(entry):
messages = []
for m in entry["request_messages"]:
text = m["content"]
preview = text if len(text) < 180 else text[:180] + " ... [truncated]"
messages.append({"role": m["role"], "content_length": len(text), "preview": preview})
return json.dumps({
"model": entry["model"],
"prompt_tokens": entry["prompt_tokens"],
"completion_tokens": entry["completion_tokens"],
"max_tokens": entry["max_tokens"],
"temperature": entry["temperature"],
"total_duration_sec": entry["total_duration_sec"],
"messages": messages,
}, indent=2)
log_text = serialize_log(log_entry)
print(log_text)
Step 3: Define the system prompt
This prompt tells the model how to reason like an inference engineer.
SYSTEM_PROMPT = """You are an LLM inference performance engineer. Analyze the request log below and explain why the call was slow. Then list concrete fixes.
Rules:
- Identify the single biggest bottleneck first.
- If prompt_tokens is high, suggest trimming system prompts or removing redundant context.
- If max_tokens is high but completion_tokens is low, suggest lowering max_tokens to reduce preallocation overhead.
- If temperature is above 0.5 for a factual task, suggest lowering it.
- If the model is large and the output is small, suggest a smaller Oxlo.ai model such as deepseek-v3.2 or qwen-3-32b.
- Keep the diagnosis under 120 words. Use bullet points.
"""
Step 4: Run the diagnostic
Now I send the serialized log to qwen-3-32b, which excels at reasoning and agent workflows.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Diagnose this slow request log:\n
```json\n{log_text}\n```
"},
],
temperature=0.2,
max_tokens=500,
)
diagnosis = response.choices[0].message.content
print(diagnosis)
Step 5: Stream the diagnosis
In production, I want to see the analysis as it arrives. Oxlo.ai supports streaming, so I will switch to chunk-based output.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
stream = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Diagnose this slow request log:\n
```json\n{log_text}\n```
"},
],
temperature=0.2,
max_tokens=500,
stream=True,
)
print("Diagnosis: ", end="", flush=True)
for chunk in stream:
content = chunk.choices[0].delta.content
if content:
print(content, end="", flush=True)
print()
Run it
Here is the complete script. Save it as debug_agent.py and run it.
import time
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are an LLM inference performance engineer. Analyze the request log below and explain why the call was slow. Then list concrete fixes.
Rules:
- Identify the single biggest bottleneck first.
- If prompt_tokens is high, suggest trimming system prompts or removing redundant context.
- If max_tokens is high but completion_tokens is low, suggest lowering max_tokens to reduce preallocation overhead.
- If temperature is above 0.5 for a factual task, suggest lowering it.
- If the model is large and the output is small, suggest a smaller Oxlo.ai model such as deepseek-v3.2 or qwen-3-32b.
- Keep the diagnosis under 120 words. Use bullet points.
"""
def capture_baseline():
start = time.perf_counter()
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in exhaustive detail, covering every major algorithm and physical implementation."},
],
max_tokens=4000,
temperature=0.9,
)
elapsed = time.perf_counter() - start
return {
"model": "llama-3.3-70b",
"request_messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in exhaustive detail, covering every major algorithm and physical implementation."},
],
"max_tokens": 4000,
"temperature": 0.9,
"total_duration_sec": round(elapsed, 2),
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
}
def serialize_log(entry):
messages = []
for m in entry["request_messages"]:
text = m["content"]
preview = text if len(text) < 180 else text[:180] + " ... [truncated]"
messages.append({"role": m["role"], "content_length": len(text), "preview": preview})
return json.dumps({
"model": entry["model"],
"prompt_tokens": entry["prompt_tokens"],
"completion_tokens": entry["completion_tokens"],
"max_tokens": entry["max_tokens"],
"temperature": entry["temperature"],
"total_duration_sec": entry["total_duration_sec"],
"messages": messages,
}, indent=2)
def diagnose(entry):
log_text = serialize_log(entry)
stream = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Diagnose this slow request log:\n
```json\n{log_text}\n```
"},
],
temperature=0.2,
max_tokens=500,
stream=True,
)
print("Diagnosis: ", end="", flush=True)
for chunk in stream:
content = chunk.choices[0].delta.content
if content:
print(content, end="", flush=True)
print()
if __name__ == "__main__":
log_entry = capture_baseline()
print(f"Captured baseline: {log_entry['total_duration_sec']}s\n")
diagnose(log_entry)
When I run this, the output looks something like the following.
Captured baseline: 4.82s
Diagnosis:
- Bottleneck: max_tokens is set to 4000 but completion_tokens is only 312. The high limit forces unnecessary KV-cache allocation and extends time-to-last-token.
- Fix: Lower max_tokens to 512 for this task. On Oxlo.ai this costs the same per request regardless of token count, but latency will drop sharply.
- Fix: Reduce temperature from 0.9 to 0.3. The task is factual, so high temperature adds non-deterministic sampling overhead without benefit.
- Fix: If output length stays small, try deepseek-v3.2 or qwen-3-32b instead of llama-3.3-70b for faster turnaround.
Wrap-up
That is the whole system. A few ideas for what to build next:
- Pipe the agent into a Slack bot so it triggers automatically when a request exceeds your SLO.
- Add a regression test that re-runs the same prompt against deepseek-v3.2 and llama-3.3-70b to confirm the speedup before you merge the fix.
Top comments (0)