We are building an inference performance debugger that benchmarks Oxlo.ai endpoints and diagnoses latency bottlenecks. It is useful for engineers running agentic workflows or long-context pipelines who need to separate API latency from model throughput issues.
What you'll need
- An Oxlo.ai API key from https://portal.oxlo.ai
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai
Step 1: Time a single request
Start by measuring end-to-end latency for a standard chat call. This gives you a baseline before you add streaming or context.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
start = time.perf_counter()
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "user", "content": "Explain the difference between throughput and latency in three sentences."},
],
)
end = time.perf_counter()
print(f"Total latency: {end - start:.2f}s")
print(f"Response words: {len(response.choices[0].message.content.split())}")
Step 2: Measure time-to-first-token
Streaming reveals whether delay is in queueing or generation. I capture the moment the first chunk arrives and count subsequent chunks.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
prompt = "Write a short Python function that implements binary search."
start = time.perf_counter()
first_token_time = None
chunk_count = 0
content = ""
stream = client.chat.completions.create(
model="deepseek-v3.2",
messages=[{"role": "user", "content": prompt}],
stream=True,
)
for chunk in stream:
if first_token_time is None:
first_token_time = time.perf_counter()
if chunk.choices[0].delta.content:
content += chunk.choices[0].delta.content
chunk_count += 1
end = time.perf_counter()
print(f"Time to first token: {first_token_time - start:.2f}s")
print(f"Total time: {end - start:.2f}s")
print(f"Chunks received: {chunk_count}")
Step 3: Benchmark context scaling
Latency usually grows with prompt length. I test three sizes against a model that handles long context well, since Oxlo.ai charges per request rather than per token.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
prompts = {
"short": "Summarize the key benefits of request-based pricing.",
"medium": "Summarize the key benefits of request-based pricing. " * 50,
"long": "Summarize the key benefits of request-based pricing. " * 200,
}
for label, text in prompts.items():
start = time.perf_counter()
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[{"role": "user", "content": text}],
)
end = time.perf_counter()
print(f"{label}: {end - start:.2f}s")
Step 4: Define the debugger agent
Now I add an agent that reads the metrics and suggests concrete fixes. Here is the system prompt.
SYSTEM_PROMPT = """You are an inference performance engineer.
Analyze the provided latency metrics and return a JSON object with these keys:
- bottleneck: either "ttft", "throughput", or "context_scaling"
- severity: "low", "medium", or "high"
- recommendation: one concrete action the developer should take
Be concise. Base your answer only on the numbers provided."""
Step 5: Run the diagnosis
I feed the collected metrics into the agent and request structured JSON output so the result is machine readable.
import json
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
metrics = {
"model": "deepseek-v3.2",
"time_to_first_token_seconds": 0.8,
"total_time_seconds": 3.2,
"chunks": 120,
"prompt_length_words": 800,
}
user_message = f"Diagnose these metrics: {json.dumps(metrics)}"
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
)
diagnosis = json.loads(response.choices[0].message.content)
print(json.dumps(diagnosis, indent=2))
Run it
Putting it together, the full script runs the benchmark then asks the agent for a verdict.
import json
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are an inference performance engineer.
Analyze the provided latency metrics and return a JSON object with these keys:
- bottleneck: either "ttft", "throughput", or "context_scaling"
- severity: "low", "medium", or "high"
- recommendation: one concrete action the developer should take
Be concise. Base your answer only on the numbers provided."""
def benchmark(model, prompt):
start = time.perf_counter()
r = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
)
end = time.perf_counter()
return {
"model": model,
"total_time_seconds": round(end - start, 2),
"prompt_length_words": len(prompt.split()),
"response_length_words": len(r.choices[0].message.content.split()),
}
if __name__ == "__main__":
metrics = benchmark("llama-3.3-70b", "Explain request-based pricing and why it helps long-context workloads.")
user_message = f"Diagnose these metrics: {json.dumps(metrics)}"
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
)
print(json.dumps(json.loads(response.choices[0].message.content), indent=2))
Example output:
{
"bottleneck": "context_scaling",
"severity": "low",
"recommendation": "The prompt is moderate length and total time is healthy. If scaling further, switch to a model with a 1M context window such as DeepSeek V4 Flash on Oxlo.ai."
}
Next steps
Wire this script into your CI pipeline to catch latency regressions on every deploy. You can also extend the benchmark to test concurrent requests and verify that Oxlo.ai's lack of cold starts keeps TTFT stable under load.
Top comments (0)