When you're building applications that depend on LLM APIs, performance matters as much as the model's capabilities. I learned this the hard way back in March 2025 when our chatbot started timing out during peak hours. The model itself was fine, but we had no systematic way to measure what "fine" actually meant under load.
This guide walks through the benchmarking framework I built to test latency, throughput, output quality, and stability across different providers and models. The code examples are production-tested and ready to run.
Why Standard Benchmarks Don't Tell the Whole Story
Academic benchmarks like MMLU or HumanEval measure model capabilities, but they won't tell you if your API calls will complete in under 2 seconds during a traffic spike. What I needed was operational metrics: how fast does the model respond to my specific prompts, how many requests can I process concurrently, and does quality stay consistent when the service is under pressure?
The gap between capability and operational performance becomes critical when you're comparing providers or deciding whether to switch models. A model that scores 5% higher on MMLU but adds 800ms of latency might actually hurt your user experience.
Setting Up a Realistic Test Harness
I start every benchmark with three real prompts from our production traffic: a short query (10-20 tokens), a medium task (100-150 tokens), and a complex instruction (300+ tokens). Synthetic prompts miss the edge cases that matter.
Here's the baseline test runner I use:
import time
import asyncio
from typing import List, Dict
import httpx
class ModelBenchmark:
def __init__(self, api_key: str, base_url: str):
self.api_key = api_key
self.base_url = base_url
self.results = []
async def single_request(self, prompt: str, model: str) -> Dict:
"""Measure single request latency and token throughput."""
start = time.perf_counter()
async with httpx.AsyncClient(timeout=30.0) as client:
try:
response = await client.post(
f"{self.base_url}/chat/completions",
headers={"Authorization": f"Bearer {self.api_key}"},
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 500
}
)
response.raise_for_status()
data = response.json()
elapsed = time.perf_counter() - start
tokens = data["usage"]["completion_tokens"]
return {
"latency": elapsed,
"tokens": tokens,
"tokens_per_second": tokens / elapsed if elapsed > 0 else 0,
"success": True,
"error": None
}
except Exception as e:
return {
"latency": time.perf_counter() - start,
"success": False,
"error": str(e)
}
async def run_sequential(self, prompt: str, model: str, n: int = 10):
"""Baseline sequential performance."""
results = []
for _ in range(n):
result = await self.single_request(prompt, model)
results.append(result)
await asyncio.sleep(0.1) # Small delay between requests
return self._calculate_stats(results)
def _calculate_stats(self, results: List[Dict]) -> Dict:
"""Calculate percentile latencies and success rate."""
successful = [r for r in results if r["success"]]
if not successful:
return {"success_rate": 0.0, "error": "All requests failed"}
latencies = sorted([r["latency"] for r in successful])
throughputs = [r["tokens_per_second"] for r in successful]
return {
"success_rate": len(successful) / len(results),
"p50_latency": latencies[len(latencies) // 2],
"p95_latency": latencies[int(len(latencies) * 0.95)],
"p99_latency": latencies[int(len(latencies) * 0.99)],
"mean_throughput": sum(throughputs) / len(throughputs),
"total_requests": len(results)
}
This captures the metrics that actually matter in production: p95 latency (what most users experience), throughput in tokens per second, and success rate.
Testing Concurrent Load
Sequential tests show baseline performance, but real applications send concurrent requests. I found that some providers maintain consistent latency up to 20 concurrent requests, then degrade sharply. Others throttle aggressively after 10.
async def run_concurrent(self, prompt: str, model: str, concurrency: int = 10, total: int = 100):
"""Test performance under concurrent load."""
semaphore = asyncio.Semaphore(concurrency)
async def bounded_request():
async with semaphore:
return await self.single_request(prompt, model)
tasks = [bounded_request() for _ in range(total)]
results = await asyncio.gather(*tasks)
return self._calculate_stats(results)
I run this test at 5, 10, 20, and 50 concurrent requests to find the point where latency starts climbing. For our use case, a provider that maintains sub-2s p95 latency at 20 concurrent requests meets our needs. Anything below 10 concurrent before degradation is a red flag.
Measuring Output Quality Under Load
Latency means nothing if the model starts producing garbage when stressed. I test quality by running the same prompt 50 times under load and checking for consistency.
from difflib import SequenceMatcher
async def quality_under_load(self, prompt: str, model: str, concurrency: int = 10, n: int = 50):
"""Check if output quality degrades under concurrent load."""
semaphore = asyncio.Semaphore(concurrency)
responses = []
async def get_response():
async with semaphore:
result = await self.single_request(prompt, model)
if result["success"]:
# You'd extract response text here from your actual API response
return result.get("response_text", "")
return None
responses = await asyncio.gather(*[get_response() for _ in range(n)])
responses = [r for r in responses if r is not None]
if len(responses) < 2:
return {"error": "Insufficient successful responses"}
# Calculate pairwise similarity
similarities = []
for i in range(min(10, len(responses) - 1)):
similarity = SequenceMatcher(None, responses[i], responses[i + 1]).ratio()
similarities.append(similarity)
return {
"mean_similarity": sum(similarities) / len(similarities),
"successful_responses": len(responses),
"total_attempts": n
}
For factual queries, I expect similarity above 0.8. For creative tasks, 0.4-0.6 is normal. The key is detecting when a provider starts returning errors disguised as completions or when token limits get enforced inconsistently.
Stability Testing Over Time
On October 15, 2025, I watched one provider's p95 latency jump from 1.2s to 4.8s over a 30-minute window, then drop back down. That taught me to test stability across different times of day.
async def stability_test(self, prompt: str, model: str, duration_minutes: int = 30):
"""Monitor performance variation over time."""
interval = 60 # Test every minute
iterations = duration_minutes
stats_over_time = []
for i in range(iterations):
print(f"Stability check {i + 1}/{iterations}")
stats = await self.run_sequential(prompt, model, n=5)
stats["timestamp"] = time.time()
stats_over_time.append(stats)
if i < iterations - 1:
await asyncio.sleep(interval)
# Calculate variance
latencies = [s["p95_latency"] for s in stats_over_time if "p95_latency" in s]
if not latencies:
return {"error": "No successful measurements"}
mean_lat = sum(latencies) / len(latencies)
variance = sum((x - mean_lat) ** 2 for x in latencies) / len(latencies)
return {
"mean_p95": mean_lat,
"std_dev": variance ** 0.5,
"min_p95": min(latencies),
"max_p95": max(latencies),
"measurements": len(latencies)
}
A standard deviation above 30% of the mean suggests unreliable performance. I want providers where the worst p95 is at most 2x the best p95 during normal operation.
Comparing Across Providers
When testing multiple providers, I keep the prompts and parameters identical. The easy way to do this is using a unified API that handles the provider differences for you. At RollTok, we built exactly that: one endpoint switches between over 60 models from different providers, so benchmark code doesn't need provider-specific logic.
async def compare_models(self, prompt: str, models: List[str]):
"""Compare performance across different models."""
results = {}
for model in models:
print(f"Testing {model}...")
results[model] = {
"sequential": await self.run_sequential(prompt, model, n=20),
"concurrent_10": await self.run_concurrent(prompt, model, concurrency=10, total=50),
"concurrent_20": await self.run_concurrent(prompt, model, concurrency=20, total=50),
}
await asyncio.sleep(2) # Breathing room between model tests
return results
I track results in a simple JSON file and visualize trends over time. The goal isn't finding the "best" model universally, it's finding which model meets your latency, cost, and quality requirements for your specific workload.
What I Learned
Three things surprised me during a year of running these benchmarks:
The model with the best accuracy often had the worst latency variance. Consistency matters more than peak performance for user-facing applications.
Concurrent request handling varies wildly between providers. Some are built for high concurrency, others clearly expect sequential usage patterns.
Time of day affects performance more than I expected. Testing only during off-peak hours gave me overly optimistic results. I now run stability tests during peak US and EU hours before committing to a provider.
Running Your Own Benchmarks
Start with your actual production prompts. Run sequential tests to establish baseline latency, then concurrent tests at 2x, 5x, and 10x your expected peak load. Add a 30-minute stability window during your busiest hours.
The code above is a starting point. You'll want to add retry logic, better error categorization, and cost tracking (tokens × price per token). I keep a spreadsheet where each row is a benchmark run: date, model, concurrency level, p95 latency, success rate, and estimated cost per 1000 requests.
For quick experiments, using a unified API service helps because you can test dozens of models without managing separate API keys and rate limits for each provider. We document the format at docs.rolltok.com if you want to adapt this code.
The real value isn't in the exact numbers but in tracking how they change when you modify prompts, switch models, or scale load. Benchmark early, benchmark often, and trust measurements over marketing claims.
Top comments (0)