We open-sourced a benchmark for AI gateway overhead today. It tests the latency, memory and cost a gateway adds on top of the upstream LLM, measured through the lens of a coding agent.
github.com/BerriAI/ai-gateway-bench
Why this exists
Every AI gateway claims to be fast. Some claim "54x faster." Nobody shows their work.
Most gateway benchmarks are garbage. They test against live LLM endpoints so you're measuring network jitter and provider load, not the gateway. They run on tiny instances with no horizontal scaling. They cherry-pick one metric and hide the rest.
We wanted something anyone can run and verify. So we built it.
How it works
Every gateway points at the same local deterministic Rust mock. No network, no provider variance. What's left is pure gateway overhead:
overhead = latency(client -> gateway -> mock) - latency(client -> mock directly)
Gateways under test: LiteLLM (Rust), LiteLLM (Python v1), Portkey, Bifrost.
Yes, we included our own Python gateway and it's the slowest. That's the point. If we hid our worst numbers, why would you trust the good ones.
What it measures
These aren't chatbot metrics. A coding agent (Claude Code, Codex) runs a tight loop: stream a plan, emit tool calls, apply edits, repeat. Thousands of turns per session. A few ms of gateway overhead compounds into real wall-clock lag.
| Metric | Why it matters |
|---|---|
| TTFT + inter-chunk latency | Agent streams every turn, stutter is felt directly |
| Tool-call argument reassembly | Reordered args break an edit |
| Overhead vs prompt size (1k-100k) | Agents paste whole files |
| p99 under concurrency | Tail latency when 64 agents hit one gateway |
| Memory (peak RSS, idle, growth) | Deploy cost and OOM risk |
| Session overhead (30 turns) | What a developer actually feels |
Results
p99 added latency (n=5000 per endpoint, single host):
- LiteLLM Rust: 0.7 ms
- Portkey: 2.3 ms
- Bifrost: 4.5 ms
- LiteLLM Python v1: 257.7 ms
Peak memory:
- LiteLLM Rust: 21.8 MB
- Portkey: 90.4 MB
- Bifrost: 199.1 MB
- LiteLLM Python v1: 329.5 MB
Estimated cost per 1M requests:
- LiteLLM Rust: $0.000175
- Bifrost: $0.001008
- Portkey: $0.001042
- LiteLLM Python v1: $0.015354
RPS per USD/hour:
- LiteLLM Rust: 283,833
- Bifrost: 63,246
- Portkey: 17,964
- LiteLLM Python v1: 4,296
30-turn coding session overhead (Claude Code + Codex control loops):
- LiteLLM Rust adds ~40ms to a full session
- Bifrost adds ~150ms
- LiteLLM Python v1 adds ~950ms
The moat metric
Most benchmarks report averages. The number that actually matters for agents is p99 inter-chunk latency held flat as concurrency rises, paired with 1:1 chunk fidelity.
This is structurally hard to fake. It requires no output buffering, no per-request re-serialization on the stream path, and no runtime pauses under load. A garbage-collected runtime will show periodic jitter spikes under load that a mean hides.
Run it yourself
git clone https://github.com/BerriAI/ai-gateway-bench
cd ai-gateway-bench
python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
# start the deterministic Rust mock
cargo run --release -p mock-upstream
# start a gateway (see gateways/<name>/README.md)
# run a scenario
locust -f scenarios/streaming_turn/locustfile.py --headless -u 16 -r 16 -t 30s \
--host http://127.0.0.1:<gateway-port>
All raw data is committed as CSVs in results/. Charts are generated from the data, not hardcoded. Add your own gateway and run the same scenarios.
What we learned building this
Gateway overhead is noise compared to model latency. Even our Python gateway adds 258ms at p99, and the actual model call is 500ms-30s. The Rust gateway adds 0.7ms.
But for coding agents running 30+ turns per session, that noise compounds. And when you have 64 concurrent agents, the tail matters more than the average.
The benchmark also exposed that Portkey's OSS gateway currently returns HTTP 500 for streaming Anthropic Messages requests. That's not a performance issue, it's a functionality gap. We labeled it honestly in the charts.
Fork it, add your gateway, open a PR. The more gateways tested under the same conditions, the better for everyone.
Top comments (0)