TL;DR: We had 11,247 lines of Python middleware handling per-tenant LLM cost attribution, rate limiting, and provider failover. Replaced about 60% of it with Bifrost's virtual keys and governance features. Some honest gaps remain, which is why this is a writeup and not a sales pitch.
The setup we inherited
Nexus Labs runs enterprise agent automation. Each customer gets isolated workloads. Each workload makes between 200 and 50,000 LLM calls per day across OpenAI, Anthropic, Bedrock, and Vertex.
When I joined, we had a Python middleware doing four things at once: API key rotation per provider, per-tenant rate limits in Redis, cost attribution via request tagging, and fallback logic when a provider returned 429s.
11,247 lines of Python. Three engineers had touched it. Two had left. One of them had encoded their team-internal pricing assumptions inline. Every model deprecation became a sprint.
What we actually needed
Three things, in priority order:
- Per-customer spend caps that don't require a deploy to update.
- Provider failover that survives Anthropic going down for 23 minutes (it did, last March).
- Cost data we don't have to reconstruct from CloudWatch logs.
I evaluated three gateways before picking one. Here is the comparison after running each through a 2-week eval against our actual traffic shape.
| Feature | Bifrost | LiteLLM | Portkey |
|---|---|---|---|
| Per-tenant virtual keys with budgets | Native | Plugin/config | Native |
| Self-host without external deps | Yes | Yes | Limited |
| OpenAI-compatible API for all providers | Yes | Yes | Yes |
| Built-in Prometheus metrics | Yes | Yes (newer) | Hosted preferred |
| Semantic caching | Yes | Yes | Yes |
| MCP gateway | Yes | No | Limited |
| Built-in web UI for config | Yes | Limited | Cloud-first |
LiteLLM was the real contender. Larger community, more battle-tested in production for some workload shapes. Where it lost for us: setting up hierarchical budgets across customer to team to workload tiers required more YAML wrangling than we wanted, and the failover behavior on streaming requests was less predictable under our tests.
Portkey was strong on dashboards. We didn't want a hosted dependency for our cost control path.
What changed
The piece that surprised me most was the virtual keys model. From the docs (governance/virtual-keys), every tenant gets a virtual key. The key carries the budget cap, rate limit, allowed providers, and allowed models. Our orchestrator stopped caring about provider routing entirely.
Config that replaced 4,200 lines of Python:
virtual_keys:
- id: vk_acme_prod
customer_id: acme_corp
budget:
max_per_month_usd: 12000
reset_duration: monthly
rate_limit:
requests_per_minute: 600
allowed_providers:
- openai
- anthropic
- bedrock
fallbacks:
- provider: openai
model: gpt-4o
- provider: anthropic
model: claude-sonnet-4-6
- provider: bedrock
model: anthropic.claude-sonnet-4-6
Our orchestrator now does one thing: pick a virtual key based on tenant. Send the request. Done.
The numbers
Before:
- 11,247 LOC in
gateway_middleware/ - p95 added latency from middleware: 47ms
- Mean time to add a new model: 2 days (testing, rollout, monitoring)
After 4 months:
- 4,108 LOC remaining (mostly business logic we still need)
- p95 added latency from Bifrost in front: 8ms
- Mean time to add a new model: under an hour
The latency number was the biggest surprise. Bifrost is Go. Our middleware was Python doing synchronous Redis calls. We knew that was a problem. Solving it wasn't on the roadmap.
Trade-offs and Limitations
This isn't free.
Migration was harder than the docs suggest. Our cost attribution data didn't map cleanly. We had legacy fields like team_internal_billing_code baked into every log. Mapping these to virtual key metadata took a full sprint, and the team still grumbles about it.
Semantic caching is risky for our workload. Our agents call LLMs with tool results embedded in prompts. Two prompts that look 92% similar can require very different responses. We disabled semantic caching for the agent path. Enabled it only for our content generation path, where we saw a 31% hit rate.
MCP gateway integration is newer than the rest. We use it for filesystem access from a customer-facing automation agent. Works fine. But debugging when a tool call fails requires more log digging than the rest of the platform.
No native cost-anomaly alerting yet. Budget caps work. But "this customer's usage spiked 3x in 2 hours" is still wired up via Prometheus alerts and PagerDuty by hand. Portkey has this in their hosted product. If real-time anomaly alerts are your top requirement, weight that.
What I'd tell a peer team
If you have one provider and one customer, you don't need this. Use the provider's SDK.
If you have 3+ providers, multiple customer tiers, and someone on your team has written class CostTrackingMiddleware more than once, evaluate. Spin up the Docker container (quickstart). Point staging traffic at it for a week. Look at the metrics. Decide.
The model is the easy part. Cost attribution is the part that wakes you up at 2am when a customer's bill is wrong.
Top comments (0)