If you are exposing LLMs to multiple users, clients, or internal teams, you are sitting on a ticking time bomb.
Without strict tenant-level cost attribution and hard API limits, a single rogue user—or a poorly written unhandled script loop—can rack up catastrophic OpenAI or Anthropic bills overnight. Managing these costs is currently a dangerous black box for most engineering teams.
In this guide, I'll walk you through building a Multi-Tenant LLM Gateway that completely eliminates this risk. We will cover OpenTelemetry (OTel) cost attribution, dynamic rate limiting, multi-provider fallbacks, and visualizing it all in a Streamlit dashboard.
1. The Bottleneck: Why Commercial Solutions Fall Short
When engineers realize they need to control LLM costs, they usually look at commercial AI gateways. While platforms like Helicone or Portkey are incredibly powerful, they introduce a few bottlenecks:
- Pricing at Scale: Paying a percentage of your AI spend or a per-request fee to your gateway quickly becomes more expensive than the LLMs themselves.
- Missing Hard Limits: Many tools offer alerting, but not synchronous hard limits that instantly cut off a tenant mid-loop before the damage is done.
- Vendor Lock-in & Black Box Analytics: Tying your cost attribution to a proprietary dashboard makes it difficult to join tenant AI costs with your primary database (e.g., PostgreSQL billing tables).
By building an internal gateway, you own the traffic, the data, and the circuit breakers.
2. The Architecture: Step-by-Step Breakdown
Our gateway acts as a reverse proxy between your users/applications and the upstream LLM providers.
Here is the flow:
-
Request Interception: The gateway (built with Python/FastAPI) receives a standard OpenAI-compatible chat completion request. It extracts the API key to identify the
tenant_id. - Hard Limit & Redis Check: Before forwarding, it queries Redis. If the tenant has exceeded their daily token limit or USD budget, the request is blocked immediately (HTTP 429).
- Prompt Sanitization & Routing: The gateway cleans the payload (stripping out malicious injections or restricted domains) and determines the provider.
- Multi-Provider Fallback: If OpenAI fails (e.g., 502 Bad Gateway), it seamlessly rewrites the prompt payload and fails over to Anthropic or a Local LLM.
-
OTel Instrumentation: Every request generates an OpenTelemetry span containing the
tenant_id,provider,model,prompt_tokens, andcompletion_tokens. - Dashboard: A Streamlit app aggregates the OTel data from your collector/database to visualize spend per tenant.
3. The Code & Logic: Prompt Structuring & Sanitization
To make this work, the gateway needs to intercept the payload, sanitize inputs (e.g., preventing scraping agents from requesting blacklisted domains), and attach OTel traces.
Here is a Python snippet demonstrating the gateway routing, domain sanitization, and OTel attribution:
import os
import json
from fastapi import FastAPI, HTTPException, Request
from opentelemetry import trace
from redis import asyncio as aioredis
import litellm
app = FastAPI()
tracer = trace.get_tracer(__name__)
redis_client = aioredis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379"))
# Domain sanitization block (e.g., for AI web scraping agents)
BLACKLISTED_DOMAINS = ["facebook.com", "instagram.com"]
def sanitize_and_structure_prompt(messages: list) -> list:
"""Sanitizes inputs and ensures structure matches fallback requirements."""
for msg in messages:
if any(domain in msg.get("content", "") for domain in BLACKLISTED_DOMAINS):
raise ValueError("Blocked domain requested in prompt payload.")
return messages
@app.post("/v1/chat/completions")
async def chat_gateway(request: Request):
payload = await request.json()
tenant_api_key = request.headers.get("Authorization", "").replace("Bearer ", "")
# 1. Identify Tenant & Check Hard Limits in Redis
tenant_id = await redis_client.hget("api_keys", tenant_api_key)
if not tenant_id:
raise HTTPException(status_code=401, detail="Invalid API Key")
cost_this_month = float(await redis_client.get(f"spend:{tenant_id}") or 0.0)
tenant_limit = float(await redis_client.get(f"limit:{tenant_id}") or 10.0) # $10 limit
if cost_this_month >= tenant_limit:
raise HTTPException(status_code=429, detail="Tenant Hard Limit Reached.")
# 2. Sanitize Inputs
try:
sanitized_messages = sanitize_and_structure_prompt(payload["messages"])
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
# 3. Execute with OTel Tracing & Multi-Provider Fallback
with tracer.start_as_current_span("llm_completion") as span:
span.set_attribute("tenant.id", tenant_id.decode("utf-8"))
try:
# LiteLLM handles the multi-provider fallback routing automatically
response = litellm.completion(
model=payload.get("model", "gpt-4o-mini"),
messages=sanitized_messages,
fallbacks=["anthropic/claude-3-haiku-20240307"],
metadata={"tenant_id": tenant_id}
)
# 4. Extract tokens & calculate cost
prompt_tokens = response.usage.prompt_tokens
completion_tokens = response.usage.completion_tokens
cost = litellm.completion_cost(completion_response=response)
# Update Redis & OTel Span
await redis_client.incrbyfloat(f"spend:{tenant_id}", cost)
span.set_attribute("llm.cost", cost)
span.set_attribute("llm.usage.prompt_tokens", prompt_tokens)
span.set_attribute("llm.usage.completion_tokens", completion_tokens)
return response
except Exception as e:
span.set_attribute("error", True)
span.record_exception(e)
raise HTTPException(status_code=500, detail="LLM Routing Failed")
4. Deployment & Performance: Timeouts & Rate Limits
When dealing with LLM APIs, network timeouts are your biggest enemy. If OpenAI hangs for 60 seconds, your users experience terrible latency.
To deploy this reliably, you should wrap the entire architecture in a docker-compose.yml that spins up:
- The FastAPI Gateway (with
gunicornanduvicornworkers). - A Redis instance for sub-millisecond rate limit checks.
- An OpenTelemetry Collector to batch and export spans asynchronously (so tracing doesn't add latency to the user request).
- A Streamlit dashboard connecting to your metrics datastore (like Prometheus or Postgres).
Handling Timeouts: In the gateway layer, configure an aggressive read timeout (e.g., timeout=15). If the primary model fails to return the first byte within 15 seconds, immediately abort the connection and execute the fallback routing to Anthropic or your local backup model.
5. Conclusion & Ready-to-Use Workflow
Building an OTel-instrumented API gateway is the only way to scale AI features across multiple tenants without risking your startup's runway. With a Redis-backed hard limit, you can confidently give API access to your users or agents knowing your max exposure is hard-coded.
You can easily recreate this architecture using the Python snippets and logic detailed above.
However, if you want to skip the trial-and-error and get a turnkey, production-ready package—complete with the full Docker Compose setup, multi-provider fallback logic, Streamlit cost dashboard, test fixtures, and raw configurations—you can grab the Multi-Tenant LLM Gateway Pro package below:
- Instant Access on Whop: Get it here
- Direct Download on Gumroad: Get it here (Use promo code **EARLYBIRD* for 20% off!)*
Drop any questions about OTel implementation or LiteLLM routing in the comments!
Top comments (1)
a small model on one gpu does that tier at a flat cost, and since litellm already routes by model name, tenants never notice the switch. i help out on nobodywho, weigh that. it's for godot games. vllm or llama-server fit a gateway.