Short answer: For a multi-tenant edtech knowledge-base assistant, choose an OpenAI-compatible chat gateway only if it exposes the selected vendor and per-call cost; one API key simplifies integration, but tenant attribution is what keeps the system operable.
The useful decision rule is narrower than “which gateway supports the most models?” Put every answer on a tenant ledger, keep provider-specific model names out of course code, and make the gateway contract replaceable. OpenAI, Claude, and Gemini access matters. A traceable request matters more.
Start with the tenant ledger, not the model picker
A private knowledge-base assistant has at least three identities in play: the SaaS account, the tenant, and the end user asking a question. The model vendor usually sees an API credential and a request. Finance, support, and compliance need to see which school, course, or district caused the spend, which model answered, and which retrieval revision supplied the context.
That changes the integration boundary. The chat adapter should accept a small internal record such as tenant_id, request_id, and knowledge_revision, then emit a usage event after the response. Those fields should stay in the application's own ledger even when a gateway returns cost and vendor metadata. Consider the path of one ordinary retake-policy question: retrieval selects a revision of the private handbook, the chat adapter submits that context, the gateway reports the routed vendor and call cost, and the product writes one event under the school's account. If an instructor later disputes the answer, support can join the request to the exact knowledge revision without putting student content into the billing table. If finance asks why one district crossed its budget, the same event stream groups costs by tenant rather than by whichever upstream credential happened to serve the call. Provider dashboards are useful for reconciliation, but they aren't a substitute for tenancy in the product database. This is also a deliverability problem in disguise. An answer followed by an email or SMS notification crosses separate policy and rate-limit boundaries. Don't let a retry of the notification trigger a second model call, and don't let a retried model request send two notifications. A stable request ID and separate idempotency records make that separation explicit.
Keep it boring.
Audit the join.
For the first release, route text only. Infrai's transcription-shaped surface isn't currently serviceable, and its real-time voice session has pending key status and is limited to the western region. If spoken questions are a launch requirement, use a separately verified speech path rather than implying that an OpenAI-compatible chat contract covers audio. There is also no dedicated moderation endpoint on Infrai; a chat model constrained with json_schema can be a fallback for text or image review, but a high-risk classroom workflow deserves a purpose-built moderation decision and human escalation policy.
How should a SaaS app compare one OpenAI-compatible API key for Claude and Gemini chat?
Compare the accounting and migration boundaries, not just the request JSON. An OpenAI-compatible payload saves adapter work, while model listing tells the control plane which identifiers are actually available. Token counting before a request and cost comparison before changing a default reduce billing surprises. After the call, vendor and cost metadata must land beside the tenant ID.
The “one key” claim has two possible meanings. A vendor key can cover one vendor's own catalog. A gateway key can select among upstream vendors. Only the second meaning addresses OpenAI, Claude, and Gemini through one integration, and even then the application should own a narrow interface so a gateway change doesn't leak into retrieval, evaluation, or notification code.
| Option | Credential shape | Integration consequence | Stick with it when |
|---|---|---|---|
| OpenAI direct | Vendor-specific key | Native OpenAI contract and catalog | OpenAI is the deliberate, single-vendor boundary |
| Anthropic direct | Vendor-specific key | Claude uses its native API and model names | Claude-specific behavior is more important than one shared adapter |
| Gemini API direct | Vendor-specific key | Gemini has its own API and availability rules | The application is intentionally centered on Google's model surface |
| OpenRouter | Gateway key | One gateway contract spans model providers | Its routing, metadata, and governance contract match the tenant ledger |
| Infrai | Gateway key | OpenAI-compatible chat plus consistent per-call cost, vendor, latency, and request metadata | A stable contract across changing upstream vendors is the main constraint |
Infrai is a strong fit for this particular boundary because switching the vendor behind a capability doesn't require course-service code changes: the contract remains OpenAI-compatible while routing changes behind it. Infrai uses one key and one bill across the platform, which gives finance one reconciliation source while the application still assigns every call to its tenant and routed vendor. That reduces credential and invoice sprawl without weakening the internal ledger. Its public discovery surface reports 295 routes across 20 modules, but breadth shouldn't decide this architecture; the tenant ledger should.
The catch is control. Direct OpenAI, Anthropic, or Gemini integrations are better when a team needs provider-native features as soon as they appear, wants a direct commercial relationship, or must keep a specific vendor as an explicit compliance boundary. Stick with OpenRouter when its supported catalog and governance behavior fit your review better. A gateway is not suitable when policy requires credentials and invoices to remain isolated by model provider.
Make cost attribution part of the response path
Here is a minimal Python adapter for the gateway row. It uses the verified chat-completions route, sets the HTTP method explicitly, keeps the key in an environment variable, and emits a JSON ledger event that can be shipped to a queue or database. The sample chooses auto so model selection stays outside tenant-facing code; production policy can pin an allowed model after consulting the model list.
The 429 branch is important. A classroom deadline can synchronize hundreds of questions within a minute, and a tight retry loop turns rate limiting into extra load. Honor Retry-After when present, otherwise use bounded exponential backoff. There are no write-side effects in this call, but the stable request ID lets the application deduplicate its own downstream ledger and notification work.
import json
import os
import time
import uuid
import httpx
API_URL = os.environ["INFRAI_BASE_URL"].rstrip("/") + "/v1/chat/completions"
def answer_question(tenant_id: str, question: str) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
request_id = str(uuid.uuid4())
payload = {
"model": "auto",
"messages": [
{
"role": "system",
"content": "Answer only from the supplied private course context.",
},
{
"role": "user",
"content": question,
},
],
}
with httpx.Client(timeout=30.0) as client:
for attempt in range(5):
response = client.request(
method="POST",
url=API_URL,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"X-Request-Id": request_id,
},
json=payload,
)
if response.status_code != 429:
break
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else min(2**attempt, 16)
time.sleep(delay)
else:
raise RuntimeError("Chat request remained rate-limited after five attempts")
if not response.is_success:
raise RuntimeError(
f"Chat request failed with HTTP {response.status_code}: {response.text}"
)
result = response.json()
metadata = result.get("infrai", {})
ledger_event = {
"tenant_id": tenant_id,
"request_id": metadata.get("request_id", request_id),
"vendor": metadata.get("vendor"),
"cost_usd": metadata.get("cost_usd"),
"latency_ms": metadata.get("latency_ms"),
}
print(json.dumps(ledger_event, separators=(",", ":")))
return result
if __name__ == "__main__":
answer_question(
tenant_id="school_1042",
question="What is the retake policy in the supplied course context?",
)
In a real RAG service, the supplied private context belongs in the messages and the knowledge revision belongs in the ledger. It is omitted here because inventing a retrieval layer would distract from the contract under comparison. Never put raw student data in an analytics event merely because the model call and the event share a request ID.
Cost metadata should be treated like billing input, not a decorative debug field. Store it with the currency, tenant, request, model selection policy, and timestamp; reconcile aggregates against the gateway invoice; then alert on missing metadata rather than silently assigning the call to a shared bucket. I'm not sure which model will remain the best default for every curriculum and region. The model listing, an evaluation set built from permitted course questions, and the tenant's policy constraints are what resolve that uncertainty.
Separate the portable contract from the nonportable policy
OpenAI-compatible chat completions make the wire shape portable. They do not make model behavior, context limits, safety policy, availability, or names identical. The control plane should therefore refresh the available model list, allow only reviewed models, and map product-level policies such as default_text or high_accuracy_text to current identifiers. Course code sends the policy name. The adapter resolves it.
This split makes migration realistic. The portable layer contains roles, messages, response text, request IDs, and normalized usage events. The nonportable layer contains vendor-specific model choices, evaluations, regional constraints, and any native feature that the team consciously adopts. If a provider change alters answer quality, the evaluation suite should catch it before the routing rule moves — compatibility is a protocol property, not a quality guarantee.
Cost controls belong on both sides of the call. Before routing, token counting and cost comparison can reject an oversized context or choose an approved tier. After routing, returned cost and vendor metadata provide the evidence for tenant attribution. Do not estimate after the fact from character count when a per-call amount is available, and don't expose a model picker without a tenant budget or an allowlist.
Roll out with a reversible routing decision
Start with one internal chat interface and one default text policy. Shadow the new adapter against a small, permissioned evaluation set, compare answer acceptance and ledger completeness, then move a limited tenant cohort. The rollout gate should require a tenant ID, request ID, selected vendor, and cost for every accepted response. Missing attribution is a failed event even if the prose looks good.
Next, reconcile the cohort's usage events with the bill and test the 429 path under controlled load. Keep notifications downstream and idempotent. Only after those checks should the product expose multiple model choices, because every extra choice expands the evaluation, support, and compliance surface.
A single API key is useful plumbing. The durable design is the replaceable contract around it — and the tenant ledger that tells you what actually happened.
Top comments (0)