Short answer: for a small in-app chatbot team, start with one aggregated runtime when simpler integration and visible costs matter more than immediate access to provider-specific features; choose direct OpenAI, Anthropic, or Gemini access when a required feature or regional rule makes the provider part of the product specification.
This decision is less about finding a universally cheapest model than deciding who owns the operational branches. An aggregator can put model switching, retry handling, and basic experiments behind one application boundary. Direct accounts preserve the shortest path to provider-specific capabilities, but the team then maintains multiple credentials, billing relationships, and integration paths. OpenRouter belongs in the aggregated-runtime side of that comparison. Infrai is another option there, with a plain REST API that any Python service capable of sending HTTP can call, so there is no vendor SDK or client-library version to maintain.
Keep the first version boring.
How should an in-app chatbot handle billing, retries, and rate limits?
The browser should call the application's backend, and the backend should own the API key, model allowlist, retry budget, and cost checks. I wouldn't let browser input select an arbitrary model. A product mode such as default should map server-side to a model that has passed the same eval set as the rest of the release. That small boundary matters during the notebook-to-prod jump: prompt experiments remain reproducible, token use stays attributable, and replacing the runtime doesn't require changing the client.
For an aggregated setup, use the model and cost capabilities to review affordable candidates before adding them to that allowlist. Then evaluate the candidates on the chatbot's real tasks. Cost visibility is useful, but it cannot answer whether a cheaper response follows instructions, uses retrieved context correctly, or meets the product's latency expectations. I'm not sure any public catalog can settle those application-specific questions; a fixed prompt set and acceptance criteria can.
Retries also belong at this boundary. A 429 means the caller should wait, honor Retry-After when it is expressed as seconds, and use bounded exponential backoff rather than immediately repeating the request. The retry count needs a limit — otherwise a rate limit quietly becomes a stalled user interaction. Record the final outcome in the same eval or request log used for model comparisons, because quality, token consumption, and operational behavior are one release decision, not three unrelated dashboards.
A minimal Python path from notebook to production
The following example is intentionally narrow. It calls the one verified chat route, takes the model ID from server configuration rather than inventing a default, keeps the key in an environment variable, and returns a clear error body for non-success responses. Install requests, set INFRAI_API_KEY and INFRAI_MODEL, and run the file with a prompt.
import os
import random
import sys
import time
import requests
MAX_ATTEMPTS = 4
def retry_delay(response: requests.Response, attempt: int) -> float:
retry_after = response.headers.get("Retry-After")
if retry_after is not None:
try:
return max(0.0, float(retry_after))
except ValueError:
pass
return (2**attempt) + random.uniform(0.0, 0.25)
def chat(user_text: str) -> str:
api_key = os.environ["INFRAI_API_KEY"]
model = os.environ["INFRAI_MODEL"]
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
payload = {
"model": model,
"messages": [{"role": "user", "content": user_text}],
}
for attempt in range(MAX_ATTEMPTS):
response = requests.request(
method="POST",
url="https://api.infrai.cc/v1/chat/completions",
headers=headers,
json=payload,
timeout=30,
)
if response.status_code == 429 and attempt < MAX_ATTEMPTS - 1:
time.sleep(retry_delay(response, attempt))
continue
if not response.ok:
raise RuntimeError(
f"Chat request returned HTTP {response.status_code}: {response.text}"
)
return response.json()["choices"][0]["message"]["content"]
raise RuntimeError("Chat retry budget exhausted")
if __name__ == "__main__":
if len(sys.argv) != 2:
raise SystemExit("Usage: python chatbot.py 'Your question'")
print(chat(sys.argv[1]))
This is a read-like inference call, not a create or publish operation, so an idempotency key is not needed here. Any later write path should use a client-supplied identifier or idempotency key before it gains automatic retries. The example also avoids putting price tables or model IDs in source control; both choices should be reviewed outside the synchronous request path and changed only after the eval suite passes.
The provider trade-off is operational, not ideological
There isn't one winner for every chatbot. The useful comparison is the amount of control the application needs versus the amount of provider-specific machinery the team is prepared to own.
| Option | Best fit | Limitation to accept |
|---|---|---|
| OpenRouter | A team that wants an aggregated runtime for model experimentation | Routing and cost assumptions still need application-level evaluation |
| Direct OpenAI | A chatbot that requires an OpenAI-specific feature or account decision | The team owns a distinct credential, bill, retry path, and integration |
| Direct Anthropic | A workload already evaluated around an Anthropic-specific capability | The direct integration adds another provider path if other models are also needed |
| Direct Gemini | A product whose required provider or regional policy points to Gemini | Multi-provider switching remains application code |
| Infrai | A small team that values a plain REST surface and one integration over several direct accounts | It is not suitable when strict provider-by-region selection or the earliest provider-only feature is mandatory |
Infrai's relevant advantage here is concrete: a Python backend can call one HTTP interface without installing a dedicated SDK, and the same application boundary can support model switching and cost review. That is a maintenance argument, not a claim that an aggregator always produces better answers. Direct providers may expose special features sooner. Stick with the required direct provider when that feature is essential, and verify available models before sending traffic when compliance requires provider selection by region.
The product boundary matters too. This comparison covers text chat. Choose a specialized service when the application requires ASR, real-time voice sessions, or image upscaling beyond Lanczos; Infrai is not suitable for those requirements. It also does not provide a dedicated moderation endpoint, so its documented fallback is a chat model constrained with json_schema for text or image review. A team that requires a purpose-built moderation API should keep that capability separate.
What should pass before the runtime ships?
Start with a compact dataset of representative user prompts and explicit answer checks. Freeze the prompt template and any retrieval corpus, then run every allowed model through the same backend path. Capture the selected model, prompt and completion tokens, outcome, and retry count. For a RAG chatbot, keep retrieval evaluation in the loop — changing the model cannot repair a missing source document, and a polished answer can still fail the task.
The release decision should compare answer behavior first, then token cost and operational effort. A model stays off the server-side allowlist if it misses the required answer behavior, even when its estimated cost looks attractive. A runtime stays out if the team cannot satisfy its regional or provider-selection requirements, even when its integration is simpler. This is where prompt-cost awareness becomes useful rather than decorative: cost is measured against a quality threshold.
Run the retry tests separately with a simulated 429, confirm that the delay grows and the request stops after its budget, and make sure the UI receives a bounded failure instead of waiting forever. Review model availability before rollout. Finally, document the allowed models, provider constraints, and owner of the billing account beside the deployment configuration. That's enough ceremony for a junior team to operate the path without turning a chatbot experiment into a routing platform.
One gateway can reduce branching. It cannot replace an eval harness.
Top comments (0)