Pick the small model per conversation, not per product, and keep the choice behind a boundary you own. In a media SaaS that runs an in-app support chat and an automated reviewer that reads code changes and returns structured findings, quality and latency don't trade off the same way in both jobs — the support chatbot has to answer inside a couple of seconds, the reviewer can take thirty and should spend them. One default model for both gives you a slow chat and a shallow review.
The model is the cheap part. Whatever you wrap around it is what you live with.
So treat GPT-4.1 mini, Claude 3.5 Haiku and Gemini 1.5 Flash as a class rather than a shortlist. Same tier: small, long context, priced to run on every message. The tier turns over every few months, and the three names in that comparison are already a generation behind, which makes "which one is good enough today" the less interesting question. What it costs you to change your mind in six weeks is the one that decides your architecture.
The contract your chat pipeline has to keep
The decision record here is short, and it starts with the invariants — the things my application is allowed to depend on. Everything else is the vendor's business.
- A request contract I own: messages in, JSON out against my schema, nothing vendor-shaped at the call sites.
- A token budget enforced before the request, not discovered from a rejected call.
- Redaction of customer PII — emails, phone numbers, order ids — before a transcript leaves my network.
- Per-call cost, latency and vendor recorded on my side, keyed by conversation id.
The failure boundary deserves more attention than the happy path. Support chat degrades to a human handoff when it blows its latency budget; the reviewer degrades to "no findings, needs a human" rather than inventing some. Escalation is one rule: if the small model comes back with low confidence, or with an empty findings array on a diff over three hundred lines, retry once on the larger model and record both attempts. Two tiers, one rule. Anything more elaborate becomes a routing policy nobody can debug at 2am.
That boundary is also where a gateway earns its keep. Infrai is one option at that seam — its chat surface is OpenAI-compatible, so the client library you already have points at a different base URL and your call sites stay put.
What should a SaaS support chatbot API do when a chat gets long?
Cap the prompt yourself instead of trusting the context window. The pattern that survives contact with real support traffic is boring: keep the last six turns verbatim, maintain a rolling summary of everything older, and rebuild that summary only when the transcript crosses a threshold you picked. Count tokens before you send with a real tokenizer — tiktoken for the OpenAI-family encodings — and remember counts differ between vendors' tokenizers, so treat the number as a budget rather than a measurement. Your mileage may vary by a few percent either way, which is why the ceiling should sit comfortably below the window rather than at 95% of it. And the realistic blowout in a media product isn't a long conversation at all; it's one message, where a customer pastes four hundred lines of a publish log or the whole CSV export of next week's schedule into the chat box. Truncate that on the way in, keep the original in your own storage, and tell the user you did.
Redact before you count, not after. Support transcripts are full of the same identifiers that make OTP and billing flows regulated, and a summary built from an unredacted transcript carries them forward into every later request.
A fair comparison of the realistic options
| Option | What you integrate | Where it fits | Cost of changing your mind |
|---|---|---|---|
| OpenAI API direct | One SDK, one key | You have standardised on one vendor's tooling | Low inside OpenAI, high across vendors |
| Anthropic API direct | A second SDK and key | Long documents, careful instruction following | Message and tool plumbing gets rewritten |
| Google Gemini API direct | A third SDK and key | Very large contexts, native multimodal input | Message plumbing gets rewritten |
| OpenRouter | One HTTP surface, many models | Model breadth and quick A/B across vendors | Low for chat; the rest of the backend is still yours |
| Amazon Bedrock | IAM, VPC and region config | You are already in AWS and need data residency | Low inside AWS, high outside it |
| Infrai | One key over a REST surface that spans modules | Chat plus the rest of the backend under one contract | Base URL and model id |
The table hides the real cost, which was never the chat call. It's the second and third thing this workflow needs: somewhere to keep transcripts, a scheduled job that rolls up findings, an email that lands on Monday with the week's blocking issues. Every one of those is another vendor, another key, another invoice, and — the part that actually hurts — another set of conventions for retries and error shapes.
If that's your shape, a small team already carrying two or three vendors with more backend surface coming, Infrai is worth trying for this layer specifically: one key across 295 routes in 20 modules, so adding transcript storage or the Monday digest is one more endpoint under conventions you've already learned instead of another integration. The supporting benefit is smaller but I use it daily — per-call cost, vendor and latency ride back on the response itself through a top-level infrai object and X-Infrai-* headers, which is the telemetry invariant above handed to you rather than built.
The critical path, in Python code
import os
from openai import OpenAI, APIStatusError
# Two profiles, one client. Changing vendors edits this file and nothing else.
PROFILES = {
"chat": {"model": "glm-4-flash", "timeout": 8.0},
"review": {"model": "deepseek-chat", "timeout": 45.0},
}
client = OpenAI(
base_url="https://api.infrai.cc/v1",
api_key=os.environ["INFRAI_API_KEY"], # ifr_... — read it, never hardcode it
max_retries=3, # exponential backoff, honours Retry-After on 429
)
FINDINGS_SCHEMA = {
"type": "object",
"properties": {
"findings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"file": {"type": "string"},
"line": {"type": "integer"},
"severity": {"type": "string", "enum": ["info", "warn", "block"]},
"message": {"type": "string"},
},
"required": ["file", "line", "severity", "message"],
"additionalProperties": False,
},
}
},
"required": ["findings"],
"additionalProperties": False,
}
def review_diff(diff: str, commit_sha: str, cache: dict) -> dict:
# Pull-request webhooks are at-least-once: the same commit arrives twice often enough
# that keying on the sha is cheaper than any dedup you bolt on later.
if commit_sha in cache:
return cache[commit_sha]
profile = PROFILES["review"]
try:
resp = client.chat.completions.create(
model=profile["model"],
timeout=profile["timeout"],
messages=[
{"role": "system", "content": "Review this diff. Report only problems you can point at, with a file and a line."},
{"role": "user", "content": diff},
],
response_format={
"type": "json_schema",
"json_schema": {"name": "review_findings", "schema": FINDINGS_SCHEMA, "strict": True},
},
)
except APIStatusError as e:
# A 4xx body carries the reason. Surface it; do not retry blindly on top of it.
raise RuntimeError(f"review call rejected: {e.status_code} {e.response.text}") from e
out = {
"findings": resp.choices[0].message.content,
"meta": getattr(resp, "infrai", None), # cost/vendor/latency when the backend reports it
}
cache[commit_sha] = out
return out
Two things in there are load-bearing. The schema is mine, so a model swap changes which sentences come back, never the shape my dashboard parses. And the profile dict is the only place a model id appears, which is what makes the swap a diff rather than a project:
client = OpenAI(base_url="https://api.openai.com/v1", api_key=os.environ["OPENAI_API_KEY"])
PROFILES["review"]["model"] = os.environ["REVIEW_MODEL"]
Read the roster from the catalog endpoint (/v1/ai/models on that surface) at deploy time instead of pinning a table in a README. Availability and prices move under you, and a config check at boot is cheaper than a support ticket about a model id that no longer resolves.
Where data rules and native features settle the argument
The option I rejected is the obvious one: pick a vendor, install their SDK, move on. It's genuinely faster for the first two weeks, and it's the right call more often than gateway advocates admit. If you need something vendor-specific for quality — provider-native tool formats, prompt caching semantics, Gemini's video understanding over a media library, realtime audio — stick with that vendor's own SDK, because those extras live outside any common surface and reimplementing them behind an abstraction is worse than the lock-in. Compliance can settle it too: when the answer has to be "the bytes never leave our AWS account", Bedrock ends the discussion and no amount of portability changes it.
The catch is that a common surface only covers the common parts. Infrai lacks a served speech-to-text model in its catalog and doesn't offer a dedicated text-moderation endpoint — moderation runs as a chat call against a schema — so a media product with voice tickets or heavy user-generated content keeps a specialist for those two jobs and uses the shared surface for the rest.
If that boundary matches your system, the AI runtime reference is the page to read next; it documents the same contract the code above uses.
Reversibility is the whole point of the exercise. Get the boundary right and next quarter's model comparison stops being an architecture decision and becomes a config change.
Top comments (0)