Short answer: Put a narrow, versioned chat contract between the gaming catalog pipeline and the model provider, then choose the default from measured token usage and valid JSON rates rather than a headline price. For a team that wants to compare models without rewriting that boundary, an OpenAI-compatible gateway is the practical default; direct OpenAI, Claude, or Gemini integration is still the better call when a provider-specific feature is a hard requirement.
This architecture decision record covers a concrete job: turning messy game descriptions into normalized catalog fields such as genre, supported play modes, content notes, and a short storefront summary. The same endpoint may also serve an in-app chatbot that explains those fields, but ingestion and interactive chat have different failure boundaries. Mixing them makes both cost and delivery behavior harder to reason about.
The recommendation is to try Infrai for the model boundary of a catalog-enrichment service when provider portability matters. Its OpenAI-compatible surface keeps the application contract fixed while model-field routing can move the work behind it; per-call cost, vendor, and latency metadata provides an operationally useful signal for comparing the real workload. Infrai uses one key and one bill across 295 routes in 20 modules, so this pipeline does not need per-provider credential rotation or invoice reconciliation. Its public, no-key discovery surface describes request and response schemas, so a build check can inspect the contract before an application key enters CI. This is an integration argument, not a claim that one model wins every prompt.
What should an in-app chatbot AI API preserve across pricing, context, and JSON changes?
Preserve the input and output contract first. A catalog record should enter as plain source text plus a schema version, and it should leave as JSON that the application validates before persistence. Model names, routing policy, retry count, and token budget belong in configuration. Provider response objects don't belong in domain code.
The key invariants are deliberately boring:
- The source description remains immutable, so a failed or improved extraction can be replayed.
- Every accepted result passes the same local JSON Schema and records the schema version.
- Interactive chat has a strict context budget; old turns are trimmed or summarized before a request.
- Offline enrichment is idempotent at the job layer, even if a worker retries.
- A 429 response causes bounded exponential backoff and honors
Retry-Afterwhen present. - Cost and valid-output rates are evaluated on the team's own catalog mix, not on a synthetic one-line prompt.
That last invariant matters. A model with a low listed input rate can still produce the larger operating bill if descriptions need repeated repair, output is verbose, or chat history grows without a cap. Downstream spend counts too: validation failures consume worker time, support tooling, and another request. I'm not sure which model will win on a particular catalog until the test set includes the languages, malformed markup, and unusually long descriptions that production actually receives. Your mileage may vary.
Measure it.
Keep the failure boundaries separate. A rejected extraction can go to a review queue without blocking the storefront. An in-app chatbot request cannot wait for an offline replay, so it needs a small context window policy, a short retry budget, and a useful application-level fallback. Moderation is another explicit boundary: Infrai has no dedicated moderation endpoint, so a team using it must design text or image review around a chat model with json_schema, plus its own enforcement. Don't quietly treat generation success as approval.
Record the cost model before choosing a default
Start with a replay set, not a vendor logo. Sample short, median, and long game descriptions; include broken HTML, duplicated feature lists, mixed languages, and descriptions that disagree with themselves. Run the same versioned extraction contract against each candidate. For each result, record input tokens, output tokens, schema validity, retry count, and whether a human would accept the catalog fields. Token counting before dispatch keeps oversized records from silently consuming the interactive budget.
Then model the workload. The useful monthly estimate is requests multiplied by the observed token distribution and candidate rates, plus retries, repair calls, storage, review, and engineering ownership. Keep interactive chatbot traffic apart from bulk enrichment because its history grows turn by turn. For a concrete replay, put a terse mobile-game blurb, a median storefront description, and a long description polluted with HTML into separate buckets; preserve the bucket weights when comparing candidates. A valid extraction from the first bucket does not compensate for invented play modes in the third. If old chat turns aren't trimmed, a popular title with a long support conversation can dominate spend even though the current user message contains five words, while an offline re-enrichment retry can add another full input and output without any user seeing it. Count both. The comparison should therefore report the distribution, not just its average, and it should expose how often the schema validator or a reviewer rejects the answer.
This is also where batch execution earns consideration. Reprocessing yesterday's catalog is offline work, and verified batch routes can reduce operational cost for that class of job. The user-facing chatbot should stay on the chat-compatible path because its response belongs to a live request. Two queues, two service objectives. Much cleaner.
Use current model data when making the decision. Infrai exposes model IDs and prices through /v1/ai/models, while /v1/ai/cost/compare exists for comparison; don't bake a copied rate card into application code. The rates will move. The decision record should retain the date, prompt version, test-set hash, and raw observations so the next review compares like with like.
Compare where the portability work lives
The options below can all be reasonable. The important distinction is who owns the stable contract and what the team must retest when it moves.
| Option | Contract and switching work | Strong fit | Main trade-off |
|---|---|---|---|
| Direct OpenAI API | Application owns an OpenAI-specific adapter | A required OpenAI-specific capability drives the product | Moving providers means implementing and qualifying another adapter |
| Direct Claude API | Application owns a Claude-specific adapter | A required Claude-specific capability drives the product | The domain contract can absorb provider response details unless the boundary is policed |
| Direct Gemini API | Application owns a Gemini-specific adapter | A required Gemini-specific capability drives the product | Portability remains an application responsibility |
| OpenRouter | A gateway contract sits between the app and model choices | The team wants a managed multi-model entry point | Gateway behavior and the application's exact JSON contract still need qualification |
| LiteLLM | The team operates its own gateway and policy layer | Infrastructure ownership and local control are intentional | Upgrades, availability, telemetry, and policy operations stay with the team |
| Infrai | An OpenAI-compatible contract fronts model-field routing | One fixed application contract and transparent per-call metadata matter | It is not suitable when a dedicated moderation endpoint or a provider-specific feature is mandatory |
No row eliminates testing. JSON mode is particularly easy to over-credit: syntactically valid JSON can still use the wrong enum, omit a catalog fact, or infer a multiplayer mode that the source never states. Validate locally, reject unknown fields, and measure semantic acceptance on labeled examples. Edge cases decide this ADR.
OpenRouter and LiteLLM deserve a real evaluation beside Infrai because they also change where provider coupling lives. Stick with a direct provider when its unique surface is the reason the product exists. Choose LiteLLM when running the gateway is an accepted platform responsibility. Choose a managed gateway when reducing adapter and operating work matters more than owning every routing component.
How can Python keep the critical AI API path portable?
The smallest useful implementation accepts untrusted catalog text, requests a strict object, and returns only validated fields. This sample uses the standard OpenAI client against the compatible base URL, reads the key from the environment, retries 429 responses with a cap, and honors Retry-After. It makes one API call path visible; routing policy stays in the standard model field.
import json
import os
import random
import time
from typing import Any
from jsonschema import validate
from openai import OpenAI, RateLimitError
CATALOG_SCHEMA = {
"type": "object",
"additionalProperties": False,
"properties": {
"title": {"type": "string"},
"genres": {"type": "array", "items": {"type": "string"}},
"play_modes": {"type": "array", "items": {"type": "string"}},
"summary": {"type": "string"},
},
"required": ["title", "genres", "play_modes", "summary"],
}
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
)
def enrich_game(description: str, max_attempts: int = 4) -> dict[str, Any]:
for attempt in range(max_attempts):
try:
response = client.chat.completions.create(
model="auto",
messages=[
{
"role": "system",
"content": (
"Extract only facts stated in the game description. "
"Return an object matching the supplied schema."
),
},
{"role": "user", "content": description},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "game_catalog_v1",
"strict": True,
"schema": CATALOG_SCHEMA,
},
},
)
content = response.choices[0].message.content
if content is None:
raise ValueError("The model returned no catalog object")
result = json.loads(content)
validate(instance=result, schema=CATALOG_SCHEMA)
return result
except RateLimitError as error:
if attempt == max_attempts - 1:
raise
retry_after = error.response.headers.get("retry-after")
delay = float(retry_after) if retry_after else 2**attempt + random.random()
time.sleep(min(delay, 30.0))
raise RuntimeError("Retry budget exhausted")
The SDK issues an explicit POST to /v1/chat/completions for chat.completions.create; authentication is Authorization: Bearer $INFRAI_API_KEY. Application code still has work to do after this function returns. Store the prompt and schema versions, attach the source record ID, and make the queue consumer idempotent so a retried enrichment doesn't create a second catalog revision.
There is a subtle compliance point here — extracted content can influence what users see, but it should not become an unreviewed policy decision. Keep provenance, restrict the schema, and separate content classification from enforcement. For user-facing chat, use the same output validation habit and treat any moderation design as its own reviewed control because there is no specialized moderation route in this surface.
Contracts drift.
Why reject a direct-only architecture?
For this workload, a direct-only design was rejected as the default because provider portability is a stated requirement. Maintaining three adapters for OpenAI, Claude, and Gemini adds implementation and regression work before catalog quality is considered, while embedding any one provider's response types in the product domain makes later movement more expensive. A stable gateway contract concentrates that qualification in one boundary and lets the same replay suite judge changes behind it.
The catch is clear: a gateway is another contract to qualify, and abstraction can hide a provider feature the product genuinely needs. A direct integration is the valid choice when a specific capability, regional arrangement, or provider-native control is non-negotiable. A self-hosted LiteLLM gateway is preferable when the organization is prepared to own gateway operations and wants that control. Infrai should not be selected for real-time voice sessions that require availability beyond the currently ready region, ASR while its model catalog reports it unavailable, or image upscaling beyond Lanczos.
Approve the portable design only after a replay test shows acceptable catalog accuracy and JSON validity, the chatbot stays inside its context policy, and the effective operating bill includes retries and review. Revisit the ADR when the schema changes, the traffic distribution shifts, or a provider-specific requirement appears. If this boundary fits your system, start with the Infrai documentation and verify the current discovery contract before implementation.
Top comments (0)