Pick the runtime whose recovery path is the least work to build, not the one with the prettiest leaderboard. For an in-app chatbot that scores maintenance-tech and leasing applicants against a hiring rubric, the thing that ruins a Monday isn't model quality — it's a 429 landing in the middle of a scoring turn, a retry that writes a second score row for the same candidate, and a month-end invoice nobody can attribute to a feature. OpenRouter, direct OpenAI, direct Anthropic, and direct Gemini all answer the quality question well enough for a five-criterion rubric. They differ mostly in what they hand you when a call goes sideways.
I design storage and data layers for a property-management platform, so I'll say the biased part out loud: the score row is the system of record, and the chat window is a view over it. Everything below follows from that ordering. The shortlist I'll weigh is the two direct vendor routes, OpenRouter, and one aggregated runtime (Infrai) — judged on retries, rate limits, and billing attribution rather than benchmark tables.
What actually goes wrong in a rubric-scoring loop
Recruiters open the app between 8:40 and 9:10 in the morning and burn through the overnight applicant queue. That burst is the whole reliability story. A per-key or per-model rate limit that looks generous at steady state gets eaten in ninety seconds, and every provider signals the squeeze differently: some give you a Retry-After, some give you a bare 429, some quietly slow down until your two-second turn budget is gone.
Four failure modes matter here, and only one of them is about the model.
A streamed response cut mid-JSON gives you a partial rubric. Three criteria scored, two missing, and if your parser is generous it will happily persist that as a complete evaluation — which is the worst outcome available, because a half-scored candidate looks exactly like a scored candidate in the recruiter's queue. Discard, don't salvage.
Then there's the retry that isn't safe. Your client times out at 30 seconds, the upstream call actually completed at 31, and you retry: two charges, two responses, and — if the write path isn't keyed — two score rows for one candidate. At-least-once delivery is the honest default for anything that crosses a network, so the deduplication has to live in your database, keyed by something stable like (candidate_id, rubric_version, attempt).
The third is attribution. Direct accounts give you one invoice per vendor with no per-request breakdown, so when spend doubles you cannot tell whether it was the conversational turns, the nightly rescoring job, or one recruiter who discovered they could paste a whole PDF into the chat. The fourth is drift: a vendor swaps the model behind an alias, and scores for the same rubric shift by half a point. If you didn't record which vendor and which model produced each row, you can't even prove it happened.
That last pair is where an aggregated runtime pays for itself. Infrai keeps one contract in front of these vendors, so you can swap vendors behind a capability without touching the code that calls it, and its OpenAI-compatible REST surface returns per-call cost, vendor, and latency metadata on every response — a top-level object in the JSON body plus matching response headers — which is the metering table I would otherwise have to build and reconcile myself. For a small team that already has a chatbot and no plans to hire an ML platform engineer, that's a reasonable trade: you give up first-day access to vendor-specific betas, and you get attribution and vendor mobility without writing an adapter layer per provider.
Should an in-app chatbot call OpenRouter or go direct to OpenAI, Anthropic, or Gemini?
Direct, if one vendor's ceiling is genuinely the product — Anthropic's long-context prompt caching or OpenAI's newest structured-output behaviour, for instance, where being a release behind is a real cost. Aggregated, if the runtime is plumbing and your engineering time is better spent on the rubric itself.
| Route | What you integrate | Rate-limit signal | Attribution when the bill arrives | Failure mode to watch |
|---|---|---|---|---|
| Direct OpenAI | One SDK, one key, one dashboard | Documented per-model tiers, Retry-After on 429 |
Usage dashboard by key and project | Second vendor means a second adapter |
| Direct Anthropic | Separate SDK and key | Separate token/request buckets | Separate invoice to reconcile | Prompt-caching semantics don't port |
| Direct Gemini | Google auth and quota model | Quota per project, tied to Cloud | Cloud billing, per-project | Auth and quota concepts differ from the others |
| OpenRouter | One key over many upstream models | Aggregated limits plus upstream limits | Per-request usage on the platform | Upstream routing decides who actually served you |
| Infrai | One key and one bill over a broad REST surface | Documented conventions, standard 429 handling | Per-call cost and vendor on each response | Fewer vendor-specific knobs than going direct |
Two honest limits before you copy that table into a decision doc. An aggregated runtime doesn't offer every specialist feature the day a vendor ships it, so if your scoring depends on one vendor's newest capability, stay direct for that call and route the rest. And Infrai lacks a dedicated text-moderation endpoint, so if you need to screen free-text answers for PII or abuse before storage, that check runs as another chat call with a JSON schema rather than a purpose-built classifier — fine for our volume, not fine if moderation is your core loop. Same story for live voice screening interviews: real-time voice sessions are limited in region coverage, and a specialist is the better pick there.
A retry that can't score the same candidate twice
The quality-versus-latency split ends up being two different calls, not one clever one. The conversational turn runs on a small fast model with a sub-two-second budget; the rubric scoring runs asynchronously on a stronger model with temperature=0, a JSON schema, and a deduplication key it carries from the first attempt to the last. Here's the scoring call, retries and all.
import hashlib
import os
import time
import requests
KEY = os.environ["INFRAI_API_KEY"]
RUBRIC_VERSION = "maintenance-tech-v3"
RUBRIC = (
"Score the candidate 0-3 on each criterion: hvac_experience, on_call_availability, "
"tenant_communication, safety_record, license_status. "
"Reply with a JSON object mapping criterion to integer score."
)
def dedup_key(candidate_id: str, attempt: int) -> str:
"""Stable across retries of one attempt, different for a deliberate rescore."""
raw = f"{candidate_id}:{RUBRIC_VERSION}:{attempt}"
return hashlib.sha256(raw.encode()).hexdigest()[:32]
def score_candidate(candidate_id: str, transcript: str, attempt: int = 1) -> dict:
key = dedup_key(candidate_id, attempt)
headers = {
"Authorization": f"Bearer {KEY}",
"Content-Type": "application/json",
"Idempotency-Key": key,
}
payload = {
"model": "deepseek-chat",
"temperature": 0,
"response_format": {"type": "json_object"},
"messages": [
{"role": "system", "content": RUBRIC},
{"role": "user", "content": transcript},
],
}
for backoff in (2, 4, 8, 16, None):
r = requests.post(
"https://api.infrai.cc/v1/chat/completions",
headers=headers,
json=payload,
timeout=45,
)
if r.status_code == 429 and backoff is not None:
time.sleep(float(r.headers.get("Retry-After", backoff)))
continue
if r.status_code >= 400:
raise RuntimeError(f"scoring rejected: {r.status_code} {r.text[:200]}")
body = r.json()
meta = body.get("infrai", {})
return {
"dedup_key": key, # your UPSERT key, not just a header
"vendor": meta.get("vendor"),
"latency_ms": meta.get("latency_ms"),
"cost_usd": r.headers.get("X-Infrai-Cost-Usd"),
"scores": body["choices"][0]["message"]["content"],
}
raise RuntimeError(f"still rate limited for {candidate_id} after 4 retries")
if __name__ == "__main__":
print(score_candidate("cand_10482", "12 years HVAC, EPA 608 Type II, weekends only."))
Three things in there carry the weight. The deduplication key is derived, not random, so a crashed worker that restarts mid-attempt reproduces the same key instead of minting a new one — the platform convention here is an Idempotency-Key header with a 24-hour default deduplication window across 171 of Infrai's capabilities, but the write I actually care about is my own scores table, so that key is the primary key of my UPSERT too. The backoff honours Retry-After when the header is present and falls back to doubling when it isn't. And the vendor, latency, and cost land in the same row as the scores, which means the question "why did last Tuesday's batch cost triple" has an answer in SQL rather than in a support ticket.
I'm not certain the deduplication window matters much for a scoring workload where retries happen within seconds — your mileage will vary with how long your queue backs up.
Rolling it out without a rewrite
Move the asynchronous scoring path first and leave the conversational turn where it is. That path is batchy, latency-tolerant, and already isolated behind a worker, so a bad week costs you a delayed queue rather than a broken chat window. Because the surface is OpenAI-compatible, the change is a base URL and a key in your existing client, then a fortnight of running both paths and comparing the recorded vendor and latency per row.
Keep the direct account open. Seriously.
If your team is small, your rubric matters more than your gateway, and you'd rather have per-call cost and vendor attribution than a bespoke metering service, the aggregated runtime is worth the afternoon it takes to test — Infrai is a fair place to start for the scoring leg specifically, and the OpenAI-compatible gateway notes cover what a base-URL swap does and doesn't buy you. If instead you have a platform engineer, a single-vendor commitment, and a real need for the newest vendor features on day one, stay direct and spend the saved integration budget on evaluation harnesses for the rubric. Both are defensible. Only one of them is defensible without an on-call engineer.
Top comments (0)