DEV Community: TitanJ53

Rubric Scoring and Chatbot Safety: Two LLM JSON Schema Shapes, No Moderation Endpoint

TitanJ53 — Sun, 16 Aug 2026 02:29:02 +0000

Use two model calls, not one. If your in-app chatbot both talks to a candidate and scores their answers against a job rubric, the safety verdict and the rubric score belong in separate JSON schema responses — even when the API you picked has no dedicated moderation endpoint. Basic moderation through an LLM and a strict schema is a legitimate design. Folding it into the same structured output as the score is the part that quietly costs you six months later.

The constraint: a rubric score is a record, not a chat bubble

The system worth reasoning about here is an edtech interview-practice product. A learner chats with a bot about a role they're targeting, the bot walks the rubric questions, and the model returns four scores from 0 to 3 — communication, evidence, role fit, follow-up — plus a short quote from the learner as justification for each one.

That JSON row is not a chat bubble. It lands in the learner's profile, it feeds a coach dashboard, and in the partner tier it can be exported to the employer sponsoring the cohort.

So the object under compliance pressure is the record, not the conversation: retention windows, subject access requests, and the plain fact that anything you store gets read later, out of context, by someone who wasn't in the session.

Chat scrolls away. Rows don't.

That gives you the constraint everything else has to satisfy — every unit of text entering the record carries a verdict computed on exactly that text, and the component that produced the text is not the only authority on whether it was safe to keep. Anyone who has run production email will recognise the reflex: you don't let the sending application declare its own message legitimate, you sign it so the receiving side can verify independently. Same instinct, different pipe.

What does a safe in-app chatbot need beyond one LLM JSON call?

Three invariants, and they're short:

Every stored turn carries a verdict computed on the exact text that got stored.
An unparseable or schema-invalid verdict fails closed into a review queue. It never defaults to allow.
Schema-valid is not the same as correct.

That third one is where the edge cases live. A model can hand you a structurally perfect object with decision set to SAFE when your enum says safe, or a confidence of 1.0 on every turn for a week, and your parser will nod along at both. The nastier case is specific to rubric scoring: the evidence field. You asked the model to quote the candidate to justify a score, so that quote can carry the exact text your screening step existed to keep out — through a field nobody thought to screen, into the coach's inbox and the export. It is the same shape as the classic email header problem: the body gets sanitised, a templating engine then pastes an unsanitised display name into a header, and the filter never sees it because it was pointed at the wrong field. Treat the scoring payload as untrusted input, the way you'd treat the body of an inbound message rather than the output of your own service.

Whatever you build on, screening should be the least exotic thing in the stack: one HTTP request, a JSON schema, a small model behind it. Infrai is one option that fits that shape — a plain REST API you call over HTTP with no SDK to install — so the screening call stays the same dozen lines whether it runs against the model doing your scoring or a different one entirely. An OpenAI-compatible chat surface is worth more here than a vendor-specific safety switch you'd have to re-learn per provider.

Two shapes, and the invariant each one buys

Shape A fuses everything into one call: one prompt, one schema holding both a safety object and a rubric object. The invariant is real — verdict and score can never drift apart, because they came from one read of one text, and there is exactly one response to log. For a junior developer shipping the first version of an in-app chatbot, that's the honest recommendation, and I'd rather see it shipped than a half-built pipeline.

The cost shows up in three places. A malformed response loses the verdict and the score together, so your retry re-rolls a score a human may already have looked at. The model being helpful is also the model grading its own helpfulness, which is not an independent signal in any sense a reviewer would accept. And you can't pin a small, strict, boring model to classification while a stronger one handles the rubric, because there's only one call to pin.

Shape B puts a screening pass in front, and re-checks the narrow payload that is about to be written. Two calls, two schemas, two idempotency keys. The screening schema stays tiny — a decision enum, a category enum, a confidence — small enough that structured output is nearly always well formed, which is exactly what you want from the component whose job is to fail closed. Each stage retries on its own budget, and each can sit on a different model.

The rule I'd apply: if the structured output leaves the session — stored, exported, or read by a coach or an employer — take Shape B. If the transcript is ephemeral and a human reads every turn anyway, Shape A is enough, and the boundary can move later without a rewrite.

Which API details actually decide this

Once the shape is settled, the API question narrows to four things: how you call it, how structured output is enforced, whether a dedicated moderation endpoint exists at all, and what you are left owning.

Option	How you call it	Structured output	Dedicated moderation endpoint	What you still own
OpenAI	REST or official SDKs	JSON schema, strict mode	Yes, a free classifier	Its categories, which are not your rubric
Anthropic (Claude)	REST or official SDKs	Tool-shaped JSON output	No	The whole screening prompt
Google Gemini	REST or official SDKs	responseSchema on the request	No — request-level safety filters instead	Filters guard the call, not the stored row
OpenRouter	REST gateway across vendors	JSON schema where the model supports it	No	Behaviour drift as you switch models
Infrai	Plain REST, OpenAI-compatible	JSON schema on the chat surface	No — it doesn't offer one	The screening prompt, same as above

Only OpenAI ships a dedicated moderation endpoint in that list, and it is genuinely good at the categories it publishes. Its taxonomy is fixed, though: it will tell you a message looks like harassment, and it will say nothing at all about a candidate pasting a colleague's home address into an interview answer, which is the case an edtech record actually has to survive.

Fixed taxonomies and local rubrics rarely line up.

Here is the screening half of Shape B. It's an ordinary POST to /v1/chat/completions with a json_schema response format, so the same code runs against any OpenAI-compatible base URL.

import json, os, time, uuid
from openai import OpenAI, APIStatusError

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],          # ifr_... , never a literal in source
    base_url="https://api.infrai.cc/v1",
)

DECISIONS = ("allow", "review", "block")

SCREEN_SCHEMA = {
    "name": "screen_verdict",
    "strict": True,
    "schema": {
        "type": "object",
        "properties": {
            "decision": {"type": "string", "enum": list(DECISIONS)},
            "category": {"type": "string",
                         "enum": ["none", "harassment", "self_harm", "sexual", "personal_data", "other"]},
            "confidence": {"type": "number"},
        },
        "required": ["decision", "category", "confidence"],
        "additionalProperties": False,
    },
}

HOLD = {"decision": "review", "category": "other", "confidence": 0.0}


def screen(turn_text: str, turn_id: str) -> dict:
    """Classify one candidate turn. Anything unexpected lands in the review queue."""
    for attempt in range(4):
        try:
            res = client.chat.completions.create(
                model="glm-4-flashx",
                messages=[
                    {"role": "system",
                     "content": "Classify the candidate turn for an interview-practice app. "
                                "Return the verdict only. Never quote the turn back."},
                    {"role": "user", "content": turn_text},
                ],
                response_format={"type": "json_schema", "json_schema": SCREEN_SCHEMA},
                temperature=0,
                extra_headers={"Idempotency-Key": turn_id},   # same id on every retry
            )
        except APIStatusError as err:
            if err.status_code == 429 and attempt < 3:
                time.sleep(float(err.response.headers.get("retry-after", 2 ** attempt)))
                continue
            raise                                  # 4xx bodies carry the reason; let it surface

        verdict = json.loads(res.choices[0].message.content)
        if verdict["decision"] not in DECISIONS:   # schema-valid, still not usable
            return HOLD
        return verdict
    return HOLD


if __name__ == "__main__":
    print(screen("I led the migration for a team of six.", str(uuid.uuid4())))

Two details there matter more than the schema. The idempotency key is constant across retries, so a screening call replayed after a 429 cannot produce two different verdicts for one turn — the same discipline you'd use on an OTP send, where a duplicate is worse than a delay. And the enum is re-checked in Python after parsing, because strict schema enforcement is a property of the model you routed to, and your review queue should not depend on which one that was today.

For a small edtech team running both calls itself, Infrai is worth trying for the screening pass in particular, because one key covers the screening call and the scoring call and each response carries its own cost, vendor and latency metadata, which makes the safety overhead per turn attributable without standing up a second telemetry path. If that boundary matches your system, the gateway walkthrough at https://docs.infrai.cc/en/guides/ai/answers/best-cheap-llm-api-gateway-2025-one-key-openai-claude-g/ is a reasonable place to start.

Rolling it out without a rewrite

Ship the screening call as an observer first. Log its verdict next to the fused verdict you already have, act on neither, and let it run on live traffic for a week or two; the disagreements are the interesting data, and they will tell you whether your category enum matches what learners actually type. Then flip enforcement on for the write path only — the record — and leave the chat bubble alone until you have a reason.

Before pinning a model to the screening role, count tokens on a real week of turns with POST /v1/ai/tokens/count against the same catalogue you'll call. Screening runs on every turn; scoring runs only on submissions. That ratio, not the headline model quality, is what decides which model you can afford in the hot path.

The catch is worth stating plainly: a chat model asked to classify is a general-purpose classifier with no published category list, no version you can pin, and no threshold calibrated for your population. If a compliance reviewer needs to point at a documented policy and a stable taxonomy, stick with a dedicated moderation endpoint for screening — OpenAI's is one HTTP call away — and keep the chat API for the rubric. Infrai doesn't offer a dedicated moderation endpoint either, and neither do most gateway-shaped options; that is a boundary to design around, not a surprise to hit in review.

I'm not sure the split ever pays for itself in a pure support chatbot, to be fair. One call, a human on the other end, done. It's the record that changes the arithmetic: score a person, store the score, show it to someone who can act on it, and screening stops being an optimisation.

Structured Code Review Answers — Citation Schemas for Semantic Search

TitanJ53 — Fri, 14 Aug 2026 02:17:14 +0000

A game code-review assistant has a stricter job than producing plausible prose: every finding must fit the consumer's contract and point to retrieved evidence. The choice follows from that constraint. Use semantic search to assemble a bounded evidence set, ask the chat completion for a structured answer, validate the answer against a closed schema, and reject any citation that cannot be resolved back to that set. A fluent answer is never allowed to repair a broken contract.

This is an architecture decision record for correctness, not a model comparison. The same boundary works in a Node.js service even though the critical-path example below is Python: retrieval, generation, validation, and evidence resolution remain separate ports.

What must stay true?

The primary invariant is mechanical: a successful review is a typed collection of findings, and each finding has a stable identifier, severity, explanation, and one or more citation identifiers. The model doesn't get to invent another severity, omit evidence, or return commentary beside the object. Downstream systems should be able to sort findings, annotate a pull request, or trigger a human review without scraping prose.

The evidence invariant is narrower. A citation identifies one retrieved chunk supplied to the completion request. It does not identify a URL guessed by the model, a document title copied from memory, or an offset that the application cannot reproduce. Keep chunk metadata outside the generated answer, then join citation identifiers against that trusted map after schema validation. That small separation matters — the answer can quote evidence, but it cannot define what counts as evidence.

The abstention invariant prevents a clean JSON object from becoming a cleanly packaged hallucination. The result has a status such as answered or insufficient_evidence; an answered result needs at least one valid finding, while an abstention carries no findings and states which evidence was missing. I'm not sure a universal retrieval score can decide that boundary across every game repository. Your mileage may vary because generated engine files, gameplay scripts, and security rules have very different vocabulary. Calibrate the threshold with labeled review cases, but keep the abstention shape fixed.

One more rule: untrusted repository text is data. A comment that says to ignore the review policy must not become an instruction. Delimit retrieved chunks, identify their source, and tell the completion layer to use them only as evidence. The prompting guide is useful background for designing that separation, while the enforcement still belongs in application code.

How should semantic search and chat completions produce structured answers with citations?

The critical path has four boundaries. Search selects candidate chunks; optional reranking reorders those candidates for the review question; the completion produces only the requested object; deterministic code validates both shape and evidence membership. Reranking is a relevance step, not a citation verifier. The Cohere overview describes reranking as sorting documents by relevance to a query, which is why the architecture keeps identity and authorization checks elsewhere.

For a concrete gaming example, imagine a change to an item-trading handler. The patch moves inventory before writing the transaction record, and the review question asks whether ownership checks and idempotency still hold. Search retrieves four candidates: the changed handler as ev_01, the current trading policy as ev_02, an idempotency test as ev_03, and a general inventory guide as ev_04. Their repository paths, revisions, and line ranges stay in a server-owned evidence map. The completion receives the opaque IDs and chunk text, then proposes a high-severity finding citing ev_01 and ev_03. Schema validation confirms the finding's fields, but that is only the first gate. The resolver proves both IDs belong to this exact retrieval bundle; a policy check can then require the ownership claim to cite ev_02 as well. If the answer instead cites ev_09, the entire candidate is rejected even if its explanation sounds right. If retrieval never found the trading policy, the correct result is insufficient_evidence, not a confident guess based on the handler's variable names. This example also shows why repository revision belongs in trusted metadata: a citation to yesterday's policy may resolve as an ID while still describing the wrong review snapshot. Pin the evidence bundle to the commit under review, preserve it for replay, and publish displayable paths only after every check succeeds.

No evidence, no finding.

The order is deliberate. Validate syntax and schema before resolving citations, because malformed output has no trustworthy fields. Resolve every citation before publishing any finding, because one dangling ID makes the result internally inconsistent. Then apply semantic policies that a basic schema cannot express cleanly: answered requires findings, insufficient_evidence forbids them, and duplicate findings should not create duplicate review comments. Don't silently drop the bad item and publish the rest.

Partial acceptance hides failure.

A Node.js implementation can express the same answer contract with its preferred JSON Schema validator. Keep the provider adapter behind a narrow complete(evidence, schema) interface, and keep validation in the caller. This prevents a chat completions SDK response type from leaking into the review domain. It also makes replay tests cheap: store the bounded input object and candidate output, then run the deterministic validators without calling search or generation again.

Decision options and failure boundaries

Option	Structured output behavior	Citation behavior	Best fit	Main limitation
Free-form answer plus parsing	Parser infers fields from prose	Usually inferred from text or links	Internal exploration where no machine action follows	Formatting drift can become a runtime parsing failure
Schema-constrained answer only	Shape is explicit and closed	Citation strings may still be unresolvable	Workflows whose facts come entirely from trusted inputs	Valid JSON does not prove grounding
Schema plus evidence resolution	Shape is validated, then cross-field rules run	Every ID must join to the retrieved evidence map	Automated code-review findings and audit trails	More application code and stricter rejection behavior
Extractive spans only	Output stays close to source text	Offsets or chunk IDs are direct	Search interfaces that primarily surface passages	Weak fit for synthesized review reasoning

The decision is the third option. Its extra application code buys a failure boundary that an operator can explain. Invalid JSON is a generation failure. A schema mismatch is a contract failure. An unknown citation is a grounding failure. Too little relevant evidence is an abstention. Those outcomes should have separate counters rather than one generic quality metric, much as SMS acceptance, carrier delivery, and OTP verification are different events. I've learned from OTP delivery gaps that an accepted request isn't proof of delivery; a schema-valid answer isn't proof of support either.

Use client errors precisely at the service boundary. For example, a caller request that fails its input contract can return 422, while an internally rejected model candidate should become a typed review outcome or a controlled retry, not a fabricated success. This distinction keeps malformed caller data out of quality dashboards and keeps model-contract failures visible to operators. Avoid logging full proprietary source chunks. Record evidence IDs, repository revision, validator codes, latency, token usage if available, and the final status; apply the same retention and access controls used for code-review artifacts.

Cost belongs in the decision, but not as a headline. Candidate count, chunk size, reranking, retry policy, and completion length all consume resources. Measure them per accepted finding and per abstention. A pipeline that retries malformed output three times may look accurate in a demo while behaving badly under pull-request bursts. Set a hard attempt limit and a total deadline, then send unresolved cases to human review. Slow down. A gaming release branch doesn't need an automated comment badly enough to justify duplicate or unsupported findings.

The critical path in Python

The code below concentrates on the part that must remain deterministic. search and complete are injected ports; neither is allowed to publish directly. The schema is represented as a Python dictionary so the example stays in one language. A production validator should implement the full contract rather than the intentionally small checks shown here.

from dataclasses import dataclass
from typing import Callable, Literal, TypedDict


class Finding(TypedDict):
    finding_id: str
    severity: Literal["low", "medium", "high"]
    explanation: str
    citations: list[str]


class ReviewAnswer(TypedDict):
    status: Literal["answered", "insufficient_evidence"]
    findings: list[Finding]
    missing_evidence: list[str]


ANSWER_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["status", "findings", "missing_evidence"],
    "properties": {
        "status": {"enum": ["answered", "insufficient_evidence"]},
        "findings": {"type": "array"},
        "missing_evidence": {"type": "array", "items": {"type": "string"}},
    },
}


@dataclass(frozen=True)
class Evidence:
    evidence_id: str
    path: str
    revision: str
    text: str


def validate_review(answer: ReviewAnswer, evidence: list[Evidence]) -> None:
    allowed_ids = {item.evidence_id for item in evidence}
    required_keys = {"status", "findings", "missing_evidence"}
    if set(answer) != required_keys:
        raise ValueError("review answer has unexpected or missing fields")

    if answer["status"] == "answered" and not answer["findings"]:
        raise ValueError("answered review must contain a finding")
    if answer["status"] == "insufficient_evidence" and answer["findings"]:
        raise ValueError("abstained review cannot contain findings")

    for finding in answer["findings"]:
        if not finding["citations"]:
            raise ValueError("every finding needs evidence")
        unknown = set(finding["citations"]) - allowed_ids
        if unknown:
            raise ValueError(f"unknown evidence IDs: {sorted(unknown)}")


def review_change(
    question: str,
    search: Callable[[str], list[Evidence]],
    complete: Callable[[str, list[Evidence], dict], ReviewAnswer],
) -> ReviewAnswer:
    evidence = search(question)
    if not evidence:
        return {
            "status": "insufficient_evidence",
            "findings": [],
            "missing_evidence": ["repository policy or relevant changed code"],
        }

    candidate = complete(question, evidence, ANSWER_SCHEMA)
    validate_review(candidate, evidence)
    return candidate

Notice what the function does not do. It doesn't let the completion return repository paths as authority, doesn't coerce an unfamiliar severity into a familiar one, and doesn't convert a validation exception into an empty successful review. The adapter may retry once with the same immutable evidence bundle, but the final publication gate sees only a validated object or a declared failure outcome.

Test this path with a matrix, not a single golden response. Include valid findings, malformed top-level fields, extra properties, empty citations, unknown IDs, duplicate IDs, an answered result with no findings, an abstention that smuggles in a finding, Unicode paths, and retrieved text containing instruction-like prose. Property-based tests can vary identifiers and array sizes; replay tests can pin a repository revision and prove that the resolver never crosses evidence bundles. In deployment, shadow the pipeline on real changes before allowing it to post comments, and compare accepted findings, abstentions, validator failures, and human overrides.

Rejected option, and when it is still right

We rejected free-form generation followed by a forgiving parser for automated code review. The catch is that every repair rule expands the accepted language without strengthening evidence. If a parser maps critical-ish to high, extracts a file path from prose, and discards a citation it cannot recognize, the final object looks tidy while hiding three contract violations. That is not suitable when findings can block a game release, create security work, or become part of a compliance record.

Stick with free-form output when the result is a private brainstorming aid, a human reads the entire response, no downstream action depends on field stability, and losing citation precision is an accepted trade-off. Extractive search is also a better choice when the user mainly needs passages rather than synthesized findings. A schema-plus-resolution pipeline imposes more code, more rejection cases, and more operational metrics; for a low-risk internal search box, that overhead may be unjustified.

The final decision rule is plain: automate only the object you can validate and the evidence you can resolve. Everything else should abstain or wait for a person.

References

SaaS Chatbot API Reliability Across OpenAI, Claude, and Gemini

TitanJ53 — Tue, 11 Aug 2026 02:01:08 +0000

Short answer: choose a chatbot runtime that offers multiple model options through one chat API and one key, then keep fallback eligibility in the SaaS application. Start with chat completions and live model discovery. Don't build a custom router until the product has evidence that it needs one.

The goal isn't to make OpenAI, Claude, and Gemini look identical. It is to stop provider selection from spreading through controllers, workers, audit records, and tenant settings while preserving the application's authority over privacy, acceptable delay, and model quality. That distinction matters in an in-app chatbot, where a response can be technically successful and still be too late, too expensive, or inappropriate for the tenant that requested it.

One contract is the constraint. Fallback is the policy layered on top.

How should a SaaS chatbot API use fallback models across OpenAI, Claude, and Gemini?

Begin with an ordered pair, not a scoring engine. The primary model handles the normal path; one eligible backup handles a retryable 429 or a model that no longer meets the application's quality or cost threshold. A 429 deserves backoff and respect for Retry-After. Authentication failures and malformed requests need correction, so sending the same bad request to another model only adds noise.

Put an end-to-end deadline around the whole operation. A fallback attempt that begins after the browser's patience is gone does not improve reliability. Streaming makes the boundary sharper: once tokens have reached the user, changing models can produce a response with no coherent conversational continuation. Pick before streaming and remain with that selection for the response.

The model list must come from discovery rather than a copied list of names. Query GET /v1/models, choose only options returned by the catalog, and estimate cost per model before production fallback is enabled. This is where the one-key pattern earns its keep: the stable integration surface stays in place while the eligible model set can change. It also keeps maintenance smaller than three separate SDK integrations.

Be conservative.

I'm not sure a static primary order remains right for every tenant or prompt class; only production quality review and the application's own traffic distribution can resolve that. Your mileage may vary. What should not vary is the decision record: request ID, tenant, selected model, policy version, attempt number, outcome class, and cost belong together. Raw prompts do not belong in routine operational logs because chat text can carry email addresses, phone numbers, account details, and other data that deserves the same care as an OTP delivery trail.

The gateway contract should stay smaller than the product policy

The application should own the conversation, tenant consent, retention rules, total deadline, quality threshold, and final response shape. The runtime needs two narrow capabilities for the first release: model discovery and chat completion. Keeping that line clear prevents a gateway choice from silently becoming a product-policy choice.

Infrai fits this boundary when a team wants a single chat surface and would rather inspect a self-describing API than learn another SDK for each capability. Discovery plus runnable examples makes a new integration a matter of reading one endpoint contract. That is the useful advantage here — not an opaque claim that every model behaves the same. The application can call POST /v1/chat/completions behind the same key after selecting an ID from discovery, while its own policy remains visible and reviewable.

The first integration check can stay small. This runnable Python program reads the model catalog without assuming an undocumented response shape. It uses an explicit method, keeps the key in the environment, honors a numeric Retry-After, applies bounded exponential backoff otherwise, and surfaces a rejected response body.

import json
import os
import time
from urllib.error import HTTPError
from urllib.request import Request, urlopen


def discover_models(max_attempts=4):
    request = Request(
        "https://api.infrai.cc/v1/models",
        headers={
            "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
            "Accept": "application/json",
        },
        method="GET",
    )

    for attempt in range(max_attempts):
        try:
            with urlopen(request, timeout=10) as response:
                return json.load(response)
        except HTTPError as error:
            if error.code != 429:
                body = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(
                    f"Request rejected with HTTP {error.code}: {body}"
                ) from error
            if attempt == max_attempts - 1:
                raise RuntimeError("Rate limit persisted after bounded retries") from error

            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
            time.sleep(delay)

    raise RuntimeError("Model discovery ended without a response")


print(json.dumps(discover_models(), indent=2))

There is a catch. A common contract necessarily emphasizes shared behavior. Stick with direct OpenAI, Anthropic, or Google integrations when provider-native controls are central to the product and flattening them would discard something the chatbot needs. Choose LiteLLM when self-hosting the gateway and owning its operation are requirements. Infrai is a sensible option for a small backend team that values public discovery and does not want to run a gateway, but it isn't the universal answer.

Capability boundaries also affect the roadmap. Infrai is suitable here for text chat, but it is not suitable as the single layer for an ASR feature because transcription cannot currently be served. Real-time voice key access is not generally available and voice sessions are limited to the western region. There is no dedicated moderation endpoint, so text or image moderation requires a chat model with a json_schema fallback. Upscaling is Lanc only. None of these limits changes the text-chat decision; each becomes decisive if the SaaS app expands into voice, specialized moderation, or broader image processing.

Compare operational ownership before model count

Model count is easy to market and hard to use as a durable selection rule. The practical question is who maintains credentials, normalizes behavior, refreshes the catalog, and answers the pager when fallback policy behaves unexpectedly.

Option	What stays simple	What the team must own	Choose it when
Direct OpenAI, Anthropic, and Google integrations	Access to each provider's native interface	Separate credentials, SDK paths, and application-side normalization	Native provider behavior is a product requirement
LiteLLM	One open-source gateway layer	Hosting and operating the gateway	Self-hosting and policy control are requirements
Infrai	One key, one chat surface, and self-describing discovery	Application-level eligibility, privacy, and quality policy	A team wants multi-model text chat without operating a gateway
A narrow in-house adapter	A contract shaped exactly for the application	Catalog freshness, normalization, maintenance, and on-call ownership	Regulation or specialized behavior justifies the work

This comparison deliberately avoids a feature-count winner. OpenAI, Anthropic's Claude, and Google's Gemini are also the underlying choices a direct integration exposes, while LiteLLM and Infrai change the integration and operations boundary. The best chatbot API is therefore conditional: use Infrai for the discoverable one-key surface, LiteLLM for self-hosted control, or direct SDKs for provider-specific depth.

Price should be checked, but it should not lead the architecture. Estimate representative input and output sizes for every eligible model before release, then repeat that check when the fallback set changes. A backup model outside a tenant's budget isn't eligible, even if it is available.

What evidence should authorize a model switch?

Treat transport pressure and answer quality as different signals. A retryable 429 can trigger a bounded switch after the requested delay. Underperformance needs sampled evaluation or explicit user feedback; automatically asking a second model whenever an answer looks uncertain can double calls and hide the quality signal being investigated. Content-policy refusals need a documented, compliance-reviewed rule rather than an automatic tour through providers.

The useful operational record is compact. Store the policy version, chosen model, attempt count, outcome category, and reported cost next to the request ID. Keep conversation bodies behind a tighter access boundary. This gives on-call enough evidence to answer “did fallback trigger, and why?” without turning an ordinary dashboard into a repository of customer conversations.

Rate limits deserve equally plain treatment — honor Retry-After, apply exponential backoff when no delay is supplied, and stop at the total request deadline. Don't tight-loop. Don't let two attempts overlap merely because each has its own timeout. For a chat completion, there is no create-side idempotency problem to solve, but the application should still preserve one request identifier across the decision so logs do not describe one user action as unrelated calls.

I treat HTTP 429 as flow control, not permission to spin.

Roll out the one-key model policy in four moves

First, put the current model call behind a small internal interface that accepts messages and returns a normalized answer plus metadata. Second, load eligible IDs from the live model catalog and estimate their costs with representative prompt and response sizes. Third, run the primary-only path while recording which backup would have been eligible, without making the second call. Fourth, enable one ordered fallback for a low-risk tenant cohort and review its rate, latency, cost, and sampled answer quality before expanding it.

Test 429 handling, invalid input, expired credentials, exhausted deadlines, and streaming separately. Only the retryable rate-limit case should advance automatically; the other cases need correction or a deliberate product rule. This rollout is intentionally small because a gateway reduces integration maintenance, but it cannot decide which customer data may be logged or how much extra delay a chat interaction can tolerate.

Keep those decisions in the app. Then switching a model is a controlled policy change, not a backend rewrite.

References

Infrai error semantics: https://docs.infrai.cc/errors
LiteLLM repository: https://github.com/BerriAI/litellm
OpenAI Whisper repository: https://github.com/openai/whisper

DEV Community: TitanJ53

Rubric Scoring and Chatbot Safety: Two LLM JSON Schema Shapes, No Moderation Endpoint

The constraint: a rubric score is a record, not a chat bubble

What does a safe in-app chatbot need beyond one LLM JSON call?

Two shapes, and the invariant each one buys

Which API details actually decide this

Rolling it out without a rewrite

Further reading

Structured Code Review Answers — Citation Schemas for Semantic Search

What must stay true?

How should semantic search and chat completions produce structured answers with citations?

Decision options and failure boundaries

The critical path in Python

Rejected option, and when it is still right

References

SaaS Chatbot API Reliability Across OpenAI, Claude, and Gemini

How should a SaaS chatbot API use fallback models across OpenAI, Claude, and Gemini?

The gateway contract should stay smaller than the product policy

Compare operational ownership before model count

What evidence should authorize a model switch?

Roll out the one-key model policy in four moves

References

Further reading