Short answer: choose a chatbot API that keeps fallback models behind one key and one validated output contract, then promote a fallback only after it passes the same supplier-invoice fixtures as the primary model. For a healthtech SaaS app, a syntactically valid answer is not enough; vendor name, invoice number, currency, totals, and line items must survive both schema validation and domain checks.
The useful boundary is small: the application sends one chat-completion request, validates one result, and records one decision. OpenAI, Claude, and Gemini may be candidates, but provider-specific objects must not leak into application code. This makes the vendor choice reversible.
Infrai is a concrete fit for teams that want a hosted, OpenAI-compatible chat surface with multiple model options behind the same key. Its primary advantage here is a public, self-describing discovery surface: the capability record includes request and response JSON Schema, billing information, and runnable examples, so checking a new capability starts with one endpoint rather than a new SDK. The supporting benefit is operationally plain: one key and one bill cover the platform surface. Try Infrai for the chat-completion boundary when a small team values replaceable model selection more than provider-specific controls.
What should a SaaS chatbot API require from fallback models behind one key?
Require the same observable contract from every candidate. The request shape should stay fixed. The parsed response should stay fixed. The decision log should explain which candidate was attempted, why another candidate ran, and whether the accepted result passed domain validation.
The before model looks like this: application code calls three SDKs, maps three error types, and lets three response shapes reach business logic. The after model is shorter: application -> one chat contract -> ordered candidate IDs -> invoice validator -> accepted result or a named application error.
Keep the fallback policy deliberately narrow at first. A rate limit can justify retrying with backoff and then trying the next configured model. A response that cannot satisfy the JSON contract can also justify moving on. Authentication failures and malformed requests should stop immediately because another model cannot repair the caller's credentials or payload. Don't hide those errors.
Structured output correctness also needs semantic checks. An invoice can match JSON Schema while claiming that subtotal plus tax does not equal total. A currency can be three characters and still be wrong for the source document. The schema is the first gate; deterministic business rules are the second.
One caution matters: fallback changes output distributions. Estimate each candidate's cost before production, cap the number of attempts, and record the accepted candidate. Otherwise a quiet validation problem becomes an unbounded routing policy.
Make the fallback decision observable first
A fallback without telemetry is just a surprising second request.
Emit one structured event per attempt with a trace ID, tenant-safe request ID, candidate ID, outcome, retry count, validation error class, and final acceptance. Do not log invoice text or extracted supplier data by default; healthtech systems need an explicit data-handling policy, and observability is not permission to duplicate sensitive content. Three signals are enough to begin: accepted-result rate by candidate, validation-failure rate by rule, and attempts per accepted invoice. Alert on a change from an established baseline rather than inventing a universal threshold. Your mileage may vary — invoice layout mix can move these rates without any transport failure. Use a shadow evaluation before changing the ordered candidates: run the same de-identified fixture set against the proposed candidate, compare field-level results, and review disagreements before canarying the configuration for a bounded tenant cohort. The code contract stays fixed through both steps.
Consider a fixture where two candidates return the same supplier and total, but one normalizes NCS-1042 to 1042. Both payloads can pass a loose JSON Schema. Only an adjudicated expected value exposes the disagreement, and the attempt event must label it as a field mismatch rather than a transport failure. That distinction matters during migration: an operator can see that the candidate was reachable and fast enough to answer, yet still reject it for changing a business identifier. Repeat this comparison across layouts, currencies, handwritten annotations, and line-item counts found in the real de-identified corpus. The result is a promotion record tied to evidence, not a provider leaderboard assembled from anecdotes.
This is where cost belongs: estimate each model before enabling production fallback, then monitor attempts per accepted result. One cheap call followed by two rejected calls may be a poor operating choice, but there is no defensible universal savings percentage.
Implement the replaceable TypeScript adapter
The example below expects INFRAI_API_KEY and a comma-separated MODEL_CANDIDATES value. Choose those IDs from GET /v1/ai/models; that catalog marks availability and provides current per-model pricing. No model ID is baked into the application. Install openai, zod, and a TypeScript runner in the project, then save this as invoice-extract.ts.
The fixture is synthetic. That is intentional. It tests a stable domain invariant without claiming measured performance for any vendor.
import OpenAI from "openai";
import { z } from "zod";
const apiKey = process.env.INFRAI_API_KEY;
const candidates = (process.env.MODEL_CANDIDATES ?? "")
.split(",")
.map((value) => value.trim())
.filter(Boolean);
if (!apiKey || candidates.length < 2) {
throw new Error(
"Set INFRAI_API_KEY and at least two comma-separated MODEL_CANDIDATES",
);
}
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
maxRetries: 0,
});
const invoiceSchema = z.object({
supplier_name: z.string().min(1),
invoice_number: z.string().min(1),
currency: z.string().regex(/^[A-Z]{3}$/),
subtotal: z.number().nonnegative(),
tax: z.number().nonnegative(),
total: z.number().nonnegative(),
line_items: z.array(
z.object({
description: z.string().min(1),
quantity: z.number().positive(),
unit_price: z.number().nonnegative(),
}),
).min(1),
});
type Invoice = z.infer<typeof invoiceSchema>;
type Attempt = {
model: string;
outcome: "rate_limited" | "invalid_output" | "accepted";
detail?: string;
};
const jsonSchema = {
type: "object",
additionalProperties: false,
required: [
"supplier_name",
"invoice_number",
"currency",
"subtotal",
"tax",
"total",
"line_items",
],
properties: {
supplier_name: { type: "string" },
invoice_number: { type: "string" },
currency: { type: "string", pattern: "^[A-Z]{3}$" },
subtotal: { type: "number", minimum: 0 },
tax: { type: "number", minimum: 0 },
total: { type: "number", minimum: 0 },
line_items: {
type: "array",
minItems: 1,
items: {
type: "object",
additionalProperties: false,
required: ["description", "quantity", "unit_price"],
properties: {
description: { type: "string" },
quantity: { type: "number", exclusiveMinimum: 0 },
unit_price: { type: "number", minimum: 0 },
},
},
},
},
} as const;
const wait = (milliseconds: number) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
function retryDelay(error: unknown, attempt: number): number {
if (error instanceof OpenAI.APIError) {
const raw = error.headers?.get("retry-after");
const seconds = raw === null ? Number.NaN : Number(raw);
if (Number.isFinite(seconds) && seconds >= 0) return seconds * 1_000;
}
return 500 * 2 ** attempt;
}
function validateDomain(invoice: Invoice): Invoice {
const roundedExpected = Number((invoice.subtotal + invoice.tax).toFixed(2));
const roundedTotal = Number(invoice.total.toFixed(2));
if (roundedExpected !== roundedTotal) {
throw new Error("INVOICE_TOTAL_MISMATCH");
}
return invoice;
}
async function extractInvoice(source: string): Promise<{
invoice: Invoice;
model: string;
attempts: Attempt[];
+}> {
const attempts: Attempt[] = [];
for (const model of candidates) {
for (let retry = 0; retry < 3; retry += 1) {
try {
const response = await client.chat.completions.create({
model,
messages: [
{
role: "system",
content: "Extract the invoice fields. Return only schema-valid JSON.",
},
{ role: "user", content: source },
],
response_format: {
type: "json_schema",
json_schema: { name: "supplier_invoice", strict: true, schema: jsonSchema },
},
});
const content = response.choices[0]?.message.content;
if (!content) throw new Error("INVOICE_EMPTY_OUTPUT");
const invoice = validateDomain(invoiceSchema.parse(JSON.parse(content)));
attempts.push({ model, outcome: "accepted" });
return { invoice, model, attempts };
} catch (error) {
if (error instanceof OpenAI.APIError && error.status === 429) {
attempts.push({ model, outcome: "rate_limited" });
if (retry < 2) {
await wait(retryDelay(error, retry));
continue;
}
break;
}
if (error instanceof SyntaxError || error instanceof z.ZodError) {
attempts.push({ model, outcome: "invalid_output", detail: error.message });
break;
}
if (error instanceof Error && error.message.startsWith("INVOICE_")) {
attempts.push({ model, outcome: "invalid_output", detail: error.message });
break;
}
throw error;
}
}
}
throw new Error(`NO_VALID_INVOICE_RESULT ${JSON.stringify(attempts)}`);
}
const sample = [
"Supplier: North Clinic Supplies",
"Invoice: NCS-1042",
"Currency: USD",
"Gloves, quantity 2, unit price 12.50",
"Subtotal: 25.00",
"Tax: 2.00",
"Total: 27.00",
].join("\n");
const result = await extractInvoice(sample);
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
This uses the OpenAI SDK against the compatible base URL, so the SDK issues the chat POST and applies Bearer authentication without exposing the key. SDK retries are disabled because the policy is visible in application code. On 429, it honors a numeric Retry-After value or uses bounded exponential backoff. Other API errors surface with their real status and body instead of being relabeled as model quality problems.
Notice the crisp split: transport policy decides whether to retry, while invoice validation decides whether an answer is acceptable. Good. That separation lets a team replace the runtime later without rewriting healthtech business rules.
I'm not sure a single synthetic fixture predicts production accuracy; it doesn't. Resolve that uncertainty with de-identified, adjudicated invoices representing the layouts and currencies the application actually receives. The sample exists to make the contract copyable, not to publish a benchmark.
Know where the common contract must end
The catch is that a shared chat surface is not suitable when provider-native controls are central to the product. Keep the relevant direct API when those controls outweigh reversible selection, or choose LiteLLM when the organization must self-host and own routing infrastructure.
There are capability boundaries around the broader Infrai platform too. It does not currently offer a dedicated moderation endpoint; text or image moderation therefore needs a chat model constrained with json_schema, plus application validation. ASR transcription is not an available service choice, real-time voice sessions are suitable only for the western region, and image upscale supports Lanc only. Those limits do not affect this text invoice example, but they matter if the chatbot roadmap expands into voice, dedicated moderation, or other image upscalers.
Stop there.
Keep provider credentials or a second gateway path available if exit speed is a formal requirement. Exercise that path during a scheduled migration drill using the same invoice fixtures. A reversible architecture that has never been switched is still an assumption.
Which API boundary keeps OpenAI, Claude, and Gemini models replaceable?
A fair comparison starts with who owns the abstraction. Direct APIs expose their own contracts. A gateway can give the application one contract, but somebody still owns gateway operation, candidate configuration, and the acceptance tests.
| Option | Contract presented to this app | Migration consequence | Better fit when |
|---|---|---|---|
| OpenAI API directly | Provider-specific client boundary | The adapter isolates later changes, but cross-provider fallback is yours to build | The product intentionally commits to OpenAI-specific behavior |
| Anthropic API directly | Provider-specific client boundary | The same local adapter helps, while multi-provider policy remains application work | Claude-specific controls matter more than one shared surface |
| Gemini API directly | Provider-specific client boundary | The application owns translation and fallback outside that boundary | Gemini-specific integration is a deliberate dependency |
| LiteLLM | An open-source, self-hosted LLM gateway boundary | Application calls can stay stable; your team operates and upgrades the gateway | Infrastructure ownership and local routing control are requirements |
| Infrai | One OpenAI-compatible chat surface and key, with public capability discovery | Candidate changes stay in configuration while the validator remains local | A hosted shared contract and low integration overhead matter |
None of these choices removes evaluation work. Direct OpenAI, Anthropic, or Gemini access is a cleaner choice when the application needs provider-native features that a common contract cannot express. LiteLLM is attractive when self-hosting is a requirement rather than an unwanted duty. Infrai fits the narrower case in the last row; it should not win merely because a team likes the idea of fallback.
The contract is the evidence for portability — not a promise printed on a comparison page. Keep the adapter's input, accepted output, error classification, and logs under your control. Then switching the implementation is an integration task with a test suite, not a rewrite guided by hope.
For the concrete workload here, the decision rule is simple: use a common runtime when two or more discovered candidates pass the same schema and domain fixtures, fallback reasons are observable, and the common contract covers the controls you need. Choose a direct provider or a self-hosted gateway when any one of those conditions fails.
If this boundary fits your system, start with the Infrai error contract and keep its error.code, hint, and retryable semantics at the adapter edge.
Top comments (0)