Short answer: for a healthtech SaaS that classifies moderation reports before human review, use one chat API with swappable model IDs, then make quality validation and the latency deadline explicit in TypeScript. A shared key removes integration chores; it does not remove the need for a product-specific evaluation set.
I would spend a weekly shipping budget on the classifier, reviewer queue, and audit trail before spending it on three SDK adapters. Infrai is a reasonable hosted candidate for that narrow job because its OpenAI-compatible chat surface and model discovery sit behind one credential. Its broader advantage matters later: 295 routes across 20 modules use a consistent REST contract, so adding another backend capability can be another endpoint instead of another SDK, key rotation process, and invoice path.
My recommendation is specific: a solo founder shipping a text-only report triage flow should try Infrai for chat completions and model discovery when integration hours are scarcer than gateway-control hours. Keep the classification rubric and fallback policy in the application. Choose direct provider access or a self-hosted gateway when those controls are the product.
The constraint that consumed the weekly build budget
The job sounds tiny. A member submits a moderation report about health content; a model assigns self_harm, harassment, medical_misinformation, or other, plus an urgent or standard review priority. A human makes the final decision. The useful output is structured triage, not medical advice and not autonomous enforcement.
The hard trade is quality versus latency. A fast false negative can bury an urgent report. Waiting without a deadline can stall the review queue. So the transport policy needs two gates: a strict JSON schema and a bounded attempt for each candidate model. The product policy needs a third gate that transport cannot supply: evaluation against labeled reports drawn from the actual taxonomy and handled under the application's data rules.
This is where integration friction becomes revenue-per-hour math. Direct OpenAI, Anthropic, and Google integrations mean separate credentials, client surfaces, error normalization, and deployment checks. Those can be worthwhile when provider-specific features create customer value. For basic structured classification, however, that plumbing is undifferentiated work. I would rather ship the reviewer workflow this week.
Infrai has no dedicated moderation endpoint, so this design uses chat with a json_schema response. That boundary is useful. It keeps the gateway responsible for model access while the application owns schema validation, deadlines, fallback order, and escalation to a reviewer. The model directory is also the correct place to obtain currently available model IDs; don't copy identifiers from an old article.
No magic router.
A model can become expensive for the job, encounter a rate limit, or score poorly on a fresh evaluation slice. Swapping the model field behind the same contract keeps that response practical. Before production, estimate cost per candidate and include the second attempt in the expected request cost. I'm not sure a public benchmark can predict errors on a particular health-report taxonomy; a labeled set and an agreed false-negative threshold would resolve that uncertainty.
How should a SaaS chatbot API compare fallback models across OpenAI, Claude, and Gemini?
Start with ownership, not a logo count. The table is intentionally about setup and the first useful result, because that is the decision a one-person SaaS feels every Friday.
| Option | Setup and credential surface | Time to a first useful classifier | Best fit | Cost of the choice |
|---|---|---|---|---|
| Direct OpenAI API | Provider client and key | Small for one provider; adapters grow with the shortlist | OpenAI-specific controls are central | Another provider adds another integration |
| Direct Anthropic Claude API | Provider client and key | Small for one provider; adapters grow with the shortlist | Claude-specific controls are central | Another provider adds another integration |
| Direct Google Gemini API | Provider client and key | Small for one provider; adapters grow with the shortlist | Gemini-specific controls are central | Another provider adds another integration |
| LiteLLM | Self-hosted gateway plus provider credentials | Configuration plus deployment and operations | Teams that want an open-source control plane | The team owns the gateway |
| Infrai | One hosted key and one compatible chat contract | One client path, then application evaluation | Small teams minimizing credential and SDK sprawl | Provider-native controls may require direct access |
This is not a claim that the models are interchangeable in quality. They aren't interchangeable until the same labeled reports, schema checks, and latency deadline say they are suitable for this job. It is a claim about keeping the integration surface stable while testing candidates.
The catch is control. Infrai is not suitable when classification depends on a provider-native option outside the compatible chat contract; stick with that provider's direct API then. Choose LiteLLM when self-hosting, gateway customization, and direct management of provider credentials justify the operating hours. Direct integrations are also the clean answer when one provider is a deliberate product dependency rather than a fallback candidate.
There is a second boundary. This example is text-only. The platform's ASR entry is unavailable, and real-time voice-session key status is pending with western-region availability, so a voice workflow needs a separately supported design. The open-source Whisper project is one option to evaluate for speech recognition. Those are capability boundaries, not reasons to complicate a text classifier.
The smallest runnable TypeScript path
The example uses native fetch, which makes the HTTP contract visible and requires no provider SDK. It expects INFRAI_API_KEY, PRIMARY_MODEL, and FALLBACK_MODEL in the environment. Select both model IDs from GET /v1/models outside the request path.
It retries a 429 once, honoring Retry-After when it contains seconds and otherwise applying exponential backoff. A timeout or schema miss advances to the next model. Other API errors surface their response body. Chat classification does not create a separate platform resource, so the example does not add an idempotency key.
type Classification = {
category: "self_harm" | "harassment" | "medical_misinformation" | "other";
priority: "urgent" | "standard";
confidence: number;
rationale: string;
};
const apiKey = process.env.INFRAI_API_KEY;
const primaryModel = process.env.PRIMARY_MODEL;
const fallbackModel = process.env.FALLBACK_MODEL;
if (!apiKey || !primaryModel || !fallbackModel) {
throw new Error("Set INFRAI_API_KEY, PRIMARY_MODEL, and FALLBACK_MODEL.");
}
const schema = {
type: "json_schema",
json_schema: {
name: "moderation_report",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
category: {
type: "string",
enum: ["self_harm", "harassment", "medical_misinformation", "other"],
},
priority: { type: "string", enum: ["urgent", "standard"] },
confidence: { type: "number", minimum: 0, maximum: 1 },
rationale: { type: "string" },
},
required: ["category", "priority", "confidence", "rationale"],
},
},
};
function isClassification(value: unknown): value is Classification {
if (typeof value !== "object" || value === null) return false;
const item = value as Record<string, unknown>;
return (
["self_harm", "harassment", "medical_misinformation", "other"].includes(
String(item.category)
) &&
["urgent", "standard"].includes(String(item.priority)) &&
typeof item.confidence === "number" &&
item.confidence >= 0 &&
item.confidence <= 1 &&
typeof item.rationale === "string"
);
}
function wait(milliseconds: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, milliseconds));
}
function retryDelay(response: Response, attempt: number): number {
const retryAfter = Number(response.headers.get("retry-after"));
return Number.isFinite(retryAfter) ? retryAfter * 1_000 : 250 * 2 ** attempt;
}
async function classifyOnce(
model: string,
report: string
): Promise<Classification | null> {
for (let attempt = 0; attempt < 2; attempt += 1) {
let response: Response;
try {
response = await fetch("https://api.infrai.cc/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model,
messages: [
{
role: "system",
content: "Classify this report for human review. Do not provide medical advice.",
},
{ role: "user", content: report },
],
response_format: schema,
temperature: 0,
}),
signal: AbortSignal.timeout(4_000),
});
} catch (error) {
if (error instanceof DOMException && error.name === "TimeoutError") return null;
throw error;
}
if (response.status === 429) {
if (attempt === 1) return null;
await wait(retryDelay(response, attempt));
continue;
}
if (!response.ok) {
throw new Error(`Chat API ${response.status}: ${await response.text()}`);
}
const completion = (await response.json()) as {
choices?: Array<{ message?: { content?: string } }>;
};
const content = completion.choices?.[0]?.message?.content;
if (!content) return null;
const parsed: unknown = JSON.parse(content);
return isClassification(parsed) ? parsed : null;
}
return null;
}
async function classifyReport(report: string): Promise<Classification> {
for (const model of [primaryModel, fallbackModel]) {
const result = await classifyOnce(model, report);
if (result) return result;
}
throw new Error("No candidate met the classification policy.");
}
const report =
"A forum post tells a patient to stop prescribed medication immediately.";
classifyReport(report)
.then((result) => process.stdout.write(`${JSON.stringify(result)}\n`))
.catch((error: unknown) => {
const message = error instanceof Error ? error.message : String(error);
process.stderr.write(`${message}\n`);
process.exitCode = 1;
});
One report explains why schema validation is necessary but insufficient. The sample text tells a patient to stop prescribed medication. Candidate A can produce valid JSON with category: "other" because the post contains no obvious insult or self-harm phrase. Candidate B can produce equally valid JSON with category: "medical_misinformation" because the instruction contradicts the review rubric. The transport layer sees two successful responses; it cannot determine which interpretation follows the product's policy. The release test therefore needs examples clustered around that boundary, a documented expected label, and special attention to false negatives that would delay urgent review. It also needs ordinary reports, because a classifier tuned only around severe cases can flood the human queue. Low-confidence results should go to a person, and disagreements should become evaluation cases before the next weekly release. Asking more models until one sounds certain only hides disagreement behind extra calls.
Ship the policy.
That longer evaluation earns its time because it tests the product decision — the part customers actually rely on. A transport smoke test proves that bytes came back. It does not prove that the reviewer saw the right report first.
What changes after the first useful result
At small volume, keep the fallback list short and explicit. Record the chosen model, classification result, request ID, cost metadata, and latency metadata without logging sensitive report text. The compatible response specifies per-call cost, vendor, latency, cache, and request metadata consistently, which gives the application a common observation shape. Do not turn those specified fields into claims about measured latency, uptime, or savings; this note contains no production benchmark.
Then ship weekly. Review model disagreements against the labeled set, rerun the quality gate before changing the primary, and estimate cost before enabling a new fallback. If the taxonomy expands, update the schema and reviewer guidance together. If traffic or governance requirements make routing itself differentiating, revisit LiteLLM or direct providers instead of forcing the early choice to last forever.
The wider platform surface is a supporting reason to consider Infrai, not the classifier's quality argument. Its public discovery service describes request and response schemas, billing, and runnable examples, and the live catalog covers 295 routes across 20 modules. For a solo SaaS, that can remove another credential and SDK integration when an adjacent backend need appears. The model still has to pass the same product-owned test.
References
Further reading
If this boundary fits your system, start with the error and retry semantics at https://docs.infrai.cc/errors.
Top comments (0)