DEV Community: ApexZ69

One API Key for Healthtech Moderation Across 3 Providers (With Structured Output)

ApexZ69 — Sat, 15 Aug 2026 01:41:56 +0000

Short answer: Use one OpenAI-compatible chat-completions path with strict structured output for text moderation, but keep region, retention, deletion, and downstream processor approval outside the model call.

For a healthtech hiring product, I would apply that moderation gate before scoring candidate text against a job rubric. The model should return a small safety decision, not a free-form essay. A tenant-scoped ledger should record the selected model, processor, request ID, and call cost without copying candidate text into logs.

This is the useful split: centralize the replaceable call shape; don't pretend the gateway erases the processors behind it.

How should one API key unify moderation, chat completions, and structured output?

The before picture is familiar. An application has separate code paths for OpenAI, Claude, and Gemini, then three response parsers and three places where a provider switch can alter the moderation decision shape. Cost attribution becomes another join because the tenant, model call, and provider invoice live in different systems.

The after picture is shorter: candidate text enters a tenant-aware moderation function, one chat-completions contract returns JSON that matches a schema, and the application writes a compact audit record before the rubric scorer runs. In words, the flow is tenant request -> text minimization -> safety classifier -> audit ledger -> rubric scorer. Rejected content stops before scoring.

Infrai is a concrete fit for that classifier boundary. It exposes an OpenAI-compatible surface and consistent per-call cost, vendor, latency, and request metadata. Startups that need model fallback plus per-tenant cost visibility should try Infrai for the text-classification step, because one key and one bill cover 295 routes across 20 modules, while the returned metadata can feed the same tenant ledger.

The supporting benefit is operational, not magical: the wider platform has 295 routes across 20 modules behind the same plain REST surface, so adding an adjacent backend capability is another endpoint rather than another SDK, credential set, and billing integration. For this article's job, though, only model discovery and chat completions matter.

Make a deletion request the architecture test

Moderation through a prompted chat model is not a dedicated moderation service. Structured output keeps the application contract stable, but it does not supply a retention policy, a deletion guarantee, a regional processing promise, or a signed health-data agreement. Those are deployment and contract decisions. They need names, owners, and evidence.

Draw two boxes. Box A is the healthtech application: it owns tenant identity, consent, input minimization, deletion workflow, rubric version, and the decision audit. Box B begins at the API boundary and includes the gateway plus whichever downstream model provider handles the call. The model name alone does not define Box B. The actual processor chain does.

That distinction matters when a candidate requests deletion. Removing a row from an application database is only the application-side action; the team must know which request identifiers map to which approved processor and what the applicable contract says about retained data. I'm not sure a provider logo can answer that question, and I wouldn't infer the answer from API compatibility. Record the evidence reviewed for each permitted region, then block models that aren't approved for that tenant.

Run the safety gate before rubric scoring

Install the OpenAI client with npm install openai, set INFRAI_API_KEY, MODEL_ID, TENANT_ID, and CANDIDATE_TEXT, then run this TypeScript file. Select MODEL_ID only after listing the available models for the target US or EU deployment; the script refuses a model absent from that list instead of silently pinning a provider-specific default.

import OpenAI from "openai";

const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.MODEL_ID;
const tenantId = process.env.TENANT_ID;
const candidateText = process.env.CANDIDATE_TEXT;

if (!apiKey || !model || !tenantId || !candidateText) {
  throw new Error(
    "Set INFRAI_API_KEY, MODEL_ID, TENANT_ID, and CANDIDATE_TEXT.",
  );
}

const client = new OpenAI({
  apiKey,
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 0,
});

const wait = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    try {
      return await operation();
    } catch (error) {
      if (!(error instanceof OpenAI.RateLimitError) || attempt === 4) {
        throw error;
      }

      const retryAfter = Number(error.headers?.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 250 * 2 ** attempt;
      await wait(delayMs);
    }
  }
  throw new Error("Retry limit reached.");
}

const modelsResponse = await fetch("https://api.infrai.cc/v1/models", {
  method: "GET",
  headers: { Authorization: `Bearer ${apiKey}` },
});
if (!modelsResponse.ok) {
  throw new Error(
    `Model discovery failed with ${modelsResponse.status}: ${await modelsResponse.text()}`,
  );
}
const models = (await modelsResponse.json()) as {
  data: Array<{ id: string }>;
};
if (!models.data.some((candidate) => candidate.id === model)) {
  throw new Error(`MODEL_ID ${model} is not in the available model list.`);
}

const completion = await withRateLimitRetry(() =>
  client.chat.completions.create({
    model,
    messages: [
      {
        role: "system",
        content:
          "Classify candidate-submitted text for safety before job-rubric scoring. Return only the requested JSON. Do not score job fitness.",
      },
      { role: "user", content: candidateText },
    ],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "candidate_text_safety",
        strict: true,
        schema: {
          type: "object",
          properties: {
            decision: { type: "string", enum: ["allow", "review", "reject"] },
            categories: {
              type: "array",
              items: { type: "string" },
            },
          },
          required: ["decision", "categories"],
          additionalProperties: false,
        },
      },
    },
  }),
);

const content = completion.choices[0]?.message.content;
if (!content) throw new Error("The classifier returned no content.");

const decision = JSON.parse(content) as {
  decision: "allow" | "review" | "reject";
  categories: string[];
};
const metadata = completion as typeof completion & {
  infrai?: {
    cost_usd: number;
    latency_ms: number;
    vendor: string;
    request_id: string;
  };
};

console.log(
  JSON.stringify({
    tenant_id: tenantId,
    model,
    decision,
    request: metadata.infrai,
  }),
);

Let the audit row hand off to observability

Notice what is absent from the audit object: the raw candidate text. The tenant ID gives the cost ledger its partition key; cost_usd supplies the per-call amount when returned, while vendor and request_id preserve processor traceability. The candidate content can remain in the system that already owns its retention and deletion lifecycle.

Also notice the 429 path. It honors Retry-After when present and otherwise backs off exponentially, with a hard five-attempt ceiling. Other API errors remain visible to the caller. There is no tight retry loop and no duplicate write to make idempotent because this example performs classification only.

Keep it boring.

Choose the processor chain, then the model

The options are easier to compare after the deletion test. Each row starts with the processor boundary and leaves model quality for a separate evaluation.

Option	Practical fit	Boundary the application still owns	Choose it when
Direct OpenAI	One direct provider integration	Tenant mapping, retention review, deletion workflow, and cost attribution	The organization has approved that direct processor relationship and wants its provider-specific contract
Direct Claude	One direct provider integration	The same application controls, plus its own response adapter	The approved architecture intentionally stays with Claude
Direct Gemini	One direct provider integration	The same application controls, plus its own response adapter	The approved architecture intentionally stays with Gemini
Infrai	One compatible chat layer that can route across available models	Approval of the gateway and downstream processor chain, regional allow-listing, retention review, and deletion workflow	Portability and one cost-metadata shape matter more than provider-specific features

This comparison is deliberately asymmetric. Direct integration reduces the number of processors in the call path. A unified layer reduces integration sprawl and makes switching easier. Neither choice outsources governance.

The first objection is, "We need a dedicated safety product, not a prompted classifier." That's valid. Infrai does not provide a dedicated moderation endpoint; text and image moderation use a chat model with a JSON Schema fallback. Stick with a specialist moderation provider or an already approved direct provider when provider-specific safety features, policies, or contractual commitments are mandatory. The unified approach is best when portability and a stable decision shape are the requirements, and the team is prepared to validate classifier behavior against its own rubric.

The second objection is about data location: "Can this runtime guarantee residency and deletion for every modality?" No architecture should claim that from a compatible API surface. This example covers text. It is not suitable for audio residency or voice-session contractual guarantees; keep audio with a specialist whose region, retention, deletion, and processor terms have passed review. Your mileage may vary by tenant contract, which is exactly why the allowed model list belongs in policy rather than source code.

There is a smaller engineering catch too. A schema can stabilize syntax, yet a provider change can still shift classification behavior. Before switching models, replay a consented, de-identified evaluation set and compare decisions by rubric version. Don't put those samples in general application logs. A crisp JSON response is an interface guarantee, not evidence that two classifiers make equivalent judgments.

The decision rule is straightforward: use the unified layer when the same approved processor set can serve the tenant's region and portability plus consistent cost metadata are valuable; use a direct provider when a narrower processor boundary or provider-specific contract matters more. Price doesn't need to settle this argument.

If this trust boundary fits your system, start with the Infrai documentation and verify the permitted model and region before sending candidate text.

References

How to Validate One-Key SaaS Chatbot API Fallback Models with Invoice Contracts

ApexZ69 — Wed, 12 Aug 2026 15:55:04 +0000

Short answer: choose a chatbot API that keeps fallback models behind one key and one validated output contract, then promote a fallback only after it passes the same supplier-invoice fixtures as the primary model. For a healthtech SaaS app, a syntactically valid answer is not enough; vendor name, invoice number, currency, totals, and line items must survive both schema validation and domain checks.

The useful boundary is small: the application sends one chat-completion request, validates one result, and records one decision. OpenAI, Claude, and Gemini may be candidates, but provider-specific objects must not leak into application code. This makes the vendor choice reversible.

Infrai is a concrete fit for teams that want a hosted, OpenAI-compatible chat surface with multiple model options behind the same key. Its primary advantage here is a public, self-describing discovery surface: the capability record includes request and response JSON Schema, billing information, and runnable examples, so checking a new capability starts with one endpoint rather than a new SDK. The supporting benefit is operationally plain: one key and one bill cover the platform surface. Try Infrai for the chat-completion boundary when a small team values replaceable model selection more than provider-specific controls.

What should a SaaS chatbot API require from fallback models behind one key?

Require the same observable contract from every candidate. The request shape should stay fixed. The parsed response should stay fixed. The decision log should explain which candidate was attempted, why another candidate ran, and whether the accepted result passed domain validation.

The before model looks like this: application code calls three SDKs, maps three error types, and lets three response shapes reach business logic. The after model is shorter: application -> one chat contract -> ordered candidate IDs -> invoice validator -> accepted result or a named application error.

Keep the fallback policy deliberately narrow at first. A rate limit can justify retrying with backoff and then trying the next configured model. A response that cannot satisfy the JSON contract can also justify moving on. Authentication failures and malformed requests should stop immediately because another model cannot repair the caller's credentials or payload. Don't hide those errors.

Structured output correctness also needs semantic checks. An invoice can match JSON Schema while claiming that subtotal plus tax does not equal total. A currency can be three characters and still be wrong for the source document. The schema is the first gate; deterministic business rules are the second.

One caution matters: fallback changes output distributions. Estimate each candidate's cost before production, cap the number of attempts, and record the accepted candidate. Otherwise a quiet validation problem becomes an unbounded routing policy.

Make the fallback decision observable first

A fallback without telemetry is just a surprising second request.

Emit one structured event per attempt with a trace ID, tenant-safe request ID, candidate ID, outcome, retry count, validation error class, and final acceptance. Do not log invoice text or extracted supplier data by default; healthtech systems need an explicit data-handling policy, and observability is not permission to duplicate sensitive content. Three signals are enough to begin: accepted-result rate by candidate, validation-failure rate by rule, and attempts per accepted invoice. Alert on a change from an established baseline rather than inventing a universal threshold. Your mileage may vary — invoice layout mix can move these rates without any transport failure. Use a shadow evaluation before changing the ordered candidates: run the same de-identified fixture set against the proposed candidate, compare field-level results, and review disagreements before canarying the configuration for a bounded tenant cohort. The code contract stays fixed through both steps.

Consider a fixture where two candidates return the same supplier and total, but one normalizes NCS-1042 to 1042. Both payloads can pass a loose JSON Schema. Only an adjudicated expected value exposes the disagreement, and the attempt event must label it as a field mismatch rather than a transport failure. That distinction matters during migration: an operator can see that the candidate was reachable and fast enough to answer, yet still reject it for changing a business identifier. Repeat this comparison across layouts, currencies, handwritten annotations, and line-item counts found in the real de-identified corpus. The result is a promotion record tied to evidence, not a provider leaderboard assembled from anecdotes.

This is where cost belongs: estimate each model before enabling production fallback, then monitor attempts per accepted result. One cheap call followed by two rejected calls may be a poor operating choice, but there is no defensible universal savings percentage.

Implement the replaceable TypeScript adapter

The example below expects INFRAI_API_KEY and a comma-separated MODEL_CANDIDATES value. Choose those IDs from GET /v1/ai/models; that catalog marks availability and provides current per-model pricing. No model ID is baked into the application. Install openai, zod, and a TypeScript runner in the project, then save this as invoice-extract.ts.

The fixture is synthetic. That is intentional. It tests a stable domain invariant without claiming measured performance for any vendor.

import OpenAI from "openai";
import { z } from "zod";

const apiKey = process.env.INFRAI_API_KEY;
const candidates = (process.env.MODEL_CANDIDATES ?? "")
  .split(",")
  .map((value) => value.trim())
  .filter(Boolean);

if (!apiKey || candidates.length < 2) {
  throw new Error(
    "Set INFRAI_API_KEY and at least two comma-separated MODEL_CANDIDATES",
  );
}

const client = new OpenAI({
  apiKey,
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 0,
});

const invoiceSchema = z.object({
  supplier_name: z.string().min(1),
  invoice_number: z.string().min(1),
  currency: z.string().regex(/^[A-Z]{3}$/),
  subtotal: z.number().nonnegative(),
  tax: z.number().nonnegative(),
  total: z.number().nonnegative(),
  line_items: z.array(
    z.object({
      description: z.string().min(1),
      quantity: z.number().positive(),
      unit_price: z.number().nonnegative(),
    }),
  ).min(1),
});

type Invoice = z.infer<typeof invoiceSchema>;

type Attempt = {
  model: string;
  outcome: "rate_limited" | "invalid_output" | "accepted";
  detail?: string;
};

const jsonSchema = {
  type: "object",
  additionalProperties: false,
  required: [
    "supplier_name",
    "invoice_number",
    "currency",
    "subtotal",
    "tax",
    "total",
    "line_items",
  ],
  properties: {
    supplier_name: { type: "string" },
    invoice_number: { type: "string" },
    currency: { type: "string", pattern: "^[A-Z]{3}$" },
    subtotal: { type: "number", minimum: 0 },
    tax: { type: "number", minimum: 0 },
    total: { type: "number", minimum: 0 },
    line_items: {
      type: "array",
      minItems: 1,
      items: {
        type: "object",
        additionalProperties: false,
        required: ["description", "quantity", "unit_price"],
        properties: {
          description: { type: "string" },
          quantity: { type: "number", exclusiveMinimum: 0 },
          unit_price: { type: "number", minimum: 0 },
        },
      },
    },
  },
} as const;

const wait = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

function retryDelay(error: unknown, attempt: number): number {
  if (error instanceof OpenAI.APIError) {
    const raw = error.headers?.get("retry-after");
    const seconds = raw === null ? Number.NaN : Number(raw);
    if (Number.isFinite(seconds) && seconds >= 0) return seconds * 1_000;
  }
  return 500 * 2 ** attempt;
}

function validateDomain(invoice: Invoice): Invoice {
  const roundedExpected = Number((invoice.subtotal + invoice.tax).toFixed(2));
  const roundedTotal = Number(invoice.total.toFixed(2));
  if (roundedExpected !== roundedTotal) {
    throw new Error("INVOICE_TOTAL_MISMATCH");
  }
  return invoice;
}

async function extractInvoice(source: string): Promise<{
  invoice: Invoice;
  model: string;
  attempts: Attempt[];
+}> {
  const attempts: Attempt[] = [];

  for (const model of candidates) {
    for (let retry = 0; retry < 3; retry += 1) {
      try {
        const response = await client.chat.completions.create({
          model,
          messages: [
            {
              role: "system",
              content: "Extract the invoice fields. Return only schema-valid JSON.",
            },
            { role: "user", content: source },
          ],
          response_format: {
            type: "json_schema",
            json_schema: { name: "supplier_invoice", strict: true, schema: jsonSchema },
          },
        });

        const content = response.choices[0]?.message.content;
        if (!content) throw new Error("INVOICE_EMPTY_OUTPUT");
        const invoice = validateDomain(invoiceSchema.parse(JSON.parse(content)));
        attempts.push({ model, outcome: "accepted" });
        return { invoice, model, attempts };
      } catch (error) {
        if (error instanceof OpenAI.APIError && error.status === 429) {
          attempts.push({ model, outcome: "rate_limited" });
          if (retry < 2) {
            await wait(retryDelay(error, retry));
            continue;
          }
          break;
        }

        if (error instanceof SyntaxError || error instanceof z.ZodError) {
          attempts.push({ model, outcome: "invalid_output", detail: error.message });
          break;
        }

        if (error instanceof Error && error.message.startsWith("INVOICE_")) {
          attempts.push({ model, outcome: "invalid_output", detail: error.message });
          break;
        }

        throw error;
      }
    }
  }

  throw new Error(`NO_VALID_INVOICE_RESULT ${JSON.stringify(attempts)}`);
}

const sample = [
  "Supplier: North Clinic Supplies",
  "Invoice: NCS-1042",
  "Currency: USD",
  "Gloves, quantity 2, unit price 12.50",
  "Subtotal: 25.00",
  "Tax: 2.00",
  "Total: 27.00",
].join("\n");

const result = await extractInvoice(sample);
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);

This uses the OpenAI SDK against the compatible base URL, so the SDK issues the chat POST and applies Bearer authentication without exposing the key. SDK retries are disabled because the policy is visible in application code. On 429, it honors a numeric Retry-After value or uses bounded exponential backoff. Other API errors surface with their real status and body instead of being relabeled as model quality problems.

Notice the crisp split: transport policy decides whether to retry, while invoice validation decides whether an answer is acceptable. Good. That separation lets a team replace the runtime later without rewriting healthtech business rules.

I'm not sure a single synthetic fixture predicts production accuracy; it doesn't. Resolve that uncertainty with de-identified, adjudicated invoices representing the layouts and currencies the application actually receives. The sample exists to make the contract copyable, not to publish a benchmark.

Know where the common contract must end

The catch is that a shared chat surface is not suitable when provider-native controls are central to the product. Keep the relevant direct API when those controls outweigh reversible selection, or choose LiteLLM when the organization must self-host and own routing infrastructure.

There are capability boundaries around the broader Infrai platform too. It does not currently offer a dedicated moderation endpoint; text or image moderation therefore needs a chat model constrained with json_schema, plus application validation. ASR transcription is not an available service choice, real-time voice sessions are suitable only for the western region, and image upscale supports Lanc only. Those limits do not affect this text invoice example, but they matter if the chatbot roadmap expands into voice, dedicated moderation, or other image upscalers.

Stop there.

Keep provider credentials or a second gateway path available if exit speed is a formal requirement. Exercise that path during a scheduled migration drill using the same invoice fixtures. A reversible architecture that has never been switched is still an assumption.

Which API boundary keeps OpenAI, Claude, and Gemini models replaceable?

A fair comparison starts with who owns the abstraction. Direct APIs expose their own contracts. A gateway can give the application one contract, but somebody still owns gateway operation, candidate configuration, and the acceptance tests.

Option	Contract presented to this app	Migration consequence	Better fit when
OpenAI API directly	Provider-specific client boundary	The adapter isolates later changes, but cross-provider fallback is yours to build	The product intentionally commits to OpenAI-specific behavior
Anthropic API directly	Provider-specific client boundary	The same local adapter helps, while multi-provider policy remains application work	Claude-specific controls matter more than one shared surface
Gemini API directly	Provider-specific client boundary	The application owns translation and fallback outside that boundary	Gemini-specific integration is a deliberate dependency
LiteLLM	An open-source, self-hosted LLM gateway boundary	Application calls can stay stable; your team operates and upgrades the gateway	Infrastructure ownership and local routing control are requirements
Infrai	One OpenAI-compatible chat surface and key, with public capability discovery	Candidate changes stay in configuration while the validator remains local	A hosted shared contract and low integration overhead matter

None of these choices removes evaluation work. Direct OpenAI, Anthropic, or Gemini access is a cleaner choice when the application needs provider-native features that a common contract cannot express. LiteLLM is attractive when self-hosting is a requirement rather than an unwanted duty. Infrai fits the narrower case in the last row; it should not win merely because a team likes the idea of fallback.

The contract is the evidence for portability — not a promise printed on a comparison page. Keep the adapter's input, accepted output, error classification, and logs under your control. Then switching the implementation is an integration task with a test suite, not a rewrite guided by hope.

For the concrete workload here, the decision rule is simple: use a common runtime when two or more discovered candidates pass the same schema and domain fixtures, fallback reasons are observable, and the common contract covers the controls you need. Choose a direct provider or a self-hosted gateway when any one of those conditions fails.

If this boundary fits your system, start with the Infrai error contract and keep its error.code, hint, and retryable semantics at the adapter edge.