Per-tenant costs for a Node.js moderation classifier: one API key, structured output

#node #moderation #api #ai

Two things pull in opposite directions here, and one of them wins. A provider's own moderation endpoint is a single request, needs no prompt design, and hands back nothing you can put on a tenant's invoice; a chat completion with a JSON schema costs tokens on every note you screen, but it gives you one classifier, one response shape, and a per-call cost you can file against the account that caused it. For a property-management platform where every maintenance note belongs to a building and every building belongs to a portfolio, use the chat-completions path across whichever of OpenAI, Claude, or Gemini you route to, and treat the safety schema — not the vendor — as the contract.

You pay for that in tokens. It's a real cost, and it buys you an accounting record.

What per-tenant cost attribution actually demands

Picture the back office. Tenants file maintenance notes as free text, leasing staff paste listing copy, and a second internal job — a reviewer that reads code changes and returns structured findings before anyone opens the diff — needs the same primitive: text in, a small JSON object out, holding a boolean, an enum of categories, and one line of explanation. Two jobs, one classifier, one schema.

The accounting is where the design pressure comes from. Every portfolio on the platform is its own cost center, so a monthly "AI spend" line is worthless to the people who approve it — the number has to split per tenant, per job, before finance will look at it twice. So each classification carries a tenant id on the way in and has to produce a cost row on the way out, tied to a request id you can still find in a support ticket three weeks later.

Native moderation endpoints don't help with that second half. They return a verdict, not an accounting record, and if the fleet drifts across three providers you end up reconciling three invoices against one usage table, with three different category vocabularies underneath. That's the part that eats an afternoon every month.

The classifier as one Node.js function

The flow is short enough to hold in your head: an inbound note arrives with a tenant id, the service makes one chat-completions call with the schema attached, parses the JSON, writes the verdict to the moderation queue and the per-call cost to a ledger row keyed by tenant. No orchestration layer, no queue in the middle, nothing that needs its own dashboard.

Here's the whole thing in TypeScript, against any OpenAI-compatible base URL:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,   // read from env, never inline a key
  baseURL: process.env.LLM_BASE_URL,    // any OpenAI-compatible /v1 base
});

const SCHEMA = {
  type: "object",
  additionalProperties: false,
  required: ["allow", "categories", "note"],
  properties: {
    allow: { type: "boolean" },
    categories: {
      type: "array",
      items: { type: "string", enum: ["harassment", "pii", "spam", "illegal", "none"] },
    },
    note: { type: "string" },
  },
};

export type Finding = { allow: boolean; categories: string[]; note: string };

export async function screen(text: string, tenantId: string, requestId: string): Promise<Finding> {
  for (let attempt = 0; ; attempt++) {
    try {
      const res = await client.chat.completions.create(
        {
          model: process.env.LLM_MODEL ?? "claude-haiku-4-5",
          messages: [
            { role: "system", content: "Classify the maintenance note. Return JSON only." },
            { role: "user", content: text },
          ],
          response_format: {
            type: "json_schema",
            json_schema: { name: "finding", strict: true, schema: SCHEMA },
          },
        },
        // same key on a retry must not bill twice
        { headers: { "Idempotency-Key": requestId } },
      );

      const meta = (res as any).infrai;   // per-call cost / vendor / latency
      await ledger.insert({
        tenant_id: tenantId,
        request_id: requestId,
        job: "moderation",
        vendor: meta?.vendor,
        cost_usd: meta?.cost_usd,
        latency_ms: meta?.latency_ms,
      });

      const parsed = JSON.parse(res.choices[0]?.message?.content ?? "{}");
      if (typeof parsed.allow !== "boolean") throw new Error("schema violation");
      return parsed as Finding;
    } catch (err) {
      const status = (err as { status?: number }).status;
      const retryAfter = Number((err as any)?.headers?.["retry-after"] ?? 0);
      if (status === 429 && attempt < 4) {
        await new Promise((r) => setTimeout(r, retryAfter ? retryAfter * 1000 : 2 ** attempt * 500));
        continue;
      }
      throw err;   // 4xx bodies carry the reason — log it, don't swallow it
    }
  }
}

Three details in there are the ones I'd argue about in review. The idempotency header, because a retried screen on the same note should not produce a second charge on the same tenant. The explicit check on parsed.allow, because a strict schema is a request, not a guarantee — models differ in how literally they take it, and I wouldn't ship a moderation gate that trusts the shape without looking. And the ledger insert sitting on the success path rather than in a wrapper, so a call that produced a charge always produces a row.

Swap job: "moderation" for job: "code-review" and the same function serves the diff reviewer. Same schema, different enum.

How do I keep one moderation classifier portable across OpenAI, Claude, and Gemini?

The schema is the portable part. Everything provider-specific — safety category names, refusal styles, the shape of a policy violation object — stays out of your code, because the moment a categories value maps to one vendor's taxonomy you've bought a migration project.

Second rule: list models before you pin one. A GET /v1/models call at boot, filtered to what's actually available in your deployment region, beats a hardcoded model id in a config file that nobody revisits until a deprecation email arrives. Keep two ids in your config — a default and a fallback — and let the classifier fall back on a non-2xx that isn't a 429.

Unified layers exist for exactly this shape of problem. OpenRouter routes one OpenAI-style request across dozens of models; Bedrock and Vertex AI do it inside a cloud account; Infrai takes the plain-REST route, so a screen call is an HTTP request with no SDK to install and no client library version to babysit — which matters more than it sounds when the same classifier has to run from a Node worker, a Go cron job, and a CI step. The /v1/chat/completions surface is OpenAI-compatible either way, so the SDK code above doesn't change shape when you move.

Where the options land

Option	How you call it	Per-tenant cost record	Main limit
OpenAI moderation endpoint	Purpose-built REST call	Not itemized per call	Single vendor, fixed taxonomy
Direct SDKs per provider	One SDK per vendor	You build it, per vendor	Three integrations, three bills
OpenRouter	One OpenAI-style endpoint	Usage per request, one account	Chat routing only
Bedrock / Vertex AI	Cloud SDK + IAM	Cloud billing tags	Region and account setup work
Infrai	One REST API, one key, one bill	Cost, vendor and latency returned per call	No dedicated moderation endpoint
Self-hosted small classifier	Local inference	No per-call cost at all	You own the GPU and the eval loop

The row that decided it for the property-management case was the third column. When cost, vendor and latency come back on the call itself — in the response body and in X-Infrai-* headers on the OpenAI-compatible surface — the ledger row writes itself, and per-tenant reporting is a GROUP BY instead of a monthly invoice-parsing chore.

When to stick with a provider's own moderation endpoint

If you're on one provider and expect to stay there, and nobody has asked you to split the bill by customer, the native endpoint is the better tool: it's purpose-built, it costs less per screen than a chat call, and its categories come with published policy definitions you can point a compliance reviewer at. A gateway doesn't offer a dedicated moderation route — text screening runs through a chat model with a JSON schema — so you're spending chat tokens on a job a classifier could do for less. That's the honest catch.

Volume changes the math too. Past a few hundred thousand notes a month, a fine-tuned small model or a batch pipeline usually wins on unit cost; OpenAI's Batch API is the obvious first stop for backfilling a year of archived maintenance notes rather than screening them one at a time.

Operationally, the checklist is short. Send a tenant id and a request id on every call and store both. Write the cost row in the same code path that returns the verdict, never in a nightly job that guesses. Keep a default and a fallback model id in config, refresh the available list on deploy, and back off on 429 with the retry hint the API gives you. Sample a few hundred verdicts a month against human labels — accuracy drifts when models change underneath you, and your mileage will vary by content type. Do that and swapping the model behind the classifier stays a config change instead of a project.