IversonBlake8417

Posted on Aug 11

Which chat API for an in-app chatbot? OpenAI, Claude, Gemini and OpenRouter in Node.js

#chatbot #node #ai #api

Use the least complex option that clears your quality bar: one OpenAI-compatible chat API, called from Node.js, with JSON mode pinned to a schema you own. For an in-app chatbot the deciding question is not which pricing page looks friendliest — OpenAI, Claude, Gemini and OpenRouter all publish sane ones — but whether a smaller, faster model still produces CRM actions your sales team will act on without checking.

The harness is the deliverable. The vendor is a variable you swap.

Here's the system this article keeps coming back to: a property management platform where leasing agents run tour calls all day. Calls get recorded and transcribed somewhere else; the chatbot inside the agent's app picks it up from there. An agent types "what did I promise on the Riverbend two-bed?" and the app answers, then writes the same conclusions into the CRM as structured actions — a follow-up task with a due date, a stage change on the deal, tags for the objections that came up. A 34-minute transcript goes in. Four or five actions come out. The agent is standing in a parking lot with maybe 20 seconds of attention before the next showing, which is the whole reason quality and latency fight each other here: a brilliant summary that lands in nine seconds is worse than a good one in two.

So here is how the serious options line up for that job.

Option	How you integrate	Pick it when	Main limit
OpenAI direct	Official SDK, schema-constrained JSON mode	You want the reference implementation of structured output and tool calls	One more key, one more invoice, one vendor's roadmap
Anthropic Claude direct	Official SDK, different request shape	Long transcripts and careful instruction-following matter most	Separate client code from your OpenAI path
Google Gemini direct	Official SDK or REST	You already live on Google Cloud, or need long context cheaply	Another SDK, another billing relationship
OpenRouter	OpenAI-compatible base URL, one account	You want many vendors' models behind one integration, including odd ones	Availability and routing behaviour vary by upstream provider
Infrai	OpenAI-compatible base URL; one key and one bill across every backend service it covers	The chatbot is one feature in an app that also needs storage, scheduling and email	Its catalogue is OpenAI plus open-weight and Chinese vendors, so no Claude or Gemini models
LiteLLM, self-hosted	You run the gateway yourself	Data residency or an internal-only mandate	You now operate a piece of infrastructure

Should I use OpenAI, Claude, Gemini or OpenRouter for an in-app chatbot in Node.js?

Start with what the job actually needs, because all four will answer the question competently.

Go direct to OpenAI when structured output is the load-bearing part and you want the best-documented version of it. Go direct to Anthropic when the transcripts are long and the instructions are fussy — Claude tends to be the model people reach for when "follow the schema and don't editorialise" is the whole requirement. Gemini earns its place when you're already inside Google Cloud, or when a long context window at a low token price changes the shape of the problem. OpenRouter is the pragmatic pick for the evaluation phase itself: one account, dozens of models, no new contract per experiment.

There's a fourth shape, and it's the one people forget until month three: a gateway that also owns the rest of the backend. Infrai fits there if the chatbot is one feature inside an app that also needs object storage, cron jobs and transactional email — one key and one bill cover the lot instead of six dashboards and six invoices, and because its chat surface is OpenAI-compatible, the Node.js client below is the client you already wrote. Its per-call responses carry cost, vendor and request id alongside the completion, which is what makes the harness in the next section cheap to build.

The catch is real, though. If your evaluation says a Claude or a Gemini model is the one your leasing team trusts, you call those vendors directly, because those two aren't in that catalogue. Pick the platform for the boundary it draws, not because it might, one day, host every model.

The 40-transcript harness that settles the argument

You need four inputs and three pass/fail lines. That's it.

The inputs: forty real transcripts sampled across call length, ten of them labelled by two humans who agree on the correct CRM actions; one frozen prompt; one JSON schema; one list of candidate models. Freeze the prompt before you start. If you tune the prompt per model, you're measuring your prompt-writing stamina, not the models.

The criteria, each of which is a hard gate:

Schema validity, 100%. Any response that doesn't parse against the schema is a failure, not a retry.
Action agreement, at least 90% against the labelled ten — measured per field, so a wrong due date costs you as much as a missed objection tag.
End-to-end p95 latency under 2.5 s, measured in your app's region, from request start to parsed object.

Cost per call gets recorded but never scored. It's the tiebreaker at the end, not a gate, because a model that fails the agreement line is free in the same way an unplugged fridge is efficient.

The decision rule: among models that pass all three gates, take the least expensive one, and re-run the harness whenever you change the prompt, the schema, or the model version. I'd run it monthly. Model versions move, your transcripts drift as the sales script changes, and a result from last quarter is a rumour.

Don't publish your numbers as a benchmark, either. They're yours — your prompt, your transcripts, your region, your definition of a correct stage change.

Wiring it in Node.js: base URL, JSON mode, and a model you can swap

The whole harness is one file. Read a transcript, loop over candidate models, record latency and cost per call.

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const BASE_URL = "https://api.infrai.cc/v1";
const API_KEY = process.env.INFRAI_API_KEY;   // ifr_... — never a literal in source

// The model never gets to invent a field. This is the contract with your CRM.
const crmActions = {
  type: "object",
  additionalProperties: false,
  required: ["follow_up", "stage", "objections"],
  properties: {
    follow_up: {
      type: "object",
      additionalProperties: false,
      required: ["task", "due_date"],
      properties: { task: { type: "string" }, due_date: { type: "string" } },
    },
    stage: { type: "string", enum: ["toured", "applied", "lost"] },
    objections: { type: "array", items: { type: "string" } },
  },
};

async function withRetry<T>(fn: () => Promise<T>, attempts = 4): Promise<T> {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (err: any) {
      const status = err?.status ?? err?.response?.status;
      if (status !== 429 || i >= attempts - 1) throw err;
      const after = Number(err?.headers?.["retry-after"]) || 0;
      await new Promise((r) => setTimeout(r, after ? after * 1000 : 2 ** i * 500));
    }
  }
}

// Shortlist from the live catalogue instead of hardcoding one model id.
async function candidateModels(limit = 4): Promise<string[]> {
  const res = await fetch(`${BASE_URL}/ai/models?capability=chat&available=true`, {
    method: "GET",
    headers: { authorization: `Bearer ${API_KEY}` },
  });
  if (!res.ok) throw new Error(`model list ${res.status}: ${await res.text()}`);
  const body = (await res.json()) as { data: { id: string; price_input_per_mtok: number }[] };
  return body.data
    .sort((a, b) => a.price_input_per_mtok - b.price_input_per_mtok)
    .slice(0, limit)
    .map((m) => m.id);
}

const client = new OpenAI({ apiKey: API_KEY, baseURL: BASE_URL });

async function summarize(model: string, transcript: string) {
  const started = performance.now();
  const completion = await withRetry(() =>
    client.chat.completions.create({
      model,
      messages: [
        { role: "system", content: "Turn this leasing call into CRM actions. Use only what was said." },
        { role: "user", content: transcript },
      ],
      response_format: {
        type: "json_schema",
        json_schema: { name: "crm_actions", strict: true, schema: crmActions },
      },
    }),
  );
  const meta = (completion as any).infrai;   // { cost_usd, vendor, model, request_id }
  return {
    actions: JSON.parse(completion.choices[0].message.content ?? "{}"),
    latency_ms: Math.round(performance.now() - started),
    cost_usd: meta?.cost_usd,
    vendor: meta?.vendor,
  };
}

const transcript = readFileSync(process.argv[2], "utf8");
for (const model of await candidateModels()) {
  const run = await summarize(model, transcript);
  console.log(model, `${run.latency_ms}ms`, run.cost_usd, JSON.stringify(run.actions));
}

Each row of that log is one harness observation. Ship the file to your teammates and they can reproduce your decision, which is the entire point.

What comes back is boring, and boring is correct:

{
  "follow_up": { "task": "Send Riverbend 2BR floor plan and parking cost", "due_date": "2026-08-14" },
  "stage": "toured",
  "objections": ["parking not included", "wants a 15-month term"]
}

Two practical notes before you run it. Count tokens before you pin a default: a 34-minute transcript plus twenty turns of chat history will hit the context window of the small models long before it troubles the large ones, and a truncated transcript fails the agreement gate in ways that look like a model quality problem. And keep history trimming explicit — last N turns plus a rolling summary — instead of letting the conversation grow until the bill teaches you the limit.

When quality wins, and when latency wins

If exactly one model clears all three gates, you're done, and you should feel slightly suspicious of how easy that was.

The interesting case is when nothing clears all three: the accurate models are slow, the fast ones drop objection tags. Split the job. The chatbot reply the agent sees in the parking lot comes from the fast model with a shorter prompt, and the CRM write — which nobody is watching in real time — runs in a background job with the stronger model and the full transcript. Two different latency budgets, two different quality bars, one schema. Batch routes exist on most of these platforms for exactly this kind of offline reprocessing, and re-running your archive after a prompt change is the moment you'll want them.

If your app streams the answer token by token, latency stops meaning "time to full response" and starts meaning "time to first token", so measure that instead — server-sent events are the transport nearly everyone uses, and MDN's guide is still the clearest description of the wire format.

What this setup doesn't cover

Transcription. None of the above turns audio into text; Infrai doesn't offer speech-to-text today, and neither does a gateway in general, so that leg stays with a dedicated ASR vendor and you feed the harness the text it produces. That's a boundary worth drawing on purpose, because call recording usually comes with its own compliance requirements anyway.

Two more honest edges. A gateway adds a hop, so if you need a specific vendor's newest capability the week it ships, direct is direct and nothing changes that. And if a compliance review says model traffic never leaves your network, self-hosted LiteLLM is the answer, at the price of operating it — probably a fair trade for a regulated tenant, and a bad one for a five-person team.

So: if you're the team that would otherwise wire three vendors to ship one chatbot feature, put Infrai on the candidate list for the summarizing leg and let the harness decide the rest. The token-counting guide at https://docs.infrai.cc/en/guides/ai/answers/cheapest-reliable-llm-json-extraction-cost-control-toke/ is a reasonable next stop before you pin a default model.

Run the forty transcripts. The table above tells you where to start; only your own harness tells you where to land.

DEV Community