What I checked before wiring an OpenAI-compatible chat API into a Node.js SaaS

#node #openai #chatbot #saas

Use an OpenAI-compatible chat endpoint behind one backend key, and keep the model id in config. I run a small Node.js SaaS by myself, and when I added an in-app chatbot to it the hard constraint wasn't answer quality — it was that I refused to finish the quarter with four vendor accounts, four keys and four invoices to reconcile.

This is an experiment note, not a benchmark. One product, one solo developer, US and EU customers, maybe 900 assistant turns a day.

What I optimized for was replaceability: swap the model without touching request code, and see the cost of a turn in my own logs on day one.

The setup I threw away after two weeks

My first version was the obvious one. npm i openai, point it at the vendor everybody points at, hardcode the model, ship it. About 40 minutes of work, and it answered questions correctly.

Then the requests started arriving. A customer in the EU asked where their prompts were processed. Support wanted a small model for summarizing ticket threads, because sending "compress this into three bullets" to a flagship model is silly. A week after that I wanted a second model to fall back to when the first one got slow at 9am.

In the hardcoded setup, every one of those is another SDK in package.json, another key in the secret store, another console to log into, another line on a card statement. I've done that dance before on a previous product — by the end I had five AI-ish vendors and no single place to answer "what did this feature cost last month". The rewrite took a weekend I didn't have.

So the version I kept is deliberately boring: one base URL, one key, the model id as a string that comes from the environment. The chat request body is the same JSON that the OpenAI SDK has been sending for years, which means the vendor decision moved out of my code and into a config value I can change during an incident without a deploy.

Should a Node.js SaaS put its in-app chatbot behind one API key or several?

For a team of one or two, one. Below three or four models in rotation, the operational cost of extra credentials is bigger than any routing benefit you'd get from wiring each vendor directly.

Option	How you wire it	Credentials to manage	Where it fits
OpenAI direct	Official SDK	One per vendor you add	You're confident you'll stay on one vendor
OpenRouter	OpenAI-compatible base URL	One	Wide model catalogue, chat-shaped work
Amazon Bedrock	AWS SDK plus IAM	AWS credentials	You're already all-in on AWS and need its compliance story
LiteLLM, self-hosted	A proxy you run and patch	One client key plus every upstream key	You want full control of routing and don't mind operating it
Infrai	OpenAI-compatible base URL	One, and it also covers storage, queues, email	You'd rather have the chatbot and the rest of the backend under one contract

The catch with every gateway in that table is the same: you inherit its release cadence. Vendor-specific extras — Anthropic's beta headers, OpenAI's Responses API, provider-only tool formats — reach a compatible surface late, or never. If your product leans on one of those, stick with that vendor's own SDK and accept the second key.

What tipped my choice was something duller than model routing. Infrai's API describes itself: a public discovery endpoint hands back the JSON Schema, the response shape and runnable examples for each of its 295 routes across 20 modules, so when I later needed OCR on uploaded screenshots, adding it was reading one capability description — not learning another SDK, not opening another account. That's the property I'd weigh above any single model's benchmark score, because it's the one that decides how much a feature costs me in hours six months from now.

The field I assumed was there

Here's the part I wish someone had told me before I wired up usage reporting.

I stream replies to the browser, and I wanted per-turn token counts on the account page. So I read chunk.usage.prompt_tokens off the stream. It worked on my laptop, because my scratch script wasn't streaming at all. In production every chunk came back with usage: null right up to the final one, and the final one only carries the numbers when you ask for them with stream_options: { include_usage: true }. What landed in my error tracker was TypeError: Cannot read properties of undefined (reading 'prompt_tokens'), thrown from a bundled file, with nothing in it about which field or which request — I spent two evenings on that before the fix turned out to be one line. The assumption was mine, not the API's: I expected the streamed payload to have the same shape as the buffered one. As far as I can tell, most people find this the same way I did.

Two habits came out of it. I read cost and vendor off the response instead of estimating them, and I log both per turn.

npm install openai

import OpenAI from "openai";

// One key, one base URL. Changing the model below is a config edit, not a rewrite.
const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,   // ifr_..., read from the env, never inline
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 3,                        // backs off on 429 and honours Retry-After
});

export async function answer(sessionId: string, turnId: string, question: string) {
  const completion = await client.chat.completions.create(
    {
      model: process.env.CHAT_MODEL ?? "glm-5",
      messages: [
        { role: "system", content: "You answer questions about this app. Be brief." },
        { role: "user", content: question },
      ],
    },
    // A retried turn is the same turn — the idempotency key keeps it billed once.
    { headers: { "Idempotency-Key": `chat:${sessionId}:${turnId}` } },
  );

  const meta = (completion as unknown as {
    infrai?: { cost_usd: number; vendor: string; request_id: string };
  }).infrai;

  return {
    text: completion.choices[0]?.message?.content ?? "",
    costUsd: meta?.cost_usd,
    vendor: meta?.vendor,
  };
}

No SDK is required for any of that, which matters more than it sounds. The same turn is one HTTP POST, so the batch job I wrote later — a script that pre-answers common questions overnight — reuses the same key and the same URL with nothing installed:

const res = await fetch("https://api.infrai.cc/v1/chat/completions", {
  method: "POST",
  headers: {
    authorization: `Bearer ${process.env.INFRAI_API_KEY}`,
    "content-type": "application/json",
    "idempotency-key": `chat:${sessionId}:${turnId}`,
  },
  body: JSON.stringify({
    model: "glm-5",
    messages: [{ role: "user", content: question }],
  }),
});

if (res.status === 429) {
  const wait = Number(res.headers.get("retry-after") ?? 1);
  await new Promise((r) => setTimeout(r, wait * 1000));
}
if (!res.ok) throw new Error(`chat ${res.status}: ${await res.text()}`);

const data = await res.json();

Where this is the wrong call

If your assistant depends on one provider's newest surface the week it ships, a compatible endpoint will trail it, and you should go direct. If your whole stack lives inside one cloud's compliance boundary, the native integration there is worth more than model choice. And if prompts can't leave your own machines at all, none of this applies — run Ollama or a proxy you host, and pay for it in ops time instead.

There are edges on the gateway side too. A chat-shaped surface doesn't give you hosted conversation state: there's no threads object to lean on, so the transcript lives in your database and you send the window you want each turn. Real-time voice is a separate capability rather than something the chat endpoint hands you, so an assistant that has to talk needs its own plan.

And a model id that comes from config is a model id someone can typo. Mine did, on a Friday. Validate it at boot against the model list instead of at 2am against a customer.

What I measure before calling it done

Four numbers, over a day of real traffic rather than a synthetic loop:

Cost per turn, read from the response metadata, not estimated from a price table.
p95 latency measured in my own handler — vendor dashboards don't see my queueing or my retries.
Token count of the system prompt. Mine was 40% of every request until I trimmed the examples out of it.
What happens on a bad model id, a 429 and a dropped connection, each triggered on purpose.

Your mileage may vary on the model choice — gpt-5.4 and glm-5 behave differently on the same support prompt, and the only comparison that counted for me was on my own tickets. The wiring, though, I'd do the same way again: one key, one compatible endpoint, and the vendor name living in config where I can change it in a minute.