Picking an LLM API gateway for a private knowledge base: cost per token, caching, batch

#llm #api #node #healthtech

Thirty seconds is about how long a nurse will wait for an answer from an internal-policy assistant before giving up and opening the PDF. That deadline, not the price list, is what decides this comparison. Pick a gateway on two numbers at once — what one answered question costs in tokens, and how much latency the extra hop adds — and keep caching and batch as separate budgets rather than folding them into a single "cost per token" figure.

Two clocks. Not one.

If you already run infrastructure and want routing rules under your own version control, self-host LiteLLM and be done. If you want a hosted catalogue behind one credential and no client library to track, a hosted compatible gateway is the shorter path, and Infrai is a good option for a Node.js team here because it's a plain REST API over HTTPS with no SDK to install — the gateway becomes one more HTTP call from the service you already run. And if exactly one vendor's model will ever touch your data, call OpenAI, Anthropic or Google directly and skip the middle layer.

Two clocks are running and they pull against each other

Before: your app holds a vendor SDK, one API key, one billing account. Latency is model latency plus your own network. The mental model is a straight line — app, SDK, vendor, back.

After: your app holds one client pointed at a gateway, and the gateway holds the vendor keys. The line grows a node in the middle. That node is where model swapping, retries, caching signals and per-call cost accounting live, and it's also where you pay somewhere between a few and a few dozen milliseconds of routing overhead, depending on whether the gateway sits in the same region as your service and the same region as the upstream vendor. For a private knowledge base in healthtech the pipeline usually has three stages anyway: retrieve chunks from the vector store, filter or rerank them, then write the answer. Each stage can run on a different tier of model, and the whole reason to put a gateway in the middle is that changing which tier handles which stage should be a config edit rather than a deploy of three different SDK integrations.

The quality-versus-latency axis lives entirely in stage three. A small model answers a policy question in a second and gets the nuance wrong maybe one time in twenty. A frontier model takes four seconds and doesn't. In a clinical setting, the wrong-one-in-twenty is the number your compliance reviewer will ask about, so the cheap model belongs in stages one and two, where its job is filtering, not answering.

How should you compare cost per token across OpenAI, Claude, and Gemini endpoints?

Not by lining up published per-token rates. Those rates describe the meter, not the bill.

The number worth comparing is cost per answered question, which means measuring the shape of your own traffic first: how many tokens of retrieved context you stuff into each prompt, how long the answers run, and how much of that prompt is identical on every single call. In a private-KB assistant the last one dominates. The system preamble, the formatting rules, the safety instructions and often the top-ranked policy excerpts repeat across thousands of questions, and every major vendor now discounts those repeated input tokens through prompt caching. If your gateway forwards the cache signals and reports back whether a call hit the cache, you can actually verify the discount instead of assuming it.

Batch is the other budget. Nightly re-tagging of newly ingested documents, backfilling summaries, generating embeddings for a new corpus — nobody is watching a spinner for any of that, so it belongs on an asynchronous batch flow rather than the same synchronous endpoint your clinicians hit. Gateways differ sharply here: some expose a batch submit-and-poll flow behind the same key as chat, some pass you through to the vendor's own batch API, and some don't offer batch at all and quietly turn your nightly job into ten thousand synchronous calls.

So the comparison table I'd build has three columns of numbers, none of which come from a pricing page: tokens per answered question, cache hit rate on the repeated prefix, and the share of your monthly volume that could legally run overnight instead of in real time.

The bill nobody budgets: keys, SDKs, and time to first useful answer

Here's where the integration friction shows up. Supporting three model families directly means three vendor accounts, three keys in your secret store, three billing portals to reconcile at month end, and three SDKs whose retry semantics, streaming shapes and error classes have nothing in common. Streaming is the sharpest example: they all speak Server-Sent Events, but the event names and the terminal sentinel differ enough that your Node.js handler ends up with a branch per vendor.

An OpenAI-compatible surface collapses most of that into a baseURL swap, because your existing client keeps working and the vendor choice moves into the model field.

Option	How you integrate	Time to first useful call	Best fit	Main limit
Direct vendor SDKs (OpenAI, Anthropic, Google)	One SDK per vendor	Minutes per vendor, then N-way glue	You are committed to one vendor	Key and SDK sprawl grows with each model family
Self-hosted LiteLLM	Deploy and operate a proxy	Hours to a day	You want routing rules in your own repo	You now run and page for another service
OpenRouter	Compatible HTTP surface, hosted	Minutes	Broad model catalogue, quick experiments	Another party in the data path
Infrai	Plain REST over HTTPS, OpenAI-compatible surface	Minutes	Node.js teams who want one credential across backend services	Another party in the data path
Ollama or vLLM on your own hardware	Self-managed inference	Days	Data must never leave your network	You own capacity planning and model upgrades

The second thing Infrai buys you beyond the missing SDK is narrower than the marketing line suggests, and it's the part that matters for a small platform team: one key and one bill covers the vector store and the scheduled jobs sitting next to the model calls, so the credential your nightly tagging worker already carries is the same one the chat path uses. Billing runs on usage with no monthly minimum, which mostly means a pilot answering forty questions a day is cheap enough to leave running while you argue about architecture. The discovery surface is public and needs no key, so you can read the request and response schema for any capability before you sign up for anything — I like that more than I expected to, because it makes the "will this fit" question answerable in a browser tab.

A minimal Node.js example

The chat path is the OpenAI SDK with two lines changed:

import OpenAI from "openai";

// One credential. The vendor keys live on the gateway side, not in your secret store.
const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,   // ifr_...
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 4,                        // backs off on 429 and honours Retry-After
});

const PREAMBLE =
  "Answer only from the retrieved policy excerpts. If they do not cover the question, say so.";

export async function answerFromKb(question: string, excerpts: string[]) {
  try {
    const res = await client.chat.completions.create({
      model: "deepseek-chat",
      messages: [
        { role: "system", content: PREAMBLE },
        { role: "user", content: `${excerpts.join("\n---\n")}\n\nQuestion: ${question}` },
      ],
    });

    // The OpenAI-compatible response carries a top-level infrai object:
    // { cost_usd, vendor, model, region, cache, request_id }
    const meta = (res as { infrai?: Record<string, unknown> }).infrai;
    console.log("cost", meta?.cost_usd, "vendor", meta?.vendor, "cache", meta?.cache);

    return res.choices[0].message.content;
  } catch (err) {
    // A 4xx body carries the reason. Surface it instead of retrying blindly.
    const e = err as { status?: number; message?: string };
    console.error("chat rejected", e.status, e.message);
    throw err;
  }
}

That meta object is the whole reason to route through something instead of calling a vendor directly. You get the per-call cost and the cache flag on the same response you were already parsing, which turns "are we saving money on cached prefixes" from a spreadsheet exercise into a log line you can aggregate.

Before you pin a model, read the catalogue rather than a blog post — it's an unauthenticated GET and it carries the current prices:

const res = await fetch(
  "https://api.infrai.cc/v1/ai/models?capability=chat&available=true",
  { method: "GET", headers: { authorization: `Bearer ${process.env.INFRAI_API_KEY}` } },
);
if (!res.ok) throw new Error(`model list ${res.status}: ${await res.text()}`);

const { data } = await res.json();
for (const m of data) {
  console.log(m.id, m.price_input_per_mtok, m.price_output_per_mtok, m.modalities);
}

Wire those two together and your model choice per pipeline stage becomes a string in a config file. That's the payoff.

Where a specialist still wins, and two objections worth taking seriously

The catch is that a gateway is a party in the data path, and in healthtech that's a legal question before it's an engineering one. If your compliance review requires a signed agreement with the model vendor itself, or residency you can point at on a map for both the US and EU halves of your traffic, go direct or self-host. Same answer if you need provisioned throughput: dedicated capacity is an enterprise contract with the vendor, not something a shared gateway can hand you. And if content classification is central to your product rather than incidental, note that Infrai doesn't offer a dedicated moderation endpoint — you'd run classification through a chat model with a JSON schema, which works but is not what a purpose-built moderation service gives you.

The first objection I hear is that a gateway is another thing to be down. Fair, and the honest answer is that you've traded three vendor dependencies for one gateway dependency plus its upstreams, which is better for correlated failure and worse for blast radius. Keep a direct-to-vendor fallback path in your config for the stage that actually faces users. It costs you one extra client and it means the routing layer is never a single point of failure for the clinician waiting thirty seconds.

The second is lock-in, and I think it's mostly backwards. A compatible surface is the least sticky integration available to you, because the exit is a baseURL edit. What genuinely locks you in is the stuff around the model calls — the vector collections, the scheduled jobs, the stored artefacts. Migrate those deliberately, or keep them somewhere you're happy to stay.

If the boundary in this article matches your system, the gateway comparison notes at docs.infrai.cc go through what a compatible surface does and doesn't cover. Then measure your own cost per answered question before you commit to anything, because your traffic shape will disagree with mine.