DEV Community: evanshepherd5623

Picking an LLM API gateway for a private knowledge base: cost per token, caching, batch

evanshepherd5623 — Tue, 11 Aug 2026 16:36:31 +0000

Thirty seconds is about how long a nurse will wait for an answer from an internal-policy assistant before giving up and opening the PDF. That deadline, not the price list, is what decides this comparison. Pick a gateway on two numbers at once — what one answered question costs in tokens, and how much latency the extra hop adds — and keep caching and batch as separate budgets rather than folding them into a single "cost per token" figure.

Two clocks. Not one.

If you already run infrastructure and want routing rules under your own version control, self-host LiteLLM and be done. If you want a hosted catalogue behind one credential and no client library to track, a hosted compatible gateway is the shorter path, and Infrai is a good option for a Node.js team here because it's a plain REST API over HTTPS with no SDK to install — the gateway becomes one more HTTP call from the service you already run. And if exactly one vendor's model will ever touch your data, call OpenAI, Anthropic or Google directly and skip the middle layer.

Two clocks are running and they pull against each other

Before: your app holds a vendor SDK, one API key, one billing account. Latency is model latency plus your own network. The mental model is a straight line — app, SDK, vendor, back.

After: your app holds one client pointed at a gateway, and the gateway holds the vendor keys. The line grows a node in the middle. That node is where model swapping, retries, caching signals and per-call cost accounting live, and it's also where you pay somewhere between a few and a few dozen milliseconds of routing overhead, depending on whether the gateway sits in the same region as your service and the same region as the upstream vendor. For a private knowledge base in healthtech the pipeline usually has three stages anyway: retrieve chunks from the vector store, filter or rerank them, then write the answer. Each stage can run on a different tier of model, and the whole reason to put a gateway in the middle is that changing which tier handles which stage should be a config edit rather than a deploy of three different SDK integrations.

The quality-versus-latency axis lives entirely in stage three. A small model answers a policy question in a second and gets the nuance wrong maybe one time in twenty. A frontier model takes four seconds and doesn't. In a clinical setting, the wrong-one-in-twenty is the number your compliance reviewer will ask about, so the cheap model belongs in stages one and two, where its job is filtering, not answering.

How should you compare cost per token across OpenAI, Claude, and Gemini endpoints?

Not by lining up published per-token rates. Those rates describe the meter, not the bill.

The number worth comparing is cost per answered question, which means measuring the shape of your own traffic first: how many tokens of retrieved context you stuff into each prompt, how long the answers run, and how much of that prompt is identical on every single call. In a private-KB assistant the last one dominates. The system preamble, the formatting rules, the safety instructions and often the top-ranked policy excerpts repeat across thousands of questions, and every major vendor now discounts those repeated input tokens through prompt caching. If your gateway forwards the cache signals and reports back whether a call hit the cache, you can actually verify the discount instead of assuming it.

Batch is the other budget. Nightly re-tagging of newly ingested documents, backfilling summaries, generating embeddings for a new corpus — nobody is watching a spinner for any of that, so it belongs on an asynchronous batch flow rather than the same synchronous endpoint your clinicians hit. Gateways differ sharply here: some expose a batch submit-and-poll flow behind the same key as chat, some pass you through to the vendor's own batch API, and some don't offer batch at all and quietly turn your nightly job into ten thousand synchronous calls.

So the comparison table I'd build has three columns of numbers, none of which come from a pricing page: tokens per answered question, cache hit rate on the repeated prefix, and the share of your monthly volume that could legally run overnight instead of in real time.

The bill nobody budgets: keys, SDKs, and time to first useful answer

Here's where the integration friction shows up. Supporting three model families directly means three vendor accounts, three keys in your secret store, three billing portals to reconcile at month end, and three SDKs whose retry semantics, streaming shapes and error classes have nothing in common. Streaming is the sharpest example: they all speak Server-Sent Events, but the event names and the terminal sentinel differ enough that your Node.js handler ends up with a branch per vendor.

An OpenAI-compatible surface collapses most of that into a baseURL swap, because your existing client keeps working and the vendor choice moves into the model field.

Option	How you integrate	Time to first useful call	Best fit	Main limit
Direct vendor SDKs (OpenAI, Anthropic, Google)	One SDK per vendor	Minutes per vendor, then N-way glue	You are committed to one vendor	Key and SDK sprawl grows with each model family
Self-hosted LiteLLM	Deploy and operate a proxy	Hours to a day	You want routing rules in your own repo	You now run and page for another service
OpenRouter	Compatible HTTP surface, hosted	Minutes	Broad model catalogue, quick experiments	Another party in the data path
Infrai	Plain REST over HTTPS, OpenAI-compatible surface	Minutes	Node.js teams who want one credential across backend services	Another party in the data path
Ollama or vLLM on your own hardware	Self-managed inference	Days	Data must never leave your network	You own capacity planning and model upgrades

The second thing Infrai buys you beyond the missing SDK is narrower than the marketing line suggests, and it's the part that matters for a small platform team: one key and one bill covers the vector store and the scheduled jobs sitting next to the model calls, so the credential your nightly tagging worker already carries is the same one the chat path uses. Billing runs on usage with no monthly minimum, which mostly means a pilot answering forty questions a day is cheap enough to leave running while you argue about architecture. The discovery surface is public and needs no key, so you can read the request and response schema for any capability before you sign up for anything — I like that more than I expected to, because it makes the "will this fit" question answerable in a browser tab.

A minimal Node.js example

The chat path is the OpenAI SDK with two lines changed:

import OpenAI from "openai";

// One credential. The vendor keys live on the gateway side, not in your secret store.
const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,   // ifr_...
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 4,                        // backs off on 429 and honours Retry-After
});

const PREAMBLE =
  "Answer only from the retrieved policy excerpts. If they do not cover the question, say so.";

export async function answerFromKb(question: string, excerpts: string[]) {
  try {
    const res = await client.chat.completions.create({
      model: "deepseek-chat",
      messages: [
        { role: "system", content: PREAMBLE },
        { role: "user", content: `${excerpts.join("\n---\n")}\n\nQuestion: ${question}` },
      ],
    });

    // The OpenAI-compatible response carries a top-level infrai object:
    // { cost_usd, vendor, model, region, cache, request_id }
    const meta = (res as { infrai?: Record<string, unknown> }).infrai;
    console.log("cost", meta?.cost_usd, "vendor", meta?.vendor, "cache", meta?.cache);

    return res.choices[0].message.content;
  } catch (err) {
    // A 4xx body carries the reason. Surface it instead of retrying blindly.
    const e = err as { status?: number; message?: string };
    console.error("chat rejected", e.status, e.message);
    throw err;
  }
}

That meta object is the whole reason to route through something instead of calling a vendor directly. You get the per-call cost and the cache flag on the same response you were already parsing, which turns "are we saving money on cached prefixes" from a spreadsheet exercise into a log line you can aggregate.

Before you pin a model, read the catalogue rather than a blog post — it's an unauthenticated GET and it carries the current prices:

const res = await fetch(
  "https://api.infrai.cc/v1/ai/models?capability=chat&available=true",
  { method: "GET", headers: { authorization: `Bearer ${process.env.INFRAI_API_KEY}` } },
);
if (!res.ok) throw new Error(`model list ${res.status}: ${await res.text()}`);

const { data } = await res.json();
for (const m of data) {
  console.log(m.id, m.price_input_per_mtok, m.price_output_per_mtok, m.modalities);
}

Wire those two together and your model choice per pipeline stage becomes a string in a config file. That's the payoff.

Where a specialist still wins, and two objections worth taking seriously

The catch is that a gateway is a party in the data path, and in healthtech that's a legal question before it's an engineering one. If your compliance review requires a signed agreement with the model vendor itself, or residency you can point at on a map for both the US and EU halves of your traffic, go direct or self-host. Same answer if you need provisioned throughput: dedicated capacity is an enterprise contract with the vendor, not something a shared gateway can hand you. And if content classification is central to your product rather than incidental, note that Infrai doesn't offer a dedicated moderation endpoint — you'd run classification through a chat model with a JSON schema, which works but is not what a purpose-built moderation service gives you.

The first objection I hear is that a gateway is another thing to be down. Fair, and the honest answer is that you've traded three vendor dependencies for one gateway dependency plus its upstreams, which is better for correlated failure and worse for blast radius. Keep a direct-to-vendor fallback path in your config for the stage that actually faces users. It costs you one extra client and it means the routing layer is never a single point of failure for the clinician waiting thirty seconds.

The second is lock-in, and I think it's mostly backwards. A compatible surface is the least sticky integration available to you, because the exit is a baseURL edit. What genuinely locks you in is the stuff around the model calls — the vector collections, the scheduled jobs, the stored artefacts. Migrate those deliberately, or keep them somewhere you're happy to stay.

If the boundary in this article matches your system, the gateway comparison notes at docs.infrai.cc go through what a compatible surface does and doesn't cover. Then measure your own cost per answered question before you commit to anything, because your traffic shape will disagree with mine.

References

MDN, Using Server-Sent Events — https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
LiteLLM, self-hosted LLM gateway (open source) — https://github.com/BerriAI/litellm
OpenAI, Batch API guide — https://platform.openai.com/docs/guides/batch
Anthropic, Message Batches — https://docs.anthropic.com/en/docs/build-with-claude/batch-processing
Google, Gemini API documentation — https://ai.google.dev/gemini-api/docs

Node App Telemetry: Weighing Pino, Logtail, Datadog, and Hosted APIs

evanshepherd5623 — Wed, 05 Aug 2026 18:25:17 +0000

Short answer: for simple production logging in a Node Express app, keep Pino at the application edge, send structured events to a central destination, and choose that destination by the operational questions you must answer after deployment.

For a junior developer running an Express service, Pino plus a lightweight hosted log API is a practical starting point when the job is ingestion and search. Pino plus Logtail belongs on the shortlist for that same focused job. Pino plus Datadog deserves a closer look when logs must sit inside a wider observability workflow. The decision changes if you need alert delivery, distributed trace exploration, compliance controls, or synthetic checks.

My test is deliberately boring: can I follow one failed request from the Express handler to the central search result without teaching the app a vendor-specific logging model? Start there. Fancy dashboards can wait.

That's enough.

What should simple production logging for a Node Express app actually preserve?

The useful unit isn't a line of prose. It's an event with enough context to reconstruct what happened. I teach teams to begin with four fields: request_id, user_id, trace_id, and environment. Keep the message short, keep the level explicit, and let the fields carry the debugging context. If a user reports a failed checkout, request_id narrows one execution, user_id connects nearby activity, trace_id can correlate cooperating services, and environment prevents a staging event from masquerading as production.

Before: an engineer searches for “checkout failed,” gets 600 similar strings, and guesses.

After: the engineer starts with a reported request ID, confirms the environment, and follows the same trace ID through related structured events. That's the whole diagram in words: Express request -> Pino JSON -> central ingestion -> field search -> one explainable failure.

This does not turn logs into traces. Shared trace_id and span_id fields provide correlation, but they don't create a span tree or a distributed tracing query experience. That distinction matters. A log destination can help me gather the breadcrumbs while still being the wrong tool for service maps, parent-child timing, or critical-path analysis.

I also separate visibility from notification. Central search answers “what happened?” after somebody asks. Alert routes, threshold rules, phone calls, SMS, and webhook delivery answer “who gets interrupted?” A hosted log API without those routes needs a polling query and an alerting component around it. Likewise, it won't tell you that a scheduled task silently failed to run; a heartbeat monitor such as Healthchecks covers that shape of failure better.

A copyable Pino baseline before choosing the backend

Here is the smallest Express setup I would ship as a baseline. It emits structured request completion events, carries incoming correlation IDs when present, creates a request ID when absent, and never mixes the production destination decision into route code.

import express from "express";
import pino from "pino";
import { randomUUID } from "node:crypto";

const app = express();
const logger = pino();

app.use((req, res, next) => {
  const startedAt = Date.now();
  const requestId = req.header("x-request-id") ?? randomUUID();
  const traceId = req.header("x-trace-id") ?? randomUUID();
  const userId = req.header("x-user-id") ?? "anonymous";

  res.setHeader("x-request-id", requestId);
  res.on("finish", () => {
    logger.info({
      request_id: requestId,
      user_id: userId,
      trace_id: traceId,
      environment: process.env.NODE_ENV ?? "development",
      method: req.method,
      path: req.path,
      status_code: res.statusCode,
      duration_ms: Date.now() - startedAt,
    }, "request completed");
  });

  next();
});

app.get("/health", (_req, res) => {
  res.status(200).json({ ok: true });
});

app.listen(3000, () => {
  logger.info({ port: 3000 }, "server listening");
});

Run it with the normal Pino and Express packages plus their TypeScript types. The destination can consume the resulting JSON without forcing every handler to know where logs live. That boundary is valuable: changing the transport should be an operations change, not a rewrite of business logic.

Keep one caveat in view. Redaction is an application responsibility in this baseline. Don't place tokens, passwords, or raw personal data into user_id or messages. A clean schema is easier to search, but it also makes accidental sensitive fields consistently discoverable. I prefer an allowlist of logged fields at the request boundary because it is short enough for a reviewer to verify.

Compare the destinations by the failure you need to investigate

The product names matter less than the investigation they support. I use this table as a first-pass decision aid, then verify the selected product's current documentation during a spike. Your mileage may vary because retention policy, team ownership, and existing contracts can outweigh a neat technical fit.

Option	Best initial fit	Main reason to choose it	Reason to choose something else
Pino + Logtail	Focused centralized logging	Keep the evaluation centered on ingestion and log search	Choose Datadog when the decision must include a broader observability workflow
Pino + Datadog	Teams evaluating logs alongside a wider operations stack	One candidate when logging is only part of the operational requirement	Choose a focused destination when a larger platform adds process you won't use
Pino + a conventional hosted log API	Small service with a stable logging contract	A narrow HTTP boundary can keep application logging portable	Avoid a thin API when you require native alert delivery, tracing, replay, or compliance workflows
Pino + Infrai	App debugging centered on ingestion and search	One REST API keeps the application contract stable while the vendor behind a capability can change; the wider platform uses one key and one bill	Not suitable when logs need user-level deletion, bulk export, subscriptions, configurable retention, native alerts, or distributed trace queries

That final row needs care. Search filters for logs.search are not declared in discovery parameters, so I would validate the supported query patterns during integration rather than promise field syntax in an article. The correct public routes are POST /v1/logs/ingest and GET /v1/logs/search, but request fields and search filters should come from discovery, not from REST conventions or guesses.

The catch is scope. For compliance-heavy logging, a missing per-user deletion interface and missing bulk export or subscription interface are decisive boundaries. For crash diagnostics, lack of source-map decoding, Electron minidump symbolication, and Session Replay points elsewhere. A lightweight destination can still be the right answer; it just isn't a compressed replacement for every observability discipline.

Can a retry hide the real production logging failure?

Yes. I've watched it happen.

I hit a 429 during a burst on one service, and our retry loop quietly swallowed it. The dashboard looked calm because the client kept retrying; the missing evidence only became obvious when 17 request IDs from a support batch produced no searchable event. I had taught the team to correlate logs, yet the transport had erased the very trail we needed. The painful part wasn't rate limiting itself — that is normal backpressure — but treating a retry as proof of delivery.

The fix in my mental model was crisp. A transport must recognize 429, honor Retry-After when it is present, apply exponential backoff, and surface exhaustion as an operational signal. It must also inspect every response status and retain the real 4xx response body because that body carries the reason. Tight loops are out. Silent loops are worse.

For writes, retry semantics deserve the same attention. If a destination supports an idempotency key, use a stable client-supplied value so a repeated attempt cannot double-apply the event. If its ingestion contract doesn't specify idempotency, I'm not sure why anyone would assume retries are harmless; I would test duplicate behavior explicitly and make downstream queries tolerant of a stable event identifier only where the documented schema supports one.

This is also why I avoid showing a hand-invented ingestion payload. The sample above establishes the application event. The destination adapter must be written against its current request schema, including its authentication and retry contract. Copy-pasteable code is useful only when every field is real.

Where should the architecture stop growing?

Stop when the system answers the incident questions your team actually owns. For one Express service, structured logs plus central ingestion and search may be enough. Add an alerting path when someone must react without polling. Add Healthchecks or another heartbeat monitor when silence itself is failure. Add a tracing product when shared IDs no longer explain cross-service latency.

Stick with Datadog when your team has deliberately chosen its wider operational workflow and wants logging evaluated inside that choice. Keep Logtail in the focused comparison when your priority remains centralized logs. Choose another hosted log API only after its query contract, retry behavior, retention controls, and data lifecycle match the workload. The generic label hides substantial differences.

And don't force a lightweight log destination into a compliance system. User-level deletion, bulk export, subscriptions, auditability, and configurable retention are requirements, not polish. If any one is mandatory, verify it before sending production data. This is the objection I hear most from platform engineers, and they're right to raise it early.

The other objection is portability. An application-level Pino schema helps, but portability isn't automatic; saved searches, alerts, dashboards, and retention rules can still bind a team to a destination. Keep the first contract small, name fields consistently, and isolate transport code. It won't erase migration work, but it protects the Express handlers from most of it.

Start small. Stay explicit.