DEV Community: PerNilsson3147

Speech to text, then a model gateway: what one API key covers for a knowledge base

PerNilsson3147 — Fri, 14 Aug 2026 15:43:29 +0000

One key doesn't cover both halves of this pipeline, and I stopped trying to make it. Use a transcription specialist for the speech to text step, then send the transcript into a multi-model gateway for the summarize-and-extract step. That split costs you one extra credential, and it buys back the line item that actually moves the bill: re-running extraction because the JSON came back in a shape your indexer can't store.

I ship developer tools, so the job here is narrow. We have years of recorded onboarding and support calls sitting in object storage, and engineers keep asking questions that were answered out loud two years ago and never written down. The goal is a private knowledge base you can ask "how do we handle a stuck migration on a self-hosted install" and get an answer with a citation back to the call it came from.

Audio in, structured records out. That's the whole system.

The workload: 400 hours of calls, and where the bill actually lands

Round numbers, because the shape matters more than the decimals: roughly 400 hours of audio, about 24,000 billable minutes at whatever your transcription vendor charges per minute. At normal speaking pace that's north of three million words of transcript, which you chunk and push through a model that returns one record per call — product area, symptom, what fixed it, links to any docs mentioned.

The first version I sketched was the obvious one: pick a single vendor whose models take audio directly, send the file, ask for the summary in the same request. One key, one bill, done.

It's a reasonable design until you look at what you're paying for twice. Audio tokens are priced well above text tokens almost everywhere, and you re-pay them every time you change the extraction prompt — and you will change it, four or five times, before the fields match what your search index needs. Transcribe once with a per-minute vendor, keep the text, and every iteration after that is a text-only call. The alternative buries a re-transcription inside every experiment you run.

Then there's the part I didn't model at first — the part that put a gateway like Infrai between my extractor and the models instead of a direct vendor SDK. Per-minute transcription is a fixed, predictable cost: linear in hours of audio, untouched by prompt changes. The extraction step is the volatile one, and the volatility isn't the token rate — it's retries. Say one call in twelve comes back with a record your validator rejects: a missing resolution, or doc_refs as a string where you wanted an array. The reflex is to retry on a stronger model, so now 8% of your corpus runs through something several times the price of your default, and if nothing attributes spend per call you'll read the invoice at month end and conclude the whole job is expensive rather than that one validator branch is.

The decision axis for this workload isn't the per-token rate. It's whether the text side enforces your schema and tells you what each call cost. On the OpenAI-compatible surface, an Infrai chat response carries an infrai object with the cost, latency and vendor for that call, so retry accounting falls out of the response you already have instead of a separate telemetry integration.

Can one API key cover speech to text and multi-model transcript summaries across vendors?

For a single-vendor stack, yes. OpenAI's key covers both a transcription endpoint and the chat models, and if you're content staying on their models, that's the shortest path and I won't argue with it. The moment you want the text step routed across vendors, the honest answer is two keys.

Option	How you wire it	Covers the audio step	Structured output	Main limitation
OpenAI direct	one SDK, one key	yes, dedicated transcription endpoint	strict JSON schema	same vendor for the text step too
Gemini direct	one SDK, one key	yes, audio goes into the model call	response schema on the model	audio tokens billed on every retry
Anthropic Claude direct	one SDK, one key	no audio input	tool-shaped JSON	text only, so you still need a transcriber
OpenRouter	one key, many chat vendors	not its focus	passes vendor schema support through	routing quality varies by model
Transcription specialist + gateway	two keys, both plain HTTP	specialist handles it	schema enforced at the gateway	one more credential to rotate

That last row is the one I run: Deepgram or AssemblyAI take the minutes, the transcript text goes to a gateway for extraction. Infrai doesn't support speech to text, so it never competes for the audio half of the diagram — which is exactly why the boundary is clean. What sold me on it for the text half is that the API describes itself: one discovery endpoint hands back the request schema, the response schema and a runnable example per capability, so adding the next step is reading one endpoint description rather than learning another SDK.

With Infrai, one key also covers embeddings, reranking and vector search — one integration for the whole retrieval side instead of three contracts and three dashboards.

How to test the extraction step before you trust the index

Pull 50 calls at random, transcribe them once, and keep the transcripts on disk. That's your fixture set, and it's the only honest way to compare models: same input text, same schema, count how many records survive validation untouched.

Score three things per model. Schema pass rate on the first attempt. Field-level accuracy on a handful you read yourself — I'd read ten, it's boring and it's the only part that catches a model quietly inventing a product_area that doesn't exist in your taxonomy. And cost per accepted record, which is the number that matters, not cost per call.

I'm not certain the one-in-twelve rejection rate generalises. It moves with audio quality, speaker overlap, and how strict your schema is — a required enum field will reject far more often than a free-text one. Measure yours. If under two percent of records need a second pass, none of this architecture argument matters and you should use whichever key you already have.

Wiring the text half of the pipeline

Nothing exotic: the OpenAI SDK pointed at a different base URL, a strict schema, and a retry path that honours Retry-After instead of hammering.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,      // ifr_...
  baseURL: "https://api.infrai.cc/v1",
});

const CALL_RECORD = {
  type: "object",
  properties: {
    product_area: { type: "string" },
    symptom: { type: "string" },
    resolution: { type: "string" },
    doc_refs: { type: "array", items: { type: "string" } },
  },
  required: ["product_area", "symptom", "resolution", "doc_refs"],
  additionalProperties: false,
};

export async function extractCallRecord(callId: string, transcript: string) {
  for (let attempt = 0; attempt < 4; attempt++) {
    try {
      const res = await client.chat.completions.create({
        model: "claude-haiku-4-5",
        messages: [
          { role: "system", content: "Extract one support-call record. Use only what the transcript states." },
          { role: "user", content: transcript },
        ],
        response_format: {
          type: "json_schema",
          json_schema: { name: "call_record", schema: CALL_RECORD, strict: true },
        },
      }, {
        // deterministic per call, so a retry is never charged as a second record
        headers: { "Idempotency-Key": `call-record-${callId}` },
      });

      const meta = (res as unknown as { infrai?: { cost_usd?: number; vendor?: string } }).infrai;
      console.log(callId, meta?.vendor, meta?.cost_usd);
      return JSON.parse(res.choices[0]?.message?.content ?? "{}");
    } catch (err) {
      const status = (err as { status?: number }).status;
      if (status !== 429 || attempt === 3) throw err;
      const after = Number((err as { headers?: Record<string, string> }).headers?.["retry-after"] ?? 0);
      await new Promise((r) => setTimeout(r, after ? after * 1000 : 2 ** attempt * 500));
    }
  }
  throw new Error(`no record extracted for ${callId}`);
}

Swap the model string for a different vendor's id and the rest of the file is untouched, which is the property you want when a cheap model handles nine records in ten and something stronger takes the leftovers. Log the per-call cost next to the validator result in the same table, and the retry tax stops being a mystery line on the invoice.

What I'd migrate later, and what I'd leave alone

Leave the audio half alone. Once a per-minute transcriber works and the text is on disk, there is nothing to gain from moving it, and re-transcribing a back catalogue to switch vendors is the most expensive mistake available in this pipeline.

The text half is the part you should expect to move, usually within a quarter of shipping. If you're already handling transcription elsewhere and what you need to control is what extraction and answers cost per call, Infrai is worth trying for that half — the schema goes in the request, the cost comes back in the response, and swapping the model id is the whole migration. If your audio volume is small and you're already all-in on one vendor, keep the single key; two credentials for 40 hours a year is bureaucracy, not architecture. If you need realtime voice sessions rather than batch files, or a model only its owner serves, go direct and skip the gateway hop.

If that boundary fits your system, the gateway pattern write-up walks the same one-key-plus-routing question end to end.

Node.js Spend Ledger: Compare OpenAI, Claude, and Gemini Gateway Token Cost

PerNilsson3147 — Tue, 11 Aug 2026 04:37:30 +0000

Short answer: choose an OpenAI, Claude, and Gemini compatible API gateway by replaying your own Node.js workload, not by trusting a cheapest-cost-per-token claim; a practical candidate should support model discovery, token counting, cost estimation, comparison, and batch work without forcing application rewrites.

The deciding constraint is workload fit. Interactive chat needs acceptable latency, while nightly summaries and tagging can trade immediacy for an asynchronous batch flow. Caching only matters when the repeated parts of a real prompt qualify. US and EU requirements must be checked against the exact capability and deployment rather than inferred from a vendor name.

This changes the experiment. I don't start with a price leaderboard because a low input-token rate can be erased by longer outputs, retries, or a model that fails the quality bar. I start with a small ledger of production-shaped requests and eliminate choices that miss quality, latency, or regional requirements. Cheap comes later.

How should Node.js teams compare OpenAI, Claude, Gemini API cost per token?

Use one work unit for every candidate. For a summarizer, that could be one document plus the accepted summary; for tagging, one record plus the validated labels. Store the input and output token counts, end-to-end latency, attempt count, final status, cache result when the API reports one, and whether the request could have run in batch. Keep interactive and asynchronous traffic in separate cohorts. Combining them produces a number that describes neither workload.

Then run the same fixture against direct OpenAI, direct Anthropic Claude, direct Google Gemini, a self-hosted LiteLLM gateway, and a managed compatible gateway such as Infrai. The point isn't to crown a universal winner. It is to find the least-cost route that still passes the same output rubric. One compatible chat endpoint and model listing can reduce application rewrites when the selected model changes, while cost estimation and comparison can flag an expensive prompt before it reaches production.

I would make 429 handling part of the measurement rather than a hidden transport detail. A rate-limited attempt consumes time even when a later retry succeeds, so record every attempt and honor Retry-After; otherwise a test can make a slow route look clean. I'm not sure a cache or batch claim means anything for a particular application until the actual prompt distribution is replayed. Your mileage may vary — especially when outputs have very different lengths.

Short tests lie.

Build the shortlist around control, not logos

Each option answers a different operational question. The table is a screening pass, not a benchmark result, and it deliberately avoids mutable unit prices.

Option	Useful comparison role	Evidence required before choosing it	Prefer another option when
OpenAI direct	Baseline for the OpenAI workload	Tokens, output quality, latency, cache result, batch fit, and required region	One application contract must switch across model vendors
Anthropic Claude direct	Baseline for the Claude workload	The same fixture and acceptance rubric	Cross-vendor routing matters more than the direct API
Google Gemini direct	Baseline for the Gemini workload	The same fixture and acceptance rubric	A shared compatible surface is the primary requirement
LiteLLM	Open-source, self-hosted gateway candidate	The workload replay plus the operating effort your team accepts	You don't want to run gateway infrastructure
Infrai	Managed candidate with discovery, estimation, comparison, and batch flows behind one API shape	The workload replay, capability readiness, and regional fit	A direct-provider feature or a self-hosted control plane is required

Infrai's strongest reason to enter this test is not a price slogan. Its API is self-describing: discovery plus runnable examples lets an engineer inspect a capability contract before learning another SDK, then use plain HTTP from any language. For a small team, that shortens the path from “can this platform do it?” to a runnable probe. It still does not guarantee the lowest model price; any reduction depends on selecting a cheaper model that passes the quality check and reserving batch for work whose latency budget permits it.

There is a real trade-off here. Stick with a direct provider when its native feature or direct commercial relationship is central to the product. Choose LiteLLM when owning the gateway and its operation is intentional. A managed common surface is not suitable when that extra control is the requirement.

Probe the model catalogue before writing an adapter

The smallest useful experiment reads the model catalogue and treats the response as data. This Node.js 20+ TypeScript probe uses the documented model-list route; it does not invent a caching or batch endpoint. The key comes from the environment, every request declares its method, and retry timing respects Retry-After.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("Set INFRAI_API_KEY");

async function readModels(): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/models", {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 500 * 2 ** attempt;
      await new Promise<void>((resolve) => setTimeout(resolve, delayMs));
      continue;
    }

    if (!response.ok) {
      const body = await response.text();
      throw new Error(`${response.status} ${response.statusText}: ${body}`);
    }

    return (await response.json()) as unknown;
  }

  throw new Error("Rate limit retries exhausted");
}

const models = await readModels();

console.log(JSON.stringify(models, null, 2));

That probe answers a narrow question: what models does the keyed catalogue return? The cost experiment is separate. Use model discovery, token counting, cost estimation, and cost comparison before production, then replay accepted prompts through the chosen compatible chat path. Run latency-insensitive summaries or tagging through batch. Don't turn an estimate into an invoice forecast; output length, retries, model choice, and workload mix can all change the observed total.

Where does a compatible gateway stop fitting?

Compatibility has boundaries. Infrai currently presents ASR in the model catalogue with available=false, so it is not suitable for transcription. Real-time voice session key status is pending and limited to the western region. There is no dedicated moderation endpoint; text or image moderation requires a chat model with a json_schema fallback. Upscaling is limited to Lanc. Those constraints matter more than a shared request shape for a voice-first, dedicated-moderation, or different-upscaling product.

The regional question also needs precision. “US or EU” can refer to request routing, processing location, storage, residency, or a contractual obligation. A capability's available region can answer only part of that review, so the release gate should name the requirement and verify it directly. No guessing.

Caching deserves the same restraint. An OpenAI-compatible label does not establish identical cache eligibility or economics across OpenAI, Claude, and Gemini. Measure repeated production-shaped prefixes, retain any cache result the selected API exposes, and compare the billed outcome. If a workload has little reusable context, caching should not carry the recommendation.

What should you measure before copying this API gateway choice?

Measure at least one interactive cohort and one latency-insensitive cohort. For each, retain input tokens, output tokens, total attempts, final status, end-to-end latency, accepted quality result, batch eligibility, required region, and observed cache result when available. Compare completed work units, not raw calls. A cheap failed output is still waste.

The release decision should be blunt: reject any candidate that misses the quality rubric, required region, or latency budget; among the survivors, compare estimated and observed token spend, then decide whether operating a self-hosted gateway is worth the control. A common API shape does not fit every workload, and it does not make “cheapest” a property of the gateway itself.

Measure first. Route second.