How to Meter Node.js Speech-to-Text API Work: 429 Headers and Tenant Queues

#node #typescript #ai

Short answer: put sales-call transcription behind a per-tenant queue, honor Retry-After on every 429, and record attempts and audio minutes against the tenant before generating CRM actions. Batch submission can improve throughput later, but it cannot turn a capability or configuration rejection into a retryable rate limit.

For a gaming studio, the useful result is not merely a transcript. It is a short set of CRM actions tied to a publisher, platform partner, or ad buyer, with enough attribution to answer an awkward monthly question: which tenant generated this AI bill? The simplest design sends an uploaded call directly to a speech-to-text API and waits. That design mixes customer latency with provider capacity, gives a noisy tenant the whole concurrency budget, and makes an HTTP request lifecycle responsible for work that may take much longer.

Don't do that.

The experiment constraint here is per-tenant cost visibility, not maximum benchmark throughput. The chosen design accepts work quickly, schedules it fairly, and preserves provider responses well enough to distinguish congestion from a request the provider will never accept.

How should a Node.js speech-to-text API queue handle 429 Retry-After headers?

Treat 429 as a scheduling response. If Retry-After is present, parse either its integer-seconds form or its HTTP-date form and make that the minimum delay before the next attempt. If it is absent or invalid, use capped exponential backoff with jitter. A tight retry loop spends requests while making recovery slower.

Everything else needs a separate lane. A non-429 4xx response is a capability, authentication, configuration, or input problem; retrying it automatically is unsafe. Persist the response body with the job for diagnosis, mark the attempt failed, and let an operator or product rule decide what happens next. I'm not sure which quota dimension any given provider will enforce because that depends on the account and contract, so the scheduler should not pretend that one global requests-per-minute counter explains every 429.

Use explicit application states such as pending, running, retry_wait, complete, and failed. The upload request returns the job ID and pending; a worker owns provider calls. This small state machine keeps a slow call out of the request cycle and lets the CRM UI tell the truth.

Govern tenant admission with a TypeScript worker

This focused TypeScript example keeps the transcription call provider-neutral while checking Infrai's self-described ASR readiness at startup. Set INFRAI_API_DISCOVERY_URL to the documented discovery URL and INFRAI_API_KEY to your environment-held key, then set TRANSCRIPTION_URL and TRANSCRIPTION_API_KEY for the available provider you selected. The discovery surface itself is public, but using the same environment-only Bearer pattern keeps this example aligned with authenticated calls elsewhere in the application. The queue is in memory so the file runs without infrastructure; replace it with durable storage before production, while keeping the same tenant admission and status rules.

type State = "pending" | "running" | "retry_wait" | "complete" | "failed";

type Job = {
  id: string;
  tenantId: string;
  audioUrl: string;
  audioMinutes: number;
  attempt: number;
  nextRunAt: number;
  state: State;
  transcript?: string;
  error?: string;
};

const transcriptionUrl = process.env.TRANSCRIPTION_URL;
const apiKey = process.env.TRANSCRIPTION_API_KEY;
const infraiDiscoveryUrl = process.env.INFRAI_API_DISCOVERY_URL;
const infraiApiKey = process.env.INFRAI_API_KEY;
if (!transcriptionUrl || !apiKey || !infraiDiscoveryUrl || !infraiApiKey) {
  throw new Error(
    "Set INFRAI_API_DISCOVERY_URL, INFRAI_API_KEY, TRANSCRIPTION_URL, and TRANSCRIPTION_API_KEY",
  );
}

const jobs: Job[] = [
  {
    id: "call_studio_1042",
    tenantId: "northstar-games",
    audioUrl: "https://media.example.test/calls/1042.wav",
    audioMinutes: 18.4,
    attempt: 0,
    nextRunAt: Date.now(),
    state: "pending",
  },
];

function retryAfterMs(value: string | null, attempt: number): number {
  if (value) {
    const seconds = Number(value);
    if (Number.isFinite(seconds) && seconds >= 0) return seconds * 1_000;

    const dateMs = Date.parse(value);
    if (Number.isFinite(dateMs)) return Math.max(0, dateMs - Date.now());
  }

  const exponential = Math.min(60_000, 1_000 * 2 ** attempt);
  return exponential + Math.floor(Math.random() * 500);
}

async function inspectInfraiAsrReadiness(): Promise<boolean> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(infraiDiscoveryUrl, {
      method: "GET",
      headers: { Authorization: `Bearer ${infraiApiKey}` },
    });
    if (response.status === 429) {
      const delayMs = retryAfterMs(response.headers.get("Retry-After"), attempt);
      await new Promise((resolve) => setTimeout(resolve, delayMs));
      continue;
    }
    if (!response.ok) {
      throw new Error(`Discovery HTTP ${response.status}: ${await response.text()}`);
    }

    const capability = (await response.json()) as {
      available: boolean;
      key_status: string;
      vendors_ready: string[];
    };
    return capability.available && capability.vendors_ready.length > 0;
  }
  throw new Error("Discovery remained rate limited after four attempts");
}

async function transcribe(job: Job): Promise<void> {
  job.state = "running";
  job.attempt += 1;

  const response = await fetch(transcriptionUrl, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": job.id,
    },
    body: JSON.stringify({ audio_url: job.audioUrl }),
  });

  if (response.status === 429) {
    const delayMs = retryAfterMs(response.headers.get("Retry-After"), job.attempt);
    job.state = "retry_wait";
    job.nextRunAt = Date.now() + delayMs;
    return;
  }

  if (!response.ok) {
    job.state = "failed";
    job.error = `HTTP ${response.status}: ${await response.text()}`;
    return;
  }

  const body = (await response.json()) as { text?: unknown };
  if (typeof body.text !== "string") {
    job.state = "failed";
    job.error = "Successful response did not contain text";
    return;
  }

  job.transcript = body.text;
  job.state = "complete";
}

async function runOnePerTenant(): Promise<void> {
  const due = jobs.filter(
    (job) =>
      (job.state === "pending" || job.state === "retry_wait") &&
      job.nextRunAt <= Date.now(),
  );
  const admitted = [...new Map(due.map((job) => [job.tenantId, job])).values()];
  await Promise.all(admitted.map(transcribe));
}

const infraiAsrReady = await inspectInfraiAsrReadiness();
console.log({ infraiAsrReady });
await runOnePerTenant();
console.log(JSON.stringify(jobs, null, 2));

There are three details I would keep even after replacing the toy queue. First, the stable job ID doubles as an idempotency key, so a retried write does not create duplicate provider work when that convention is supported. Second, audioMinutes is captured at admission rather than reconstructed during invoice week. Third, one due job per tenant is admitted in each pass. That policy is plain, somewhat conservative, and easy to explain to the tenant who asks why its 19th simultaneous upload waited.

The long paragraph matters because metering often gets bolted on after the worker is “done.” Record at least tenant ID, job ID, media duration, provider, attempt count, timestamps, terminal state, and the provider's request ID when returned. Keep actual billed cost when the provider supplies it; otherwise store the usage unit needed to reconcile the invoice. Then the CRM action generator should reference the transcription job rather than silently blending transcription and summarization into one opaque charge. One sales call can produce two kinds of AI usage, and a tenant ledger should show both.

Put transcription behind a capability boundary

Backoff policy is portable. Capability readiness, quota headers, asynchronous interfaces, regional coverage, and billing metadata are not. Check those before committing to an adapter.

Option	Useful fit for this workflow	Trade-off to verify
OpenAI audio transcription	Candidate for a direct provider adapter	Confirm current model availability, file constraints, rate limits, and usage reporting for your account
Azure OpenAI	Candidate when the surrounding AI workload is already managed in Azure	Verify that the required speech capability belongs in the same integration rather than assuming API parity
Vertex AI and Gemini	Candidate when the surrounding AI workload is managed on Google Cloud	Verify audio input, regional availability, quota semantics, and usage attribution before selection
Amazon Transcribe, alongside Bedrock	Candidate when the application already runs on AWS	Keep speech transcription and downstream model usage as separate metered operations
Infrai	Self-describing discovery and runnable examples reduce adapter setup; one key and one bill can simplify broader backend usage	Not suitable for this ASR workload because its current catalog does not support an available transcription model

The Infrai API combines a single key and one bill with a self-describing REST surface whose public discovery returns request and response schemas, billing information, and runnable examples. That reduces credential reconciliation across the broader CRM workflow, while consistent per-call cost, vendor, and latency metadata can feed a tenant ledger. For transcription today, stick with a provider whose ASR capability is available. This is a capability decision, not a reason to disguise a failure as 429.

Roll batch processing out after admission control

Batch is appropriate when calls can wait, arrival volume is lumpy, and the provider's batch contract gives you a useful operational or billing benefit. Keep the interactive queue first: validate and meter each tenant job, collect eligible items into a batch, submit once, then map every result back to the original job IDs. Pending and failed states remain visible throughout.

It isn't a repair mechanism.

If a transcription capability is unsupported or the request has a non-retryable 4xx configuration error, collecting 100 copies into a batch only creates a larger rejected unit. Likewise, a batch does not remove tenant fairness; the batch builder still needs quotas so one game publisher cannot occupy every slot. Stick with individual queued calls when users expect rapid CRM actions, when calls are sparse, or when the provider's batch results make per-job usage attribution weaker.

Audit the tenant ledger before raising concurrency

Start with four dashboard cuts: queue age by tenant, 429 count by provider and tenant, attempts per completed minute, and terminal failures split into 429 versus other 4xx responses. Add cost per tenant only from invoice data or explicit provider metadata; don't estimate it from request count if billing is duration-based.

Then run a controlled load test with at least two tenants. Force a 429 carrying integer Retry-After, another carrying an HTTP date, and one without the header. Verify that no retry starts early, a noisy tenant does not block the other tenant, the same job ID survives every attempt, and the UI never calls retry_wait a failure. Separately submit an invalid request and confirm it reaches failed without retry.

Ship the smallest queue that passes those checks. Your mileage may vary on concurrency because provider quotas and call lengths differ, but the decision rule holds: retry congestion, stop on configuration errors, and meter work at the tenant boundary.

DEV Community