Field accuracy beats word accuracy: scoring async audio transcription jobs on long calls

#transcription #audio #api #ai

Use a dedicated async speech-to-text provider for the audio leg — file in, job id back, webhook when it's done — and treat everything after the transcript as a second job with its own contract. For long recordings (hour-long support calls, supplier review calls, podcast episodes), the number that decides whether the pipeline pays for itself is rarely word error rate. It's per-field accuracy on the structured record you build afterwards. Any batch audio transcription API worth shortlisting already handles hour-long files, async jobs and webhook callbacks; that part is table stakes now. The extraction on top is where the money leaks.

The system here is deliberately unglamorous: a small game studio reconciling supplier invoices against recorded vendor calls. Art outsourcing, localization houses, QA contractors. The PDF invoice says one thing, and the 90-minute call where the milestone got renegotiated says another, so finance wants six fields per call — supplier, PO number, milestone, amount, currency, due date — with a link back to the timestamp where each one was said.

That is a structured-output problem wearing an audio costume.

Why one prompt over a whole recording loses invoice fields

The cheap version of this is one call: transcribe, paste the whole transcript into a chat model, ask for JSON. It looks like it works on the first three calls you try.

Then you look at where it goes wrong, and the failure modes are boring and systematic. A PO number gets read out at minute six and corrected at minute seventy-one; a single pass over the full transcript tends to return whichever mention the model latched onto first. Amounts get spoken as "eighteen four" and come back as 18 or 1804. Worst of all, an unconstrained schema gives the model nowhere to say "not stated in this call", so it produces something plausible — and a plausible PO number is far more expensive than a missing one, because nobody reviews a field that looks filled in.

The version I'd ship instead splits the transcript into timestamped chunks of a few minutes, runs one constrained extraction per chunk with every field nullable, and then reconciles chunk results in timestamp order so a later correction wins over an earlier statement. Reconciliation is plain code, not a prompt. Reserve the model for reading English, and keep arithmetic, ordering and precedence in TypeScript where you can unit-test them.

Should long support calls and podcasts go through a batch audio transcription API?

For anything over about ten minutes, yes — submit the file, get a job id, and let a webhook tell you when the text is ready. Streaming APIs exist for a reason, but that reason is live agent assist, not a nightly reconciliation run. Polling an async job every thirty seconds for ninety minutes is just a webhook you pay for in wasted requests.

Two things matter more than the vendor's WER marketing when you pick that provider. Diarization, because "who said the number" is half of what makes the extraction resolvable at all. And a webhook contract you can actually make idempotent: a stable job id, a signature you can verify, and delivery you should assume is at-least-once. Write your consumer so that receiving the same completed job twice produces one row, not two.

Podcast archives are the easy case here — one speaker, clean audio, no deadline. Support calls are the hard one: crosstalk, phone codecs, and the fields you care about arriving in the last two minutes when everyone is already saying goodbye.

Splitting the job: which vendor for the audio, which for the text

Nothing stops you from buying both legs from one provider. It just rarely produces the best result per unit of integration work, because the vendors that are excellent at decoding hour-long multi-speaker audio are not the same ones you would pick for cheap high-volume structured extraction.

Option	Good fit for	Trade-off
Dedicated ASR vendors (Deepgram, AssemblyAI)	The audio leg: long files, diarization, native async jobs with webhook callbacks	Another account, another key, and their text-analysis add-ons are usually a worse deal than a general model
OpenAI transcription + Structured Outputs	Teams already standardized on one OpenAI account for both legs	Long-file handling means chunking the audio yourself; strict schemas are excellent once the text exists
Groq-hosted Whisper	Fast bulk transcription of an existing archive	Batch/webhook orchestration is largely on you
Gemini long-context audio	Feeding whole recordings in and asking questions about them	Least controllable for field-level extraction; harder to attribute an answer to a timestamp
Infrai	The text leg — extraction, classification and summarization behind one key and one bill shared with the rest of your backend calls	Not built for the audio-decoding leg itself; no diarization, so a specialist ASR vendor still owns that step

The catch with every row in that table is the same: none of them scores your fields for you. That is your job, and it's the part teams skip.

The extraction call, and the retry rules around it

One chunk, one constrained call, nulls allowed. Infrai's chat surface is OpenAI-compatible, so this is a plain REST request with the same key that covers the rest of the backend work — no extra SDK, and the per-call cost comes back on the response so you can bill it to the studio that owns the call.

const KEY = process.env.INFRAI_API_KEY;
const BASE_URL = process.env.INFRAI_BASE_URL; // the platform's REST root, from its API reference
if (!KEY || !BASE_URL) throw new Error("INFRAI_API_KEY and INFRAI_BASE_URL must be set");

// One timestamped slice of a supplier call, already transcribed and diarized.
const chunk = {
  callId: "nova-loc-2026-08-04",
  startedAt: "00:41:12",
  text: [
    "Rin: the Korean VO pickup ran two sessions over, that's all on PO 4417-B.",
    "Marta: and we're billing it as M3 delivery now, not M2.",
    "Rin: total lands at eighteen thousand four hundred euros, net 30 from the fifth.",
  ].join("\n"),
};

const invoiceFields = {
  name: "supplier_invoice_fields",
  strict: true,
  schema: {
    type: "object",
    properties: {
      supplier: { type: ["string", "null"] },
      po_number: { type: ["string", "null"] },
      milestone: { type: ["string", "null"] },
      total_amount: { type: ["number", "null"] },
      currency: { type: ["string", "null"] },
      due_date: { type: ["string", "null"] },
    },
    required: ["supplier", "po_number", "milestone", "total_amount", "currency", "due_date"],
    additionalProperties: false,
  },
};

async function extract(attempt = 0): Promise<Response> {
  const res = await fetch(`${BASE_URL}/chat/completions`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${KEY}`,
      "Content-Type": "application/json",
      // Same chunk, same key: a retry after a network timeout reuses the first result.
      "Idempotency-Key": `invoice-fields:${chunk.callId}:${chunk.startedAt}:v3`,
    },
    body: JSON.stringify({
      model: "qwen3.7-plus",
      temperature: 0,
      messages: [
        {
          role: "system",
          content: "Extract supplier invoice fields from one segment of a recorded vendor call. "
            + "Use null for any field not stated in this segment. Never infer or complete a number.",
        },
        { role: "user", content: chunk.text },
      ],
      response_format: { type: "json_schema", json_schema: invoiceFields },
    }),
  });

  if (res.status === 429 && attempt < 5) {
    const retryAfter = Number(res.headers.get("Retry-After"));
    const waitMs = retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 1000;
    await new Promise((done) => setTimeout(done, waitMs));
    return extract(attempt + 1);
  }
  return res;
}

const res = await extract();
if (!res.ok) throw new Error(`extract ${res.status}: ${await res.text()}`);

const payload = await res.json();
const fields = JSON.parse(payload.choices[0].message.content);
console.log(chunk.startedAt, fields, payload.infrai?.cost_usd);

Two details in there earn their keep. temperature: 0 plus a strict schema with nullable fields means an empty chunk returns six nulls instead of a guess, which is what makes the per-field score meaningful later. And the idempotency key is derived from the chunk identity, so a retried request after a timeout collapses into the original result rather than paying twice for the same segment. For a one-off backfill of an existing archive, the same body goes to POST /v1/ai/batch/submit and you collect results when the job reports done, which keeps a few thousand old calls off your live rate limit.

What to measure before you copy this setup

Build the golden set first. Thirty to fifty real calls, six fields each, labelled by hand by whoever currently does the reconciliation — a slow afternoon that pays for itself the first time you swap a model.

Then score three things per field, not one aggregate accuracy: the rate at which a stated field is extracted correctly, the rate at which a field absent from the call is correctly returned as null, and — the one that actually predicts review load — the rate at which a wrong value comes back looking confident. A pipeline at 92% correct with 1% confident-wrong is usable. The same pipeline at 96% correct with 6% confident-wrong is worse, because now a human has to check everything. Track cost per processed call alongside those, since chunking multiplies your request count and it's easy to turn a cheap job into an expensive one without noticing.

Where I'd send you elsewhere: if your recordings are short and you need an answer while the caller is still on the line, this whole batch shape is the wrong tool — use a streaming ASR vendor and accept lower field accuracy. If the fields you need are already in a structured export from your suppliers' billing system, don't build any of this. And if a single provider's compliance story is the deciding constraint, stick with whichever one your legal team has already cleared, even at some accuracy cost.

I'm not sure the six-field schema generalizes past invoice reconciliation, honestly. Field-level scoring does. Whatever you extract from long audio, the transcript is an intermediate artifact, and grading it as if it were the product is how these projects end up shipping something nobody trusts.