DEV Community

mT41vB6
mT41vB6

Posted on

Schema-First Speech-to-Text Uploads: MP3 and WAV Moderation Clips as Structured Records

#api #python #audio #moderation

Use two calls, not one: send the file to a speech-to-text API that returns a transcript with timestamps, then classify that transcript in a second call whose output is validated against a JSON Schema before it reaches a human queue. The deciding constraint is auditability, not latency. An endpoint that takes an MP3 and hands back a moderation category is delightful right up until the first appeal, when someone asks which words in the clip produced that decision and you have nothing but a label.

The system here is a media platform's report intake. Users flag clips — voice notes, podcast segments, streamed audio ripped into MP3 or WAV — and a machine pass has to file each report as a structured record before a reviewer ever presses play.

Volume is why the machine pass exists. The review clock is why its output has to be correct in a boring, checkable way: under the EU's Digital Services Act, a platform that receives a notice has to act on it without undue delay and then tell the reporter what it decided and why. A record that says category: "hate" and nothing else cannot become a statement of reasons. A record carrying a category, a severity, and two evidence spans with millisecond offsets into the audio can.

The invariants an audio report intake has to hold

Everything downstream — retries, backfills, appeals, the eventual argument with legal — gets easier if the intake refuses to violate five things.

  • The audio is content-addressed. A SHA-256 of the bytes is the file's identity; the same clip mass-reported by 400 accounts is one transcription, not 400.
  • The transcript is keyed by (digest, model_id, params). Change the model and you get a new row, not a silent overwrite of the evidence a past decision rested on.
  • Every classification record validates against a versioned schema before it is queued. No exceptions, no partial writes.
  • Nothing is ever dropped. A clip that cannot be transcribed goes to a human with the raw audio attached and a reason string, which is a worse outcome than automation but a defensible one.
  • Origin region decides processing region, and it is decided once, at intake, from the reporter's context — not later, by whichever worker happened to pick up the job.

That last one is the invariant teams bolt on last and regret most. Region has to be a property of the report, carried on every hop, because the audio, the transcript, and the classification are three different pieces of personal data with three different retention clocks, and reconstructing "where was this allowed to be processed" after the fact is unpleasant work.

What should a speech-to-text upload API return before a human sees the file?

Four fields, and a transcription API that omits any of them will cost you a workaround somewhere else in the pipeline.

You need the transcript text, obviously. You need segment or word timestamps, because evidence spans are the difference between a reviewable record and a vibe. You need the detected language, because a clip in Portuguese scored by an English-language policy prompt produces a schema-valid record that is simply wrong. And you need the decoded duration and byte count the server actually received, which is the cheapest truncation check in existence.

That fourth one catches the failure I would design against first. Multipart uploads fail in a way that looks like success: the connection drops mid-body, the server assembles a well-formed part from what arrived, and a 9-minute clip becomes a 40-second one whose transcript ends in the middle of a sentence. The record validates. The severity comes out low. Nobody notices until the reporter escalates. Comparing the byte count you sent against the byte count the server decoded turns that class of bug into a loud error at intake, and it costs one integer comparison.

The other edge cases in this format zoo are less dramatic and more frequent. Variable-bitrate MP3 files without a Xing or VBRI header report wildly wrong durations to naive parsers, so trust the decoder's duration, not the header's. WAV is a container, not a codec — a .wav file can hold 24-bit PCM, or A-law telephony audio at 8 kHz, and the mono 8 kHz case is exactly what phone-recorded abuse reports look like. Long silences and hold music are a well-documented source of spurious transcript text in autoregressive speech models, which matters here because a hallucinated sentence in an empty clip can produce a confident classification of nothing at all. Guard it with a voice-activity check before the classifier ever sees the text.

Comparing three intake shapes for MP3 and WAV clips

Intake shape What you operate Where it breaks first Reasonable when
One request: upload, transcribe and classify inline Almost nothing A 40-minute clip outlives your HTTP timeout, and retries re-upload the whole file Short clips, low report volume, one region
Upload to object storage, queue a job, receive a webhook A queue, a webhook receiver, a dead-letter path Webhook delivery, replay and ordering Mixed clip lengths, steady volume, audit requirements
Self-hosted speech model next to a hosted classifier GPUs, model versions, capacity headroom Capacity planning and upgrade windows Strict residency, or audio you are not permitted to send out

Most media moderation intakes belong in the middle row, and the honest reason is not technical elegance. It is that the middle row separates the two calls, which is what lets you re-run classification over a stored transcript when the policy taxonomy changes — a thing that happens two or three times a year and would otherwise mean re-uploading and re-billing every clip in the archive.

The examples below are Python because that's where intake workers tend to live, but nothing in the contract is language-specific. In Node.js the same two calls are FormData plus fetch with the identical headers, and the schema validation step is any JSON Schema validator you already trust.

The critical path in code

Upload first. Content hash, idempotency key, region pin, declared length, and a retry loop that honours Retry-After instead of inventing its own backoff:

import hashlib
import time
import requests

ASR_ENDPOINT = "https://asr.eu.example.internal/transcribe"
RETRYABLE = {429, 500, 502, 503, 504}

def transcribe(path: str, report_id: str, region: str) -> dict:
    raw = open(path, "rb").read()
    digest = hashlib.sha256(raw).hexdigest()
    headers = {"Idempotency-Key": f"{report_id}:{digest}", "X-Processing-Region": region}
    files = {"file": (f"{digest}.mp3", raw, "audio/mpeg")}
    form = {"timestamps": "segment", "language": "auto", "declared_bytes": str(len(raw))}

    for attempt in range(4):
        r = requests.post(ASR_ENDPOINT, headers=headers, files=files, data=form, timeout=180)
        if r.status_code in RETRYABLE:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        r.raise_for_status()
        out = r.json()
        if out["received_bytes"] != len(raw):
            raise TruncatedUpload(f"{report_id}: server read {out['received_bytes']} of {len(raw)}")
        return {"digest": digest, **out}
    raise Unavailable(f"{report_id}: no transcript after 4 attempts, escalating to human review")
Enter fullscreen mode Exit fullscreen mode

The idempotency key is report_id plus digest rather than a random UUID per attempt, so a retry after a network drop returns the original job instead of paying for a second transcription of identical bytes.

Then the part that actually decides whether this pipeline is trustworthy. The classifier's job is to emit one object; the intake's job is to disbelieve it until it validates:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["schema_version", "report_id", "category", "severity", "evidence", "needs_human"],
  "properties": {
    "schema_version": { "const": "2026-02-11" },
    "report_id": { "type": "string", "minLength": 1 },
    "category": { "enum": ["harassment", "hate", "self_harm", "sexual", "violence", "spam", "none"] },
    "severity": { "type": "integer", "minimum": 0, "maximum": 3 },
    "language": { "type": "string", "pattern": "^[a-z]{2}(-[A-Z]{2})?$" },
    "evidence": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["start_ms", "end_ms", "quote"],
        "properties": {
          "start_ms": { "type": "integer", "minimum": 0 },
          "end_ms": { "type": "integer", "minimum": 0 },
          "quote": { "type": "string", "minLength": 1 }
        }
      }
    },
    "needs_human": { "type": "boolean" }
  }
}
Enter fullscreen mode Exit fullscreen mode

additionalProperties: false and a closed enum are doing the heavy lifting. A model that invents category: "misinformation" because the clip sounds like misinformation is not being helpful; it's putting a value into a queue whose consumers switch on seven known strings.

Structural validity isn't correctness, though, so the second gate checks the record against the transcript it claims to describe:

from jsonschema import Draft202012Validator

VALIDATOR = Draft202012Validator(REPORT_SCHEMA)

def accept(record: dict, transcript: dict) -> dict:
    VALIDATOR.validate(record)
    duration = transcript["duration_ms"]
    spoken = transcript["text"].lower()

    for span in record["evidence"]:
        if not 0 <= span["start_ms"] < span["end_ms"] <= duration:
            raise SpanOutOfRange(span)
        if span["quote"].lower() not in spoken:
            raise QuoteNotInTranscript(span)

    if record["severity"] >= 2 or record["language"] not in REVIEWED_LOCALES:
        record["needs_human"] = True
    return record
Enter fullscreen mode Exit fullscreen mode

The quote check is the one I would not skip. An evidence span whose text never appears in the transcript is the clearest signal available that the classification was produced from something other than the audio, and it is a plain substring test rather than a model-graded evaluation.

On failure, retry the classification exactly once with the validator's error message appended to the input, then stop. A record that fails twice gets needs_human and goes to a person with the transcript attached. Repair loops that run three or four times mostly produce records that are valid and confidently wrong, which is the worst possible output for a queue humans are trusting to be pre-sorted.

The single-call design I rejected, and when it is the right one

The rejected option is the appealing one: post the MP3 to a multimodal endpoint and read a category out of the response. Fewer moving parts, one integration to maintain, no transcript storage, no schema file to version.

It's a good fit for internal tagging, prototypes, and anything with no appeal path — auto-labelling an archive so editors can search it, say. Stick with it while the output is advisory.

The catch is that it destroys the artefact you need when the decision is contested. Without a stored transcript you can't re-run last quarter's clips against this quarter's taxonomy, you can't show a reviewer which seconds of audio triggered the flag, and you can't separate a transcription problem from a classification problem when accuracy drops. That trade-off is fine at low stakes and unacceptable once the record feeds a queue that produces enforcement.

Batch tiers deserve a similar caveat. Queued batch processing is a genuinely good fit for archive-wide re-classification after a taxonomy change, where a multi-hour turnaround is irrelevant. It isn't a good fit for live notice-and-action intake, where the same clip needs a first-pass record in minutes.

I'm not sure the two-call split is right at very small scale. If you handle a handful of reports a day, the human is faster than the pipeline and the schema work is overhead — your mileage may vary, and the honest test is whether anyone has ever asked you to explain a decision after the fact. Once someone has, the transcript stops being an implementation detail and becomes the record.

References

Top comments (0)