The hard part of speech-to-text moderation is not parsing a transcript. It is deciding what happens when a burst of gaming reports arrives and the transcription API answers with HTTP 429. Short answer: put transcription behind a durable queue, honor Retry-After when it is present, use bounded jittered backoff when it is absent, and reserve batch transcription for work that can tolerate extra latency.
That choice protects both sides of the quality-versus-latency trade-off. A moderator-facing report may need a fast provisional result; an overnight voice-chat audit can wait. Treating those jobs as one workload is how a small rate limit becomes a system-wide incident.
Queue first.
What should a speech-to-text API queue do after 429 headers?
A 429 is a scheduling signal, not permission to replay the request in a tight loop. The worker should read the Retry-After header, delay the attempt, and keep the job visible as pending. If the header is missing or unusable, bounded exponential backoff with jitter gives workers different wake-up times. The queue also needs a maximum attempt count and a dead-letter state, so one stubborn audio file cannot consume the entire worker pool.
The web request should create a job and return its ID. It should not upload audio, sleep, and wait while an upstream limit decides the schedule. A worker owns attempts, timestamps, and the final state: pending, complete, or failed. A result write should be idempotent. If a worker crashes after transcription succeeds but before acknowledging the queue message, the next delivery must not create a second moderation record.
That last transition deserves more design attention than its four status labels suggest. Imagine a report containing a 38-second clip: the worker sends it, receives a transcript, stores the text, and then loses its lease before acknowledging the message. A second worker will quite reasonably try the same job. The durable record therefore needs a stable job key, an atomic “result already exists” check, and a rule for what happens when two attempts return different text. Keep the first accepted result, retain attempt metadata for review, and make the moderation pipeline consume the stable record rather than the worker's in-memory response. This is also where observability belongs: log the job ID, attempt number, response status, queue age, and completion timestamp, while keeping the audio and transcript access controls separate from retry logs. The point is not to make duplicates impossible; it is to make duplicate delivery harmless and diagnosable.
Here is the policy boundary I would test first. The sender is intentionally an adapter: the retry code knows HTTP behavior, while the provider-specific audio request stays elsewhere.
from dataclasses import dataclass
from collections.abc import Callable
import random
import time
@dataclass(frozen=True)
class Response:
status_code: int
retry_after: str | None = None
class TerminalRequestError(RuntimeError):
pass
def retry_delay(header: str | None, attempt: int) -> float:
if header is not None:
try:
return max(0.0, float(header))
except ValueError:
pass
return min(30.0, 2.0 ** attempt)
def send_with_policy(
send: Callable[[], Response],
max_attempts: int = 5,
sleep: Callable[[float], None] = time.sleep,
) -> Response:
for attempt in range(max_attempts):
response = send()
if 200 <= response.status_code < 300:
return response
if response.status_code != 429:
raise TerminalRequestError(
f"terminal HTTP status {response.status_code}"
)
delay = retry_delay(response.retry_after, attempt)
sleep(delay + random.uniform(0.0, 0.25))
raise TerminalRequestError("retry budget exhausted")
responses = iter([
Response(429, "1"),
Response(429),
Response(200),
])
result = send_with_policy(lambda: next(responses), sleep=lambda _: None)
assert result.status_code == 200
The important behavior is narrow. A non-429 4xx should usually become a terminal job failure until the application changes the request or selects a supported capability. Retrying every client error creates noise and hides bad input. A 5xx or network timeout may deserve its own transient policy, but that policy should be explicit rather than smuggled into the 429 branch.
Do not confuse delay with progress.
One small test catches a surprising amount: return 429 with Retry-After: 1, then 429 without the header, then 200, and assert that the worker sleeps twice before completing. Add a separate test for a 4xx and assert that the sender runs once. These are eval cases, not production benchmarks. They belong in the same harness as transcript-quality checks because a perfect transcript delivered after an unusable delay is still a failed moderation workflow.
When does batch transcription improve the moderation workflow?
Batch processing helps when the work has a deadline instead of a conversation. Examples include rechecking last week's flagged voice clips, building an evaluation corpus, or transcribing archived reports before a model review. The queue can meter batch submission and status polling just like individual jobs, while the application records one terminal outcome per input.
Batch does not remove rate limits. It changes scheduling and progress reporting. A batch job still needs an ID, an ownership record, a polling interval, a timeout, and reconciliation for items that never reach a result. Retrying a whole batch after an ambiguous network failure can duplicate work; prefer item-level idempotency where the API contract permits it, and make result persistence idempotent regardless.
For live moderation, the latency budget is stricter. A moderator may accept a partial signal first and a fuller transcript later, but that is a product decision, not a retry trick. The system should label provisional text clearly and prevent it from becoming the sole basis for an irreversible action. Quality controls belong beside the queue, not after it.
A practical decision rule for quality versus latency
Start with two lanes. The interactive lane has a short queue-age target, a small bounded retry budget, and an explicit fallback state for human review. The bulk lane accepts longer queue age, uses batch submission where it matches the provider contract, and spends its time budget on larger evaluation sets. Both lanes share authentication boundaries, result schemas, observability, and deduplication.
The metrics should make the trade-off visible: queue age, time to first transcript, end-to-end completion time, 429 count, retry count, terminal status class, duplicate-result count, and transcript error rate on a labeled sample. For a gaming system, slice those measures by clip length, language, and report type. Otherwise a good average can conceal the exact category moderators care about.
I would also track token and prompt cost after transcription, especially when transcripts feed summarization or RAG. Keep ASR evaluation separate from retrieval evaluation. If both stages change in one release, a single pass rate cannot tell you which change helped.
There is a real limit to this design. A queue is not suitable when the product promise requires a transcript inside a synchronous request and no provisional state is acceptable; in that case, choose a lower-latency speech path and validate its quality before launch. Batch is not suitable for urgent review. A self-hosted gateway is not suitable for a team that cannot operate another durable service. Stick with the simpler direct adapter when the workload is small enough to observe and replay safely.
Your mileage may vary with audio length, language mix, concurrency, and moderation policy. I'm not sure which speech model will win on a particular game's slang without a representative labeled set, and guessing would be less useful than running that eval. The architecture earns its keep when it makes that uncertainty measurable without turning every 429 into a user-visible failure.
Top comments (0)