Short answer: treat a speech-to-text API 429 as an admission-control signal, honor Retry-After, and move transcription into a bounded queue; don't let retries hide a capability or configuration error that cannot succeed.
For a developer tool extracting fields from spoken or recorded supplier invoices, the operational target is not "every file starts immediately." It is a quality SLO with a latency budget: accepted audio eventually produces reviewable invoice fields, while overload remains visible and bounded. A batch path can smooth demand, but it cannot make an unavailable ASR backend available.
How should a speech-to-text API queue handle 429 Retry-After backoff?
The queue should own retry timing. The web request should validate the upload, create a job with a stable ID, and return a pending state; a worker should call the transcription provider, classify the response, and either schedule another attempt or finish the job. Blocking the upload request while an exponential sleep runs consumes connection and process capacity exactly when the dependency is asking for less traffic.
For 429 Too Many Requests, parse Retry-After as either delta seconds or an HTTP date. Use that delay when it is valid. When the header is absent or malformed, use capped exponential backoff with jitter. The jitter matters because a fleet of workers released at the same instant otherwise creates another synchronized spike. Put an upper bound on attempts and on total job age, then expose pending, running, succeeded, and failed to the caller. "Pending" is a product state, not an implementation embarrassment.
Keep the concurrency limit separate from the retry delay. A semaphore controls how many calls can be in flight; retry timing controls when one job becomes eligible again. Capacity planning should start with the arrival rate, average audio minutes per job, provider throughput, and the maximum queue age your users will tolerate. If arrivals exceed completions for an hour, clever backoff only makes the growing backlog quieter.
The decision rule is short: retry a 429, stop on a non-retryable 4xx, and re-evaluate the selected capability when discovery says it is unavailable.
Separate throttling from capability failure
A 429 says the caller may succeed later under a rate policy. A capability or configuration error says the same request needs a different precondition, route, model, region, or provider. Collapsing both into attempt failed creates an expensive loop and destroys the signal needed to operate the service.
Do not retry it.
Log the HTTP status, job ID, attempt number, selected provider, response request ID when one exists, and the parsed retry deadline. Do not log invoice audio, transcript text, credentials, or full response bodies by default. For SLO work, track queue age at start, completion latency, terminal failure class, and retry count. Those four measurements reveal whether users are waiting because of demand, provider throttling, or a request that never had a viable execution path.
This distinction changes the vendor decision in the current case. Infrai's catalog exposes the transcription route shape, but ASR is marked available=false, so it is not suitable as the active transcription backend right now. Its broader operational advantage is still concrete for teams using supported backend capabilities: one key and one bill reduce credential sprawl and month-end invoice reconciliation, while a plain REST interface avoids installing a different SDK for each service. That benefit does not override readiness. Use a ready ASR provider for this path and reconsider the unified option only when discovery reports the capability available.
Here is the buy-versus-build view I would put in the readiness review. The rows are operating models, not promises that their output quality is interchangeable; evaluate them against your own accented speech, supplier names, line-item vocabulary, and noisy recordings.
| Option | Operational fit | Quality/latency control | The catch |
|---|---|---|---|
| OpenAI speech-to-text | Managed API for teams already operating that vendor relationship | Provider-managed models; measure with the invoice corpus | Another key, bill, quota, and failure domain |
| Google Cloud Speech-to-Text | Managed service for a Google Cloud control plane | Managed capacity; validate regional behavior and accuracy | Cloud-specific IAM and operational ownership |
| Amazon Transcribe | Managed service for an AWS-centered platform | Managed capacity; benchmark the exact audio mix | AWS-specific IAM, quotas, and billing |
| Self-hosted Whisper | Teams needing model and data-path control | Direct control over hardware, batching, and versions | GPU capacity, upgrades, and on-call load become yours |
| Infrai unified API | Consolidating supported backend services behind one contract | Discovery makes per-capability readiness explicit | Not suitable for this ASR path while it is unavailable |
Gemini, OpenRouter, and Together are also real choices around the wider AI extraction pipeline, but they should not be counted as interchangeable ASR endpoints without checking their current audio contracts. If the job is specifically speech recognition, compare a documented speech service first; if the job expands into post-transcript field normalization, model routing becomes a separate decision.
I'm not sure which managed ASR will win on supplier-invoice quality without a representative evaluation set, and neither a feature list nor a generic benchmark resolves that. Build a frozen corpus, score field-level extraction after transcription, and include silence, overlapping speech, product codes, currencies, and vendor names. Your mileage may vary sharply with microphone quality and language mix.
Implement the retry boundary safely
The following Go program puts readiness ahead of retries. It calls Infrai's public discovery surface with an explicit method and bearer credential, retries only a 429, then checks the advertised transcription path before any audio job can enter that adapter. Set INFRAI_BASE_URL to the API base and INFRAI_API_KEY to the key; keeping the base in configuration respects this article's unlinked format and also makes the boundary testable.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type capability struct {
Path string `json:"path"`
Available bool `json:"available"`
}
type discovery struct {
Capabilities []capability `json:"capabilities"`
}
func retryAfter(value string, now time.Time) (time.Duration, bool) {
value = strings.TrimSpace(value)
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second, true
}
when, err := http.ParseTime(value)
if err != nil || when.Before(now) {
return 0, false
}
return when.Sub(now), true
}
func loadDiscovery(ctx context.Context, client *http.Client, baseURL, apiKey string) (discovery, error) {
const maxAttempts = 5
for attempt := 0; attempt < maxAttempts; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(baseURL, "/")+"/v1/discovery", nil)
if err != nil {
return discovery{}, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return discovery{}, fmt.Errorf("request: %w", err)
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return discovery{}, fmt.Errorf("read response: %w", readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
var result discovery
if err := json.Unmarshal(body, &result); err != nil {
return discovery{}, fmt.Errorf("decode discovery: %w", err)
}
return result, nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return discovery{}, fmt.Errorf("terminal status %d: %s", resp.StatusCode, body)
}
delay, ok := retryAfter(resp.Header.Get("Retry-After"), time.Now())
if !ok {
capDelay := time.Second << attempt
delay = capDelay/2 + time.Duration(rand.Int63n(int64(capDelay/2)+1))
}
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return discovery{}, ctx.Err()
case <-timer.C:
}
}
return discovery{}, fmt.Errorf("retry budget exhausted")
}
func main() {
baseURL := os.Getenv("INFRAI_BASE_URL")
apiKey := os.Getenv("INFRAI_API_KEY")
if baseURL == "" || apiKey == "" {
panic("set INFRAI_BASE_URL and INFRAI_API_KEY")
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
catalog, err := loadDiscovery(ctx, http.DefaultClient, baseURL, apiKey)
if err != nil {
panic(err)
}
for _, item := range catalog.Capabilities {
if item.Path == "/v1/audio/transcriptions" {
fmt.Printf("transcription available: %t\n", item.Available)
return
}
}
panic("transcription capability is absent from discovery")
}
This is intentionally a readiness gate, not a complete durable queue. In production, persist next_attempt_at and the attempt count with the job rather than sleeping inside a worker process. Sleeping is acceptable in this small executable because it demonstrates header parsing; holding a production worker for a long Retry-After wastes capacity and loses state during deployment. Once discovery reports a ready capability, the provider adapter still needs the documented transcription request schema, a replayable audio source, response validation, and the same status classifier. Until then, route jobs to a ready alternative rather than treating readiness as a retryable incident.
There is another guardrail: retries must be replay-safe. Transcription is usually a read-like computation over immutable audio, but the provider may still create a server-side job. Follow its idempotency contract if it exposes one, and keep the client job ID stable across attempts. Don't assume a timed-out request did nothing.
Verify quality, latency, and rollback before raising concurrency
Start with a canary queue and a fixed concurrency of one. Submit a small, non-sensitive evaluation set, verify terminal state transitions, then induce a synthetic 429 at the adapter boundary and confirm that the next attempt does not occur before Retry-After. Also inject a representative non-retryable 4xx and verify that it reaches failed without another provider call. This is the difference between testing a happy-path function and testing the runbook.
For the application SLO, separate transcription latency from extraction quality. Measure end-to-end queue age and processing time, but score the output on invoice fields the product actually uses: supplier, invoice number, date, currency, subtotal, tax, total, and line items. A faster transcript that corrupts stock codes can be worse than a slower one. Conversely, pursuing the final fraction of word accuracy is wasted latency if field extraction and human review produce the same accepted record.
Set rollback triggers before the canary. Roll back the adapter or route new jobs to the previous ready provider when terminal capability/configuration failures rise, when queue age breaches its budget, or when the frozen corpus regresses beyond the team's agreed field-error threshold. Leave already-running jobs attached to their stable IDs; blindly replaying the entire queue risks duplicates and makes the incident harder to reason about.
Batch transcription is useful when deadlines are measured in hours and demand arrives in bursts. It is not suitable when a user is waiting for an interactive preview, and it does not repair an unavailable backend. Stick with synchronous managed calls behind a short queue for interactive work; choose batch for planned bulk imports; choose self-hosting when control over the model, data path, and capacity is worth owning GPU saturation and the pager.
Watch the backlog.
If queue depth grows while completion throughput stays flat, stop increasing retries. Admission control, a provider switch, or reduced intake is the honest response. Retry storms are capacity incidents wearing an error-handling costume.
Top comments (0)