DEV Community

onyxcross5743
onyxcross5743

Posted on

Speech-to-Text API Rate Limits: Shared Retry-After Admission for CRM Calls

#speechtotext #backend #go

Short answer: When a speech-to-text API returns a 429 rate limit, treat its Retry-After header as shared queue state, reserve batch transcription for delay-tolerant calls, and commit each transcript-to-CRM transition under an idempotency key.

The dangerous response to a speech-to-text API 429 is to make every logistics sales-call worker sleep and retry independently. The shared control preserves the provider's timing signal without turning a temporary rate limit into duplicate follow-ups, missing quotes, or an audit trail nobody can reconcile.

This is an accounting problem wearing an audio badge. A recording may be uploaded once, submitted more than once, transcribed once, and mapped into several CRM actions; therefore, “exactly once” cannot mean exactly one network request. It has to mean that repeated delivery converges on one observable business result. The distinction matters when a call produces a promised rate sheet, a pickup-window change, and a next-call date, because replaying all three actions is materially worse than replaying a read.

How should a speech-to-text API queue handle 429 Retry-After headers?

Parse Retry-After in both forms defined for HTTP: either a non-negative delay in seconds or an HTTP date. Then store the resulting not_before time against the throttling scope shared by workers — for example, an account and model class — rather than merely sleeping the worker that received the response. RFC 9110 defines the field's two forms, while RFC 6585 defines 429 Too Many Requests and says the response may include Retry-After; neither specification tells a client that a retry is automatically safe, so replay safety remains an application obligation.

Pause admission, not progress.

A worker that already owns a completed transcript should be allowed to validate and commit it while new submissions wait. Conversely, a process-local timer is insufficient when ten replicas draw from the same quota: nine replicas can't see the first one's cooldown and can continue the burst. The durable record should contain the request key, audio object identity, attempt count, state, next eligible time, and the exact response class that caused the transition. Record the raw header for audit, but schedule from a parsed instant; clocks and malformed inputs need an explicit policy.

When the header is absent or unusable, capped exponential backoff with jitter is a defensible fallback, not a substitute for queue-level control. I'm not sure any fixed cap is universally correct: the right value depends on the provider's quota window, the age at which a sales-call summary stops being operationally useful, and the queue's oldest-job latency. Resolve that uncertainty with documented service limits and production telemetry. A deadline should ultimately move stale work to a reviewable terminal state rather than retry forever.

The ledger between audio and CRM state

The queue entry is only half the design. Give each uploaded recording a stable business key, such as a tenant-scoped call ID plus the audio content digest, and carry that key through submission, transcript validation, and CRM mutation. Do not use a random retry ID: randomness distinguishes attempts when the requirement is to recognize sameness.

Model the workflow as monotonic transitions: received, eligible, submitted, transcribed, validated, and committed, with retry_wait returning only to eligible. Each transition appends an audit event containing the prior state, new state, attempt number, timestamp, and reason. The current row supports fast scheduling; the append-only events explain how it got there. If the process dies after the remote service accepts audio but before the local acknowledgement is stored, reconciliation searches by the stable request key or checks the recorded operation before allowing another business-side commit.

Exactly-once delivery is the wrong promise.

Exactly-once effect is the useful target: at-least-once queue delivery plus an atomic compare-and-set on the business key. For CRM actions, persist the validated action set and an outbox record in the same database transaction, then let a separate dispatcher deliver that outbox. A uniqueness constraint on (tenant_id, call_id, action_version) turns duplicate workers into a harmless conflict instead of duplicate customer contact. Retain the event history according to the organization's audit and privacy policy; call audio and transcripts can contain personal or commercially sensitive information, so indefinite retention should never be the accidental default.

A Go scheduler with header-aware backoff

The following core deliberately leaves persistence behind an interface. In production, DeferScope and job state must share durable storage visible to every worker; an in-memory map would make the central guarantee disappear at process restart. The transport receives a stable idempotency key, while the scheduler honors a valid server instruction before calculating fallback delay.

package transcription

import (
    "context"
    "errors"
    "math/rand/v2"
    "net/http"
    "strconv"
    "strings"
    "time"
)

var ErrRateLimited = errors.New("speech service rate limited")

type Job struct {
    ID             string
    TenantID       string
    AudioObjectKey string
    Attempt        int
}

type Queue interface {
    Ack(ctx context.Context, jobID string) error
    RetryAt(ctx context.Context, jobID string, at time.Time, reason string) error
    DeferScope(ctx context.Context, tenantID string, until time.Time) error
}

type Transcriber interface {
    Submit(ctx context.Context, audioObjectKey, idempotencyKey string) (*http.Response, error)
}

func retryAfter(value string, now time.Time) (time.Time, bool) {
    value = strings.TrimSpace(value)
    if seconds, err := strconv.ParseInt(value, 10, 64); err == nil && seconds >= 0 {
        return now.Add(time.Duration(seconds) * time.Second), true
    }
    if at, err := http.ParseTime(value); err == nil && at.After(now) {
        return at, true
    }
    return time.Time{}, false
}

func fallback(attempt int, now time.Time) time.Time {
    if attempt > 6 {
        attempt = 6
    }
    ceiling := time.Second * time.Duration(1<<attempt)
    return now.Add(time.Duration(rand.Int64N(int64(ceiling) + 1)))
}

func Run(ctx context.Context, q Queue, api Transcriber, job Job, now time.Time) error {
    response, err := api.Submit(ctx, job.AudioObjectKey, job.TenantID+":"+job.ID)
    if err != nil {
        return err
    }
    defer response.Body.Close()

    if response.StatusCode == http.StatusTooManyRequests {
        at, ok := retryAfter(response.Header.Get("Retry-After"), now)
        if !ok {
            at = fallback(job.Attempt, now)
        }
        if err := q.DeferScope(ctx, job.TenantID, at); err != nil {
            return err
        }
        return q.RetryAt(ctx, job.ID, at, ErrRateLimited.Error())
    }

    if response.StatusCode < 200 || response.StatusCode >= 300 {
        return errors.New("unexpected transcription response")
    }
    return q.Ack(ctx, job.ID)
}
Enter fullscreen mode Exit fullscreen mode

Notice what the code does not do: it doesn't hold a worker goroutine through the cooldown, and it doesn't infer that every non-success response is retryable. The queue owns time. A lease protects active work, but its duration should not encode the backoff interval; otherwise a lease timeout can make a second worker submit the same audio while the first attempt is merely waiting.

Testing needs more than a happy-path unit test. Use a fake clock to cover Retry-After: 17, a future HTTP date, a past date, missing input, and concurrent 429 responses that race to extend one scope deadline. Verify the invariant that the deadline never moves backward. Then fault-inject a crash after submission, after transcript persistence, and between the CRM row and outbox dispatch; every replay must produce one action version and a complete transition history. Property tests are particularly useful for the parser and state machine because combinations of attempts, clocks, and duplicate delivery are where tidy examples stop being representative.

When should batch transcription replace immediate retries?

Batching is appropriate when completion latency is flexible, recordings are already immutable in object storage, and the provider exposes a documented asynchronous batch mechanism. It can smooth admission and reduce connection churn, but it does not erase request limits, idempotency, or reconciliation. A batch needs a manifest with one stable key per recording; on partial completion, resubmit only entries whose state remains uncommitted, never the whole manifest by habit.

The catch is operational delay. Batch transcription is not suitable when a dispatcher must update a delivery exception during the call or when the CRM action has a deadline shorter than the batch completion window. Keep an immediate queue for urgent calls and place ordinary post-call summaries in a controlled batch lane when the service contract supports it. If no documented batch contract exists, stick with bounded single-record submissions rather than inventing one by sending a larger audio payload.

Use a small decision table in the runbook, because the choice should survive personnel changes:

Constraint Immediate queued request Asynchronous batch
Action needed within minutes Better fit, with shared admission control Poor fit unless completion bounds satisfy the deadline
Partial replay One recording per ledger entry Requires per-item manifest state
Quota response Honor shared Retry-After deadline Honor the batch service's documented limits
Audit question Trace one request key Trace manifest and item keys

Cost belongs in that same decision record, but published unit prices alone are not enough. Measure billable audio duration, storage retention, failed-attempt treatment, queue age, worker occupancy, and the engineering cost of reconciliation. Your mileage may vary — especially when call duration has a long tail — so a representative replay corpus is more useful than a single average call.

How can a team roll out the control loop without losing evidence?

Start in observe-only mode: parse Retry-After, calculate the proposed scope deadline, and compare it with actual queue behavior without changing admission. Next, enable shared deferral for one tenant cohort while retaining the old scheduler as a rollback path. Track the rate of 429 responses, oldest eligible-job age, attempts per committed transcript, scope-cooldown duration, duplicate-key conflicts, and time from audio receipt to CRM commit. None of those metrics alone is a success criterion; the reconciliation invariant is zero distinct committed action versions for the same business key.

Then introduce the outbox boundary and deliberately replay historical jobs in a non-production CRM target. Confirm that duplicate delivery changes audit counters but not business state. Only after that invariant holds should batching become a second admission lane. This order is slower than adding a sleep call, yet it gives operators a trail they can inspect when an account manager asks why a promised follow-up appeared late.

The final design rule is compact: obey the server's timing signal, coordinate it across workers, and make every downstream effect replay-safe. Backoff reduces pressure. The ledger preserves correctness.

References

Top comments (0)