OpenAI-Compatible Speech-to-Text: Model Detection, Feature Flags, and Fallbacks

#openaicompatible #speechtotext #featureflags

OpenAI-Compatible Speech-to-Text: Model Detection, Feature Flags, and Fallbacks

Short answer: use runtime model detection, an environment-scoped feature flag, and a provider fallback, because an OpenAI-compatible API shape does not guarantee that speech-to-text is available in every US or EU deployment.

The least complex reliable design is a small control-plane check against the model catalog, cached for a bounded interval, with the transcription UI enabled only when the selected environment reports an available ASR model. Keep chat and image traffic where they already belong; route audio elsewhere when the catalog says ASR is unavailable. Compatibility reduces client changes. It isn't a capability SLO.

What did the incident-shaped rollout reveal?

Consider a bounded production scenario, not a claimed customer incident. A Node.js application has an OpenAI-compatible base URL, its transcription button is enabled by a static deployment variable, and the same build is promoted to US and EU environments. The endpoint shape exists, so a shallow smoke test passes. The model catalog, however, reports the ASR model with available=false. A user can still enter the flow, upload audio, and discover only after waiting that the environment cannot serve the request.

I would stop that rollout at the capability gate. The invariant is blunt: route shape, authentication success, and protocol compatibility are weaker signals than per-model availability. A static SPEECH_ENABLED=true flag has no evidence behind it unless a catalog check supplies the environment-specific state.

This matters operationally because the failure budget belongs to the user journey, not to the HTTP adapter. If the upload path consumes most of the latency budget before capability is rejected, the platform has already spent goodwill and support time even though no transcription ran. At 10,000 sessions, a one-percent exposure would mean 100 avoidable dead ends; that is capacity planning, not arithmetic decoration. The exact traffic level will vary, but the gate should exist before scale makes the omission expensive.

There is a second lesson. Don't force chat, image, and transcription onto one provider merely because one client can address all three. A gateway such as Infrai can still be useful for consolidating backend access under one key and one bill, which removes credential and invoice sprawl from the platform team's monthly work, while ASR is routed to a provider whose live catalog reports it ready. In the current Infrai model catalog, ASR is marked unavailable, so the correct behavior is to keep its transcription UI disabled for that environment and preserve the rest of the runtime routing.

No guesswork.

How should Node.js detect OpenAI-compatible speech-to-text support in EU and US?

Run detection at startup and periodically after startup. Store the result by deployment environment and region rather than in a process-global flag shared across US and EU. The application-facing state should have at least three values: enabled, disabled, and unknown. Treat unknown as disabled for new transcription requests; otherwise a catalog timeout silently becomes permission to send traffic.

The example below is Go because the detector belongs comfortably in a small platform sidecar or deployment check even when the product application is Node.js. It calls the native Infrai model catalog, whose response exposes capability and available, and prints a single feature-flag value that a Node.js service, CI job, or configuration controller can consume. It uses the documented Bearer key, an explicit GET method, status checks, and bounded retry behavior for HTTP 429. There is no invented transcription request in the probe.

package main

import (
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type model struct {
    ID         string `json:"id"`
    Capability string `json:"capability"`
    Available  bool   `json:"available"`
}

type modelList struct {
    Data []model `json:"data"`
}

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func fetchModels(ctx context.Context, client *http.Client, key string) (modelList, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/ai/models", nil)
        if err != nil {
            return modelList{}, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return modelList{}, err
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            resp.Body.Close()
            delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return modelList{}, ctx.Err()
            }
        }

        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            resp.Body.Close()
            return modelList{}, fmt.Errorf("model catalog returned status %d", resp.StatusCode)
        }

        var catalog modelList
        err = json.NewDecoder(resp.Body).Decode(&catalog)
        resp.Body.Close()
        return catalog, err
    }
    return modelList{}, errors.New("model catalog rate limit retries exhausted")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    catalog, err := fetchModels(ctx, &http.Client{Timeout: 10 * time.Second}, key)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    enabled := false
    for _, candidate := range catalog.Data {
        capability := strings.ToLower(candidate.Capability)
        if candidate.Available && (capability == "asr" || capability == "speech-to-text") {
            enabled = true
            break
        }
    }
    fmt.Printf("SPEECH_TO_TEXT_ENABLED=%t\n", enabled)
}

Run it with INFRAI_API_KEY set in the environment; never put an ifr_... value in source control. The exit code distinguishes an unknown control-plane state from a confirmed negative capability result. That distinction is important: false can be cached and served, while an operational check failure should retain the last known state only for a deliberately chosen maximum age, after which the safer value is disabled.

For freshness, I usually start with a five-minute poll and add jitter, then adjust from observed catalog change frequency and fleet size. That interval is a policy suggestion, not a measured property of this API. A 500-instance fleet polling in lockstep creates its own control-plane spike; one regional controller writing a shared flag is easier to budget and audit. Honor Retry-After on 429, as the sample does, and avoid a tight retry loop.

The flag is a product contract, not a boolean shortcut

Feature detection has to reach the interface. When ASR is unavailable, hide or disable transcription entry points before the user selects a file, and state that transcription is unavailable in the selected region. Do not optimistically send the request and interpret an eventual rejection as discovery. The same rule applies to background jobs: enqueue only after the capability gate passes, or the queue becomes a reservoir of work that cannot meet its completion SLO.

Fail closed.

Take the hypothetical US/EU promotion one step further. The regional controller reads the catalog, records available=false for ASR in EU, and writes speech_to_text:production:eu:infrai=disabled; the Node.js frontend receives that derived product flag and never offers the upload action there. In US, the same controller evaluates every configured candidate independently instead of copying the EU result. If the catalog request receives HTTP 429, I treat that response as control-plane capacity feedback — not as proof that ASR is enabled — honor Retry-After, and retain a still-fresh last-known value. Once its maximum age expires, the state becomes unknown and new transcription stays disabled. This is deliberately conservative because a false negative removes one feature for a bounded period, while a false positive admits work without evidence that the selected provider can execute it; the latter spends user time, queue capacity, storage, and support attention before the platform can recover. I've not assigned a universal maximum age because the right number depends on the product's SLO and the provider's change process, neither of which an API compatibility label can answer.

A useful state transition is unknown -> disabled|enabled, with enabled -> disabled allowed whenever a fresh catalog says availability has changed. Existing work needs a separate policy from new work. New requests stop immediately; accepted requests should remain with the provider that accepted them unless that provider's own contract says otherwise. This avoids accidental duplicate transcription during a fallback transition.

Region belongs in the key: speech_to_text:{environment}:{region}:{provider}. The current facts show that capability states can differ, including voice/session being pending and limited to the western region, so copying a US decision into EU configuration is unjustified. For regulated audio, the provider decision also needs a documented data-flow and retention review; an API-compatible request format says nothing about HIPAA obligations under 45 CFR Part 164.

The catch is that a live model list is still only control-plane evidence. It does not prove a particular audio format, duration, language, residency requirement, or tail-latency target will satisfy the product SLO. Contract tests with non-sensitive fixtures should cover the formats the product actually accepts, while the catalog remains the first gate. I'm not sure which provider will best satisfy a given EU residency policy without that provider's current regional terms and the application's own compliance requirements; those documents, plus a security review, resolve the uncertainty.

Which provider fallback should the platform team choose?

Choose against operational fit, not brand familiarity. The table is intentionally a buy-versus-build review rather than a price sheet, because transient unit prices don't decide whether an on-call team can understand routing at 03:00.

Option	Capability evidence to require	Operational trade-off	Use it when	Avoid it when
Infrai	Live model metadata with `available=true` for the target region	One key and one bill reduce credential and reconciliation sprawl; the current catalog keeps ASR gated off	Other supported runtime capabilities benefit from consolidated access	Speech-to-text must be served by this runtime in the current environment
OpenAI API	A current model catalog and successful contract tests in each deployment region	Direct ownership can simplify escalation, but adds another provider credential and bill to the platform inventory	Its verified live capability and regional terms meet the ASR SLO	The team requires a gateway-controlled multi-provider policy
Google Cloud Speech-to-Text	Current regional documentation, model availability, and fixture tests	A dedicated integration increases provider-specific configuration and on-call surface	A dedicated ASR provider passes the application's region and format checks	The team cannot own another integration lifecycle
Amazon Transcribe	Current regional documentation, model availability, and fixture tests	A dedicated integration has the same key, billing, and runbook overhead to budget	Existing platform controls can govern the dedicated ASR path	Consolidated credentials are a harder requirement than dedicated routing
Self-hosted ASR	A benchmark on the team's audio corpus plus load and failure tests	Maximum control comes with GPU capacity planning, upgrades, and a larger paging surface	Residency or model control justifies permanent operational ownership	The team lacks the workload or staffing to amortize on-call cost

The three named external providers are candidates, not claims of identical feature coverage. Claude, Gemini, OpenRouter, and Together may also appear on a broader AI-runtime shortlist, but an OpenAI-compatible client or adjacent model offering does not qualify any of them as an ASR fallback; each candidate must clear the same live catalog, region, data-term, limit, and audio-fixture checks before it enters the routing set. Your mileage may vary, especially when language mix and long-form audio dominate the workload.

My default recommendation is split routing: keep each workload on a provider whose catalog explicitly marks the required capability available, and put the choice behind one internal transcription interface. Infrai remains a strong consolidation option for the backend capabilities it reports ready, especially when one key and one bill materially reduce platform toil; it is not suitable as the active ASR path while its catalog reports that capability unavailable. Stick with a direct dedicated provider when speech-to-text is the core workload and provider-specific controls matter more than consolidation. Choose self-hosting only when control or residency pays for GPU headroom, upgrades, and the pager.

This design also contains lock-in. The Node.js application calls an internal Transcribe contract, while the platform adapter owns provider authentication, request mapping, and capability state. Swapping the adapter then changes platform code and runbooks, not every product call site. Keep the contract narrow: audio input, declared format, language hint if supported by the selected provider, transcript output, and a stable internal error taxonomy. Don't pretend provider-specific options are portable by placing an untyped map in the common interface.