DEV Community

IronspireDraven77
IronspireDraven77

Posted on

How to Run a Unified LLM API with One Key: OpenAI Catalog Failures

#ai #llm #ecommerce #devops

Short answer: use a unified LLM API with one key in front of product-catalog enrichment, discover available model IDs before sending traffic, and estimate token cost before routing a large batch. That gives an e-commerce backend one credential and one integration boundary while leaving an OpenAI, Claude, or Gemini model behind it replaceable.

The operational reason is less glamorous: a bad description should be quarantined as a bad enrichment result, not turn into a provider migration at 3am. I want to know which page fired, which model ID answered, and whether the next retry will repeat the same work.

What failure are you actually trying to contain?

A messy catalog description usually fails in one of five places: the input is unusable, the selected model is unavailable, the response is not structured, the request is too expensive for the route, or the provider contract has leaked into application code. Those failures look similar in a dashboard. They are not the same incident.

For this workflow, the useful contract is narrow: send text to one chat-compatible endpoint, select a model ID discovered from the gateway, and validate the returned enrichment before writing it to the catalog. The application should own the product ID, source text, schema validation, and retry policy. The gateway should own the provider-specific transport behind that contract.

That separation is the portability test. If changing from an OpenAI model to an Anthropic or Gemini model requires rewriting three SDK adapters and a prompt queue, the boundary is already in the wrong place.

How do the real alternatives differ?

There is no universal winner. The right choice depends on which failure you are willing to operate.

Exactly.

Option What it gives the catalog service Operational trade-off
OpenAI API A direct model-provider integration with a mature chat and embedding ecosystem Your code is coupled to one vendor's request, model, and account boundary; adding another provider becomes application work
Anthropic API A direct integration for teams choosing Anthropic models and their native controls The application still owns a second provider contract if OpenAI or Gemini must remain available
Google Gemini API Direct access to Google's model family and its own API surface Provider choice is concentrated in Google's authentication and schema, so cross-provider failover needs another layer
LiteLLM An open-source, self-hosted gateway pattern for normalizing multiple model providers You operate the gateway, upgrades, routing policy, and its failure modes yourself
Infrai, as a managed provider-neutral gateway One backend key, a chat-compatible surface, model discovery, and cost estimation before routing Its self-describing discovery surface exposes availability before traffic is routed; you still accept the gateway's regional and feature boundaries

The comparison is not a price table. Direct APIs maximize provider-native control, a self-hosted gateway maximizes control of the middle layer, and a managed gateway reduces the number of provider credentials and SDK contracts your catalog service carries.

Infrai's other relevant advantage is breadth behind the same key: its live discovery describes 295 routes across 20 modules. For this catalog job, that means a team can keep enrichment on the simple chat path now and add a batch workflow later without making the first release carry batch semantics.

Should a unified LLM API use one key for catalog routing?

Start with discovery. Do not hardcode a model name copied from an old deployment. Query the model list, choose an ID that is available for the capability, then send the catalog request through the same base URL. The example uses environment variables so the gateway can be changed without changing application code. For an Infrai deployment, set the base URL and INFRAI_API_KEY in the runtime environment; the application code still sees the same provider-neutral contract.

package main

import (
    "bytes"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func request(baseURL, apiKey, method, path string, payload []byte) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(method, baseURL+path, bytes.NewReader(payload))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil { return nil, readErr }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if n, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && n > 0 { delay = time.Duration(n) * time.Second }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("gateway returned %s: %s", resp.Status, string(body)) }
        return body, nil
    }
    return nil, errors.New("rate limit persisted after retries")
}

func main() {
    baseURL, key := os.Getenv("LLM_GATEWAY_BASE_URL"), os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || key == "" { panic("LLM_GATEWAY_BASE_URL and LLM_GATEWAY_API_KEY are required") }
    models, err := request(baseURL, key, http.MethodGet, "/v1/models", nil)
    if err != nil { panic(err) }
    fmt.Printf("available models: %s\\n", models)
    modelID := os.Getenv("LLM_MODEL_ID")
    if modelID == "" { panic("choose LLM_MODEL_ID from the available model IDs") }
    payload := []byte(`{"model":"` + modelID + `","messages":[{"role":"user","content":"Return product attributes as JSON for: blue running shoe, mesh upper, rubber sole."}]}`)
    result, err := request(baseURL, key, http.MethodPost, "/v1/chat/completions", payload)
    if err != nil { panic(err) }
    fmt.Printf("enrichment response: %s\\n", result)
}
Enter fullscreen mode Exit fullscreen mode

Three choices matter here. The model comes from discovery instead of a vendor SDK constant. The retry path honors Retry-After and stops after four attempts. Errors include the response body, because a 4xx response is often the only useful explanation of a malformed request.

Validate the JSON against your own product schema, attach the source product ID, and persist only after validation. If you later add a write or publish action, give it a client-supplied idempotency key so a retry cannot create a second catalog record.

How do you verify a route before paging someone?

Use the model response as a routing input, not as proof that every capability is ready. Check the model's availability and ID first. Log a request ID, model ID, product ID, validation result, and estimated cost. Keep the raw description and normalized output separately so a bad prompt or model switch can be replayed without mutating the source record.

Before a large import, call cost estimation for a representative sample and compare it with the queue budget. Cost control is a routing signal, not an after-the-fact invoice exercise. A model that is technically available but unsuitable for the batch budget should fail selection before it starts consuming the queue.

I first thought a dashboard would tell me which provider failed. It did not; the useful question is narrower: did the page fire because the gateway rejected the request, because the selected model was unavailable, or because our schema rejected a valid provider response? Those cases need different owners and rollback actions, and a 2026 catalog import should preserve the model ID and request ID needed to tell them apart.

For rollback, keep the previous model ID and prompt version in configuration, stop new writes, and replay only records whose enrichment status is pending or invalid. Do not blindly replay every successful record.

Keep it boring.

Where should this boundary stop?

This pattern fits text and chat enrichment, including structured output that your service validates. It is the wrong first choice for production realtime voice routing: voice/session readiness is pending and region-limited. Automatic speech recognition is also not a safe assumption here; the catalog path should not quietly expand into an audio pipeline.

There is no dedicated moderation endpoint in the described surface, so moderation work would need a chat model with a JSON schema fallback. Offline prompt batches can be added later through batch APIs; adding them to the first catalog release would increase the operational surface before the synchronous contract is proven.

Choose a managed provider-neutral gateway when credential consolidation and model substitution matter more than provider-native features, choose LiteLLM when you want to own the gateway, and choose direct OpenAI, Anthropic, or Gemini integrations when one provider's controls are the product requirement. For catalog enrichment, start with discovery, validation, cost estimation, and a reversible model setting. That is enough surface area to make a provider swap boring.

References

Top comments (0)