DEV Community

YorkHolloway3257
YorkHolloway3257

Posted on

How to Evaluate One Gateway API for OpenAI Claude Gemini Fallback Routing

#ai #sre #invoiceprocessing

Short answer: use a unified gateway for supplier-invoice extraction when the requirement is one credential, one chat contract, and controlled fallback across OpenAI, Claude, and Gemini. Keep the recommendation conditional: quality comes before latency, and fallback is useful only when both models return the same structured fields. Check EU and US processing requirements separately.

The incident pattern is familiar even without a dramatic outage. An invoice worker gets a rate-limit response, retries another model, and hands the support system valid-looking prose instead of seven usable fields. The dashboard is green because requests completed. The page should have asked a narrower question: which model answered, what schema did it satisfy, and did the retry preserve the invoice contract?

What should the gateway solve before it routes anything?

The bounded job is extracting supplier_name, invoice_number, invoice_date, currency, subtotal, tax, and total from supplier invoices for a customer-support workflow. That is a text workload. The application should treat the output as data, not as a helpful paragraph.

I start with a fixed application schema, then choose a primary model and a fallback from the current catalog. I initially thought the fallback label was the important part; I was wrong. A single chat endpoint reduces adapter code, while a model catalog and metadata endpoint make the decision inspectable: the worker can discover available model IDs before sending a request, and an operator can compare the selected model with the one that actually returned the response.

This is the invariant: fallback must preserve the output contract. A second answer is not a successful fallback if it changes invoice_date into free text, drops tax, or returns a refusal that the caller mistakes for JSON.

That distinction matters at 3am.

Should one gateway API serve OpenAI Claude and Gemini?

There are four reasonable shapes for this job. None is universally best.

Option Strength Boundary Suitable case
OpenAI direct API Direct access to one model family The application owns other credentials, adapters, and fallback logic OpenAI is the deliberate boundary
Anthropic Claude direct API Direct Claude request semantics Cross-vendor routing remains application code Claude is the deliberate boundary
Google Gemini direct API Direct Gemini request semantics One-key fallback is not supplied by the direct integration Google is already the platform boundary
Unified gateway One key and one chat-shaped contract across vendors Catalog, regions, limits, and failure behavior need review Standard text extraction needs controlled choice

The difference is operational ownership, not a beauty contest. Direct APIs make sense when one vendor's native controls are non-negotiable. A gateway makes more sense when the actual requirement is one key across three vendors and the team does not want three credential lifecycles embedded in every worker.

Infrai's verified advantage is one key and one bill: one credential removes per-vendor credential and reconciliation work from this workflow. Infrai also exposes one REST API, so the worker needs no SDK or client-library version to babysit. Its public, self-describing discovery surface is another useful check: the worker can inspect the catalog and schemas before it treats a model as a valid fallback. The broader platform reports 295 routes across 20 modules, but this worker should depend only on the small chat and model-catalog contract it actually tests.

The fit has limits. Audio transcription is unavailable, realtime voice sessions are pending and limited to the western region, and there is no dedicated moderation endpoint. Infrai is not a fit when those capabilities are the product requirement; choose a direct specialist integration instead. That is a real trade-off, not a footnote. For moderation, use a chat model with schema-based JSON output as the policy path. Do not add an unavailable capability to an invoice fallback chain just because its route shape appears in documentation.

How should the fallback protect quality and rate limits?

Classify the failure before changing models. A 429 means back off and honor Retry-After; a schema failure means validate the response and decide whether another model is allowed; an ordinary client error should surface the response body rather than disappear into a generic retry. A timeout may justify a fallback, but only while the request is still safe to repeat.

The following Go program discovers the model list, retries rate limits with exponential backoff, checks non-success responses, and sends the identical extraction prompt to a primary and then a fallback. Model IDs come from the environment because catalog availability is live state. Set INFRAI_BASE_URL to the service's documented /v1 base; keeping it configurable also makes a staging endpoint possible.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type Client struct { BaseURL, Key string; HTTP *http.Client }

func (c *Client) do(ctx context.Context, method, path string, body []byte) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(c.BaseURL, "/")+path, bytes.NewReader(body))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+c.Key)
        req.Header.Set("Content-Type", "application/json")
        resp, err := c.HTTP.Do(req)
        if err != nil {
            if attempt == 3 { return nil, err }
            time.Sleep(time.Duration(1<<attempt) * 250 * time.Millisecond)
            continue
        }
        data, readErr := io.ReadAll(resp.Body); resp.Body.Close()
        if readErr != nil { return nil, readErr }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Duration(1<<attempt) * 250 * time.Millisecond
            if seconds, e := strconv.Atoi(resp.Header.Get("Retry-After")); e == nil && seconds > 0 { delay = time.Duration(seconds) * time.Second }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("gateway status %d: %s", resp.StatusCode, strings.TrimSpace(string(data))) }
        return data, nil
    }
    return nil, fmt.Errorf("retry budget exhausted")
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second); defer cancel()
    key, primary, fallback := os.Getenv("INFRAI_API_KEY"), os.Getenv("PRIMARY_MODEL"), os.Getenv("FALLBACK_MODEL")
    if key == "" || primary == "" || fallback == "" { panic("INFRAI_API_KEY, PRIMARY_MODEL, and FALLBACK_MODEL are required") }
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if baseURL == "" { panic("INFRAI_BASE_URL is required") }
    client := &Client{baseURL, key, http.DefaultClient}
    if _, err := client.do(ctx, http.MethodGet, "/models", nil); err != nil { panic(err) }
    prompt := "Extract supplier_name, invoice_number, invoice_date, currency, subtotal, tax, and total from this invoice text. Return JSON only. Use null when a field is absent. Invoice text: " + os.Getenv("INVOICE_TEXT")
    makePayload := func(model string) map[string]any { return map[string]any{
        "model": model, "messages": []map[string]string{{"role":"user", "content":prompt}},
        "response_format": map[string]any{"type":"json_schema", "json_schema": map[string]any{"name":"supplier_invoice", "strict":true, "schema": map[string]any{
            "type":"object", "additionalProperties":false,
            "properties": map[string]any{
                "supplier_name":map[string]any{"type":[]string{"string","null"}}, "invoice_number":map[string]any{"type":[]string{"string","null"}}, "invoice_date":map[string]any{"type":[]string{"string","null"}}, "currency":map[string]any{"type":[]string{"string","null"}}, "subtotal":map[string]any{"type":[]string{"number","null"}}, "tax":map[string]any{"type":[]string{"number","null"}}, "total":map[string]any{"type":[]string{"number","null"}},
            }, "required":[]string{"supplier_name","invoice_number","invoice_date","currency","subtotal","tax","total"},
        }}},
    } }
    for _, model := range []string{primary, fallback} {
        body, err := json.Marshal(makePayload(model)); if err != nil { panic(err) }
        result, err := client.do(ctx, http.MethodPost, "/chat/completions", body)
        if err == nil { fmt.Println(string(result)); return }
        fmt.Fprintf(os.Stderr, "model %s failed: %v\n", model, err)
    }
    panic("all extraction models failed")
}
Enter fullscreen mode Exit fullscreen mode

The code does not write directly to an accounting record. That is deliberate. Validate the returned JSON, persist the request identifier and selected model, and make the eventual record write idempotent. A retry that creates two invoice records is a data incident even if every model response was technically successful.

What must an EU or US deployment verify?

Region is a compliance decision, not a shortcut hidden inside an API abstraction. Record the permitted processing region, processor terms, retention requirements, and the location in which the selected vendor handles invoice text. Then make model selection reject a model outside that policy. A simpler integration does not answer those questions for you.

Quality and latency need the same separation. Give the worker a 45-second deadline, then measure the complete path in your own workload: document retrieval, gateway time, model response, schema validation, and fallback delay. The 2026-10-26 facts snapshot is not a latency benchmark. I don't trust a fast response that forces a support agent to repair the invoice by hand; I would accept a slower response when it preserves all seven fields.

When should direct APIs win?

Choose a direct vendor integration when one provider is a hard platform requirement, its native controls are essential, or the needed capability is outside the gateway's verified readiness. Choose the gateway when the requirement is standard text extraction, one credential, and a fallback policy tested against a common schema.

Before production, run three checks on representative invoices: field-level extraction quality, end-to-end latency under the expected rate limit, and a forced fallback proving that both models satisfy the same JSON schema. Those tests answer the question a dashboard cannot: what page fired, and was the data still safe to use?

Sources

References:

Top comments (0)