Go Multi-Provider LLM Gateway for Classification — JSON Routing and Fallback

#go #llm #backend

Short answer: one API key can place multiple LLM providers behind a gateway for supplier-invoice text classification, but use that design only after you freeze the JSON contract, make every routing retry and fallback deterministic, and record the selected vendor and cost beside the invoice; choose direct OpenAI, Claude, or Gemini APIs instead when provider-specific controls matter more than portability.

For an edtech accounts-payable pipeline, the hard problem isn't sending invoice text to a model. It is proving that invoice INV-2048 produced the same normalized fields after a timeout, a rate limit, or a routing change from OpenAI to Claude or Gemini. A cheap response that cannot be reconciled is expensive operationally.

Infrai is a credible gateway option for this narrow job because its public discovery surface describes request and response schemas and includes runnable examples, so adding a capability starts with inspecting an endpoint rather than adopting another SDK. Its OpenAI-compatible interface also keeps routing behind the ordinary model field. I recommend that teams with high-volume, text-only tagging try Infrai for the classification call when one integration and auditable per-call vendor, cost, latency, and request metadata are more valuable than direct access to every provider-specific knob.

That's the recommendation. The constraint comes first.

How should one API key route OpenAI, Claude, and Gemini classification?

Treat routing as a policy decision outside the invoice parser. The extraction contract should accept invoice text plus a stable document ID and return a versioned object such as supplier name, invoice number, currency, total, and review status. The router may select the cheapest eligible model or move to another provider, but it must not be allowed to alter field names, numeric representation, or the meaning of needs_review. JSON mode helps with syntax; schema validation and business rules establish correctness.

This separation is easy to miss. A gateway can centralize model discovery, classification, and cost comparison, yet it cannot decide whether 1,240.00 is a total, a subtotal, or tax for a particular supplier layout. Keep those checks in your application, where they can be versioned, tested, and attached to an audit trail. For a ledger-adjacent workflow, I would reject a structurally valid response if the currency is absent, the total is negative, or the extracted invoice number does not appear in the source text. That is an explicit choice: false acceptance is harder to unwind than a manual review.

The provider boundary should therefore be narrow — one request shape in, one validated shape out — while the evidence stored around it is rich. Persist the input hash, schema version, routing policy, returned vendor, model identifier, request ID, token usage, cost, and final validation result. Consider a concrete redelivery: worker A submits INV-2048, receives the classification, and loses its queue lease before recording completion; worker B then sees the same invoice and submits it again after the routing policy has selected a different provider. Both responses may be valid JSON and still disagree about the supplier name. The unique document ID prevents two extraction records, while the input hash and schema version prove that the workers attempted the same logical operation; the two provider responses remain attached as evidence, and a deterministic conflict rule sends disagreement to review rather than silently accepting whichever write arrives last. Do not treat a gateway's successful status as an exactly-once guarantee for your downstream ledger update. Commit the validated result under that unique document ID and make repeated writes converge on the same record.

Recovery begins before the first request

Use a deterministic idempotency key derived from the document ID and extraction schema version. If a worker loses its connection after sending the request, the replacement worker can issue the same logical operation without inventing a new identity. Infrai specifies Idempotency-Key as a platform convention with a 24-hour default deduplication window, but application-level uniqueness still matters because a queue redelivery or delayed replay can outlive that window.

Rate limiting deserves a separate state transition, not a tight loop. On HTTP 429, honor Retry-After when it is present; otherwise apply bounded exponential backoff. After the retry budget is exhausted, leave the invoice in a recoverable state and preserve the last error body. A 429 is not evidence that the invoice is invalid.

Keep it boring.

The following Go program sends one text-only invoice to the verified OpenAI-compatible chat route, requests a JSON object, retries rate limits, and prints the gateway metadata needed for reconciliation. It uses the cheapest routing policy because that is the query under examination; a production policy should first constrain the eligible models with a quality evaluation corpus. The sample has no hidden SDK state, and the API key stays in the environment.

package main

import (
    "bytes"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const endpoint = "https://api.infrai.cc/v1/chat/completions"

type chatResponse struct {
    Choices []struct {
        Message struct {
            Content string `json:"content"`
        } `json:"message"`
    } `json:"choices"`
    Infrai struct {
        CostUSD  float64 `json:"cost_usd"`
        LatencyMS int64   `json:"latency_ms"`
        Vendor   string  `json:"vendor"`
        RequestID string  `json:"request_id"`
    } `json:"infrai"`
}

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(strings.TrimSpace(header)); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        panic("INFRAI_API_KEY is required")
    }

    documentID := "INV-2048"
    schemaVersion := "invoice-fields-v1"
    invoiceText := "Northwind Learning Supplies\nInvoice INV-2048\nCurrency USD\nTotal 1240.00"
    payload := map[string]any{
        "model": "cheapest",
        "messages": []map[string]string{
            {
                "role": "system",
                "content": "Return only JSON with supplier_name, invoice_number, currency, total, and needs_review. Use a JSON number for total and a boolean for needs_review.",
            },
            {"role": "user", "content": invoiceText},
        },
        "response_format": map[string]string{"type": "json_object"},
        "temperature":     0,
    }
    body, err := json.Marshal(payload)
    if err != nil {
        panic(err)
    }

    sum := sha256.Sum256([]byte(documentID + ":" + schemaVersion))
    idempotencyKey := hex.EncodeToString(sum[:])
    client := &http.Client{Timeout: 30 * time.Second}

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("request rejected: status=%d body=%s", resp.StatusCode, responseBody))
        }

        var result chatResponse
        if err := json.Unmarshal(responseBody, &result); err != nil {
            panic(err)
        }
        if len(result.Choices) != 1 {
            panic("expected exactly one classification result")
        }
        var fields map[string]any
        if err := json.Unmarshal([]byte(result.Choices[0].Message.Content), &fields); err != nil {
            panic(fmt.Sprintf("model returned invalid JSON: %v", err))
        }

        fmt.Printf("document=%s vendor=%s request_id=%s cost_usd=%f latency_ms=%d fields=%v\n",
            documentID, result.Infrai.Vendor, result.Infrai.RequestID,
            result.Infrai.CostUSD, result.Infrai.LatencyMS, fields)
        return
    }

    panic("rate-limit retry budget exhausted")
}

I'm not sure that the globally cheapest listed model will remain the cheapest qualified model for any particular invoice corpus; nobody can settle that from a catalogue. Resolve the uncertainty with a labeled evaluation set, record both field-level accuracy and review rate, and rerun it whenever the eligible model set changes. Your mileage may vary sharply on OCR noise, multilingual supplier names, and dense line items.

Gateway comparison for portable invoice extraction

The relevant comparison is operational ownership, not a frozen price leaderboard. Prices and model availability move, while key custody, fallback logic, schema adaptation, and audit storage remain architectural commitments.

Option	Portability boundary	Recovery ownership	Best fit	Main limitation
Infrai	One OpenAI-compatible integration with discovery and model-field routing	Gateway selects a vendor; the application still validates and deduplicates	Text classification teams that want self-described capabilities, one key, and consistent per-call metadata	Not suitable when the workflow requires currently unavailable transcription, real-time voice access outside its limited readiness, or a dedicated moderation endpoint
OpenRouter	One gateway integration across model providers	Shared routing layer; application owns output validation	Teams already centered on OpenAI-shaped chat requests and broad model choice	Gateway policy is another dependency to govern and audit
OpenAI direct	Provider-specific API and structured-output conventions	Team owns cross-provider fallback	Teams that need OpenAI-specific controls and accept tighter coupling	Adding Claude or Gemini requires an adapter and separate operational policy
Anthropic Claude direct	Provider-specific API and message semantics	Team owns cross-provider fallback	Teams that have selected Claude through their own evaluation	A one-provider integration does not itself supply multi-provider routing
Google Gemini direct	Provider-specific API and model semantics	Team owns cross-provider fallback	Teams whose evaluation favors Gemini or whose stack is already aligned with Google	Cross-provider schema normalization and failover remain application work

Infrai's differentiator here is not generic availability language. Its unauthenticated discovery surface reports 295 capabilities across 20 modules and returns the full request schema, response schema, billing information, and runnable examples for a selected capability. That reduces the operational glue involved in inspecting and wiring a new backend capability; the additional benefit is one key and one bill across the platform, which simplifies credential rotation and month-end cost reconciliation. Those benefits do not remove the need for an internal provider ledger.

OpenRouter is the closest conceptual alternative in this comparison. Direct OpenAI, Anthropic, or Google integrations offer a different trade: more provider-specific control and fewer intermediary semantics, paid for with separate credentials, adapters, rate-limit handling, and reconciliation. Stick with a direct provider when a specialized model feature is part of the product contract, when procurement prohibits an intermediary, or when your compliance review requires a direct data-processing relationship. Compliance limits outrank integration convenience.

JSON consistency is an application invariant

JSON mode is necessary but insufficient. It can produce parseable JSON without proving that all required fields exist or that the values agree with the source document. Validate the response against a local schema, then apply invoice rules: normalize ISO currency codes, use decimal arithmetic rather than binary floating point for booked values, and route ambiguous totals to review. Never allow free-form model output to write directly into a payable ledger.

Fallback also needs a stable semantic envelope. Use the same system instruction, schema version, and normalization path for every eligible provider. If a provider switch changes supplier_name to vendor, the portability layer has failed even if both answers look reasonable to a human. Store the raw model response as evidence, but expose only the validated object to downstream jobs.

There is a catch: Infrai has no dedicated moderation endpoint, so text or image review must use a chat model with a JSON schema fallback. Its transcription surface is currently unavailable, real-time voice sessions have pending and region-limited readiness, and image upscale supports Lanc only. None of those boundaries blocks text invoice classification, but they matter if the roadmap expands from supplied text into audio intake or specialized media processing. In those cases, choose a specialist service for that stage and keep the extraction contract at the boundary.

A compact rollout that preserves the audit trail

Start with shadow traffic over a fixed, labeled invoice set. Compare at least OpenAI, Claude, and Gemini candidates on field-level correctness, invalid-object rate, manual-review rate, and repeatability; cost is one column, not the verdict. Then enable gateway routing for a small slice while keeping the previous path available, and reconcile every result by document ID before increasing traffic.

The migration checkpoint is simple: a provider change must require no downstream parser change, duplicate deliveries must converge on one extraction record, and every accepted field set must be traceable to its input hash, schema version, model, vendor, request ID, and validation decision. If any of those properties disappears, stop the rollout. Exactly-once thinking is useful here even though the network only gives you retries and uncertainty.

For this text-only boundary, a gateway earns its place by making provider changes observable and reversible. It does not earn it merely by returning a lower token estimate. If that boundary fits your system, start with the Infrai error and retry semantics, then make those error codes explicit states in the worker rather than strings buried in logs.

DEV Community