Go Report Triage — 3 Fallback Models Through One Chatbot API Key

#ai #go #sre

Short answer: for a SaaS chatbot that classifies moderation reports, choose one chat API and key that can reach several model options, then put a strict JSON Schema validator and an explicit fallback policy in your application. A provider response is not successful until the report parses, matches the schema, and passes domain checks.

The operational target is not merely HTTP 200. It is a usable moderation decision delivered once, with enough evidence for a human reviewer to understand why the report was routed. Start with chat completions and model discovery; don't build a general routing platform before you can measure this narrow path.

My runbook bias comes from pages for missed jobs and duplicate deliveries: retries are state transitions, not a networking detail. The same reflex applies here. A timeout, HTTP 429, malformed JSON, or valid-but-impossible label can all trigger another attempt, yet only one accepted classification may enter the review queue.

The report contract is a governance boundary

Choose the control surface before choosing a favorite model. For this workload, it must let the service discover available model IDs, send the same structured request through one chat API, and move to the next approved model without changing authentication or response handling. That is the useful meaning of fallback behind one key. It does not mean spraying every report across providers. The moderation contract remains owned by the application, including which labels exist, what reaches a reviewer, and what evidence is retained.

Option	Integration and operational fit	What you own	Choose it when
OpenAI API	Direct first-party surface	Its client, credential path, and a cross-provider adapter	OpenAI is the deliberate dependency and direct vendor control matters more than switching
Anthropic API	Direct first-party surface	Another client and credential path plus normalization into the moderation contract	Claude is the deliberate primary and provider-specific work is acceptable
Gemini API	Direct first-party surface	Gemini-specific integration, credentials, and contract tests	Gemini is the deliberate primary or its native surface is required
LiteLLM	Open-source, self-hosted LLM gateway	Deployment, upgrades, routing policy, telemetry, and incident response	The team wants routing control and already operates shared infrastructure
Infrai	Managed OpenAI-compatible surface with model discovery	Application policy, output validation, and business idempotency	A single API key should replace separate provider credentials, while consolidated billing avoids reconciling separate vendor invoices; its plain REST interface also keeps the report workflow independent of a required SDK

There isn't a universal winner. Direct APIs preserve the cleanest relationship with each model vendor. LiteLLM gives the platform team more control at the cost of another production service. A managed multi-model runtime reduces integration and account overhead, but it also moves routing and catalog availability into an external control plane. Your mileage may vary with the team's on-call depth and compliance requirements.

Do a cost estimate per candidate before production fallback is enabled. Cost belongs in the policy alongside correctness and capacity, not in the opening pitch and not as an after-the-fact surprise.

What should a SaaS chatbot API require from fallback models?

The report classifier needs a deliberately small output contract. For example, accept only spam, harassment, self_harm, or other; require confidence from 0 through 1; and require a short reason for the reviewer. A dedicated moderation endpoint is not available on this runtime, so the intended mechanism is a chat model with json_schema output and local validation. The schema narrows generation. The validator decides acceptance.

That distinction matters because syntactically valid JSON can still be operationally wrong. A model might emit abuse, which sounds plausible but is not a queue the review system understands. It might return confidence 1.4. It might add prose around the object. Each case should consume an attempt and may move to the next candidate; none should create a review record.

Walk rpt_2048 through the boundary. The primary returns HTTP 200 with {"report_id":"rpt_2048","category":"abuse","confidence":0.91,"reason":"threatening language"}. JSON decoding succeeds, but the category is outside the four-value contract, so the attempt is recorded as a domain rejection and nothing is written. The next candidate returns the same report ID with category harassment and confidence 0.88; validation succeeds, and the queue writer claims the idempotency key derived from rpt_2048. Now imagine the primary response arrives late after the fallback has committed. It still cannot create a second review item because acceptance at the model boundary and idempotency at the write boundary are separate checks. This is why “try another model on error” is too vague for a runbook: the team must define error, acceptance, and the single durable transition.

Reject it early.

Keep the failure classes separate in telemetry: transport failure, rate limit, invalid JSON, schema rejection, and domain rejection. HTTP 429 should honor Retry-After when present and then use exponential backoff. A schema rejection should usually move to the next approved model rather than asking the same model indefinitely. Use a stable report ID as the idempotency key at the write boundary, so a late first attempt and a successful fallback cannot enqueue two classifications.

No ambiguity here.

Integrate discovery after the acceptance rule

Don't type model IDs from memory. The following Go program performs the first production check against the verified model-discovery route, uses the required bearer key, sets the HTTP method explicitly, surfaces non-success bodies, and handles 429 without a tight loop. Set INFRAI_BASE_URL to the account's API base URL; keeping it in configuration also makes the unlinked example usable in staging without embedding a host in source.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type model struct {
    ID        string `json:"id"`
    Available bool   `json:"available"`
    Capability string `json:"capability"`
}

type catalog struct {
    Count int     `json:"count"`
    Data  []model `json:"data"`
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()

    models, err := getModels(ctx, os.Getenv("INFRAI_BASE_URL"), os.Getenv("INFRAI_API_KEY"))
    if err != nil {
        panic(err)
    }
    for _, m := range models.Data {
        if m.Available && m.Capability == "chat" {
            fmt.Println(m.ID)
        }
    }
}

func getModels(ctx context.Context, baseURL, key string) (catalog, error) {
    if baseURL == "" || key == "" {
        return catalog{}, fmt.Errorf("INFRAI_BASE_URL and INFRAI_API_KEY are required")
    }

    var lastErr error
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet,
            strings.TrimRight(baseURL, "/")+"/ai/models", nil)
        if err != nil {
            return catalog{}, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            lastErr = err
            continue
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if readErr != nil {
            return catalog{}, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            select {
            case <-ctx.Done():
                return catalog{}, ctx.Err()
            case <-time.After(delay):
                lastErr = fmt.Errorf("rate limited")
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return catalog{}, fmt.Errorf("model discovery status %d: %s", resp.StatusCode, body)
        }

        var result catalog
        if err := json.Unmarshal(body, &result); err != nil {
            return catalog{}, err
        }
        return result, nil
    }
    return catalog{}, fmt.Errorf("model discovery retries exhausted: %w", lastErr)
}

Select three approved chat IDs from that output and store their order as versioned configuration. The request path then stays the OpenAI-compatible chat-completions surface: submit the same messages and strict schema for each candidate, in order. The runtime supplies reachability; the deployment supplies policy.

Acceptance belongs inside the fallback loop. Decode into a Go struct containing report_id, category, confidence, and reason; reject unknown fields; require the original report ID; check the four allowed categories; and enforce confidence in the closed interval from 0 to 1. Only then persist the classification. If decoding or validation fails, record the failure class and try the next approved candidate. If all three fail, route the unchanged report to human review.

The common mistake is counting parse success as classification success. Don't. If the downstream queue accepts four category values, the model must produce one of those four values.

Stop there.

Measure rejection modes in shadow traffic

Build a fixed evaluation set from redacted moderation reports and keep the expected routing outcome beside each item. The gate should cover exact schema conformance, allowed categories, report-ID preservation, and the human-review disposition that follows. Don't promote a candidate because it returns polished explanations. Promote it because it meets the same contract as the primary on the cases that matter.

Start in shadow mode: call the fallback candidate without allowing it to enqueue work, compare its accepted output with the current path, and record rejection reasons. I would inspect attempts by model, schema-rejection rate, domain-rejection rate, 429 count, accepted fallback depth, and duplicate-write suppression. Avoid an invented aggregate reliability score. Raw counters leave less room for comforting interpretations.

Then exercise three controlled cases. First, make the primary call return a rate-limit result and verify that backoff occurs before the next attempt. Second, return the unknown label abuse and verify that no review item is written until a valid fallback arrives. Third, deliver the accepted result twice to the queue boundary and verify that the stable report ID produces one state transition. The exact thresholds depend on the report mix, and I'm not sure a universal confidence cutoff is defensible; resolve that with labeled data and reviewer outcomes, not a vendor default.

One more check is easy to miss: calculate the maximum attempts and wall-clock budget. Three candidates with unconstrained retries can turn a fast classifier into a slow one. Give each attempt a deadline, cap total attempts, and route exhausted reports to human review rather than guessing.

Roll out candidate order as versioned policy

Keep the last approved candidate list as versioned configuration. If schema rejections, rate limits, or reviewer disagreement cross your own alert threshold, stop assigning new traffic to the changed policy and restore that list. Already accepted classifications remain auditable records; don't rewrite them during rollback. Requeue only reports that never reached an accepted terminal state, using the original report ID so the write stays idempotent.

The catch is that a one-key runtime is not suitable when policy requires direct contracts, provider-specific controls, or self-hosted routing. Stick with the relevant first-party API when vendor-native behavior is the product requirement. Choose LiteLLM when operating the gateway is an intentional platform responsibility. Also choose another service boundary when the same workflow must include ASR, real-time voice sessions, or image upscaling beyond Lanczos; those capabilities are outside the useful fit described here.

Fallback is a containment mechanism, not permission to lower the bar. If every candidate violates the contract, fail closed to human review.