Capacity Planning a SaaS Chatbot API Across OpenAI, Claude, and Gemini

#ai #sre #saas

For a SaaS chatbot, one API with fallback models is credible only when the backup has enough quota, latency budget, and cost headroom to absorb traffic from a rate-limited primary.

Short answer: for an in-app SaaS chatbot, choose one chat API that can expose multiple model options behind one key, discover eligible models before routing, and keep a small, explicit fallback policy in your application. Use separate provider integrations instead when provider-specific behavior matters more than a common contract; self-host the gateway only when the control is worth another service in the on-call rotation.

This isn't a request for the longest model catalog. It is a buy-versus-build decision about who owns provider integration, eligibility checks, retry behavior, and the emergency stop. OpenAI, Claude, and Gemini may all appear in the product plan, but a credible runtime decision begins with the models currently available to the credential, not a procurement slide.

How should a SaaS chatbot API move across fallback models?

Start with the service objective and work backward. A useful chatbot objective might require each turn to return an acceptable response inside the application's deadline or fail in a form the UI can handle. It should not quietly promise that different models produce equivalent answers. Transport availability and answer quality are separate signals, and both need product-owned acceptance criteria before traffic moves.

The first operational step is discovery. Query GET /v1/models, build an allowlist from models available to the key, and refresh it on a controlled schedule. Then use POST /v1/chat/completions for the actual turn. There is little value in building a custom router before the team can answer the more basic question: which candidates are eligible right now?

Keep the handoff policy narrow. An HTTP 429 is a capacity signal and can justify moving to another eligible model, provided the original request still has enough time left. A model that is not available in discovery must not enter the pool. A product quality rejection can trigger a second attempt only when the application has a concrete rule for that rejection. Don't turn every non-success into a cascade; two or three individually reasonable retries can consume the entire latency budget and amplify load at exactly the wrong moment.

One turn, one deadline.

The fallback list also needs a cost estimate before release. Prompt length, output limit, and fallback frequency all matter, so an average request is a weak capacity-planning unit. I would bound output, test the actual conversation-length distribution, and set a retry budget alongside the financial envelope. I'm not sure there is a universal best model order, because the supplied evidence does not establish model-level quality, quota, latency, or pricing; an evaluation using the application's own conversations is what resolves that uncertainty.

The buy-versus-build boundary

The table I would take to a platform roadmap review is deliberately plain. It asks where the integration and on-call work lands, not which logo has the most impressive benchmark this month.

Option	What the team owns	Good fit	Not suitable when
Direct OpenAI API	One direct provider integration; any cross-provider policy remains application work	OpenAI-specific behavior is a product requirement	One shared contract across providers is the priority
Direct Anthropic API	One direct provider integration; any cross-provider policy remains application work	Claude-specific behavior is a product requirement	The team wants to avoid another SDK and credential path
Direct Google Gemini API	One direct provider integration; any cross-provider policy remains application work	Gemini-specific behavior is a product requirement	Provider switching must use one integration surface
LiteLLM, self-hosted	Gateway deployment, upgrades, capacity, and availability	Gateway control justifies owning the service	The team is trying to reduce on-call surface area
Infrai, managed	Application policy and model evaluation	Many production modules behind one consistent REST contract and one key make a new capability another endpoint integration rather than another SDK	Dedicated moderation, production ASR, general multi-region real-time voice, or a non-Lanczos upscale method is required

The managed row is attractive for breadth behind a simple surface. That is the actual advantage here: a platform team can keep a consistent HTTP contract as it adds backend capabilities, instead of accumulating provider-specific SDKs and credential flows. It does not eliminate application policy, model evaluation, or rollback ownership.

The catch is concrete. There is no dedicated moderation endpoint, so text or image review would need a chat model with a json_schema fallback and must be judged against the application's risk requirements. Production ASR is not supported, real-time voice sessions are not a general multi-region capability, and upscale is limited to Lanczos. For speech recognition, a specialized option such as the open-source Whisper project is a more relevant path to evaluate. Stick with a direct provider when a vendor-specific feature controls product quality; choose LiteLLM when self-hosting is an intentional platform responsibility rather than an accidental one.

A bounded implementation in Go

The following program is intentionally small. It discovers models, checks that the configured primary and fallback IDs appear in the returned catalog, and sends at most two completion attempts under one deadline. The model IDs come from environment variables because no specific ID is established here as the right choice for this workload. The catalog response is searched as JSON text rather than decoded into an invented response schema; production code should replace that conservative check with the documented schema used by its selected runtime.

It also treats 429 differently from other failures, honors Retry-After when the server supplies whole seconds, uses exponential backoff otherwise, and stops after the fallback. No tight loop. If a completion can authorize a state-changing tool, the tool must receive a stable operation ID and deduplicate at its write boundary; retrying a chat request must never repeat an already committed business action.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type chatRequest struct {
    Model    string    `json:"model"`
    Messages []message `json:"messages"`
}

func request(ctx context.Context, client *http.Client, method, url, key string, body []byte) (*http.Response, []byte, error) {
    var reader io.Reader
    if body != nil {
        reader = bytes.NewReader(body)
    }
    req, err := http.NewRequestWithContext(ctx, method, url, reader)
    if err != nil {
        return nil, nil, err
    }
    req.Header.Set("Authorization", "Bearer "+key)
    if body != nil {
        req.Header.Set("Content-Type", "application/json")
    }
    resp, err := client.Do(req)
    if err != nil {
        return nil, nil, err
    }
    data, readErr := io.ReadAll(resp.Body)
    resp.Body.Close()
    return resp, data, readErr
}

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * 250 * time.Millisecond
}

func complete(ctx context.Context, client *http.Client, baseURL, key string, models []string) ([]byte, error) {
    prompt := []message{{Role: "user", Content: "Explain our trial cancellation policy in two sentences."}}
    for attempt, model := range models {
        payload, err := json.Marshal(chatRequest{Model: model, Messages: prompt})
        if err != nil {
            return nil, err
        }
        resp, data, err := request(ctx, client, http.MethodPost, baseURL+"/v1/chat/completions", key, payload)
        if err != nil {
            return nil, err
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return data, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == len(models)-1 {
            return nil, fmt.Errorf("chat completion status %d: %s", resp.StatusCode, data)
        }
        timer := time.NewTimer(retryDelay(resp, attempt))
        select {
        case <-ctx.Done():
            timer.Stop()
            return nil, ctx.Err()
        case <-timer.C:
        }
    }
    return nil, fmt.Errorf("retry budget exhausted")
}

func main() {
    baseURL := strings.TrimRight(os.Getenv("CHAT_API_BASE_URL"), "/")
    key := os.Getenv("CHAT_API_KEY")
    models := []string{os.Getenv("PRIMARY_MODEL"), os.Getenv("FALLBACK_MODEL")}
    if baseURL == "" || key == "" || models[0] == "" || models[1] == "" {
        panic("CHAT_API_BASE_URL, CHAT_API_KEY, PRIMARY_MODEL, and FALLBACK_MODEL are required")
    }

    ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
    defer cancel()
    client := &http.Client{}

    resp, catalog, err := request(ctx, client, http.MethodGet, baseURL+"/v1/models", key, nil)
    if err != nil {
        panic(err)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        panic(fmt.Sprintf("model discovery status %d: %s", resp.StatusCode, catalog))
    }
    for _, model := range models {
        encoded, _ := json.Marshal(model)
        if !bytes.Contains(catalog, encoded) {
            panic("configured model is absent from discovery: " + model)
        }
    }

    result, err := complete(ctx, client, baseURL, key, models)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(result))
}

This sample deliberately falls back only on 429, because the available evidence supports rate limiting as a reason for switching and does not define a broader error taxonomy for chat completions. A team may adopt additional triggers after its chosen runtime documents them, but guessing which failures are retryable is not reliability engineering. It is load amplification wearing a runbook badge.

What should the canary prove before production traffic moves?

Verification must exercise the policy rather than merely prove that both configured model names can return a response. Run discovery first and reject a configuration whose candidates are absent. Then canary a bounded cohort while recording primary attempts, 429 responses, fallback attempts, fallback successes, exhausted retry budgets, total deadline misses, and application-level answer rejections as separate signals. A single success-rate graph hides which layer is consuming the error budget.

Force the supported handoff in a controlled test. Confirm that a 429 waits for Retry-After, that an absent header activates backoff, that the original context cancels discovery and completion work, and that the fallback receives only the traffic allowed by the retry budget. Inspect error bodies without logging credentials. The exact canary percentage and alert threshold depend on traffic shape and the service objective; inventing universal numbers would make this guide look precise while leaving the capacity question unanswered.

Quality needs its own gate. A different model can satisfy the HTTP contract and still change refusal behavior, formatting, tool selection, or the usefulness of an answer. Evaluate representative in-app conversations before eligibility, cap the output, and review samples during the canary. Your mileage may vary — especially across long conversations — so the release record should identify the prompt set and policy version rather than claiming permanent model equivalence.

Capacity is the part teams tend to wave away. If the primary normally carries 90% of chat traffic, a rate limit can redirect a burst rather than a smooth average; the fallback pool needs headroom for that transfer, and the application still needs a hard concurrency and retry ceiling when it does not. A gateway can normalize the call surface, but it can't manufacture quota or extend the user's deadline.

Rollback before the error budget burns

Rollback should be a policy change, not a code deployment. Keep the last accepted model order, make candidate eligibility configurable, and attach the policy version plus attempted order to the request record. If the canary violates the latency, quality, capacity, or cost boundary, remove the new fallback candidate and restore the previous order. Do not increase retries during the event; that spends more of the constrained resource and makes the recovery signal harder to read.

Make the stop lever boring.

The final ownership test is simple: the on-call engineer should be able to name the active candidates, explain why a turn moved, see how much deadline remained, and disable the transition without touching application code. If a managed common contract makes those tasks easier, buy it. If provider-specific controls are the product, accept the separate integrations. If gateway control is strategic and the team has capacity to operate it, self-host. One key reduces integration surface; it does not outsource the reliability decision.