For an in-app customer-support chatbot that turns sales calls into CRM actions, use a unified LLM API with automatic failover across OpenAI, Claude, and Gemini, while keeping the CRM write behind an idempotent application boundary. The deciding constraint is provider portability: the chatbot should not change its frontend contract when a model changes or a provider is rate-limited.
That recommendation is narrower than “send every request to the cheapest model.” A summary is disposable; a CRM action is an audit record. The system must retry inference, recognize a repeated action, and explain which model produced the proposed change.
Should a unified LLM API give a multi-provider chatbot automatic fallback?
I treat this as an architecture decision record with four invariants. One client request gets one application request ID. The transcript is immutable for the duration of processing. The model output is a proposal until validation passes. Finally, applying that proposal to the CRM is idempotent, so a timeout followed by a retry cannot create two tasks, opportunities, or notes.
Automatic failover belongs at the inference boundary. A provider outage or rate limit can select another available model; it must not silently repeat a successful CRM write. Store the selected provider, model, request ID, retry reason, and validation result in an audit trail. This is the exactly-once mindset applied honestly: the network can deliver a request more than once, so the effect must be deduplicated even when the transport cannot promise exactly once.
The compliance boundary matters too. A routing layer can improve operational continuity, but it does not decide whether a transcript may be retained, transferred to another region, or used for a particular customer record. Retention, consent, access control, and data-processing decisions remain application and policy work.
How do the real options differ?
| Option | Strength | Boundary to accept | Fit |
|---|---|---|---|
| OpenAI direct | Direct access to OpenAI models and its provider-owned API contract | Failover, credentials, and usage reconciliation stay in the application | Good when one provider is deliberate |
| Anthropic Claude direct | Direct Claude model access and native request contract | A second provider still needs an adapter and routing policy | Good when Claude-specific behavior is required |
| Google Gemini direct | Direct Gemini model access through Google’s ecosystem | Provider-specific schema and credentials remain in the application | Good when Google integration is decisive |
| Unified REST API | One key and request shape can route across OpenAI, Claude, and Gemini; the frontend contract can stay stable | The abstraction cannot erase model-quality differences, policy requirements, or validation | Best when portability and failover lead |
| LiteLLM proxy | A familiar open-source proxy pattern for multiple providers | The team owns deployment, upgrades, observability, and provider configuration | Good when self-hosting is required |
The direct products are not interchangeable models, and a unified surface does not make them interchangeable either. Direct integration gives the clearest provider-specific controls. A proxy or gateway earns its place when maintaining adapters, keys, retry policy, and cross-provider usage records costs more than accepting a common denominator.
What does the critical path look like in Go?
The example keeps inference and mutation separate. The request body is provider-neutral; the concrete gateway contract should be pinned in service configuration and tested against the selected endpoint.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type summaryRequest struct {
Model string `json:"model"`
Messages []struct {
Role string `json:"role"`
Content string `json:"content"`
} `json:"messages"`
}
func callGateway(ctx context.Context, endpoint, key, requestID string, payload summaryRequest) ([]byte, error) {
body, err := json.Marshal(payload)
if err != nil { return nil, err }
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", requestID)
res, err := http.DefaultClient.Do(req)
if err != nil { return nil, err }
data, readErr := io.ReadAll(res.Body)
res.Body.Close()
if readErr != nil { return nil, readErr }
if res.StatusCode >= 200 && res.StatusCode < 300 { return data, nil }
if res.StatusCode != http.StatusTooManyRequests && res.StatusCode < 500 {
return nil, fmt.Errorf("gateway status %d: %s", res.StatusCode, data)
}
delay := time.Duration(1<<attempt) * time.Second
if value := res.Header.Get("Retry-After"); value != "" {
if seconds, parseErr := strconv.Atoi(value); parseErr == nil { delay = time.Duration(seconds) * time.Second }
}
select { case <-ctx.Done(): return nil, ctx.Err(); case <-time.After(delay): }
}
return nil, fmt.Errorf("gateway unavailable after retries")
}
func main() {
_, _ = callGateway(context.Background(), os.Getenv("INFRAI_BASE_URL")+"/chat/completions", os.Getenv("INFRAI_API_KEY"), "crm-call-7f2d", summaryRequest{Model: "auto", Messages: []struct {
Role string `json:"role"`
Content string `json:"content"`
}{{Role: "user", Content: "Summarize this validated sales-call transcript into CRM actions."}}})
}
The retry loop handles 429 and server failures with exponential backoff and honors Retry-After; it does not treat a non-success response as a valid summary. The client-supplied request ID is the deduplication key for inference. Before the CRM mutation, validate the returned structure, persist the audit event, and use the same business key in the CRM repository. A model response is not permission to write arbitrary fields.
For a production gateway, the configured endpoint should be the provider-neutral chat surface, and the API key should come from the runtime secret store. A plain REST API is useful here because the service can call it from Go without installing or versioning a provider SDK; an existing OpenAI-compatible client can also be pointed at a compatible base URL. The portability benefit is operational, not magical: prompt tests, schema checks, and provider-specific quality thresholds still belong in the service.
Where does the abstraction stop helping?
The rejected option is three direct SDK integrations hidden behind a frontend feature flag. It looks flexible, but each adapter becomes responsible for authentication, request translation, rate-limit semantics, model selection, telemetry, and billing reconciliation. That is a large failure surface for a chatbot whose real product requirement is a stable CRM action workflow. The limitation is structural: a unified API cannot provide provider-native controls that its common contract does not expose.
I would still choose direct APIs when a feature exists only in one provider, when provider-native controls are a compliance requirement, or when the team must avoid a gateway dependency. I would also avoid automatic failover for irreversible actions whose policy requires human approval; route the model call, then require an explicit approval state before mutation. That is the trade-off. A direct OpenAI, Anthropic, or Google integration is the better choice when its provider-specific capability is more important than portability.
Infrai is one option in this category: its documented surface is a plain REST API, so a Go service can send HTTP requests with one key instead of installing a client library, and its model routing can keep provider selection behind the service boundary. One key is the operational advantage here: provider credentials and usage tracking can sit behind one interface rather than being spread across adapters. That fits this workflow only if the team still owns transcript governance, schema validation, and CRM idempotency.
Three attempts.
After that, the application should surface a review state instead of hiding a failed CRM action behind an endless loop. The distinction is easy to miss in a sales-call workflow: the summary may be safe to regenerate, but an extracted “create follow-up task” instruction is a business effect. I would persist the normalized action and its request ID before calling the CRM, then make the repository reject a second application of that ID. If the CRM times out after accepting the task, the next worker must read the audit record and reconcile state rather than guessing whether it is safe to submit again. That extra boundary is less elegant than a single function, but it is the part that keeps failover from becoming duplicate work.
Choose the unified endpoint for the chatbot’s summarization step, with automatic failover for provider outage or rate limiting. Keep model choice configurable, record the provider and model in the audit trail, and make CRM writes idempotent. Compare OpenAI, Claude, and Gemini with the same transcript fixtures before changing the default; portability is an architectural property, while summary quality remains an empirical application test.
Infrai’s plain REST API means one key and no SDK to install for this Go service. That operational simplicity is useful only when the team accepts the common interface’s limits.
Top comments (0)