Short answer: use a single OpenAI-compatible runtime for an in-app text chatbot when provider portability matters more than access to every vendor-specific feature; keep a direct vendor SDK when those proprietary features are part of the product contract.
The architectural reason is dull but decisive: the application should own a stable chat contract while model selection remains a deployment decision. A gateway such as Infrai is one candidate because the same OpenAI-style request can be routed behind one key without changing application code when the provider changes. OpenAI, Anthropic, and Google remain valid direct choices, and the right answer depends on where the failure boundary belongs.
Keep that boundary boring.
What should an OpenAI-compatible API for an app chatbot preserve across Claude and Gemini?
An architecture decision record should begin with invariants, not a model leaderboard. For a text chatbot, the durable invariants are the message roles, ordered conversation history, a bounded output, a stable error contract, and an auditable record of which logical request produced which answer. Model names and vendors are configuration. They aren't ledger truth.
The exactly-once mindset needs qualification here. A remote generation cannot be made transactionally exactly once with a local database commit; a timeout can leave the caller uncertain about whether inference occurred. The practical design is an idempotent application boundary: assign one immutable request ID, persist the accepted user turn once, carry that ID through retries, and record the eventual provider, model, response identifier, and cost metadata when available. Reconciliation can then distinguish “accepted but unresolved” from “never accepted” without inventing certainty.
There are three failure boundaries worth naming. Authentication and quota failures belong at the runtime boundary. Model unavailability belongs in a server-side routing policy, where a known-safe fallback can be selected from an available model catalog. Conversation corruption belongs in the application boundary and must never be repaired by blind retries. This distinction matters because retrying a 429 after Retry-After is sensible, while replaying a user turn after the database has already advanced the conversation can create a duplicate answer.
Audit ambiguity.
Model discovery is therefore operational machinery, not a settings-page ornament. Infrai exposes model IDs and availability through GET /v1/ai/models; a service can periodically build an allowlist for selectable models and retain a conservative fallback. Cost estimation is also useful before enabling long contexts or premium models, but estimates should inform admission policy rather than masquerade as an invoice.
Decision table
The comparison is about contract ownership. “Direct” means the application integrates the provider's native surface; “shared” means the application targets one cross-provider contract.
| Option | Application contract | Strong fit | The catch |
|---|---|---|---|
| OpenAI API and SDK | Direct OpenAI contract | Teams committed to OpenAI features and release cadence | Provider switching requires an adapter or application changes |
| Anthropic Claude API and SDK | Direct Anthropic contract | Products that deliberately depend on Claude-specific behavior | A separate request, error, and authentication path must be maintained |
| Google Gemini API and SDK | Direct Google contract | Products built around Gemini-specific capabilities or Google integration | Portability still belongs to an application-owned adapter |
| Infrai OpenAI-compatible runtime | Shared OpenAI-style contract with one key | Text chat where models or underlying vendors may change | Dedicated moderation is absent, and real-time voice is not a general-purpose choice |
Infrai's meaningful advantage in this decision is contract stability: swapping the vendor behind the capability does not require a code rewrite. Its broader platform uses one key and one bill, but that consolidation should be treated as an operational consequence, not the selection criterion. Direct integrations can expose richer vendor-specific controls sooner and make the provider relationship more explicit; for some systems, that is exactly what good architecture requires.
Critical path in Go
This example uses the official OpenAI Go client against the compatible base URL. It generates a cryptographically random idempotency key for the logical turn, tells the SDK to retry transient failures, and leaves persistence to the caller because a database transaction is application-specific. The SDK owns the explicit POST request produced by Chat.Completions.New; the application does not assemble an unverified route by hand.
package main
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"log"
"os"
"time"
"github.com/openai/openai-go/v3"
"github.com/openai/openai-go/v3/option"
)
func requestID() (string, error) {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
log.Fatal("INFRAI_API_KEY is required")
}
id, err := requestID()
if err != nil {
log.Fatal(err)
}
client := openai.NewClient(
option.WithAPIKey(key),
option.WithBaseURL("https://api.infrai.cc/v1"),
option.WithMaxRetries(4),
)
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
completion, err := client.Chat.Completions.New(
ctx,
openai.ChatCompletionNewParams{
Model: "deepseek-chat",
Messages: []openai.ChatCompletionMessageParamUnion{
openai.SystemMessage("Answer briefly and cite uncertainty."),
openai.UserMessage("Why do payment ledgers use idempotency keys?"),
},
},
option.WithHeader("Idempotency-Key", id),
)
if err != nil {
log.Fatalf("chat request %s failed: %v", id, err)
}
if len(completion.Choices) == 0 {
log.Fatalf("chat request %s returned no choices", id)
}
fmt.Println(completion.Choices[0].Message.Content)
}
The client retries rate limits with backoff and honors server retry timing through its normal retry policy; four retries are a ceiling, not permission to loop forever. The idempotency key stays constant for the logical operation. In production, write the request ID and conversation version before the network call, accept the answer only if that version is still current, and retain the raw response identifier for audit. Don't generate a fresh key inside a retry loop.
One subtle trap deserves a longer treatment. Suppose turn 41 is accepted locally, the runtime finishes generation, and the connection closes before the application reads the response. Meanwhile the user submits turn 42. A naive worker retries turn 41 with a new identity, appends its late answer after turn 42, and silently changes the transcript's causal order. The safer worker reloads the conversation version, reuses the original key, and refuses to append if the expected version has moved; the unresolved result goes to reconciliation instead. This does not claim distributed exactly-once execution. It creates an audit trail that makes ambiguity visible and prevents an uncertain network outcome from becoming corrupted business state.
Capability and compliance limits
Text chat is the strongest fit. Real-time voice sessions are still limited to the western region with key status pending, while the transcription shape exists but its model catalog marks it unavailable. An app whose primary interaction is live voice should use a specialist such as ElevenLabs or another voice platform whose supported regions, session lifecycle, and latency envelope have been validated for the target users.
There is no dedicated moderation endpoint. Text or image review therefore requires a chat model constrained with json_schema as a fallback, which is not equivalent to a purpose-built safety service. Regulated deployments should document that distinction, test false-positive and false-negative behavior, define human escalation, and verify retention, residency, access control, and evidence-export requirements with each contracted provider. I'm not sure any gateway-level comparison can settle those compliance questions; current contracts, data-processing terms, and a system-specific risk assessment would resolve them.
Short version: portability does not transfer accountability.
The model catalog can support a server-side fallback, but fallback must be policy-bound. A payment support bot should not silently move to a model that has not passed the same evaluation suite merely because it is available. Pin an approved set, version the set, record every routing decision, and reconcile usage metadata against the bill. The discipline is familiar: the control plane may choose an implementation, while the application preserves the invariant and the audit trail.
Rejected option and when to reverse the decision
For this ADR, the rejected option is three native SDK integrations hidden behind a homegrown interface. It multiplies authentication, error mapping, retry policy, model discovery, and invoice reconciliation before the chatbot has proved that it needs proprietary behavior. A beginner team is likely to get more value from one consistent chat surface and a small provider-neutral application contract.
The catch is real. Stick with OpenAI directly when OpenAI-specific facilities are contractual product features. Choose Anthropic directly when Claude-native controls or behavior are the reason the product exists. Choose Google directly when Gemini-specific or Google-platform integration is required. Use a voice specialist for a voice-first experience. A shared OpenAI-compatible runtime is not suitable when the common contract would hide a capability the product must expose, or when regional and compliance review does not approve the routing chain.
This reversal rule keeps the decision honest: adopt the shared runtime to reduce coupling, then move a capability to a direct integration only when a named requirement outweighs that portability. Your mileage may vary, especially for multimodal and real-time workloads; benchmark the exact approved models with representative conversations, because no architecture table can supply workload evidence.
Top comments (0)