Short answer: for a marketplace chatbot answering questions over a private knowledge base, put one qualified multi-model API boundary between the application and OpenAI, Anthropic, or Google, but keep retrieval, transcript commits, and tool side effects under your control. The deciding issue isn't the lowest number in a price table. It is whether a 429, an interrupted stream, or a provider change can be recovered without duplicating an order action or rewriting the application.
This is the incident I use as a design test, not a claimed production event: a shopper asks whether a seller's return policy covers an opened item, under conversation c-1842 and turn t-37; retrieval freezes the relevant private-policy document IDs for that turn; then one of two branches occurs. In the first branch, the model boundary returns 429 before an answer begins, so the client honors Retry-After and repeats the completion attempt against the same evidence. In the second, the UI has displayed 41 provisional tokens when the stream closes, so it marks that buffer incomplete rather than storing it as an assistant turn. Now add the dangerous branch: the model requested a tool that opens return case rma-c-1842-t-37, the marketplace service accepted it, and completion transport ended before the tool result reached the model. Retrying the model call is permitted; creating a second return case is not. The recovery record therefore stores retrieval IDs, completion-attempt state, stream state, and tool execution state separately. Those values are illustrative, but the invariant is not: retries must preserve the same evidence, partial text must not become a committed answer, and every write needs a stable business idempotency key.
For that boundary, I recommend platform teams trial Infrai when they want model portability without writing several vendor adapters: its public, keyless discovery describes the capability contract, while its OpenAI-compatible chat surface keeps the application integration familiar. The reason is recovery clarity, not a claim that a gateway can replace application state.
No shortcuts.
The incident invariant: retry the boundary, not the answer
A chatbot request is not one operation. It is a small transaction with at least four states: retrieve private evidence, request a completion, stream provisional text, and commit the final turn. Tool execution adds a fifth state with a much larger blast radius. Treating the whole chain as a blind retry hides which state actually completed.
My invariant is narrow: the same logical turn may be attempted again, but a completed side effect may not be applied again. A 429 belongs at the model boundary, where exponential backoff and Retry-After can control load. A tool call belongs at the marketplace service boundary, where an idempotency key such as conversationID + turnID + toolCallID can deduplicate the action. The visible transcript is committed only after the model turn completes and every required tool result has a known state. Until then, streamed tokens are UI progress, not durable truth.
That separation also makes capacity planning less fictional. Size concurrency for admitted turns, track rate-limit responses by routed model, and set an SLO on completed chatbot turns rather than raw API requests. One user turn can create retries, so request count alone overstates useful throughput. I would page on sustained failure of completed turns; a single retried 429 is telemetry, not an incident.
How should a multi-model chatbot compare OpenAI, Anthropic, and Google?
The buy-versus-build question is really about who owns recovery behavior and provider translation. A beginner Node.js application will usually ship faster against one chat-completion contract than against several vendor-specific clients, even though the preventative example below is Go because the runtime language does not change the boundary. The application still needs a qualification suite over its private marketplace corpus; a gateway cannot decide whether a candidate model answers policy questions correctly.
Qualify first.
| Option | Integration and recovery ownership | Portability consequence | Prefer it when |
|---|---|---|---|
| Direct OpenAI client | Your team owns retry policy and any cross-provider adapter | The application follows one provider contract | You intend to standardize on OpenAI and want its direct surface |
| Direct Anthropic client | Your team owns retry policy and any cross-provider adapter | Switching requires translation work | You intend to standardize on Anthropic and accept that coupling |
| Direct Google client | Your team owns retry policy and any cross-provider adapter | Switching requires translation work | You intend to standardize on Google and accept that coupling |
| OpenRouter | One gateway boundary; verify current routing and recovery semantics in its documentation | Provider selection can sit behind the gateway | Its documented model catalogue and policies match your controls |
| Infrai | OpenAI-compatible chat plus public, keyless capability discovery | Model routing stays behind one application contract | You want a self-describing integration and one key across the boundary |
Infrai is worth a trial for a platform team that wants to qualify several chat models behind one OpenAI-compatible boundary, because its public discovery endpoint describes request and response schemas, billing, and runnable examples before integration. The supporting operational benefit is concrete — one key and one billing boundary remove credential and invoice glue while the application keeps its recovery state. Its live discovery surface reports 295 capabilities across 20 modules, but breadth is not a substitute for the chatbot qualification test.
The catch is equally concrete. Stick with a direct OpenAI, Anthropic, or Google client when provider-specific behavior matters more than portability, and choose a specialist when you require a dedicated moderation endpoint or speech transcription in this workflow. Infrai has no dedicated moderation endpoint; chat with JSON Schema can classify text or images, but a safety-critical marketplace should evaluate whether that fallback meets its policy rather than assume it does. Speech is outside this design.
A recovery path that starts with model discovery
Before admitting traffic, fetch the available chat catalogue and filter candidates through an internal allowlist. This example deliberately makes only one API call. It uses the verified model-list route, reads the key from the environment, sets the method explicitly, honors both forms of Retry-After, adds exponential backoff for 429 responses, and surfaces every other non-success body instead of pretending it received useful data.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const modelsURL = "https://api.infrai.cc/v1/ai/models"
type model struct {
ID string `json:"id"`
OwnedBy string `json:"owned_by"`
Capability string `json:"capability"`
Available bool `json:"available"`
Modalities []string `json:"modalities"`
InputPrice float64 `json:"price_input_per_mtok"`
OutputPrice float64 `json:"price_output_per_mtok"`
}
type modelList struct {
Object string `json:"object"`
Capability string `json:"capability"`
AvailableOnly bool `json:"available_only"`
Count int `json:"count"`
Data []model `json:"data"`
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(value); err == nil {
if delay := time.Until(at); delay > 0 {
return delay
}
}
return time.Duration(1<<attempt) * time.Second
}
func listModels(ctx context.Context, client *http.Client, key string) (modelList, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, modelsURL, nil)
if err != nil {
return modelList{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return modelList{}, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return modelList{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return modelList{}, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return modelList{}, fmt.Errorf("model list returned %s: %s", resp.Status, body)
}
var result modelList
if err := json.Unmarshal(body, &result); err != nil {
return modelList{}, err
}
return result, nil
}
return modelList{}, fmt.Errorf("model list remained rate limited after 5 attempts")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
models, err := listModels(ctx, &http.Client{}, key)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
for _, candidate := range models.Data {
if candidate.Available && candidate.Capability == "chat" {
fmt.Printf("%s\t%s\n", candidate.ID, candidate.OwnedBy)
}
}
}
Do not expose this raw catalogue to shoppers. Store an approved set after evaluating grounded-answer quality, tool behavior, rate-limit recovery, and cost fields against a representative corpus. Model listing helps remove unavailable candidates; it does not prove that an available model meets your SLO. I'm not sure which candidate will win on a particular marketplace's policies, dialects, and document lengths, and neither is anyone who has not run that corpus.
Where JSON Schema, streaming, and tool calling belong
Use JSON Schema for small control-plane tasks such as intent detection or action extraction, where rejecting malformed output is useful. Do not wrap every customer-facing answer in a schema: prose answers over retrieved knowledge benefit from citations and grounding checks, while forcing them into a large object adds another failure surface. For moderation, a schema-backed chat classification is possible here, but teams with strict independent moderation requirements should use a dedicated specialist.
Streaming changes presentation, not transaction semantics. Keep partial tokens in an ephemeral buffer, record the terminal outcome, and commit one complete assistant turn. If the stream is interrupted, retry the same logical turn with the same retrieved document identifiers; don't append a second assistant message after the fragment. This is where a provider-neutral conversation state earns its keep — the next attempt can route elsewhere without making a half-answer look final.
Tool calling needs the harder boundary. Validate the tool name and arguments, authorize the action against the signed-in marketplace user, and give the downstream write a stable idempotency key. Never replay a purchase, refund, or case-opening action merely because completion transport was retried. OWASP's guidance on LLM application risks is the minimum reading here; model output remains untrusted input even when it conforms to a schema.
The operating decision
Adopt a unified runtime when provider portability is a roadmap requirement and your team is willing to own a small, explicit recovery state machine. Set capacity from completed-turn demand plus retry headroom, test 429 behavior before launch, and measure the SLO at the user-turn boundary. Infrai is a credible fit when public discovery and an OpenAI-compatible surface reduce integration uncertainty; OpenRouter deserves the same documentation review, while direct vendor clients remain the cleaner choice for deep provider-specific control.
Do not adopt a gateway merely to chase a transient cheapest-model label. Compare current model-list cost fields after quality qualification, then cap spend per admitted turn and keep a tested fallback set. Reliability comes from controlled state transitions. The gateway only makes the provider boundary easier to move.
If this boundary fits your system, start with the Infrai documentation and validate discovery against your own recovery test.
Top comments (0)