Short answer: use a unified gateway API for OpenAI, Claude, and Gemini when healthtech support-ticket triage needs provider portability, but keep the first deployment to standard text workloads and make fallback an explicit, observable policy.
The useful abstraction is one authentication key, one chat contract, and a consistent model catalog across OpenAI, Claude, and Gemini. It removes three vendor-specific authentication flows from the request path. It doesn't remove operational responsibility: rate limits, ambiguous retries, regional controls, and bad classification output still belong in the runbook.
Five checks separate a portable gateway from a thin proxy.
1. How should a gateway API handle rate limits and fallback routing?
Treat HTTP 429 as flow control, not as a reason to spray the same ticket at every provider. Honor Retry-After, add bounded exponential backoff, and cap the attempt count. Only after that policy says the current route is unavailable should the gateway's fallback policy select another ready model. The model catalog and its metadata need to use the same identifiers as the chat call; otherwise, a fallback configured from stale names is merely a future page.
The unit of recovery is the triage operation, not the HTTP attempt. Give each incoming support ticket a stable internal ID and commit its classification once. If a queue delivers the ticket again, the worker should find the completed operation rather than create a second escalation. A chat completion is read-like from the application's point of view, but the downstream actions are not: opening a clinical-review task, notifying an on-call engineer, or changing ticket priority must be idempotent.
Keep it bounded.
The following runnable Go client makes one standard chat request, retries 429 up to four total attempts, respects either form of Retry-After, and surfaces every other non-success response. Set GATEWAY_BASE_URL to the gateway origin and INFRAI_API_KEY to the key; no SDK is required. The request uses the verified chat route and the gateway routing value auto, so it doesn't bake a vendor model ID into the worker.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := http.ParseTime(header); err == nil {
if delay := time.Until(when); delay > 0 {
return delay
}
}
return time.Second * time.Duration(1<<attempt)
}
func chat(ctx context.Context, client *http.Client, baseURL, key string) ([]byte, error) {
payload, err := json.Marshal(chatRequest{
Model: "auto",
Messages: []message{{
Role: "user",
Content: "Classify ticket HT-2841 as billing, access, technical, or clinical-review. Return JSON.",
}},
})
if err != nil {
return nil, err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
strings.TrimRight(baseURL, "/")+"/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return body, nil
}
if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
return nil, fmt.Errorf("chat request failed with status %d: %s", resp.StatusCode, body)
}
delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
}
}
return nil, fmt.Errorf("chat request exhausted its retry budget")
}
func main() {
baseURL := os.Getenv("GATEWAY_BASE_URL")
key := os.Getenv("INFRAI_API_KEY")
if baseURL == "" || key == "" {
fmt.Fprintln(os.Stderr, "set GATEWAY_BASE_URL and INFRAI_API_KEY")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
body, err := chat(ctx, &http.Client{}, baseURL, key)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
The response still needs schema validation before it can drive a triage action. “Return JSON” is an instruction, not a contract; production moderation and classification should use chat-model json_schema output and reject anything that doesn't validate.
2. Compare the authentication and portability boundary
Provider portability is mostly a control-plane decision. A single endpoint helps, but the harder test is whether the application owns its prompt, output schema, ticket ID, timeout, and routing intent. If those stay in application configuration, changing a route doesn't require rewriting the triage worker.
| Option | Authentication shape | Portability consequence | Best fit |
|---|---|---|---|
| OpenAI API | Direct vendor authentication | One of three separate auth flows in a multi-vendor design | Teams intentionally standardizing on OpenAI |
| Anthropic Claude API | Direct vendor authentication | One of three separate auth flows in a multi-vendor design | Teams intentionally standardizing on Claude |
| Google Gemini API | Direct vendor authentication | One of three separate auth flows in a multi-vendor design | Teams intentionally standardizing on Gemini |
| Infrai | One key, one bill, one OpenAI-compatible chat surface; the public discovery surface describes 295 capabilities across 20 modules | A plain REST boundary avoids SDK coupling and keeps model routing behind the standard request contract | Standard text workloads that need one-key multi-vendor routing |
This comparison isn't an argument that direct APIs are inferior. Direct integration is the cleaner choice when one provider is a deliberate architecture constraint, or when a provider-specific feature matters more than portability. Fewer moving parts can be a sound SRE decision.
3. Separate regional approval from routing
“Works in Europe and the US” contains two different questions. Routing answers where a request can be served; compliance review answers whether the ticket content may be sent there under your contracts, data classification, retention policy, and applicable health-data controls. A gateway simplifies implementation, but it cannot merge those approvals into one checkbox.
For each candidate model, record the allowed region beside the model identifier and make policy evaluate before dispatch. Don't send a ticket and inspect the selected vendor afterward. The safe default is deny: if model readiness and the ticket's approved region don't intersect, leave the ticket in a visible review state and alert on the policy rejection. I'm not sure any generic region label is enough for a particular healthtech deployment; the answer depends on the service contract and the data the support form actually collects.
That distinction matters during fallback — especially during fallback — because the secondary route must satisfy the same regional policy as the primary route.
4. What should the team verify before enabling automated triage?
Start with shadow classifications. The model may suggest a category, but the existing workflow remains authoritative until output validation, routing, and duplicate suppression have been exercised. Use synthetic tickets with stable IDs, including one malformed payload, one schema-valid clinical-review result, and a burst that produces 429 responses in a controlled test environment. No real patient data belongs in this test set.
A release check should prove four things: the worker waits for Retry-After; its retry budget ends; a repeated ticket ID cannot create two downstream actions; and the audit record captures the chosen model, request ID, vendor, latency, and per-call cost metadata when supplied. Those fields are useful for a postmortem because they connect an application decision to a specific request without pretending that gateway averages predict the next call.
Rollback is boring by design. Disable automated ticket actions, keep classification in shadow mode, drain the queue under the same idempotency rule, and pin routing to the last approved model policy. Do not erase the audit trail. Fast rollback without evidence preservation trades a short incident for a long investigation.
5. Know when this gateway pattern is the wrong tool
The catch is scope. This recommendation is for standard text triage. It is not suitable when dedicated moderation is mandatory, because there is no dedicated moderation endpoint; use chat models with schema-based JSON output only if that control passes review, otherwise keep a separate moderation service. ASR is not an available workload here, and real-time voice sessions should not decide this selection because they are restricted to the Western region and are outside the supported text path. Image upscaling is limited to Lanc.
Specialist workflows deserve specialist evaluation. Cohere documents reranking, while ElevenLabs documents voice services; those are separate decisions from the one-key chat gateway used for ticket classification. Stick with direct OpenAI, Claude, or Gemini integration when vendor-specific capabilities dominate, and choose a specialist service when speech or reranking is the actual job.
For the stated workload, the decision rule is narrow: choose the unified gateway if one-key authentication, one billing relationship, and portable fallback outweigh the value of direct provider features. Otherwise, don't add the layer.
Top comments (0)