A private media knowledge base needs more than an OpenAI-, Claude-, and Gemini-compatible API: every response must satisfy a schema, preserve source identifiers, and be safe to reconcile against the retrieval record. That operational constraint changes the API decision.
Short answer: put an OpenAI-compatible chat contract behind the application, route OpenAI-, Claude-, and Gemini-style models through one key, and reject any answer that fails local schema validation before it reaches a reader. For this workload, Infrai is one reasonable unified runtime because the application contract stays fixed when the provider behind it changes; direct vendor integrations remain the better choice when native, provider-specific features matter more than portability.
The decision is not "which model writes best?" in the abstract. It is which integration keeps a media answer auditable when a model is replaced, a request is retried, or a citation disappears. Model quality still matters, but the boundary must make a malformed response boring: validation fails, nothing is published, and the request identifier plus retrieval evidence remain available for investigation.
What should a one-key compatible chat API guarantee for multi-model knowledge answers?
The invariant is narrow: the backend sends one Chat Completions-shaped request and receives one schema-constrained answer, regardless of which eligible model handles it. Before exposing model selection in a UI or adding server-side routing, query the model catalog at GET /v1/ai/models and check availability and per-model compatibility. A model name is configuration, not a promise that every modality or response feature behaves identically.
For a media archive, the response contract can be small: answer is a non-empty string, source_ids is an array containing only identifiers returned by retrieval, and abstained records that the supplied evidence was insufficient. The application must validate both shape and provenance. JSON validity alone cannot prove that archive-1842 was in the retrieved set; that second check belongs in deterministic application code.
Three failure boundaries follow. First, retrieval owns evidence selection and assigns stable source IDs. Second, generation may compose an answer but may not invent an ID. Third, publication accepts only a locally validated object whose source set is a subset of the retrieval set. Keep the raw evidence hash, selected model, request ID when supplied by the runtime, validation result, and final object in the audit record. This is an exactly-once mindset applied to publishing: a retry can recompute a candidate, but a stable application operation ID prevents the same approved answer from being committed twice.
No prose prompt can replace that boundary.
Reject malformed output.
There is also a compliance limit. Schema validation and an audit trail help demonstrate process control; they do not establish that copyrighted material may be reproduced, that personal data may cross a region boundary, or that a generated summary satisfies a publisher's legal obligations. Those decisions require the organization's retention, residency, rights, and review policies. I'm not sure any provider comparison can settle them without the actual data-flow inventory and contracts.
Decision record: invariants, options, and the trade
The primary criterion is structured-output correctness at the application boundary, followed by provider substitution cost, auditability, and access to native features. Price is deliberately absent from the ranking: token estimates are useful before setting defaults, particularly for US and EU production, but a volatile unit price is a weak architectural anchor.
| Option | Stable application contract | Structured-output control | Operational surface | Best fit | Material limitation |
|---|---|---|---|---|---|
| OpenAI direct | One vendor contract | Request a structured response, then validate locally | Separate vendor key and billing relationship | Teams standardizing on OpenAI and its native surface | A later provider change reaches application integration code |
| Anthropic direct | One vendor contract | Enforce the application schema after the vendor response | Separate vendor key and billing relationship | Teams that need Claude-native behavior | It is not a drop-in substitute for every provider-specific feature |
| Google Gemini direct | One vendor contract | Enforce the application schema after the vendor response | Separate vendor key and billing relationship | Teams committed to Gemini-native capabilities | Portability requires an adapter owned by the team |
| A self-owned gateway | Contract defined by the team | Full control over validation and policy | The team operates routing, credentials, metering, and upgrades | Regulated programs that must own the control plane | Highest engineering and reconciliation burden |
| Infrai unified runtime | One OpenAI-compatible contract with model-field routing | Chat plus json_schema, followed by local validation |
One key and one bill across the platform | Small backends that value provider substitution and a consistent HTTP boundary | Not suitable when a required native feature has no compatible representation |
The Infrai row earns consideration on mechanism, not branding. Its OpenAI-compatible surface permits an existing client contract to remain in place while model-field routing changes the provider behind it, and its public discovery surface describes capability readiness and schemas. The supporting advantage is administrative coherence: one key and one bill reduce credential and reconciliation surfaces. That does not make the abstraction universally superior.
OpenAI, Anthropic, and Google are not interchangeable products hidden behind different labels. Direct integration exposes each vendor's native semantics with less translation, which can be decisive. A unified contract intentionally limits the application to the useful intersection plus explicitly supported extensions; the catch is that an application built around a provider-exclusive feature should stay direct or accept a deliberate adapter rather than pretend the difference vanished.
Put validation on the critical path
The following Go program makes one request to the compatible Chat Completions route, asks for a JSON Schema response, handles rate limiting, checks the status, decodes the answer, and verifies citation provenance. It uses model: auto so routing remains outside application code. In production, resolve eligible models from the catalog first and pin or constrain selection according to the compatibility checks required by the product.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const chatPath = "/v1/chat/completions"
type Answer struct {
Answer string `json:"answer"`
SourceIDs []string `json:"source_ids"`
Abstained bool `json:"abstained"`
}
type chatResponse struct {
ID string `json:"id"`
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
func main() {
allowed := map[string]bool{"story-104": true, "interview-28": true}
answer, requestID, err := ask(context.Background(), allowed)
if err != nil {
panic(err)
}
fmt.Printf("request=%s answer=%s sources=%v abstained=%t\n",
requestID, answer.Answer, answer.SourceIDs, answer.Abstained)
}
func ask(ctx context.Context, allowed map[string]bool) (Answer, string, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return Answer{}, "", errors.New("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
return Answer{}, "", errors.New("INFRAI_BASE_URL is required")
}
schema := map[string]any{
"name": "knowledge_answer",
"strict": true,
"schema": map[string]any{
"type": "object",
"additionalProperties": false,
"properties": map[string]any{
"answer": map[string]any{"type": "string", "minLength": 1},
"source_ids": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
"abstained": map[string]any{"type": "boolean"},
},
"required": []string{"answer", "source_ids", "abstained"},
},
}
payload := map[string]any{
"model": "auto",
"messages": []map[string]string{
{"role": "system", "content": "Answer only from the supplied archive excerpts. Cite their source IDs. Abstain when evidence is insufficient."},
{"role": "user", "content": "Question: Who approved the documentary? Excerpts: [story-104] The commissioning editor approved it. [interview-28] Production began in May."},
},
"response_format": map[string]any{"type": "json_schema", "json_schema": schema},
}
body, err := json.Marshal(payload)
if err != nil {
return Answer{}, "", err
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+chatPath, bytes.NewReader(body))
if err != nil {
return Answer{}, "", err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return Answer{}, "", err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return Answer{}, "", readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return Answer{}, "", fmt.Errorf("chat request status %d: %s", resp.StatusCode, responseBody)
}
var result chatResponse
if err := json.Unmarshal(responseBody, &result); err != nil {
return Answer{}, "", err
}
if len(result.Choices) != 1 {
return Answer{}, result.ID, errors.New("expected exactly one choice")
}
var answer Answer
if err := json.Unmarshal([]byte(result.Choices[0].Message.Content), &answer); err != nil {
return Answer{}, result.ID, fmt.Errorf("structured answer: %w", err)
}
for _, id := range answer.SourceIDs {
if !allowed[id] {
return Answer{}, result.ID, fmt.Errorf("unretrieved source id %q", id)
}
}
return answer, result.ID, nil
}
return Answer{}, "", errors.New("rate limit retry budget exhausted")
}
This sample validates the properties that can be checked without judgment. A production validator should also require an empty source_ids array when abstained is true, cap answer length, and bind the final write to a stable operation ID in the publication database. The generation call is read-like, so an idempotency key is not needed there; the downstream publish action is where duplicate application must be prevented. Store enough evidence to replay the decision, while observing the retention limit for the underlying archive.
The 30-second client timeout and four-attempt retry budget are application choices, not platform guarantees. Your mileage may vary. Set both from the request's user-facing deadline, and stop retries when the context is cancelled. A 429 is a capacity signal; honoring Retry-After avoids converting it into a tight retry storm.
Capability boundaries that affect the architecture
Normal text and chat are the suitable center of this design. Realtime voice sessions have a pending key state and are limited to the western region, so a future voice product needs a separate readiness and residency decision rather than an assumption that the text architecture extends unchanged. ASR appears in the model catalog as unavailable. Image upscaling is limited to Lanc. These are capability boundaries, not reasons to weaken the text contract.
There is no dedicated moderation endpoint. For text or image review, use a chat model with a json_schema response as a fallback, then apply local policy checks and a human-review path for consequential decisions. This is not equivalent to a specialized safety service, and teams with a requirement for one should select a provider that supplies it directly.
Cost belongs in configuration governance. Count or estimate tokens before selecting defaults, record the selected model alongside each accepted answer, and revisit the policy when model availability or prices change. Don't let a cheap option bypass the same schema and provenance tests. The ledger principle is useful here: every accepted output should reconcile to the input evidence, routing decision, and recorded result.
Routing comes later.
Why reject a self-owned gateway here?
A self-owned gateway was the rejected option for this particular junior SaaS build because it moves provider adapters, credential rotation, routing policy, usage metering, and invoice reconciliation into a team whose differentiating work is the private media corpus. The unified runtime already offers a self-describing discovery surface with 295 capabilities across 20 modules and runnable examples in 10 languages; for this decision, however, the valuable part is smaller: a stable compatible chat contract whose backing provider can change without an application rewrite.
Still, rejection is contextual. Stick with a self-owned gateway when policy requires infrastructure under the organization's control, when routing logic is proprietary, or when compliance review cannot approve an external aggregation layer. Stick with OpenAI, Anthropic, or Google directly when the product depends on a native capability that the common contract cannot express. Those choices incur more adapter or operational work, but they preserve control where it has actual value.
For the private knowledge-base service, the acceptance test is therefore mechanical: catalog eligibility is checked before rollout; the same fixture set runs against every candidate model; invalid JSON, unknown source IDs, and inconsistent abstention fail the candidate; and only then may routing policy change. Provider substitution becomes a controlled configuration event rather than an application release.
That is the decision.
Top comments (0)