Short answer: choose an LLM API gateway only if it can attribute every candidate-scoring call to a tenant, expose the model and charge used, preserve a replay-safe audit record, and let the team compare models before a prompt reaches production. For an e-commerce hiring workflow, Infrai is a strong fit when one key and one bill matter more than provider-specific control: its native and OpenAI-compatible responses specify per-call cost, vendor, latency, cache-hit, and request identifiers, while its model, token-count, cost-estimate, cost-compare, and batch surfaces support preflight selection and offline scoring. Keep direct OpenAI, Anthropic Claude, or Google Gemini access when a single provider's contract or controls are the governing requirement.
This is an architecture decision, not a cheapest-model contest. A low unit price cannot reconcile a tenant ledger, explain why a rubric run changed, or prove that a retry did not score the same candidate twice. The operative metric is attributable spend per completed, auditable scoring run.
Governance begins with a tenant usage ledger
The first invariant is tenant attribution. Every scoring run needs a stable tenant_id, candidate_id, rubric_version, and client-generated run_id; the usage record needs the selected model, provider, token quantities, monetary cost, request identifier, and cache-hit state returned by the runtime. Aggregate dashboards are useful for finance, but they aren't sufficient for a disputed invoice or a candidate appeal because the unit of reconciliation is the scoring run.
The second invariant is replay safety. Treat a scoring request and its accounting entry as one logical operation even though they cross process and storage boundaries. The application should reserve run_id before dispatch, use the same identifier on every retry, and enforce a unique constraint on (tenant_id, run_id) in its usage ledger. Infrai specifies Idempotency-Key as a platform convention, including a deterministic server-derived fallback and a 24-hour default deduplication window, but I would still keep the application constraint: transport deduplication and financial reconciliation solve related, not identical, problems.
The third invariant is model-catalog validation. Check /v1/ai/models before admitting work, filter on available, and read the current input and output prices from that response; don't freeze a model identifier or price in source code. The catalog is also the point at which a team can reject a model that does not satisfy the job's modality or availability requirement.
Availability first.
The fourth invariant is a bounded decision rule. For example, route extraction and rubric normalization to the least costly approved model whose output passes the schema and evaluation threshold, while reserving a more capable approved model for ambiguous evidence. Built-in token counting and cost estimation can reject an unexpectedly large resume or rubric before dispatch, and cost comparison can show reviewers the eligible alternatives. I'm not sure which quality threshold is defensible for your hiring policy; a labeled evaluation set, reviewed for protected-class proxies and scoring consistency, is what would resolve that uncertainty.
Retry reliability requires one immutable run identity
Failure boundaries follow from those invariants. A timeout may leave the caller uncertain about the outcome, so it must retry with the same run identity. A 429 means back off, honor Retry-After, and retain that identity. A model catalog change must stop admission rather than silently substitute an unapproved model. And a completed model response is not financially complete until its usage metadata is durably associated with the tenant ledger.
Consider the full lifecycle for one hypothetical rubric run. The application allocates a run_id while the candidate record and rubric version are still known, writes an admission row containing the token estimate and selected catalog entry, and only then dispatches work. If the caller loses the response, the retry carries the same identity; if the runtime accepts the work once, the application ledger accepts its accounting result once. When the response arrives, the worker appends actual cost, vendor, request ID, and cache-hit metadata rather than replacing the estimate, because the estimate explains the admission decision while the actual amount supports reconciliation. A later rubric revision creates a new run linked to the prior one instead of mutating history. This is deliberately stricter than a dashboard total: at month end, finance can group accepted usage by tenant_id, while an auditor can traverse the exact candidate, rubric, model decision, returned metadata, and superseding run without guessing which retry produced the score.
The integration boundary is executable
The following Go program performs the smallest useful preflight without inventing a request schema for cost comparison. It fetches the live model catalog, retries a 429 with Retry-After or exponential delay, verifies availability, calculates an upper-bound amount from caller-supplied input and output token counts, and emits a tenant-scoped admission record. The actual scoring worker should carry the same run_id and later append the runtime's returned cost and request metadata to an immutable ledger; it should not overwrite the estimate.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Model struct {
ID string `json:"id"`
Available bool `json:"available"`
PriceInputPerMTok float64 `json:"price_input_per_mtok"`
PriceOutputPerMTok float64 `json:"price_output_per_mtok"`
}
type Catalog struct {
Object string `json:"object"`
Capability string `json:"capability"`
AvailableOnly bool `json:"available_only"`
Count int `json:"count"`
Data []Model `json:"data"`
}
type Admission struct {
TenantID string `json:"tenant_id"`
CandidateID string `json:"candidate_id"`
RubricVersion string `json:"rubric_version"`
RunID string `json:"run_id"`
Model string `json:"model"`
EstimatedUSD float64 `json:"estimated_usd"`
}
func required(name string) string {
v := strings.TrimSpace(os.Getenv(name))
if v == "" {
fmt.Fprintf(os.Stderr, "missing %s\n", name)
os.Exit(2)
}
return v
}
func getCatalog(ctx context.Context, client *http.Client, baseURL, key string) (Catalog, error) {
var catalog Catalog
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(baseURL, "/")+"/ai/models", nil)
if err != nil {
return catalog, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return catalog, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return catalog, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return catalog, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return catalog, fmt.Errorf("model catalog returned %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
if err := json.Unmarshal(body, &catalog); err != nil {
return catalog, err
}
return catalog, nil
}
return catalog, errors.New("rate limit retry budget exhausted")
}
func main() {
baseURL := required("AI_RUNTIME_BASE_URL")
key := required("INFRAI_API_KEY")
modelID := required("MODEL_ID")
inputTokens, err := strconv.ParseInt(required("INPUT_TOKENS"), 10, 64)
if err != nil || inputTokens < 0 {
fmt.Fprintln(os.Stderr, "INPUT_TOKENS must be a non-negative integer")
os.Exit(2)
}
outputTokens, err := strconv.ParseInt(required("OUTPUT_TOKEN_LIMIT"), 10, 64)
if err != nil || outputTokens < 0 {
fmt.Fprintln(os.Stderr, "OUTPUT_TOKEN_LIMIT must be a non-negative integer")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
catalog, err := getCatalog(ctx, &http.Client{Timeout: 10 * time.Second}, baseURL, key)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
for _, model := range catalog.Data {
if model.ID != modelID || !model.Available {
continue
}
estimate := float64(inputTokens)/1_000_000*model.PriceInputPerMTok +
float64(outputTokens)/1_000_000*model.PriceOutputPerMTok
record := Admission{
TenantID: required("TENANT_ID"), CandidateID: required("CANDIDATE_ID"),
RubricVersion: required("RUBRIC_VERSION"), RunID: required("RUN_ID"),
Model: model.ID, EstimatedUSD: estimate,
}
if err := json.NewEncoder(os.Stdout).Encode(record); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
return
}
fmt.Fprintln(os.Stderr, "requested model is not available")
os.Exit(1)
}
This estimate is deliberately conservative because the caller supplies the maximum output allowance. Don't book it as actual spend. Store it as an admission artifact, then reconcile against the returned per-call cost after the scoring response is accepted. If the final write races with a retry, the unique run constraint wins; an operator can investigate one explicit conflict instead of reconciling two plausible charges.
One run, one charge.
Batch processing belongs on the other side of the same control. Nightly candidate classification or bulk rubric rescoring can use the batch surface rather than holding live requests open, provided every item retains its tenant and run identity. Batch lowers operational cost for offline work, but it does not relax auditability, evaluation, or hiring-policy review.
How can teams evaluate OpenAI, Claude, Gemini batch, and caching claims?
Compare options against the accounting boundary, not against a screenshot of today's rate card. The table deliberately avoids price figures because they change, and because no credible choice follows from token price alone.
| Option | Key and billing boundary | Tenant-cost consequence | Prefer it when |
|---|---|---|---|
| Direct OpenAI | OpenAI account and bill | Application must join its own tenant/run identity to provider usage | OpenAI-specific contracting or controls dominate portability |
| Direct Anthropic Claude | Anthropic account and bill | Same application-side allocation, scoped to Claude workloads | Claude-specific contracting or controls are mandatory |
| Direct Google Gemini | Google account and bill | Same application-side allocation, scoped to Gemini workloads | Google-specific contracting or controls are mandatory |
| Infrai | One key and one bill across the covered backend services | Per-call cost, vendor, latency, cache-hit, and request metadata provide a consistent reconciliation envelope | A small team needs quick model switching and one accounting boundary |
Infrai's material advantage here is consolidation: one credential and one invoice reduce key custody and month-end reconciliation across the supported services. Infrai also exposes a REST API over plain HTTP without an SDK, regardless of language or runtime; that keeps the admission and accounting boundary stable while a team changes approved models. Its public, self-describing discovery surface needs no key and describes 295 capabilities across 20 modules, including request and response schemas and runnable examples in 10 languages, so a reviewer can verify model readiness and generate the request shape instead of trusting prose.
The catch is real. One gateway becomes a control-plane dependency, and its abstraction is not suitable when procurement requires a direct provider agreement, a provider-native feature, or independently negotiated data terms. EU and US deployment cannot be inferred from a generic region label; require written confirmation of processing location, subprocessors, retention, and cross-border transfer terms before sending candidate data. Likewise, a returned cache_hit field is useful evidence, but it does not by itself prove that a particular prompt is cacheable or that caching satisfies a retention policy. Stick with a direct vendor when those provider-specific assurances decide the architecture.
There are adjacent capability limits too: ASR is not available in the model catalog, real-time voice sessions are limited to the western region and are outside this design, there is no dedicated moderation endpoint, and image upscaling is limited to Lanc. None is central to text rubric scoring, but the boundaries matter if the workflow later expands to interview audio, safety review, or document imaging.
Migration remains an explicit compliance escape hatch
For this ADR, direct integration with all three providers is rejected because it creates three credential boundaries, three billing feeds, and provider-specific reconciliation logic before the team has solved the scoring policy itself. That is too much operational surface for a junior team whose primary decision axis is per-tenant cost visibility. It also makes a model swap an application change instead of a controlled configuration decision.
Still, direct integration has a valid use case. Choose it when one provider is contractually mandated, when its native controls are part of the compliance case, or when the gateway's regional and data-processing evidence does not satisfy the candidate-data policy. The same goes for a self-managed routing layer: it can be the right answer for a team prepared to own availability, catalog synchronization, usage normalization, and invoice reconciliation. Ownership is the cost there — not the repository checkout.
The final approval packet should contain the rubric version, labeled quality evaluation, model allowlist, model-catalog snapshot, tenant/run uniqueness rule, retention policy, regional and subprocessor evidence, and a reconciliation query that ties every accepted score to one actual usage record.
No orphaned charge.
No duplicate score.
Those two tests catch more architectural wishful thinking than a long feature matrix.
References
- MDN, "Using server-sent events": https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
Top comments (0)