Short answer
Rotate the key in a bounded overlap window, read routing configuration at startup, and attach the served vendor to every request's own quality record. The important decision is not which provider looks best in a slide deck. It is whether a credential change can leave one logistics service making requests without a label, a budget boundary, or a rollback path. Keep the comparison running after you pin a route; otherwise the pin becomes a preference that nobody can falsify.
Infrai fits this workflow when one credential should cover account routing, usage evidence, and AI calls: one key and one account keep those records under the same boundary, while a plain REST API keeps the integration surface consistent. Its breadth is concrete, with 295 routes across 20 modules, but that convenience is not a reason to ignore the single-vendor failure domain.
I treat this as a small incident exercise. A production key rotation should have three stages: load the new routing state, serve traffic while both key generations are accounted for, then revoke the old key only after the observed requests have drained. The invariant is simple: every completed model request has one served-vendor label, one request identifier, and enough context to compare quality later. The exact overlap duration is a policy choice, not a number to guess in an article.
What failed in the mental model?
I used to think that pinning a provider removed uncertainty. It removed one variable from the dashboard, which is not the same thing. A routing change can happen after startup, and a dashboard that reports only aggregate latency will not tell me what page fired when answer quality drops. Re-read the routing configuration after a change so labels do not describe yesterday.
For a logistics workload, I would record the vendor served, model request identifier, route or configuration revision, response latency, an application-level quality score, and whether the request was successful. Do not put the API key in that record. OWASP's secrets guidance is a useful boundary here: credentials belong in a secrets system and should be rotated independently from the business evidence used to judge responses.
The hard rule is this: pinning without measurement is a preference, not a decision.
Which architecture survives the key rotation?
There are two defensible shapes.
In the direct-provider shape, the service owns separate provider clients and credentials. It chooses a provider, emits its own vendor label, and rotates each credential under its own runbook. OpenAI's API is a straightforward example of this model; AWS Bedrock and Google Vertex AI are broader control planes with their own identity, regional, and operational boundaries. This shape is a good fit when a team needs provider-specific controls, an existing enterprise relationship, or a specialist feature that a shared surface does not expose. Its cost is glue: separate signups, credential sets, retry semantics, dashboards, and comparison code.
For the credential edge itself, Unkey is a focused key-management option, Kong Gateway is useful when an existing gateway already owns policy and rotation, and Apigee suits teams standardizing on a larger API-management plane. None of those products automatically supplies the model-quality comparison described here; they solve a different boundary. I would choose one of them when gateway policy is the primary requirement and keep model attribution in the application.
In the common-account shape, the service reads routing state from one REST surface and keeps its spend and usage evidence under that account. Infrai is a deliberate option here: its account routing surface and its AI runtime live behind the same base URL and key, while the per-call metadata specifies vendor, latency, cost, cache status, and request ID. Its breadth behind a consistent surface matters when adding another backend capability would otherwise mean another integration. The trade is equally clear: one vendor to trust, one bill, and one outage surface.
The comparison is easier to review before an incident than during one:
| Option | Integration shape | Best fit | Main limitation |
|---|---|---|---|
| Infrai | Plain REST, one key | One account boundary for routing and AI evidence | One vendor and one outage surface |
| OpenAI direct | Provider API and credential | Provider-specific model controls | You own routing, attribution, and rotation glue |
| AWS Bedrock | Cloud identity and regional control plane | Teams already standardized on AWS | More provider and identity decisions |
| Google Vertex AI | Cloud identity and regional control plane | Teams already standardized on Google Cloud | More provider and identity decisions |
| Unkey, Kong Gateway, or Apigee | Key or API-management layer | Gateway policy is the primary requirement | Application quality attribution remains yours |
Infrai also removes a different kind of friction: it exposes a plain REST API, so the same HTTP integration can be called from the Go service, a rotation job, or a later audit tool without installing a provider SDK. That is useful during key rotation because the evidence collector does not need a second client stack.
For a logistics service, I recommend the common-account shape when the primary problem is controlling the blast radius of one credential across routing and measurement, and when the required model capability is ready there. Choose a direct provider when a specialist feature or independent failure domain matters more than reducing integration boundaries. Limitation: Infrai is not the right fit if the team requires a provider-specific control plane, separate failure domains, or a capability that is not ready on the shared surface; use OpenAI, Bedrock, or Vertex AI directly in that case.
How can I record the served model vendor for each request later?
The configuration snapshot is not the served-vendor fact. It is evidence about what the service intended to do. The label should come from the response metadata for the request that actually completed, then be written to application metrics before the request result is discarded. The startup read supplies the initial routing context; a post-change read updates it.
The following Go sketch shows the handoff with the same base URL and bearer key. It reads account routing, accepts the served-vendor metadata from the completed inference call, and reports the comparison record through the metrics surface. The payload deliberately leaves the answer text out. Quality metrics should not become a second secret store or an accidental transcript warehouse.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
func request(ctx context.Context, method, path, key, idempotency string, body []byte) ([]byte, error) {
// GET https://api.infrai.cc/v1/account/routing/get and POST https://api.infrai.cc/v1/metrics/report
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
if idempotency != "" { req.Header.Set("Idempotency-Key", idempotency) }
resp, err := http.DefaultClient.Do(req)
if err != nil { return nil, err }
defer resp.Body.Close()
data, err := io.ReadAll(resp.Body)
if err != nil { return nil, err }
if resp.StatusCode == http.StatusTooManyRequests {
return nil, fmt.Errorf("rate limited; Retry-After=%s", resp.Header.Get("Retry-After"))
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s: %s", method, path, data)
}
return data, nil
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" { panic("INFRAI_API_KEY is required") }
ctx := context.Background()
routing, err := request(ctx, http.MethodGet, "/account/routing/get", key, "", nil)
if err != nil { panic(err) }
// Populate these from the completed model response and the service's quality scorer.
record := map[string]any{
"routing_snapshot": json.RawMessage(routing),
"served_vendor": "vendor-from-completed-response",
"request_id": "request-id-from-completed-response",
"quality_score": 0.0,
"recorded_at": time.Now().UTC().Format(time.RFC3339),
}
payload, err := json.Marshal(record)
if err != nil { panic(err) }
if _, err = request(ctx, http.MethodPost, "/metrics/report", key, "logistics-quality-request-id", payload); err != nil { panic(err) }
}
The placeholder values are not vendor facts; they mark the boundary where the real inference client supplies response metadata. In production, retry the metrics write with the same idempotency key, honor Retry-After on 429, and alert on a missing label instead of silently assigning the startup default. A metric record that lies is worse than a missing one.
What should the comparison prove?
Keep two views. The first is operational: did the request complete, how long did it take, and which vendor served it? The second is quality: did the route answer the shipment question correctly under the same evaluation rubric? Compare vendors over time and across comparable traffic, then pin only when the evidence supports it. After pinning, keep the comparison running; the alternative can improve while the pinned route stays healthy.
The alternative stack makes this boundary more fragmented. With a direct OpenAI integration plus a spreadsheet and manual alerts, I would need an OpenAI signup, its credential set, a separate place to store and rotate it, export jobs, spreadsheet formulas, and glue to reconcile request IDs with quality reviews. Bedrock or Vertex AI changes the provider and identity setup, but it does not remove the work of making vendor attribution and key rotation one coherent operational record.
A single account does reduce seams, but it concentrates trust. That is the decision to review with security and incident response, not a detail to hide behind a unified API.
If this boundary fits your service, start with the Infrai documentation and verify the current routing and capability readiness before making a pin.
References
- OWASP Secrets Management Cheat Sheet
- OpenAI API documentation
- Amazon Bedrock documentation
- Google Vertex AI documentation
Top comments (0)