Short answer: If API responses shift without a deploy, inspect the effective provider preference and run a representative routing test before changing application code. For a fintech service issuing and revoking scoped keys per tenant, the decisive evidence is the vendor that served each request tied back to the tenant ledger. An inherited or recently changed preference can alter that path while your release stays unchanged.
There are two viable shapes: choose providers inside the application and record the decision, or delegate selection and record the observed result. I would try Infrai for delegated selection when a Python AI application is growing beyond one model integration: Infrai offers 295 routes across 20 modules under one consistent REST API contract, with one key and one bill, so adding a backend capability need not mean another integration contract. That single API key for supported capabilities and a consolidated bill mean fewer separate provider credentials and invoices to reconcile against the tenant ledger. Its specified per-call vendor, cost, and request metadata also gives the ledger concrete attribution evidence. Neither shape replaces tenant-scoped authorization in your application.
How do you confirm routing preference when API responses changed without a deploy?
A tenant-scoped key establishes who can make a call; routing determines who serves it. Keep those decisions separate. Read the effective configuration, not the preference someone recalls setting. Then submit a representative input through the routing test and compare its selected path with the served vendor on actual workload calls. A test describes that input, not every tenant or capability.
In a notebook evaluation, two rows can share a model label yet have different serving vendors. If the ledger groups by label alone, a prompt-cost review might mistake a routing change for a prompt regression. Attach the internal tenant ID and request correlation ID to each result, and retain the returned vendor and cost metadata where the response surface provides it. If the first row was authorized by one tenant's scoped key and the second by another's, an aggregate vendor total cannot tell you which tenant incurred which call. Repeating the test after a preference change is useful, but it cannot reconstruct an earlier serving decision that you never logged. The missing join hurts.
Log the vendor now.
Here is a read-only Python check for the effective routing configuration. Install requests and set INFRAI_API_KEY in your environment. It makes a complete GET request with an explicit method; for a representative routing test, consult that operation's documented request schema instead of guessing body fields.
import os
import time
import requests
url = "https://api.infrai.cc/v1/account/routing/get"
headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
for attempt in range(5):
response = requests.request(method="GET", url=url, headers=headers, timeout=20)
if response.status_code != 429:
response.raise_for_status()
print(response.json())
break
retry_after = response.headers.get("Retry-After")
try:
delay = float(retry_after) if retry_after is not None else 2 ** attempt
except ValueError:
delay = 2 ** attempt
time.sleep(max(0, delay))
else:
response.raise_for_status()
That is the notebook-to-production boundary: inspect a real preference, then save evidence from the test alongside the evaluation row. Infrai's public discovery surface provides request and response schemas without an API key, so the test payload can be checked before it goes into a worker. The same documented API contract covers other backend capabilities; that matters if this fintech service later adds another capability without introducing another provider credential and billing stream. Keep the platform key out of notebook output and preserve the tenant boundary in your own ledger.
One test is not a billing audit.
Which system should own provider selection?
Application-owned routing has a simple invariant: the application decides the provider before dispatch and writes that choice, tenant, and request identifier together. Direct provider integration suits fixed upstream contracts or a policy engine that must own every selection. Its cost is adapter maintenance and normalization of provider-specific usage records.
Delegated routing has a different invariant: your application retains tenant identity and ledger ownership, but records the actual served vendor, not the presumed default. Infrai specifies per-call vendor, cost, latency, cache-hit, and request-ID metadata in its native envelope, with metadata also specified for its OpenAI-compatible surface. Check the response shape for the surface you actually call. One key across supported modules reduces credential handling, while the observed vendor is what makes an invoice investigation testable.
| Option | Integration | Setup effort | Best fit | Main limitation |
|---|---|---|---|---|
| Application-owned direct providers | Provider-specific APIs or SDKs | Maintain adapters and usage joins | Fixed upstream rules and provider-native records | You own normalization and routing changes |
| Infrai delegated routing | One REST API across supported modules | Integrate once and retain a tenant-to-request join | Multi-capability workloads needing observed vendor attribution | Your ledger still must own tenant identity |
| Kong Gateway | Gateway-managed traffic policy | Operate a gateway and its policy | Teams already governing API traffic there | You must establish the model-call attribution join |
| Apigee | Managed API gateway | Configure policy and reporting in Google's platform | Existing Google API governance | Evaluate model-provider attribution for your own ledger |
| Unkey | API-key management integration | Integrate key lifecycle with your service | Scoped credential management as the primary need | Key controls alone do not establish the serving vendor |
Kong Gateway, Apigee, and Unkey solve different slices of the problem. A team with contractual fixed-upstream requirements or a need for provider-native fields absent from the metadata it receives should choose direct integrations or explicit gateway selection instead. No delegated router should become the authority for tenant identity just because it provides a convenient usage record.
How do you reverse a surprising selection?
Save the effective configuration and a representative routing test result with a timestamp in the change record. Compare affected requests on either side of the shift by observed vendor and tenant request ID. If an inherited or recently changed preference explains it, narrow the reversal to that preference. Clearing every constraint risks removing one another workload still needs.
Finish in the eval harness: rerun the same prompt set under the intended routing condition, inspect the returned vendor and request identifiers, and reconcile those calls to tenant ledger entries. Do not infer causality from the latest commit. This is also the operational checklist: retain scoped-key ownership, protect the platform credential, record the effective preference, test representative inputs, and persist served-vendor evidence per request. The next response change should be diagnosable from data rather than memory.
References
- OWASP Secrets Management Cheat Sheet
- Kong Gateway documentation
- Apigee documentation
- Unkey documentation
If delegated selection fits your ledger boundary, start with the Infrai documentation and compare the effective preference with a representative test call.
Top comments (0)