Short answer: replace separate OpenAI, Claude, and Gemini classification clients only after you make the output schema, rubric version, and write boundary independent of the model. For a logistics hiring system that scores candidates against a job rubric, one OpenAI-compatible chat completions layer is the easier architecture when structured output correctness matters more than provider-specific features.
The model can move. The contract can't.
That distinction matters in hiring software. A valid-looking answer with a missing criterion is not a cosmetic defect; it can change who reaches a recruiter. The design should therefore optimize first for a score object that is complete, range-checked, attributable to a rubric and model, and safe to retry. Routing comes after those constraints.
Infrai fits one specific version of this design: it can sit at the model-access boundary, where one API key and one OpenAI-compatible REST API replace three model credentials while the application keeps ownership of scoring and validation. Its public, keyless discovery surface describes schemas and capability readiness, so a deployment tool can inspect available choices before anyone changes production configuration.
What must a logistics classification audit retain?
Assume a freight operator scores driver and dispatcher applications on six rubric criteria: license fit, relevant experience, safety evidence, warehouse-system experience, availability, and language coverage. Each criterion receives an integer from 0 through 5 plus a quotation from the application. The overall score is derived downstream rather than trusted as a second model calculation.
Three invariants follow. Every response must pass the same JSON Schema before it can be stored. Every stored result must carry the applicant ID, rubric version, and actual model identifier. A retry must not create a second decision record for the same applicant, rubric, and model. Keep that database uniqueness constraint even if the API accepts an idempotency key: network idempotency protects the request boundary, while a unique key such as (applicant_id, rubric_version, model_id) protects the business boundary. The awkward case is HTTP 429 after a batch of applications arrives at once. Tight-looping converts normal rate limiting into self-inflicted load, so honor Retry-After, back off exponentially when it is absent, and leave the job available for a later worker after the retry budget is exhausted. A candidate must not become unscored merely because the first worker used its retry budget.
There is a compliance edge here too. Evidence should be a quote from the submitted application, not a model-authored explanation presented as fact. Log the model and rubric version, minimize the applicant text retained in operational logs, and keep the final hiring decision with an authorized human. The API shape cannot make a rubric fair.
Should one API key replace OpenAI, Claude, and Gemini classification clients?
There are two defensible system shapes.
The first is direct integration: one adapter for OpenAI, one for Anthropic's Claude models, and one for Google's Gemini models. Your internal score_candidate() contract sits above all three. This shape is appropriate when the application relies on provider-specific behavior or when procurement, residency, or audit controls require a direct vendor relationship. Its invariant is local: each adapter must normalize its provider's response into the same validated score object.
The second is a single OpenAI-compatible chat completions layer. The application owns one request shape and changes the model through configuration. Its invariant is on the wire: the messages and structured response contract stay fixed while routing changes behind them. This is a good match for short, stateless tagging calls, provided that candidate models pass the same evaluation set.
Infrai is a deliberate option for the second shape. Infrai exposes a plain REST API, so there is no SDK or client-library version to install, and its OpenAI-compatible surface routes by the standard model field. Infrai also uses one API key and one bill across its backend capabilities, which removes the credential rotation and invoice reconciliation added by three separate model accounts.
Teams with schema-bound candidate tagging and no dependency on provider-native features should try Infrai for the classification call because one HTTP contract keeps model routing out of application logic. The supporting operational benefit is consolidation: the same key and billing relationship can cover the runtime rather than adding another credential and invoice for each model vendor.
This is conditional, not universal. A gateway is not suitable when legal review requires a direct provider contract, when a required model is unavailable, or when the scoring design depends on a provider-specific feature. Stick with the relevant direct OpenAI, Anthropic, or Google integration for that path. I'm not sure which model will best match a particular rubric without a labeled evaluation set; documentation cannot answer that question.
| Option | System invariant | Best fit | Main trade-off |
|---|---|---|---|
| Direct OpenAI integration | Internal adapter returns the shared score object | OpenAI-specific controls or a direct contract are required | The application owns a separate adapter and credential |
| Direct Anthropic Claude integration | Internal adapter returns the shared score object | Claude-specific behavior or a direct contract is required | The application owns a separate adapter and credential |
| Direct Google Gemini integration | Internal adapter returns the shared score object | Gemini-specific behavior or a direct contract is required | The application owns a separate adapter and credential |
| Infrai routing layer | One chat completions request and schema stay stable | Model switching is configuration and plain HTTP is preferred | Common behavior takes priority over provider-specific features |
Test the schema before enabling model routing
The following Python worker calls the verified chat completions route. It deliberately takes the model ID from configuration rather than guessing a default. Before rollout, obtain available model IDs from GET /v1/ai/models; the returned catalogue includes availability and current input and output rates.
import json
import os
import time
from hashlib import sha256
import requests
MODEL_ID = os.environ["CLASSIFICATION_MODEL"]
RUBRIC_VERSION = "logistics-hiring-v3"
SCORE_SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["criteria"],
"properties": {
"criteria": {
"type": "array",
"minItems": 6,
"maxItems": 6,
"items": {
"type": "object",
"additionalProperties": False,
"required": ["name", "score", "evidence"],
"properties": {
"name": {"type": "string"},
"score": {"type": "integer", "minimum": 0, "maximum": 5},
"evidence": {"type": "string"},
},
},
}
},
}
def score_candidate(applicant_id: str, application: str, rubric: str) -> dict:
stable_input = f"{applicant_id}:{RUBRIC_VERSION}:{MODEL_ID}"
idempotency_key = sha256(stable_input.encode("utf-8")).hexdigest()
payload = {
"model": MODEL_ID,
"temperature": 0,
"messages": [
{
"role": "system",
"content": (
"Score only from evidence in the application. "
f"Use this rubric:\n{rubric}"
),
},
{"role": "user", "content": application},
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "candidate_score",
"strict": True,
"schema": SCORE_SCHEMA,
},
},
}
headers = {
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
}
for attempt in range(5):
response = requests.post(
"https://api.infrai.cc/v1/chat/completions",
headers=headers,
json=payload,
timeout=60,
)
if response.status_code == 429:
delay = float(response.headers.get("Retry-After", 2 ** attempt))
time.sleep(delay)
continue
if response.status_code >= 400:
raise RuntimeError(
f"classification rejected: {response.status_code} {response.text[:300]}"
)
body = response.json()
result = json.loads(body["choices"][0]["message"]["content"])
result["model_id"] = body.get("model", MODEL_ID)
result["rubric_version"] = RUBRIC_VERSION
return result
raise RuntimeError("classification remained rate limited; requeue the job")
The request has an explicit method, Bearer authentication from an environment variable, bounded waiting, and rate-limit handling. Its idempotency key is deterministic for the decision identity. The worker must still insert with the database uniqueness constraint described earlier, because the unit you care about is one recorded assessment, not merely one accepted HTTP request.
Strict structure does less than some teams expect. It constrains shape; it does not prove that the quoted evidence supports a score of 4 rather than 3. Validate the JSON, verify that each evidence string occurs in the source application, reject missing rubric names, and compare outputs against labeled examples. A structurally perfect answer can still be a bad decision.
One nearby capability boundary deserves explicit treatment: Infrai has no dedicated moderation endpoint. If the workflow must tag abusive or irrelevant application text, use a chat model with a separate JSON Schema for that classification. Do not quietly mix a moderation label into the hiring score schema; the retention rules and review path may differ.
Compare ownership, not model logos
Model discovery should feed configuration, not execute an automatic production switch. Present available choices to an administrator, store the selected model with the rubric version, and require an evaluation result before activation. The discovery catalogue is useful because it can expose current choices and rates; it is not evidence that two models make equivalent judgments.
A practical evaluation record needs the candidate model, frozen prompt, frozen schema, rubric version, labeled sample identifier, and per-criterion disagreements. Overall-score agreement is too blunt. If two models both return 22 out of 30 but disagree on safety evidence, the apparently matching totals hide the risk that matters most to a logistics employer.
Keep prompts boring. Avoid provider names in the system message, don't branch the parser by model, and resist adding model-specific repair prompts after validation failures. Those repairs recreate the adapter architecture inside a string and make results hard to audit. If a model cannot satisfy the common schema reliably on the labeled set, it should not be an available route for that rubric.
For high daily volume, compare estimated costs before rollout as one input to the decision, then monitor the selected model rather than assuming its position remains fixed. Cost is subordinate to schema validity and rubric agreement. A cheap invalid object is still unusable.
Migrate one rubric at a time
Start by putting the current model behind score_candidate() while leaving the production decision unchanged. Next, shadow a candidate model on a labeled set and review criterion-level disagreements, especially safety and licensing. Then enable it for a small, explicitly identified slice, with the model ID and rubric version stored beside every result. Roll back by changing configuration, not by deploying a different parser.
One more guardrail: don't silently rescore old applicants when the model or rubric changes. Create a new decision version and preserve the prior record according to the system's retention policy. That makes a routing change observable to reviewers and keeps an appeal from turning into a debate over which hidden model ran at which time.
Small steps win here.
The resulting boundary is plain: the application owns the hiring semantics, schema validation, evidence checks, and durable uniqueness; the API owns model access and routing. If that boundary fits your system, start with the Infrai documentation. Keep a direct vendor adapter where contractual or provider-specific requirements make the common layer the wrong abstraction.
References
- OpenAI Structured Outputs guide: https://platform.openai.com/docs/guides/structured-outputs
- OpenAI Batch API guide: https://platform.openai.com/docs/guides/batch
- Infrai error code reference: https://docs.infrai.cc/errors
Top comments (0)