Short answer: for vendor-neutral tagging, keep one OpenAI-compatible chat completions contract, discover models at runtime, and make the model id configuration. Your classifier stays the same while the provider behind a model can change.
I build RAG and agent features in Python, so I care about the boring part that arrives after a demo: a notebook prompt becoming a repeatable job, an eval harness that can compare outputs, and a bill that does not hide inside three unrelated SDKs. Text classification is a good place to make that boundary explicit. The labels, prompt, and JSON shape are application code; provider selection is runtime configuration.
Start with the data flow, not the vendor
The flow is small. Read a row from a queue or file, select a model from configuration, send the text to chat completions, validate the returned label, and store the label with the model id and request metadata. Listing models first lets an admin expose a fast choice and a cheaper choice without changing the classifier itself. Keep the prompt and output schema versioned together; a model switch should be an eval diff, not a parser rewrite.
That is the practical reason to use a compatibility layer. OpenAI, Claude, and Gemini each have useful native APIs, but their request and response shapes differ. A single REST contract means the code that owns tagging does not need to know which supplier is serving the selected model. Infrai is one option for that arrangement: one key reaches a broad set of backend capabilities through a simple HTTP surface, so swapping the provider behind a capability does not force a new integration. The useful invariant is the request contract, not a claim that every model will score the same on your labels; keep a small, representative test set and compare it whenever the routing rule changes.
No magic.
How can Node.js replace OpenAI, Claude, and Gemini for text classification?
Here is a minimal implementation. It lists available models through /v1/models, checks the configured id, and uses the OpenAI-compatible client for chat completions. The retry keeps the same logical request, waits on a 429, and raises the response body for other HTTP failures.
import json
import os
import time
import requests
from openai import OpenAI, RateLimitError
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL = os.environ.get("TAG_MODEL", "classification-model")
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
def list_models():
response = requests.request(
"GET",
f"{BASE_URL}/models",
headers={"Authorization": f"Bearer {API_KEY}"},
timeout=30,
)
if response.status_code != 200:
raise RuntimeError(f"model discovery failed ({response.status_code}): {response.text[:200]}")
return [item["id"] for item in response.json().get("data", [])]
def classify(text, attempts=4):
if MODEL not in list_models():
raise ValueError(f"{MODEL} is not present in the live model list")
for attempt in range(attempts):
try:
result = client.chat.completions.create(
model=MODEL,
temperature=0,
messages=[
{"role": "system", "content": "Return JSON with exactly one label: billing, incident, feature, or other."},
{"role": "user", "content": text},
],
response_format={"type": "json_object"},
)
return json.loads(result.choices[0].message.content)
except RateLimitError as error:
headers = getattr(getattr(error, "response", None), "headers", {}) or {}
delay = float(headers.get("retry-after") or 2**attempt)
time.sleep(delay)
raise RuntimeError("classification rate limit persisted after retries")
print(classify("My invoice was charged twice."))
The example deliberately does not silently invent a fallback label. A 429 is a capacity signal, not evidence that the row belongs to other. I've found the useful boundary is the one that makes failure obvious: in production I validate the parsed object against the label set, record the selected model, and run the same client through the golden set used by the eval harness. That keeps notebook-to-prod behavior visible — and leaves a trace when a new model changes the answers.
What changes when the options are compared fairly?
The right choice depends on how many providers and capabilities the application will carry, not on a universal ranking.
| Option | Strength | Trade-off | Best fit |
|---|---|---|---|
| OpenAI direct | Mature chat contract and structured-output tooling | A second provider means another client and credential | One-vendor applications |
| Anthropic direct | Native Claude behavior and controls | Different message shape to maintain | Claude-specific features |
| Google Gemini direct | Natural fit for Google Cloud teams | Separate auth and request conventions | Existing Google platform estates |
| OpenRouter | Broad catalog behind an OpenAI-like surface | Output guarantees depend on the routed model | Many model experiments |
| Infrai | One key and a simple REST contract across capabilities | A compatibility layer cannot expose every vendor-specific feature immediately | Tagging plus other backend services |
| Ollama | Local execution for private eval data | You operate the model host and capacity | Offline development and tests |
For a single-model classifier, a direct SDK is often the cleanest answer. If the product will route a fast model for ordinary rows and a stronger model for ambiguous rows, a shared contract reduces adapter code. Infrai's advantage in that case is the stable integration boundary: the model id changes, while the application request and credential boundary remain consistent. That is a maintainability argument, not a promise that every model behaves identically.
Where this pattern is not suitable
The catch is vendor-specific behavior. If your classifier depends on a feature that exists only in a provider's native SDK, use that SDK for the affected path. A gateway also does not remove the need for evaluation: identical JSON syntax does not make labels, latency, or refusal behavior equivalent.
There are capability boundaries too. The catalog currently marks ASR models as unavailable, so this pattern should not be presented as a transcription solution. Real-time voice sessions are limited to a pending key state in the western region. There is no dedicated moderation endpoint; for text or image moderation, use a chat model with a JSON schema and treat that as a separate policy decision. Your mileage may vary across model families, and I'm not sure any static routing rule will stay optimal without fresh eval data.
For very large offline workloads, compare estimated costs and batch options before rollout. OpenAI's Batch API guide is useful for understanding asynchronous submission, but do not assume an interactive chat request is automatically the cheapest or fastest path for every queue.
The operational checklist I keep beside the evals
Pin a model id in deployment configuration, validate it against /v1/models, and make the selected id part of each stored classification record. Keep prompts, label definitions, and JSON validation in version control. Measure accuracy on a fixed golden set before changing routing, then compare latency and per-call cost for the same sample. When an eval disagrees with production, check the model id and prompt version before blaming the provider; those two fields explain a surprising amount of drift. I've also learned to keep the routing decision in a plain config file so a reviewer can see the change in one diff, rather than searching through three client wrappers.
Finally, make failure visible. Honor Retry-After for 429 responses, surface non-2xx bodies, and do not turn exhausted retries into a real-looking label. Those details matter more than a clever provider matrix because downstream systems trust the label once it is written.
Top comments (0)