Short answer: replace separate provider clients with one scoped classification API key and a strict JSON contract, keep chat completion model routing behind aliases, and require every route change to pass the same eval suite.
The deciding trade-off is control versus convenience. A Node.js application can submit the same label request every time while an internal gateway selects a chat completion model. The important part isn't the shared key. It is the eval suite and output schema that make a route change measurable and reversible.
This is a small architecture, on purpose.
If an application currently has separate OpenAI, Claude, and Gemini branches, replacing the SDK switch statements is only half the job. A useful migration also normalizes labels, refusal handling, retry ownership, observability, and the criteria for promoting a model. Otherwise, one API key merely hides three behaviors behind one URL.
How should a Node.js text classification API handle chat completions and model routing?
Treat the Node.js side as a caller of a stable internal classification contract. It sends an item identifier, text, taxonomy version, and routing profile such as interactive or batch. A Python classification worker turns that request into a chat completion, asks for a strict JSON object, validates the response, and returns a provider-neutral result. The routing layer maps the profile to a concrete model. Application code never receives a provider model ID and never chooses one.
The flow is straightforward: the product emits an item, the classifier builds a versioned prompt, the gateway routes the request, the classifier validates the returned label, and an event records the result plus the versions that produced it. That last record is what lets a team replay yesterday's examples against tomorrow's candidate route. Without it, model routing is guesswork — quick to configure, impossible to defend.
| Boundary | Credential shape | Best fit | Main limitation |
|---|---|---|---|
| Direct provider clients | One key per provider | Workloads that need native controls | Application owns divergent clients and response handling |
| Shared compatibility gateway | One scoped gateway key | Closed-schema classification across routes | Common contract may omit native features |
| Internal inference service | One internal service credential | Restricted data or self-managed models | Team owns deployment and capacity |
Keep the output contract deliberately boring. For a support-ticket example, it might contain label, confidence, and reason, with label restricted to a closed set. Confidence is useful for queues and review thresholds, but it should not be treated as calibrated probability until an eval demonstrates calibration on the application's own data. I'm not sure a model's self-reported confidence is useful at all for some taxonomies; a held-out confusion matrix resolves that question better than intuition.
Run the contract before debating routes
The following worker is a compact implementation of that boundary. It uses one endpoint and one credential, while the routing_profile chooses an alias rather than a vendor model name. The endpoint is supplied as configuration because the code should work with an internal gateway, a managed compatibility layer, or a service the team operates itself.
from __future__ import annotations
import json
import os
import urllib.error
import urllib.request
from dataclasses import dataclass
from typing import Literal
Label = Literal["billing", "bug", "feature_request", "other"]
ALLOWED_LABELS = {"billing", "bug", "feature_request", "other"}
ROUTES = {
"interactive": "classifier-fast",
"batch": "classifier-throughput",
}
@dataclass(frozen=True)
class Classification:
label: Label
confidence: float
reason: str
def validate_result(raw: object) -> Classification:
if not isinstance(raw, dict):
raise ValueError("E_CLASSIFY_SHAPE: result must be an object")
label = raw.get("label")
confidence = raw.get("confidence")
reason = raw.get("reason")
if label not in ALLOWED_LABELS:
raise ValueError(f"E_CLASSIFY_LABEL: unexpected label {label!r}")
if not isinstance(confidence, (int, float)) or isinstance(confidence, bool):
raise ValueError("E_CLASSIFY_CONFIDENCE: confidence must be numeric")
if not 0 <= float(confidence) <= 1:
raise ValueError("E_CLASSIFY_CONFIDENCE: confidence must be between 0 and 1")
if not isinstance(reason, str) or not reason.strip():
raise ValueError("E_CLASSIFY_REASON: reason must be non-empty")
return Classification(
label=label,
confidence=float(confidence),
reason=reason.strip(),
)
def classify(text: str, routing_profile: str = "interactive") -> Classification:
if routing_profile not in ROUTES:
raise ValueError(f"unknown routing profile: {routing_profile}")
schema = {
"type": "object",
"properties": {
"label": {"type": "string", "enum": sorted(ALLOWED_LABELS)},
"confidence": {"type": "number", "minimum": 0, "maximum": 1},
"reason": {"type": "string"},
},
"required": ["label", "confidence", "reason"],
"additionalProperties": False,
}
payload = {
"model": ROUTES[routing_profile],
"messages": [
{
"role": "system",
"content": (
"Classify the text using the supplied label set. "
"Base the result only on the text."
),
},
{"role": "user", "content": text},
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "classification",
"strict": True,
"schema": schema,
},
},
}
request = urllib.request.Request(
os.environ["CLASSIFIER_URL"],
data=json.dumps(payload).encode("utf-8"),
headers={
"Authorization": f"Bearer {os.environ['CLASSIFIER_API_KEY']}",
"Content-Type": "application/json",
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=20) as response:
envelope = json.load(response)
except urllib.error.HTTPError as error:
if error.code == 429:
raise RuntimeError("E_CLASSIFY_RATE_LIMIT: retry through the job queue") from error
raise RuntimeError(f"E_CLASSIFY_UPSTREAM_{error.code}") from error
content = envelope["choices"][0]["message"]["content"]
return validate_result(json.loads(content))
The schema request follows the Structured Outputs pattern: constrain the model's response, then validate again at the application boundary. Don't remove the second check. Network services change, routes can be misconfigured, and old cached results may predate the current schema. The validator turns all of those cases into explicit data-contract failures instead of letting an unfamiliar label leak into analytics.
One detail deserves extra attention. The sample sends a routing alias in the model field, so the gateway must own the alias mapping. That mapping should be reviewed and deployed like code, with a previous version available for rollback. If aliases are edited manually in an untracked dashboard, the application has gained convenience and lost reproducibility.
Let evals choose the model route
A candidate route should earn promotion on a labelled dataset that resembles production, including the awkward cases. Start with examples that distinguish adjacent labels, empty or extremely short inputs, multilingual text if the product accepts it, prompt-injection attempts, and items that genuinely belong in other. Split examples by taxonomy version because a label renamed last month cannot be scored fairly against a prompt written for the old definition. The primary report should show per-label precision and recall, a confusion matrix, invalid-output rate, refusal rate, latency distribution, and input/output token counts. Aggregate accuracy alone can conceal the failure that matters: a classifier can look excellent by predicting the dominant label while missing every rare billing escalation, and a route that improves the headline score may quietly make human-review volume unacceptable. Token counts belong next to quality because prompt edits often shift both, and a notebook experiment that omits them leaves a surprise for production. Still, don't collapse quality, latency, and token use into one magic score. Set separate acceptance thresholds, inspect the worst confusion pairs, and reject a candidate that violates any hard boundary even if its average is attractive. The minimal scorer below doesn't pretend to replace a full evaluation harness; it makes one slice of that promotion rule visible and testable before the notebook grows into a scheduled job.
No shortcuts.
from collections import Counter
from dataclasses import dataclass
@dataclass(frozen=True)
class EvalRow:
expected: str
predicted: str
valid: bool
def eval_summary(rows: list[EvalRow]) -> dict[str, object]:
if not rows:
raise ValueError("evaluation set must not be empty")
correct = sum(row.expected == row.predicted for row in rows)
invalid = sum(not row.valid for row in rows)
confusions = Counter(
(row.expected, row.predicted)
for row in rows
if row.expected != row.predicted
)
return {
"examples": len(rows),
"accuracy": correct / len(rows),
"invalid_rate": invalid / len(rows),
"confusions": {
f"{expected}->{predicted}": count
for (expected, predicted), count in sorted(confusions.items())
},
}
Run the same frozen set against the current and candidate aliases, store raw outputs, and inspect changed predictions rather than looking only at the summary. Then shadow a small production sample without using the candidate's labels in the product. This sequence catches schema drift in the notebook, distribution drift in live traffic, and operational differences before a route starts changing user-visible tags.
Batching needs its own decision. An asynchronous batch interface is a good fit for backfills and nightly tagging because the caller doesn't need an immediate label, while an interactive request needs a latency budget and bounded retries. OpenAI documents a dedicated Batch API, but adopting a provider-specific batch format creates a separate execution path that a generic chat-completion gateway may not reproduce. Keep a common result envelope and eval suite if the transport diverges. Portability lives in the contract, not in pretending every execution mode is identical.
What does one API key fail to solve?
One application credential reduces secret distribution and client configuration. It does not merge provider data-processing terms, regional availability, retention policies, quotas, or incident domains. The gateway becomes a critical dependency too, so its authentication, request logs, timeout budget, and access controls need the same scrutiny as any other production service. A shared key should be scoped to classification, stored in a secret manager, rotated, and kept out of browser code; “one” must never mean “used everywhere.”
The catch is the common contract. It works well for short text classification with a closed JSON schema, but it can hide provider-specific controls that matter for another workload. Stick with a native provider interface when a required safety control, batch facility, data boundary, or output feature isn't represented faithfully by the shared layer. Keep separate routes when legal or organizational isolation requires separate credentials. And if text cannot leave the team's network, a hosted multi-provider gateway is not suitable; use an internal inference service and accept the capacity-planning work.
Retries are another boundary. The synchronous request path should not sleep through a long series of attempts. Return a typed retryable result or enqueue the item, apply exponential backoff with jitter in one owner, and place a maximum age on the job. A 429 is operational evidence, not permission for every application instance to start its own retry loop. Invalid labels are different: retrying the same prompt against the same route may repeat the same answer, so record E_CLASSIFY_LABEL, send the item to review, and use that example to improve the next eval set.
Operate classification as a data pipeline
Before release, version the taxonomy, prompt, schema, route mapping, and eval dataset independently. Log those versions with the item ID, selected alias, latency, token usage, validation outcome, and final label, while excluding raw sensitive text unless retention rules explicitly allow it. Put dashboards on invalid-output rate, queue age, per-label volume shifts, and the share sent to human review. A sudden fall in feature_request may indicate a product trend, a prompt change, or routing drift; versioned events let an operator tell the difference.
Deployment should move from offline replay to shadow traffic, then to a limited route allocation with an automatic rollback threshold. Reclassifying old data is a migration, so write new labels beside old ones until downstream reports have been checked. Don't silently overwrite history. For interactive calls, define a timeout and a product fallback such as “unclassified”; for batch work, make jobs idempotent and resume from item-level checkpoints. Review access to the shared credential, rehearse its rotation, and verify that the previous route mapping can be restored without an application deploy.
The final production decision is compact: one classification contract, one scoped application key, model aliases owned by the routing layer, strict schema validation, and promotion by repeatable evals. This keeps provider changes inexpensive without pretending they are risk-free. More importantly, it turns model choice from a preference into an observable deployment decision.
Top comments (0)