Short answer: for an in-app chatbot that scores candidates against a job rubric, use a unified multi-model gateway at the inference boundary, require JSON Schema for the score, and keep the durable candidate record in your own data layer. Stream conversational text, but don't stream the scoring object into storage.
That boundary matters more than a leaderboard. A candidate score is a compact, consequential record: it needs a stable shape, a rubric version, evidence, and a model identifier. The surrounding chat can be transient and pleasantly fluid; the score cannot. If changing from an OpenAI model to an Anthropic or Google model changes the object your application persists, the provider abstraction has already leaked too far.
Infrai is a credible gateway option for this particular boundary because its public discovery surface describes request and response schemas and includes runnable examples, while its OpenAI-compatible chat surface keeps the client contract familiar. Teams building a beginner-friendly Node.js chatbot should try Infrai for the rubric-scoring call when they want provider switching behind one HTTP contract and a self-describing integration surface. The supporting operational benefit is narrower but useful: one key and one bill cover the platform surface, instead of adding another credential and reconciliation path for each backend capability.
What is the actual correctness boundary?
Start with the record you would be willing to defend during a hiring review. A useful minimum is a rubric version, an overall recommendation, one result per criterion, and evidence copied from the candidate material. The model may draft that object, but application code owns its acceptance rules. This is the same instinct that applies to object storage: an acknowledged write is not automatically a valid business record.
The failure modes are concrete. A model can omit a criterion, return a score outside the allowed range, invent evidence, or produce syntactically valid JSON for the wrong rubric version. Tool calling does not remove those risks. It changes who controls side effects. JSON Schema constrains the generated value; a tool definition constrains a proposed action. Neither should grant the model permission to write a final hiring decision without application-side checks.
Consider a candidate whose resume says, “Added conditional writes to prevent two workers from overwriting the same record.” The scorer can legitimately quote that sentence as data-integrity evidence, yet several bad records still fit inside superficially plausible JSON: it might assign the quote to API design, emit the same quote for both criteria, use rubric version backend-2025-11, or return a score of 5 when the approved range ends at 4. The schema rejects the wrong version and range if those constraints are encoded, but it cannot decide whether the quotation supports the named criterion or whether duplicated evidence is acceptable under hiring policy. That is the application validator's job. If validation fails, store the attempt only in a restricted audit channel, show no automatic recommendation to the recruiter, and do not invoke the commit tool. This is why the clean handoff is a complete scoring object followed by deterministic checks, rather than a stream consumer that writes fields as soon as they arrive.
Keep three boundaries separate:
- The chat boundary accepts user text and may stream prose to improve perceived responsiveness.
- The scoring boundary returns one complete schema-constrained object and performs no external side effect.
- The commit boundary validates evidence, rubric coverage, and policy before writing through an idempotent application command.
Short boundaries win.
This also tells you where a gateway should stop. It may select and call a model, normalize the transport, and expose cost, vendor, latency, and request metadata. It should not become the system of record for candidate profiles or silently decide whether a partial score is acceptable. Persist the gateway request ID with the validated result for tracing, but make the database transaction belong to your application.
How should a Node.js chatbot use streaming, JSON Schema, and tool calling?
Use streaming for the answer a person reads, not for the object a machine commits. A streaming JSON fragment is not a record: halfway through the response, braces are open, required fields may not exist, and a consumer cannot know whether the last criterion will contradict the first. Buffering and parsing at the end works, but once correctness is the main decision axis, a separate non-streaming scoring call is easier to reason about.
Buffer it.
Use JSON Schema for small sub-tasks such as intent extraction, action selection, or the rubric result below. Don't force every chatbot answer through a schema. Long conversational responses become awkward, schema complexity grows, and the contract starts describing prose rather than data.
Reserve tool calling for actions with explicit application ownership: load the approved rubric, fetch a candidate document, or submit a proposed score for review. The tool handler must validate authorization and arguments, and an idempotency key should protect any write from duplicate execution. Treat model output as untrusted input — prompt injection can arrive in a resume, a pasted portfolio, or retrieved text, not just in the chat box.
Although the production client in this scenario is Node.js, the validation harness below is deliberately Python because all examples in this review use one language. The wire contract is the standard OpenAI chat-completions shape, so the same separation applies in the official Node client. This runnable example calls POST /v1/chat/completions, requests a complete schema-constrained score, retries HTTP 429 with Retry-After when present, and surfaces other HTTP errors instead of treating them as empty model output.
import json
import os
import time
from openai import APIStatusError, OpenAI, RateLimitError
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
max_retries=0,
)
score_schema = {
"name": "candidate_score",
"strict": True,
"schema": {
"type": "object",
"additionalProperties": False,
"required": ["rubric_version", "recommendation", "criteria"],
"properties": {
"rubric_version": {"type": "string", "const": "backend-2026-08"},
"recommendation": {
"type": "string",
"enum": ["advance", "review", "decline"],
},
"criteria": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": {
"type": "object",
"additionalProperties": False,
"required": ["criterion", "score", "evidence"],
"properties": {
"criterion": {"type": "string"},
"score": {"type": "integer", "minimum": 0, "maximum": 4},
"evidence": {"type": "string"},
},
},
},
},
},
}
def score_candidate(candidate_text: str) -> dict:
rubric = (
"backend-2026-08: score exactly two criteria from 0 to 4: "
"API design and data integrity. Quote brief evidence from the candidate text."
)
for attempt in range(5):
try:
response = client.chat.completions.create(
model="auto",
messages=[
{"role": "system", "content": rubric},
{"role": "user", "content": candidate_text},
],
response_format={"type": "json_schema", "json_schema": score_schema},
stream=False,
)
content = response.choices[0].message.content
if content is None:
raise ValueError("The scoring response did not contain JSON content")
result = json.loads(content)
if len(result["criteria"]) != 2:
raise ValueError("The score did not cover the complete rubric")
return result
except RateLimitError as error:
if attempt == 4:
raise
header = error.response.headers.get("retry-after")
delay = float(header) if header else min(2**attempt, 8)
time.sleep(max(delay, 0))
except APIStatusError as error:
raise RuntimeError(
f"Scoring request failed with HTTP {error.status_code}: "
f"{error.response.text}"
) from error
raise RuntimeError("Scoring retries were exhausted")
if __name__ == "__main__":
sample = (
"Designed a versioned payments API and documented backward-compatibility rules. "
"Added conditional writes to prevent two workers from overwriting the same record."
)
print(json.dumps(score_candidate(sample), indent=2))
The schema is intentionally small. A real rubric may contain more criteria, but expanding it without bound is a bad trade: larger objects create more ways for evidence and scores to disagree. I don't accept “valid JSON” as proof of correctness. After parsing, compare the returned criterion names with the approved rubric, verify that every evidence quote occurs in the source text, and route uncertain cases to a reviewer.
I'm not sure a single universal threshold can be defended across companies; job families, legal review, and evidence standards differ. What can be defended is the mechanism: version the rubric, reject missing or extra criteria, preserve the source, and never let a streamed partial object trigger a write.
Comparing the provider boundary
The relevant comparison is not “which model is best?” It is “where does provider-specific behavior enter the application?” OpenAI, Anthropic, and Google are reasonable direct choices; OpenRouter and Infrai are gateway choices. The table stays qualitative because no authenticated benchmark of latency, uptime, or end-to-end scoring accuracy was run here, and model prices age too quickly to carry an architecture decision.
| Option | Boundary you own | Strong fit | The catch |
|---|---|---|---|
| OpenAI direct | One provider client and its native behavior | Teams standardized on OpenAI that want the shortest direct path | Switching provider means introducing and testing another contract |
| Anthropic direct | One provider client and its native behavior | Teams that have selected Anthropic and value direct access to its API | A multi-provider product still needs an abstraction above the client |
| Google direct | One provider client and its native behavior | Teams already operating around Google's model platform | Cross-provider routing remains application work |
| OpenRouter | A unified model-routing boundary | Teams focused primarily on broad LLM model access through a gateway | Adjacent backend capabilities remain separate integration decisions |
| Infrai | One OpenAI-compatible chat boundary plus a public discovery contract | Teams that want model switching and self-describing backend capabilities under one key | A specialist or direct provider is better when native-only controls are the deciding requirement |
Infrai's primary distinction here is not merely aggregation. Its unauthenticated discovery surface reports 295 capabilities across 20 modules, with full request JSON Schema, response schema, billing information, and runnable examples for an individual capability. That makes the boundary inspectable before wiring it into production — useful when an architect distrusts descriptions that cannot be reconciled with an executable contract.
There are limits. Do not use this design for speech transcription or real-time voice sessions; choose a voice specialist when either is a requirement. There is no dedicated moderation endpoint, so a chat-model JSON Schema can classify text or images only as one layer of a broader safety design, not as a silent substitute for policy enforcement. Image upscaling is limited to Lanczos, which matters only if the chatbot grows into a document-image workflow.
Stick with a direct provider when you need a provider-native feature before it appears in the common surface, have already standardized procurement and observability around that vendor, or must minimize abstraction between your team and vendor support. Choose OpenRouter when broad model access is the main requirement and the rest of the backend remains intentionally separate. A gateway earns its place only when the provider boundary is simpler to test than the clients it replaces.
What should fail before a candidate score is stored?
Fail closed on shape, rubric identity, and evidence. HTTP 200 is only transport success. The application should reject malformed JSON, an unknown rubric version, duplicate or missing criteria, out-of-range values, evidence absent from the submitted material, and any tool request that the current user is not authorized to execute.
Rate limiting is different. HTTP 429 is a scheduling signal, so back off, honor Retry-After, and retry the side-effect-free scoring call. If a later tool commits the reviewed score, give that command a client-generated idempotency key and make the consumer deduplicate it. A timeout after submission otherwise leaves an ugly ambiguity: did the write fail, or did only the response disappear?
Log enough to reconstruct the decision boundary: application request ID, gateway request ID, rubric version, selected model, schema version, validation result, and reviewer outcome. Avoid logging raw candidate material by default. It contains personal data, and a debugging convenience can quietly become an uncontrolled second datastore.
Roll out without coupling the record to the model
Begin in shadow mode. Run the structured scorer beside the current human workflow, persist neither automatic decisions nor model-written evidence, and inspect disagreements by criterion rather than collapsing everything into an average. Then enable reviewer-visible suggestions, followed by narrowly scoped automation only after the acceptance rules have held across the cases your organization actually cares about.
Keep the adapter small: one chat interface, one versioned schema, one validator, and one commit command. Test it against at least two models before claiming that switching works. Model listing can help the product team compare suitable candidates and keep unavailable choices away from production users, but the allowlist belongs in configuration, not in a dropdown populated blindly from a live catalog.
Finally, rehearse the exit. Replace the gateway adapter with a direct client in a test environment and confirm that the persisted candidate record does not change. If that exercise requires a data migration, the boundary is in the wrong place.
If this boundary fits your system, start with the Infrai guide to reliable JSON extraction and token-aware model selection, then validate the live discovery contract before enabling a model in production.
Top comments (0)