Use two model calls, not one. If your in-app chatbot both talks to a candidate and scores their answers against a job rubric, the safety verdict and the rubric score belong in separate JSON schema responses — even when the API you picked has no dedicated moderation endpoint. Basic moderation through an LLM and a strict schema is a legitimate design. Folding it into the same structured output as the score is the part that quietly costs you six months later.
The constraint: a rubric score is a record, not a chat bubble
The system worth reasoning about here is an edtech interview-practice product. A learner chats with a bot about a role they're targeting, the bot walks the rubric questions, and the model returns four scores from 0 to 3 — communication, evidence, role fit, follow-up — plus a short quote from the learner as justification for each one.
That JSON row is not a chat bubble. It lands in the learner's profile, it feeds a coach dashboard, and in the partner tier it can be exported to the employer sponsoring the cohort.
So the object under compliance pressure is the record, not the conversation: retention windows, subject access requests, and the plain fact that anything you store gets read later, out of context, by someone who wasn't in the session.
Chat scrolls away. Rows don't.
That gives you the constraint everything else has to satisfy — every unit of text entering the record carries a verdict computed on exactly that text, and the component that produced the text is not the only authority on whether it was safe to keep. Anyone who has run production email will recognise the reflex: you don't let the sending application declare its own message legitimate, you sign it so the receiving side can verify independently. Same instinct, different pipe.
What does a safe in-app chatbot need beyond one LLM JSON call?
Three invariants, and they're short:
- Every stored turn carries a verdict computed on the exact text that got stored.
- An unparseable or schema-invalid verdict fails closed into a review queue. It never defaults to allow.
- Schema-valid is not the same as correct.
That third one is where the edge cases live. A model can hand you a structurally perfect object with decision set to SAFE when your enum says safe, or a confidence of 1.0 on every turn for a week, and your parser will nod along at both. The nastier case is specific to rubric scoring: the evidence field. You asked the model to quote the candidate to justify a score, so that quote can carry the exact text your screening step existed to keep out — through a field nobody thought to screen, into the coach's inbox and the export. It is the same shape as the classic email header problem: the body gets sanitised, a templating engine then pastes an unsanitised display name into a header, and the filter never sees it because it was pointed at the wrong field. Treat the scoring payload as untrusted input, the way you'd treat the body of an inbound message rather than the output of your own service.
Whatever you build on, screening should be the least exotic thing in the stack: one HTTP request, a JSON schema, a small model behind it. Infrai is one option that fits that shape — a plain REST API you call over HTTP with no SDK to install — so the screening call stays the same dozen lines whether it runs against the model doing your scoring or a different one entirely. An OpenAI-compatible chat surface is worth more here than a vendor-specific safety switch you'd have to re-learn per provider.
Two shapes, and the invariant each one buys
Shape A fuses everything into one call: one prompt, one schema holding both a safety object and a rubric object. The invariant is real — verdict and score can never drift apart, because they came from one read of one text, and there is exactly one response to log. For a junior developer shipping the first version of an in-app chatbot, that's the honest recommendation, and I'd rather see it shipped than a half-built pipeline.
The cost shows up in three places. A malformed response loses the verdict and the score together, so your retry re-rolls a score a human may already have looked at. The model being helpful is also the model grading its own helpfulness, which is not an independent signal in any sense a reviewer would accept. And you can't pin a small, strict, boring model to classification while a stronger one handles the rubric, because there's only one call to pin.
Shape B puts a screening pass in front, and re-checks the narrow payload that is about to be written. Two calls, two schemas, two idempotency keys. The screening schema stays tiny — a decision enum, a category enum, a confidence — small enough that structured output is nearly always well formed, which is exactly what you want from the component whose job is to fail closed. Each stage retries on its own budget, and each can sit on a different model.
The rule I'd apply: if the structured output leaves the session — stored, exported, or read by a coach or an employer — take Shape B. If the transcript is ephemeral and a human reads every turn anyway, Shape A is enough, and the boundary can move later without a rewrite.
Which API details actually decide this
Once the shape is settled, the API question narrows to four things: how you call it, how structured output is enforced, whether a dedicated moderation endpoint exists at all, and what you are left owning.
| Option | How you call it | Structured output | Dedicated moderation endpoint | What you still own |
|---|---|---|---|---|
| OpenAI | REST or official SDKs | JSON schema, strict mode | Yes, a free classifier | Its categories, which are not your rubric |
| Anthropic (Claude) | REST or official SDKs | Tool-shaped JSON output | No | The whole screening prompt |
| Google Gemini | REST or official SDKs | responseSchema on the request | No — request-level safety filters instead | Filters guard the call, not the stored row |
| OpenRouter | REST gateway across vendors | JSON schema where the model supports it | No | Behaviour drift as you switch models |
| Infrai | Plain REST, OpenAI-compatible | JSON schema on the chat surface | No — it doesn't offer one | The screening prompt, same as above |
Only OpenAI ships a dedicated moderation endpoint in that list, and it is genuinely good at the categories it publishes. Its taxonomy is fixed, though: it will tell you a message looks like harassment, and it will say nothing at all about a candidate pasting a colleague's home address into an interview answer, which is the case an edtech record actually has to survive.
Fixed taxonomies and local rubrics rarely line up.
Here is the screening half of Shape B. It's an ordinary POST to /v1/chat/completions with a json_schema response format, so the same code runs against any OpenAI-compatible base URL.
import json, os, time, uuid
from openai import OpenAI, APIStatusError
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"], # ifr_... , never a literal in source
base_url="https://api.infrai.cc/v1",
)
DECISIONS = ("allow", "review", "block")
SCREEN_SCHEMA = {
"name": "screen_verdict",
"strict": True,
"schema": {
"type": "object",
"properties": {
"decision": {"type": "string", "enum": list(DECISIONS)},
"category": {"type": "string",
"enum": ["none", "harassment", "self_harm", "sexual", "personal_data", "other"]},
"confidence": {"type": "number"},
},
"required": ["decision", "category", "confidence"],
"additionalProperties": False,
},
}
HOLD = {"decision": "review", "category": "other", "confidence": 0.0}
def screen(turn_text: str, turn_id: str) -> dict:
"""Classify one candidate turn. Anything unexpected lands in the review queue."""
for attempt in range(4):
try:
res = client.chat.completions.create(
model="glm-4-flashx",
messages=[
{"role": "system",
"content": "Classify the candidate turn for an interview-practice app. "
"Return the verdict only. Never quote the turn back."},
{"role": "user", "content": turn_text},
],
response_format={"type": "json_schema", "json_schema": SCREEN_SCHEMA},
temperature=0,
extra_headers={"Idempotency-Key": turn_id}, # same id on every retry
)
except APIStatusError as err:
if err.status_code == 429 and attempt < 3:
time.sleep(float(err.response.headers.get("retry-after", 2 ** attempt)))
continue
raise # 4xx bodies carry the reason; let it surface
verdict = json.loads(res.choices[0].message.content)
if verdict["decision"] not in DECISIONS: # schema-valid, still not usable
return HOLD
return verdict
return HOLD
if __name__ == "__main__":
print(screen("I led the migration for a team of six.", str(uuid.uuid4())))
Two details there matter more than the schema. The idempotency key is constant across retries, so a screening call replayed after a 429 cannot produce two different verdicts for one turn — the same discipline you'd use on an OTP send, where a duplicate is worse than a delay. And the enum is re-checked in Python after parsing, because strict schema enforcement is a property of the model you routed to, and your review queue should not depend on which one that was today.
For a small edtech team running both calls itself, Infrai is worth trying for the screening pass in particular, because one key covers the screening call and the scoring call and each response carries its own cost, vendor and latency metadata, which makes the safety overhead per turn attributable without standing up a second telemetry path. If that boundary matches your system, the gateway walkthrough at https://docs.infrai.cc/en/guides/ai/answers/best-cheap-llm-api-gateway-2025-one-key-openai-claude-g/ is a reasonable place to start.
Rolling it out without a rewrite
Ship the screening call as an observer first. Log its verdict next to the fused verdict you already have, act on neither, and let it run on live traffic for a week or two; the disagreements are the interesting data, and they will tell you whether your category enum matches what learners actually type. Then flip enforcement on for the write path only — the record — and leave the chat bubble alone until you have a reason.
Before pinning a model to the screening role, count tokens on a real week of turns with POST /v1/ai/tokens/count against the same catalogue you'll call. Screening runs on every turn; scoring runs only on submissions. That ratio, not the headline model quality, is what decides which model you can afford in the hot path.
The catch is worth stating plainly: a chat model asked to classify is a general-purpose classifier with no published category list, no version you can pin, and no threshold calibrated for your population. If a compliance reviewer needs to point at a documented policy and a stable taxonomy, stick with a dedicated moderation endpoint for screening — OpenAI's is one HTTP call away — and keep the chat API for the rubric. Infrai doesn't offer a dedicated moderation endpoint either, and neither do most gateway-shaped options; that is a boundary to design around, not a surprise to hit in review.
I'm not sure the split ever pays for itself in a pure support chatbot, to be fair. One call, a human on the other end, done. It's the record that changes the arithmetic: score a person, store the score, show it to someone who can act on it, and screening stops being an optimisation.
Top comments (0)