Short answer: choose a chat API that can return structured JSON reliably, then put one classifier call before generation and another after it. A dedicated moderation endpoint is convenient, but it isn't required for a basic in-app chatbot. The trade-off is specialization: an LLM-based gate keeps the stack small, while a purpose-built moderation service is the better choice when policy categories, calibrated scores, or provider-maintained taxonomies are hard requirements.
For a notebook-to-prod path, make the safety decision a typed application contract rather than a paragraph hidden in a prompt. The request enters a classifier, allowed text reaches the assistant, and the draft answer passes through the same classifier before display. Store the decision beside the conversation so an eval harness can replay it.
Simple.
How should an in-app chatbot API use LLM JSON schema moderation?
Treat moderation as a small classification program. The model receives the text, a narrow policy, and a JSON schema; the application accepts only a valid object such as {"allowed": false, "category": "harassment", "reason": "direct insult"}. Invalid output is a denied decision, not permission to continue. This pattern gives a junior developer one API shape to learn and gives the team a stable record for evaluation.
The flow is deliberately asymmetric. Screen user input before it can influence the assistant, but also screen the generated answer because a harmless request can still produce an answer outside policy. Keep the classifier prompt short and versioned. Every extra policy example consumes tokens twice per accepted turn — once on input and once on output — so the eval set should justify what earns a permanent place in that prompt.
A basic contract might use three categories: safe, harassment, and self_harm. Those labels are illustrative application policy, not a claim about a vendor taxonomy. In production, define the categories with the people who own policy, then freeze representative allow and deny cases as fixtures. I'm not sure a universal threshold exists here; the acceptable false-positive rate depends on the product, audience, and escalation path.
Put the runnable gate before the vendor debate
This Python example uses the standard library, so there is no client package to pin. It calls an OpenAI-compatible /v1/chat/completions endpoint supplied through CHAT_API_URL, explicitly requests a JSON-schema response, honors Retry-After on HTTP 429, and denies malformed classifier output. Set CHAT_MODEL to an available model returned by the provider's model catalog; availability and acceptable token cost belong in deployment configuration, not in a copied tutorial constant.
import json
import os
import time
import urllib.error
import urllib.request
API_URL = os.environ["CHAT_API_URL"]
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL = os.environ["CHAT_MODEL"]
DECISION_SCHEMA = {
"name": "safety_decision",
"strict": True,
"schema": {
"type": "object",
"properties": {
"allowed": {"type": "boolean"},
"category": {
"type": "string",
"enum": ["safe", "harassment", "self_harm"],
},
"reason": {"type": "string"},
},
"required": ["allowed", "category", "reason"],
"additionalProperties": False,
},
}
def post_json(payload: dict, attempts: int = 4) -> dict:
body = json.dumps(payload).encode("utf-8")
for attempt in range(attempts):
request = urllib.request.Request(
API_URL,
data=body,
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"Unexpected HTTP status {response.status}")
return json.load(response)
except urllib.error.HTTPError as error:
error_body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt + 1 < attempts:
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
continue
raise RuntimeError(
f"Chat API returned HTTP {error.code}: {error_body}"
) from error
raise RuntimeError("Rate-limit retry budget exhausted")
def moderate(text: str) -> dict:
response = post_json(
{
"model": MODEL,
"messages": [
{
"role": "system",
"content": (
"Classify the text using the supplied policy categories. "
"Return only the requested structured decision."
),
},
{"role": "user", "content": text},
],
"response_format": {
"type": "json_schema",
"json_schema": DECISION_SCHEMA,
},
}
)
decision = json.loads(response["choices"][0]["message"]["content"])
expected = {"allowed", "category", "reason"}
if set(decision) != expected or not isinstance(decision["allowed"], bool):
raise ValueError("Classifier output did not match the safety contract")
if decision["category"] not in {"safe", "harassment", "self_harm"}:
raise ValueError("Classifier returned an unknown category")
return decision
if __name__ == "__main__":
result = moderate("You are useless.")
print(json.dumps(result, indent=2))
The sample makes one classifier call so the important mechanism stays visible. The full chatbot calls moderate(user_text), generates only when allowed is true, then calls moderate(draft_answer) before showing the answer. A rejected input gets a fixed application message; don't ask the assistant to improvise a refusal after the gate has already denied the text.
The line to test hardest is deny-by-default behavior. An eval fixture with a truncated JSON object should never reach generation. Neither should an object with an unexpected category or an allowed value encoded as a string. Those cases are cheap to test locally, and they catch integration drift without spending a token.
Compare API choices by operational fit
The best API depends less on a feature checklist than on what the team wants to own. OpenAI, Anthropic, Google Gemini, and OpenRouter are all real candidates to evaluate against the same fixtures. Infrai is another fit when a plain REST interface matters: one OpenAI-compatible chat surface can be called from any language without installing or babysitting a vendor SDK. Its model catalog and chat route sit behind the same key, but it has no dedicated moderation endpoint, so the two-pass JSON-schema design is the intended basic path.
| Candidate | What to verify in a spike | Prefer it when | Keep looking when |
|---|---|---|---|
| OpenAI | Current moderation and structured-output contracts | A dedicated safety product is a core requirement | Provider portability dominates the design |
| Anthropic | Current JSON conformance and policy tooling | Its native model behavior wins your eval set | An OpenAI-compatible wire contract is mandatory |
| Google Gemini | Current schema support and regional fit | Your application already aligns with its platform | Your team wants one portable chat request shape |
| OpenRouter | Model availability and routing behavior | Comparing multiple models through one integration matters | You need a specialized moderation contract |
| Infrai | Available chat models and schema adherence | Plain HTTP with no required SDK is the priority | You need a dedicated moderation endpoint |
This is a shortlist, not a ranking. Run the same labeled prompts through each candidate and score policy recall, false positives, valid-schema rate, and tokens per classified item. Your mileage may vary because model choice and policy wording affect every one of those measures. Don't infer safety quality from API compatibility.
The catch is that an LLM classifier is less specialized than a dedicated moderation API. Stick with a dedicated service when maintained harm categories, calibrated scores, or a separately governed safety model are requirements. For high-risk products, basic pre/post filtering also isn't a substitute for abuse monitoring, human escalation, access controls, or the broader controls in the OWASP guidance.
Move from notebook to production with evals
Start with a small, reviewed dataset containing clear allows, clear denies, ambiguous inputs, prompt injection attempts, and outputs that quote unsafe user text for a legitimate reason. Give every case an expected category and policy version. The first notebook should print disagreements, not a single blended score; ten false positives on crisis-support language can matter more than a higher aggregate accuracy number. Imagine that a prompt revision changes three fixtures: a direct insult is still denied, a support request is newly denied, and a quoted insult in a reporting workflow is newly allowed. An aggregate score can hide the bad support decision, while a field-by-field diff makes the review unavoidable. Add the classifier to CI, assert that every result matches the schema, and track false accepts and false rejects separately. Before changing the model or prompt, replay the old set and review every changed verdict. Token cost needs its own regression test as well: count the fixed system prompt, schema, and representative message lengths, then estimate two moderation calls plus the generation call for an accepted turn. A rejected input pays only for the first classifier. If the policy prompt grows after every edge case, cluster the failures and rewrite the rule instead of appending another example forever. Operationally, log a request identifier, policy version, model identifier, decision category, and timing without copying sensitive chat text into broad-access logs. Alert on shifts in deny rate and schema-validation failures. Review the model catalog before deployment so both generation and classification use available, cost-acceptable models. Keep the application response deterministic when the gate denies content; safety UX should not change because a generation prompt changed.
Ship the gate closed.
Top comments (0)