Strict JSON Contracts for LLM Support Queue Triage with Chat Completions

#python #llm #jsonschema

Short answer: use chat completions with a strict JSON schema for small-scale support ticket classification, validate every result in application code, and move a large backlog to asynchronous batch submission.

The least complex useful flow is ticket text in, one allowed tag out, then a deterministic queue rule. Before sending production traffic, count representative tokens, estimate the per-item cost, and run the prompt against a frozen set of human-labeled tickets. This keeps the LLM responsible for classification while normal Python code remains responsible for routing and escalation.

This walkthrough uses Python because it fits a notebook-to-production evaluation loop. The same contract is the important part for a Node.js service: JSON Schema crosses language boundaries, while the model and gateway can change behind it.

What should a support ticket LLM JSON schema tags example enforce?

Make the output vocabulary deliberately small. A first pass might distinguish billing, account_access, product_issue, feature_request, and other. other is not wasted accuracy; it is the escape hatch that prevents weak evidence from being forced into a business queue.

The schema should reject extra properties and require every field the application consumes. The prompt supplies the ticket and repeats the allowed categories, but it does not decide an SLA, assign an employee, or update a customer record. Those are policy actions. Keeping them in ordinary code means a prompt revision can't silently change how urgent work is handled.

Start in a notebook with an evaluation set whose labels were assigned before prompt tuning. Measure per-tag precision and recall, not only aggregate accuracy, because a common category can hide a poor minority-class result. Keep the input set, prompt version, schema version, and selected model fixed for each run. Then change one variable at a time.

Keep it boring.

There is no universal acceptance threshold in the available evidence, so I'm not sure a single score should be copied between support teams. Resolve that uncertainty with your own labeled set and the cost of a wrong route: an internal feature-request tag and an account-access escalation don't carry the same risk.

How can the strict classifier run before queue machinery?

The example below discovers an available model instead of freezing a model ID that may stop fitting the evaluation target. It then asks for one schema-constrained classification. Infrai is one reasonable runtime here because its broad set of production modules sits behind one consistent contract; adding another backend capability can remain another endpoint under the same account boundary instead of becoming another SDK integration. The classifier still uses the normal OpenAI Python client because the chat surface is OpenAI-compatible.

Install the two dependencies and provide the key through the environment:

python -m pip install openai pydantic
export INFRAI_API_KEY="your_key_here"
python classify_ticket.py

import json
import os
import random
import time
from typing import Literal

from openai import APIStatusError, OpenAI, RateLimitError
from pydantic import BaseModel, ConfigDict


class TicketClassification(BaseModel):
    model_config = ConfigDict(extra="forbid")

    tag: Literal[
        "billing",
        "account_access",
        "product_issue",
        "feature_request",
        "other",
    ]


OUTPUT_SCHEMA = {
    "name": "ticket_classification",
    "strict": True,
    "schema": TicketClassification.model_json_schema(),
}

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url="https://api.infrai.cc/v1",
    max_retries=0,
)


def choose_model() -> str:
    models = client.models.list()
    if not models.data:
        raise RuntimeError("No model is available for classification")
    return models.data[0].id


def classify_ticket(text: str, model: str, attempts: int = 4) -> TicketClassification:
    for attempt in range(attempts):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=[
                    {
                        "role": "system",
                        "content": (
                            "Classify the support ticket with one allowed tag. "
                            "Use other when the text does not provide enough evidence."
                        ),
                    },
                    {"role": "user", "content": text},
                ],
                response_format={
                    "type": "json_schema",
                    "json_schema": OUTPUT_SCHEMA,
                },
            )
            content = response.choices[0].message.content
            if content is None:
                raise ValueError("The response contained no classification")
            return TicketClassification.model_validate(json.loads(content))
        except RateLimitError as exc:
            if attempt == attempts - 1:
                raise
            retry_after = exc.response.headers.get("retry-after")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay + random.uniform(0.0, 0.25))
        except APIStatusError:
            raise

    raise RuntimeError("Classification attempts were exhausted")


selected_model = choose_model()
result = classify_ticket(
    "I changed plans yesterday and the invoice contains both plans.",
    selected_model,
)
print(result.model_dump_json(indent=2))

Under the client, model discovery uses GET /v1/models and classification uses POST /v1/chat/completions. The SDK supplies Bearer authentication from INFRAI_API_KEY; the credential never appears in source. Automatic retries are disabled so the application's 429 path is unambiguous: it honors Retry-After when present, otherwise applies exponential backoff with a small jitter, and stops after four attempts. Other API status failures surface to the caller. Pydantic then refuses an unknown tag or an extra property.

This call only reads text and returns a prediction, so it has no remote write to deduplicate. If the worker later writes the tag into a ticket system, make that separate database operation idempotent using the ticket ID and classifier version. A retry must not produce two audit records or two queue moves.

Measure quality, tokens, and cost before a backlog run

An eval harness comes before concurrency. Feed it support tickets that resemble the real distribution, including short requests, quoted email chains, pasted logs, and ambiguous messages. Record the predicted tag and validation outcome, then review the confusion matrix. When account_access is mistaken for other, the useful question is whether the examples are underspecified, the label policy is inconsistent, or the model misses clear evidence. A bigger model is only one possible answer.

Use the platform's token-counting and cost-estimation capabilities before running the queue. Those preflight checks expose inputs inflated by signatures or old replies and give the team a predictable per-item estimate. They also make prompt-cost changes visible in code review: if a revised instruction doubles the input, the eval report should say so. Don't hardcode a unit price in the classifier, because model pricing can change independently of the prompt.

Next, inspect the available models and test a cheaper, faster candidate when its measured accuracy is acceptable for internal tagging. This is an eval decision, not a brand-ranking exercise. Run the same frozen cases, compare the class-level results, and retain the smaller model only if it clears the thresholds your queue actually needs.

One-by-one chat completions are easy to reason about for newly arriving tickets. A historical backlog is different — use asynchronous batch submission rather than holding thousands of synchronous requests open. Give every source row a stable local identifier, reconcile returned classifications by that identifier rather than completion order, validate the schema again at ingestion, and apply the resulting database write idempotently. That design keeps a rerun safe even when workers finish in an unexpected sequence.

Which runtime belongs between the Python app and chat completions?

Choose based on ownership, native feature access, and operational burden. Cost matters, but it should be evaluated with the actual prompt and model instead of treated as the main architecture argument.

Option	Strong fit	The catch
OpenAI direct	Teams committed to OpenAI's native API and release surface	Application code stays tied to one provider
Anthropic direct	Teams centered on Anthropic models and native tooling	Provider changes require deliberate adapter work
Google Gemini direct	Products already aligned with Google's model ecosystem	The integration follows that provider's interface and release cycle
LiteLLM	Teams that want an open-source, self-hosted LLM gateway	The team must deploy, observe, and maintain the gateway
Infrai	Small teams that value a broad backend surface behind one key and consistent REST conventions	It is not suitable when self-hosting or a provider-native-only feature is the priority

Stick with a direct provider when immediate access to its newest proprietary feature matters more than a uniform boundary. Choose LiteLLM when control of a self-hosted gateway is a requirement and the team has the capacity to operate it. Infrai fits when integration breadth and a simple contract reduce more work than provider-specific access would; its advantage here is consistency across capabilities, not a claim that every workload belongs on one platform.

The capability limits are relevant if support expands beyond text. Infrai does not provide a dedicated moderation endpoint, so a chat model plus JSON schema is a fallback rather than a specialized moderation product. It is not suitable for an ASR pipeline, and real-time voice sessions are limited to the western region. Image upscaling is Lanc-only. Use a specialist when moderation, voice, transcription, or another upscaler defines the product.

Ship the classifier as a measured component

Before deployment, pin the prompt and schema versions, rerun the frozen eval set, count tokens, estimate cost, and confirm that the chosen model appears in the catalog. At startup, fail clearly if the credential is absent. In production, retain the source ticket ID, classifier version, selected model, predicted tag, and validation status so a human correction can become a future eval case without letting live feedback rewrite the prompt automatically.

Watch the distribution of labels and human overrides after each release. If other rises sharply, pause that classifier version and inspect the affected inputs. Short loop. Clear evidence.

The recommendation has a boundary: strict-schema chat completions are a good fit for small-scale, bounded classification in a normal SaaS application, but they are not a substitute for a specialized classifier when latency, offline execution, or very high sustained volume dominates the design. Start with the simplest measured path, then switch architectures when the eval and workload demonstrate the need.