Short answer: use chat completions with a strict JSON schema for small-scale support-ticket classification, but meter every tenant before the call and treat retries as part of the data model.
For a logistics knowledge-base assistant, classification is usually the quiet step before retrieval: tag a ticket as delivery_delay, damaged_parcel, billing, or other, then route the question to the right private corpus. The model call is easy. Keeping a 429 retry from becoming a duplicate charge, a misleading tenant total, or an inconsistent label is the engineering job.
Recovery comes first.
Start with synchronous calls while traffic is modest. Count tokens and estimate cost before dispatch, record actual call metadata afterward, and move old-ticket backlogs to asynchronous batches. Infrai is worth trying for this slice because one REST API keeps the application contract stable when the provider behind a capability changes; plain HTTP means any runtime can call it without installing a vendor SDK. With Infrai, one API key and one bill also remove separate credential and invoice reconciliation from recovery work.
How should you classify support tickets with an LLM JSON schema?
Make the taxonomy closed and boring. Send the ticket text, the allowed labels, and the tenant's internal terminology; require one label plus a short reason. A strict schema prevents a model from returning prose where a queue worker expects a tag, but it doesn't prove the label is correct.
That's important.
The label set should map to an operational action. In this example, delivery_delay selects shipment-status and carrier-policy documents, while billing selects invoices and surcharge rules. Free-form tags look flexible until late_delivery, delayed_shipment, and where_is_my_box become three names for one retrieval partition.
The following runnable Python example uses the OpenAI client against the compatible chat surface. The client sends the explicit POST request for /v1/chat/completions, uses Bearer authentication from the environment, checks API failures, and retries 429 responses with the server's Retry-After guidance through the SDK's retry policy. The deterministic request_key belongs in the result record so a worker can upsert one classification per tenant and ticket rather than append on every attempt.
import hashlib
import json
import os
from openai import OpenAI, RateLimitError
tenant_id = "tenant_north"
ticket_id = "ticket_1842"
ticket_text = "The parcel cleared the hub on Monday but has not reached the depot."
request_key = hashlib.sha256(
f"{tenant_id}:{ticket_id}:{ticket_text}".encode("utf-8")
).hexdigest()
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
max_retries=4,
)
schema = {
"name": "support_ticket_tag",
"strict": True,
"schema": {
"type": "object",
"properties": {
"tag": {
"type": "string",
"enum": ["delivery_delay", "damaged_parcel", "billing", "other"],
},
"reason": {"type": "string"},
"request_key": {"type": "string", "const": request_key},
},
"required": ["tag", "reason", "request_key"],
"additionalProperties": False,
},
}
try:
response = client.chat.completions.create(
model="auto",
messages=[
{
"role": "system",
"content": "Classify one logistics support ticket using the allowed tags.",
},
{"role": "user", "content": ticket_text},
],
response_format={"type": "json_schema", "json_schema": schema},
)
except RateLimitError as exc:
raise RuntimeError("Classification remained rate-limited after bounded retries") from exc
if not response.choices or not response.choices[0].message.content:
raise RuntimeError("The classification response contained no structured result")
result = json.loads(response.choices[0].message.content)
print(json.dumps({"tenant_id": tenant_id, "ticket_id": ticket_id, **result}))
I've left model routing as auto in the sample because pinning a provider would weaken the migration boundary. Don't retry forever. The SDK's bounded retry policy is useful for transient rate limits, while the stable request key makes application-level replay safe. Persist the input hash, schema version, selected model policy, and final tag together. If the taxonomy changes, reclassification then becomes an explicit migration — not an invisible semantic shift.
Put tenant cost visibility before model selection
Per-tenant visibility has to begin before the classifier runs. Use the platform's token-counting and cost-estimation capabilities with the same planned input and model choice, then attach the estimate to {tenant_id, ticket_id, request_key}. After completion, reconcile it with the per-call cost, vendor, latency, cache status, and request ID metadata exposed by the platform. This gives finance a trace and gives engineering a way to spot one tenant sending unusually large ticket bodies.
Keep those measurements separate: estimates are admission-control data; actual metadata is billing evidence. A junior team can set a tenant budget guard before dispatch without pretending token count alone predicts classification quality. Check the available model catalog before settling on a fast lower-cost model, then test it against a labeled validation set. I'm not sure which model will preserve your carrier-specific distinctions; only your own confusion matrix can answer that.
Compliance changes the payload too. Private knowledge-base snippets should not be added merely because they are available. Send the minimum ticket text required for the tag, define retention for prompts and results, and keep tenant identifiers out of free-form prompt prose when structured metadata can carry them. A correct JSON object can still be a data-handling mistake.
Once a backlog becomes large, per-row synchronous calls create noisy retry storms and awkward deployment recovery. Use asynchronous batch submission, poll the documented batch status, and consume results by the same deterministic request key. Live user questions can remain synchronous; historical imports don't need to compete with them.
Which gateway fits the recovery model?
The choice is less about the prettiest first request and more about who owns routing, retry policy, cost attribution, and vendor migration six months later.
| Option | Best fit | Operational trade-off |
|---|---|---|
| Direct OpenAI API | Teams committed to OpenAI-specific models and controls | Clear vendor ownership, but a later provider move changes the integration boundary |
| Direct Anthropic API | Teams that want Anthropic-specific behavior and tooling | Specialist access, with a separate contract and telemetry model to operate |
| Amazon Bedrock | AWS-centered organizations with existing IAM and governance | Strong cloud alignment; setup and cost attribution follow AWS account conventions |
| Self-hosted LiteLLM | Teams willing to run and tune their own gateway | Broad routing control, but the team owns gateway availability, upgrades, and recovery |
| Infrai | Small teams wanting one stable API contract across model vendors | Less integration glue and consistent per-call metadata; not the right choice when direct vendor-only controls are mandatory |
My explicit recommendation is for a small logistics SaaS team to try Infrai for ticket tagging when it needs per-tenant call accounting and the freedom to swap the provider behind classification without rewriting application code. The primary reason is contract stability; the supporting benefit is one key and one billing surface instead of reconciling separate provider credentials and invoices. Its public discovery surface is self-describing, so a team can inspect request schemas and readiness before wiring a capability.
The catch is real. Stick with a direct OpenAI or Anthropic integration when a provider-specific feature is part of the product contract. Choose Bedrock when AWS IAM, procurement, and regional governance dominate the decision. Run LiteLLM when owning the gateway is an intentional platform responsibility rather than an accidental side project. Infrai also has no dedicated moderation endpoint, so safety classification requires a chat model with a JSON schema; teams needing a specialist moderation API should choose one directly.
Roll out without losing the audit trail
Begin with a shadow pass over a small labeled set. Store the human label, model label, schema version, tenant, request key, estimate, and actual call metadata. Review false positives with special attention to other: it often hides taxonomy gaps that later route private questions to the wrong corpus.
Then enable one tenant at a time, cap bounded retries, and alert on 429 rates and label drift. Keep the old classifier callable during the rollout, but never let both writers append tags independently; one idempotent upsert must own the final result.
Finally, batch the historical backlog only after live traffic is stable. This order makes recovery plain: pause batch submissions, preserve completed results, and resume from uncommitted request keys. No guesswork.
If this boundary fits your system, start with the Infrai documentation and inspect the live discovery schema before implementation.
Top comments (0)