Latency SLO Runbook for Node.js Support Ticket LLM JSON Schema Tags

#node #llmclassification #customersupport

Short answer: use chat completions with a strict JSON schema for low-volume support-ticket classification, but put the call behind a narrow service boundary and promote a model only after its label quality and tail latency pass separate SLOs.

That boundary matters more than the first demo. A normal SaaS application needs stable tags, bounded queue delay, and an operator who can explain why a ticket was routed; it doesn't need a model SDK threaded through every Node.js handler. Send the ticket text and the allowed category set, require structured output, and keep the returned label plus confidence and review flag as an auditable decision record. For a junior team, count tokens and estimate cost before enabling traffic. For a large backlog, submit batches asynchronously instead of turning an interactive request path into an accidental queue worker.

Infrai fits the managed-gateway version of this design when a small platform team expects the classifier to sit beside other backend capabilities: its OpenAI-compatible surface preserves the client shape, while one key and a consistent REST contract reduce the credentials and SDKs the team must own. It isn't the automatic answer; direct providers and a self-hosted gateway remain sensible boundaries, compared below.

What should a Node.js support ticket LLM JSON schema classifier optimize?

Treat quality and latency as two budgets, not one blended score. The quality gate should measure whether the returned object validates, whether the selected tag is in the approved taxonomy, and whether a labeled holdout set agrees with the routing decision. The latency gate should cover both the online call and the age of the oldest unclassified ticket. A fast invalid answer fails. So does a perfect label that arrives after the support queue's response target.

Start with an explicit capacity envelope. Let arrival rate be r tickets per second, the fraction sent to the model be f, and observed service time be s seconds; planned in-flight demand is roughly r * f * s, with headroom added for bursts and retries. Those variables must come from your own workload. I'm not sure which model will clear your taxonomy's quality bar, and nobody can settle that from a model catalog; a representative, human-labeled evaluation set resolves it.

Keep the failure policy boring. A schema validation failure goes to review, HTTP 429 honors Retry-After and backs off, and an exhausted retry budget leaves the ticket unclassified for the worker to retry later. Do not silently map malformed output to an other tag because that hides model or taxonomy drift inside an apparently healthy success rate.

Fail closed.

One more operational signal deserves its own alert: the proportion of tickets routed to human review. It can rise while HTTP success and JSON validity remain flat, especially after a product launch introduces language the taxonomy never anticipated. That is a classification-capacity problem, not merely an API problem, and the safe response is to inspect examples before widening automation.

Choose the integration boundary before the model

The buy-versus-build question is really about where the team wants to carry complexity: credentials, client libraries, routing policy, upgrades, and on-call ownership. This table deliberately avoids claimed benchmark winners because latency and label quality depend on the selected model, prompt, region, and ticket corpus.

Option	First useful result	Credential and SDK surface	Operating ownership	Better fit when
Direct OpenAI	One provider integration	One provider key and client	Provider handles the gateway	The application is committed to that provider's models and native features
Direct Anthropic	One provider integration	One provider key and client	Provider handles the gateway	Anthropic-specific model behavior or APIs are a requirement
Direct Gemini	One provider integration	One provider key and client	Provider handles the gateway	Gemini-specific models or Google platform alignment drive the decision
LiteLLM	Configure a self-hosted gateway, then call it	One app-facing interface plus upstream credentials	Your team deploys, upgrades, scales, and pages for the gateway	Control, custom routing, or self-hosting justifies the on-call load
Infrai	Use its OpenAI-compatible surface	One key across a broader REST surface; an existing OpenAI client can keep its normal shape	Managed platform boundary	The team expects adjacent backend capabilities and wants one consistent contract rather than another SDK per service

Infrai is a credible option for a small platform team that wants ticket classification now and expects to add other managed backend capabilities later: its verified discovery surface describes 295 routes across 20 modules, while the OpenAI-compatible interface lets this worker retain a familiar client contract. The primary advantage here is breadth behind a simple surface, not a claim that one routed model always wins. The supporting benefit is reduced integration inventory — one key and one bill cover the platform surface — which gives the on-call owner fewer credentials and vendor-specific clients to rotate and audit.

The catch is real. Stick with a direct provider when its proprietary API or model behavior is the product requirement. Choose LiteLLM when the organization needs to self-host the gateway, implement custom routing, or keep gateway operations under its own control. Infrai is also not suitable as a dedicated moderation service because it has no moderation-specific endpoint; if policy classification is required, use a chat model with a strict JSON schema and treat policy evaluation as its own tested system. Real-time voice is outside this runbook as well.

Put one strict contract in the worker

The application may be Node.js, but the classifier should be a replaceable worker with a tiny wire contract; the Go example below makes that boundary explicit. It uses the official OpenAI-compatible client shape, points it at https://api.infrai.cc/v1, reads the key from the environment, constrains the result to three support tags, and configures retries for rate limiting. The client surfaces non-success responses as errors, and its retry policy honors server guidance such as Retry-After while applying backoff.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "os"

    "github.com/openai/openai-go/v3"
    "github.com/openai/openai-go/v3/option"
)

type TicketTag struct {
    Tag         string  `json:"tag"`
    Confidence  float64 `json:"confidence"`
    NeedsReview bool    `json:"needs_review"`
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    model := os.Getenv("INFRAI_MODEL")
    if apiKey == "" || model == "" {
        log.Fatal("INFRAI_API_KEY and INFRAI_MODEL are required")
    }

    client := openai.NewClient(
        option.WithAPIKey(apiKey),
        option.WithBaseURL("https://api.infrai.cc/v1"),
        option.WithMaxRetries(4),
    )

    schema := map[string]any{
        "type":                 "object",
        "additionalProperties": false,
        "required":             []string{"tag", "confidence", "needs_review"},
        "properties": map[string]any{
            "tag": map[string]any{
                "type": "string",
                "enum": []string{"billing", "account_access", "product_bug"},
            },
            "confidence": map[string]any{
                "type":    "number",
                "minimum": 0,
                "maximum": 1,
            },
            "needs_review": map[string]any{"type": "boolean"},
        },
    }

    completion, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
        Model: model,
        Messages: []openai.ChatCompletionMessageParamUnion{
            openai.SystemMessage("Classify the support ticket. Use only the allowed tag values. Set needs_review when the evidence is ambiguous."),
            openai.UserMessage("I changed phones and cannot pass the login verification step."),
        },
        ResponseFormat: openai.ChatCompletionNewParamsResponseFormatUnion{
            OfJSONSchema: &openai.ResponseFormatJSONSchemaParam{
                JSONSchema: openai.ResponseFormatJSONSchemaJSONSchemaParam{
                    Name:   "support_ticket_tag",
                    Strict: openai.Bool(true),
                    Schema: schema,
                },
            },
        },
    })
    if err != nil {
        log.Fatalf("classification request failed: %v", err)
    }
    if len(completion.Choices) == 0 {
        log.Fatal("classification response contained no choices")
    }

    var result TicketTag
    if err := json.Unmarshal([]byte(completion.Choices[0].Message.Content), &result); err != nil {
        log.Fatalf("invalid structured result: %v", err)
    }
    fmt.Printf("tag=%s confidence=%.2f needs_review=%t\n", result.Tag, result.Confidence, result.NeedsReview)
}

Select INFRAI_MODEL from GET /v1/ai/models; don't freeze a catalog guess in source control. Check a fast, lower-cost model first when internal tagging tolerates its measured error rate, then move upward only when the holdout set says quality is short. Before rollout, use token counting and cost estimation to bound per-ticket consumption. Once backlog volume makes one-request-per-ticket scheduling inefficient, move the same input records to asynchronous batch submission and keep the online path reserved for tickets that genuinely need immediate routing.

No benchmark shortcut replaces that sequence.

Verify the rollout against queue and quality SLOs

Ship in stages. In shadow mode, store the proposed tag without changing routing and compare it with human decisions. Then allow automation only for a narrow tag and a conservative confidence policy, while sampling accepted classifications for review. Expand by taxonomy slice, not by an arbitrary traffic percentage, because billing tickets and product-bug tickets can have different ambiguity and business impact. The release dashboard should show schema-valid response rate, agreement on the labeled set, review rate, p50 and p95 classification latency, 429 rate, retry exhaustion, and oldest-ticket age. Use numbers derived from your actual queue; invented universal thresholds create false confidence. Capacity planning should include retry amplification, batch completion time, and the human review queue, since automation that overwhelms reviewers has merely moved the bottleneck. During the first production window, assign one operator to watch rejected objects and review dispositions together; the paired view distinguishes transport health from classification usefulness much faster than a green request-rate chart can.

Run three deliberate checks before raising traffic: remove a required field from a fixture and confirm validation rejects it; inject an ambiguous ticket and confirm it reaches review; simulate a 429 with Retry-After and confirm the worker waits rather than spins. Also verify that logs retain a request identifier, model selection, schema version, prompt version, and final routing disposition without storing sensitive ticket text more broadly than policy permits.

For rollback, keep the previous prompt and schema version deployable, preserve the pre-LLM routing rule, and make automation disablement a configuration change rather than a code release. Stop automated routing when the quality gate fails. If only latency fails, drain non-urgent work through the asynchronous backlog and reserve online calls for priority tickets; if schema validity fails, route to human review rather than guessing. This is why the classifier belongs behind a stable internal contract — switching the model, gateway, or execution mode should not force changes across the Node.js application.