TL;DR
Use a closed label set, require one small JSON object, validate it before any side effect, and send uncertain support tickets to review. For an edtech queue, that is the least complex design that keeps an LLM useful without letting generated text become routing state. It also gives a Node.js caller a clean operational contract: accepted labels are exact, unknown labels are rejected, and a retry cannot create a second ticket action.
The model call is only one stage. Quality depends on the taxonomy and examples; latency depends on the deadline, queue policy, and review threshold. Treat those controls separately.
How should a Node.js LLM return exact JSON labels for multi-label ticket classification?
Make the output contract narrower than the prompt. A useful response for ticket T-1842 is an object with a schema version and a bounded array such as billing, login, or course_access. The consumer should reject prose, missing fields, duplicate labels, unknown labels, and more labels than the business workflow can use. Don't silently map payment_problem to billing. That feels helpful during a demo, but it hides taxonomy drift and turns a model guess into an unaudited production rule.
A Node.js service can enforce this with its normal JSON Schema validator. Keep the schema and the allowed-label set in source control, attach their version to every classification job, and return a client-side contract error such as HTTP 422 when the generated object is structurally valid JSON but violates the closed set. The same pattern works for ecommerce product tagging: replace the ticket taxonomy with approved catalog tags, but keep generated strings out of downstream filters and facets.
The request also needs an idempotency key derived from the ticket revision and taxonomy version. If a learner adds a message after classification, that is a new revision and should produce a new job. If a worker merely retries the same revision, it should observe the existing result. This distinction matters because queues commonly deliver work more than once; exactly-once business behavior comes from an idempotent state transition, not from hoping for exactly-once transport.
I've been paged by missed jobs and duplicate deliveries. The painful part was rarely the classifier itself — it was a worker that had enough information to perform a side effect but not enough durable state to prove whether that side effect had already happened. The invariant I now want is plain: no validated result, no routing write; no successful compare-and-set, no notification.
For quality versus latency, use a decision table rather than one global timeout:
| Situation | Automated action | Why |
|---|---|---|
| Allowed labels, strong evidence, within deadline | Commit labels | Fast path is bounded and auditable |
| Allowed labels, weak evidence | Send to review | A quick wrong route costs another queue hop |
| Unknown or malformed output | Record contract rejection; do not route | Generated text cannot expand the taxonomy |
| Deadline exhausted | Leave the ticket unclassified for retry or review | Late output must not mutate newer state |
| Ticket revision changed | Discard the stale result | Classification belongs to a specific input |
I'm not sure one confidence threshold will transfer between billing disputes and course-access questions; your mileage may vary. Resolve that uncertainty with labeled evaluation data from each queue, then choose thresholds against the actual cost of a wrong route and the actual cost of human review. A model's self-reported confidence is not enough evidence on its own.
The incident invariant is a state transition, not a prompt
Prompt instructions still matter. State the allowed labels, describe when multiple labels are valid, include ambiguous examples, and prohibit explanations. Yet a prompt is probabilistic input to a component, while the routing database is authoritative state. Put a deterministic boundary between them.
The job record should move through explicit states such as pending, leased, classified, review, and applied. A lease has an expiry so another worker can recover abandoned work. The apply step compares the ticket revision, taxonomy version, and prior state in one transaction. If any has changed, the result is stale.
Stop there.
This is where scheduling policy affects classification quality. Suppose the end-to-end target is two seconds. Spending the full two seconds on the model leaves no budget for queue delay, validation, storage, or cancellation. Instead, allocate a shorter model deadline and reserve time for the deterministic tail. The exact split must come from production latency distributions; inventing a universal ratio would be false precision. Watch p50, p95, and p99 separately, because a healthy median can coexist with a review queue that grows during tail-latency spikes.
Use three operational counters at minimum: contract rejections by reason, stale results by ticket revision, and apply attempts deduplicated by idempotency key. Add queue age and review age as gauges. Those signals answer different questions. Contract rejections expose a prompt or schema mismatch. Stale results expose work completing after the source changed. Deduplicated applies show transport retries being absorbed as designed. Queue age tells the on-call engineer whether the system is keeping up.
Keep raw support text out of broad logs. Store a request identifier, input hash, taxonomy version, selected labels, validation result, timings, and the final state transition. Access to ticket content should follow the same controls as the support system itself. This reduces accidental exposure while preserving enough evidence to replay a decision in a restricted environment.
One more trap: changing a label in place. Renaming course_access to content_access without a versioned migration makes old evaluations incomparable and queued jobs ambiguous. Publish a new taxonomy version, let workers finish or invalidate old jobs deliberately, and map historical analytics through an explicit migration.
Slow is smooth here.
The preventative path belongs before side effects
The following Go boundary is intentionally boring. It shows the logic a Node.js caller or worker should mirror: strict decoding, a fixed set, duplicate detection, a maximum label count, and an idempotent apply interface. The model adapter is outside this function, so no provider-specific response shape leaks into routing code.
package triage
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
)
type Result struct {
SchemaVersion string `json:"schema_version"`
Labels []string `json:"labels"`
}
type ApplyRequest struct {
IdempotencyKey string
TicketID string
TicketRevision int64
TaxonomyVersion string
Labels []string
}
type Store interface {
ApplyOnce(ctx context.Context, req ApplyRequest) (applied bool, err error)
}
var allowed = map[string]struct{}{
"billing": {},
"login": {},
"course_access": {},
"technical_issue": {},
}
func ValidateAndApply(ctx context.Context, store Store, raw []byte, req ApplyRequest) error {
decoder := json.NewDecoder(bytes.NewReader(raw))
decoder.DisallowUnknownFields()
var result Result
if err := decoder.Decode(&result); err != nil {
return fmt.Errorf("invalid classification JSON: %w", err)
}
var trailing any
if err := decoder.Decode(&trailing); !errors.Is(err, io.EOF) {
return errors.New("classification contains trailing JSON")
}
if result.SchemaVersion != req.TaxonomyVersion {
return errors.New("taxonomy version mismatch")
}
if len(result.Labels) == 0 || len(result.Labels) > 3 {
return errors.New("label count outside policy")
}
seen := make(map[string]struct{}, len(result.Labels))
for _, label := range result.Labels {
if _, ok := allowed[label]; !ok {
return fmt.Errorf("unknown label %q", label)
}
if _, ok := seen[label]; ok {
return fmt.Errorf("duplicate label %q", label)
}
seen[label] = struct{}{}
}
req.Labels = append([]string(nil), result.Labels...)
applied, err := store.ApplyOnce(ctx, req)
if err != nil {
return fmt.Errorf("apply classification: %w", err)
}
if !applied {
return nil // A retry observed the already-applied transition.
}
return nil
}
The storage method must compare the ticket revision and taxonomy version atomically. An in-memory deduplication map doesn't count.
Test this boundary without calling a model. Feed it prose around JSON, two concatenated objects, wrong schema versions, empty arrays, four labels, duplicates, and visually similar strings. Then test worker races: two leases applying the same idempotency key, a ticket revision changing during inference, and a deadline firing before commit. Only after those tests pass should an end-to-end evaluation measure classification quality.
Deploy taxonomy and prompt changes as versioned artifacts. Shadow them on a sample of already resolved tickets, compare disagreement by queue, and canary the worker while watching review age and contract rejection rate. Rollback means stopping new jobs for that version; already applied labels remain attributable to the version that produced them. That postmortem trail is worth the extra column.
When is closed-set LLM classification the wrong choice?
The catch is that exact labels solve output control, not taxonomy design. This approach is not suitable when the organization cannot agree on label definitions, when almost every ticket needs a new label, or when routing depends on facts absent from the ticket. Fix the source data or workflow first. A validator cannot manufacture evidence.
Stick with deterministic rules when a small set of explicit fields decides the route, such as plan type plus account status. Rules are faster to explain and easier to test. Use human review when the harm from a wrong label is high, especially for account security, payments, or learner-safety escalation. Consider an embeddings-based retrieval path when the main problem is finding semantically similar resolved tickets rather than assigning a small approved taxonomy; embeddings are vector representations useful for classification, search, and related tasks. That is an architectural alternative, not a reason to weaken the output contract.
Attachments create another boundary. A Node.js intake service may preprocess screenshots before a separate vision-capable stage, but image evidence should be tracked as an input revision just like text. If the attachment changes, the old classification is stale. Don't let asynchronous media processing quietly update the evidence after routing has committed.
There is also a team cost. Closed sets require taxonomy ownership, labeled examples, migrations, review operations, and on-call runbooks. For a low-volume queue, manual triage may be the cheaper and clearer system. For a high-volume queue with stable labels, automation earns its place when the review load, rejection rate, latency tails, and duplicate suppression are visible. Price per model call is a secondary input; total operational effort and the cost of wrong routing usually dominate the decision.
The finish line is not valid JSON. It is a ticket transition that is timely, attributable, repeatable, and harmless when delivered twice.
Top comments (0)