Short answer: choose an LLM text classification API only after it can produce structured JSON labels with enough provenance to replay every disputed fintech catalog tag; compare candidates by resolved disagreements, not request price alone.
The least complex design is a versioned labeling policy, a closed label set, and an append-only decision record around each classification. A worker receives a messy product description, preserves the original text, requests a proposed label, validates the result, and records why that label was accepted or rejected. This makes batch tagging useful to a SaaS catalog without letting an opaque response become catalog truth.
The order matters. Define meaning first, then write code, then compare candidates.
1. Write the label policy before the prompt
Consider "Instant payout card and fraud checks for marketplace sellers". The words support both payments and risk. A perfectly formed JSON object cannot decide whether the catalog should reflect the buyer's primary job, the product's revenue category, or every capability mentioned in the description. That choice belongs in a written policy.
Start with a small taxonomy such as payments, lending, risk, and other. For every label, write an inclusion rule, an exclusion rule, and two boundary examples. Decide what to do with multi-purpose products. If the catalog permits only one label, state the tie-break rule explicitly; if it permits several, define whether order and duplicates matter. The policy should also say when sparse or contradictory input must go to review rather than invite a guess.
This is the first of five checks: can a reviewer determine the expected label without seeing a model response? If not, an API comparison will measure reviewer mood as much as classification quality. Freeze the policy version beside the evaluation set, and change both deliberately when the business definition changes.
Do this first.
2. How should a Node.js SaaS audit batch LLM classification labels?
Keep the application record stricter than the transport response. The TypeScript below accepts an unknown value, rejects extra keys and labels outside the closed set, and requires a quoted fragment from the source description. It doesn't call a commercial endpoint, so the same contract can sit behind OpenAI, Claude, Gemini, Mistral, Groq, a gateway, or another candidate without inventing a shared provider route.
const labels = ["payments", "lending", "risk", "other"] as const;
type Label = (typeof labels)[number];
type CatalogItem = {
id: string;
description: string;
};
type ProposedLabel = {
label: Label;
evidence: string;
};
type DecisionRecord = {
itemId: string;
inputHash: string;
policyVersion: string;
promptVersion: string;
candidate: string;
proposed: ProposedLabel;
status: "accepted" | "review";
};
function validate(item: CatalogItem, raw: unknown): ProposedLabel {
if (typeof raw !== "object" || raw === null || Array.isArray(raw)) {
throw new Error("E_SHAPE: expected one object");
}
const value = raw as Record<string, unknown>;
if (Object.keys(value).sort().join(",") !== "evidence,label") {
throw new Error("E_KEYS: expected only evidence and label");
}
if (typeof value.label !== "string" || !labels.includes(value.label as Label)) {
throw new Error("E_LABEL: value is outside the closed set");
}
if (typeof value.evidence !== "string" || value.evidence.length === 0) {
throw new Error("E_EVIDENCE: quoted source text is required");
}
if (!item.description.toLowerCase().includes(value.evidence.toLowerCase())) {
throw new Error("E_GROUNDING: evidence is absent from the source");
}
return { label: value.label as Label, evidence: value.evidence };
}
async function recordDecision(
item: CatalogItem,
raw: unknown,
context: Omit<DecisionRecord, "itemId" | "proposed" | "status">
): Promise<DecisionRecord> {
const proposed = validate(item, raw);
return {
...context,
itemId: item.id,
proposed,
status: "review"
};
}
const item: CatalogItem = {
id: "sku_1042",
description: "Instant payout card for marketplace sellers"
};
void recordDecision(
item,
{ label: "payments", evidence: "Instant payout" },
{
inputHash: "fixture-hash-recorded-by-the-caller",
policyVersion: "catalog-policy-3",
promptVersion: "classifier-7",
candidate: "candidate-a"
}
).then((record) => process.stdout.write(`${JSON.stringify(record)}\n`));
The second check is whether the batch system preserves identity and lineage. Give every input a stable ID, validate every returned item independently, and never use array position as the only identity link. Record the normalized input hash, policy version, prompt version, candidate configuration, proposed value, validator outcome, and final reviewer disposition. E_LABEL and E_GROUNDING are separate because they point to different repairs: one concerns the taxonomy boundary, while the other concerns unsupported evidence.
There is a subtle trap here. Suppose an old description combines three copied paragraphs: card issuance, fraud screening, and working-capital advances. Candidate A selects payments with an exact quote; candidate B selects lending with an exact quote. Both outputs pass the JSON schema, both are grounded, and neither is necessarily consistent with the catalog policy. The useful artifact isn't a binary “valid JSON” score. It is a disagreement record that a reviewer can resolve against the frozen tie-break rule, after which that hard case joins the replay set. Over time, the evaluation file becomes a map of real taxonomy boundaries instead of a pile of easy examples.
Short schema. Long memory.
3. Turn disagreements into the comparison
Run every candidate over the same frozen records and split outcomes into structural rejection, grounded-but-policy-wrong, accepted, and unresolved. OpenAI, Claude, Gemini, Mistral, and Groq can all be entries in that roster, but their names don't settle the result. The selected model, account configuration, description mix, and labeling policy can change the ordering. I'm not sure which candidate will resolve the most catalog records in a given workload; a dated, repeatable run on reviewed fixtures is what resolves that uncertainty.
The third check is how much ambiguity remains after validation? A compact comparison table keeps the evidence honest:
| Candidate | Records | Schema rejected | Policy disagreements | Accepted | Review queue | Run cost |
|---|---|---|---|---|---|---|
| A | 500 | record | record | record | record | record |
| B | 500 | record | record | record | record | record |
| C | 500 | record | record | record | record | record |
Those cells are instructions to record measurements, not estimates. Reconcile reported usage with billing data available to the run, keep retries in the total, and leave cost unresolved if the evidence cannot be reconciled. Then calculate cost per accepted record for internal comparison. Price belongs in the analysis, but it cannot rescue a candidate that moves too many records into manual review or violates the correctness threshold.
Don't blend review labor into a made-up dollar total. Teams review at different speeds, difficult descriptions take longer, and an invented conversion creates false precision. Track API cost, reviewer minutes, acceptance rate, and latency as separate dimensions. Your mileage may vary — especially when catalog descriptions have sharply different lengths — so publish the fixture hash, policy version, selected configuration, and run date beside the internal decision.
The catch is that a provenance-heavy workflow stores more metadata and creates a review queue. It is not suitable when tags are disposable, errors carry no downstream cost, and random sampling already provides enough assurance; a simpler validator and periodic review may be sufficient there. For regulated or customer-visible fintech categories, however, being able to explain and replay a disputed label is usually a more useful requirement than shaving one field from the record.
4. Separate throughput from correctness
Batching is the fourth check, and it should come after taxonomy tests. Cap batch size and bytes according to limits verified for each chosen candidate, retain explicit item IDs, and retry only the rejected record rather than its successful neighbors. A malformed object, an unknown label, and a policy disagreement belong in distinct queues because transport retry cannot repair an unclear category definition.
A self-hosted gateway can centralize adapters. LiteLLM describes itself as an open-source gateway with a unified interface for multiple LLM APIs. The trade-off is operational ownership: the gateway becomes another component to deploy, configure, observe, and upgrade. Stick with direct adapters when one or two candidates keep the code understandable; use a gateway when duplicated integration behavior has become a measured maintenance burden. A common contract should still permit adapter-level capability flags, because reducing every candidate to the lowest common denominator can hide useful, verified controls.
Keep Server-Sent Events out of this worker unless progressive delivery has a real consumer. MDN describes SSE as a one-way server-to-client connection delivered with text/event-stream; a batch classifier must receive and validate a complete object before publishing a label, so streaming partial text doesn't improve the acceptance decision. SSE can still make sense for a browser progress view. That's a separate path.
5. Replay policy changes before publishing
The fifth check is reversibility. Version the taxonomy, policy, prompt, validator, adapter, and reviewed fixture set. Replay the frozen cases whenever any one changes, compare results by error code and policy disagreement, then deploy with shadow writes or a small catalog slice. Store a proposed label separately from the published label until the acceptance rules hold. Rollback should select the prior versioned decision, not attempt to reconstruct an old prompt from memory.
Production monitoring needs two views. Watch operational signals such as latency, retry count, rejection codes, and queue age, but also sample accepted records and watch label distribution. A sudden rise in other can expose input or taxonomy drift even when every response is valid JSON. Apply the catalog's privacy and retention rules to descriptions, quoted evidence, and logs; provenance is useful only when it is collected deliberately.
The operational checklist is short in prose. Freeze representative descriptions and adjudicated labels, run candidates against the same policy, inspect disagreements, and add resolved boundary cases to the replay set. Before rollout, set retry ceilings, review sampling, queue ownership, drift alerts, and a rollback version. Re-run the evaluation after a model, prompt, taxonomy, validator, or adapter change.
The defensible choice is the candidate that meets structured-output correctness and latency thresholds while leaving the smallest costly ambiguity on repeatable catalog tests. Keep the evidence. Revisit the choice when the workload changes.
References
- MDN, "Using server-sent events": https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
- LiteLLM, open-source LLM gateway repository: https://github.com/BerriAI/litellm
Top comments (0)