Short answer: for a fintech support queue, start with a zero-shot LLM baseline behind a strict label contract, then move stable slices to an embeddings classifier; use reranking when retrieval produces plausible but crowded candidates. This sequence is usually the least complex way to compare alternatives to fine-tuning while keeping per-tenant cost visible.
I build RAG and agent features in Python, so I care about the notebook-to-prod handoff. A tagging experiment is useful only when another engineer can reproduce the eval, inspect an abstention, and explain which tenant paid for the work. The model choice is secondary to that accounting boundary.
Three words: measure the queue.
The real problem is label policy, not model selection
Support tickets in fintech carry operational consequences. A label might route a card dispute, request an identity check, or start a billing review. A plausible-looking tag can still be wrong enough to create a bad handoff, so accuracy alone is a thin acceptance rule.
Before comparing embeddings, a classifier, a zero-shot LLM, or reranking, freeze a small label policy. Give every label a definition, positive examples, exclusions, and an abstain outcome. “Login problem” and “account locked after verification” may sound close, but a reviewer needs to know which evidence separates them. If two reviewers cannot apply the distinction consistently, more model capacity will not repair the taxonomy.
Keep a held-out set that contains common tickets, rare labels, ambiguous wording, long threads, and adversarial text copied from messages. The exact mix should reflect each tenant rather than a global average. Record the tenant identifier, ticket identifier, policy version, predicted label, confidence or score, reviewer label, latency, input and output token counts when available, and the final decision. Do not send raw ticket text to a dashboard when a redacted excerpt or hash answers the operational question.
For example, suppose one tenant calls a ticket “card payment reversed” while another uses “duplicate charge review” for a similar customer story. A global accuracy score can hide that these tenants have different routing policies. I would preserve the tenant-specific expected label, the shared taxonomy version, and the reason for any exception in the eval row, then inspect the confusion matrix separately for each tenant. If the label is allowed globally but not for that tenant, the classifier should abstain rather than manufacture a cross-tenant interpretation. That extra bookkeeping feels slow in a notebook, but it prevents a later cost dashboard from mixing model work with policy work. It also gives the reviewer a concrete question: was the prediction semantically wrong, or did the taxonomy fail to express the workflow? Those are different fixes, with different owners and different release gates. Don't collapse them into one “accuracy” column.
The first failure mode is leakage: examples, reviewer corrections, or a prompt revision slip into the evaluation set. The second is a majority-label illusion: aggregate accuracy rises while a low-volume compliance queue quietly loses recall. Report per-label precision, recall, abstention rate, and confusion pairs. A cost number without these slices is not a decision.
How should a fintech team compare embeddings, classifiers, zero-shot LLMs, and reranking for support ticket tagging?
Use the same label policy and the same held-out tickets for each route. That makes the comparison about the system rather than a favorable prompt.
| Approach | Good fit | Main trade-off |
|---|---|---|
| Zero-shot LLM | A new taxonomy, nuanced language, or fast baseline | Prompt tokens, output validation, and variable confidence |
| Embeddings plus classifier | Stable labels with enough reviewed examples | Thresholds and drift need calibration |
| Embeddings plus nearest examples | A small catalog with meaningful historical cases | Similar wording can hide a different workflow |
| Reranking after retrieval | A large catalog where candidate descriptions overlap | It depends on retrieval quality and adds a second score |
The zero-shot route should return one allowed label, a brief rationale for review, and an abstention signal in a schema your application validates. It is easy to start, which matters when the label policy is still changing. It is a poor final answer if the prompt accepts arbitrary labels or if a downstream job treats confidence as a probability without calibration.
Embeddings are useful when meaning repeats. Store vectors for reviewed examples or label descriptions, retrieve candidates, and classify only among those candidates. A Postgres deployment with the pgvector extension is one standards-friendly way to keep ticket metadata and vector search close together. Similarity is evidence, not a verdict: two tickets can be close in language while requiring different controls.
Reranking belongs after retrieval, not in place of it. Give the reranker a bounded candidate set and ask it to order relevance using the ticket and the candidate definitions. This helps when a broad first pass finds several credible labels. It is not suitable when label descriptions are vague, and it does not create missing candidates. If retrieval never returns the correct label, a better ordering model cannot select it.
A runnable Python baseline with tenant-level cost records
The first implementation can be intentionally plain. The provider adapter below is an interface, not a vendor integration; the important part is that every classification returns a result record with an ownership key and a versioned policy. A local model, an internal HTTP service, or a hosted model can implement the same boundary.
from dataclasses import dataclass
from typing import Protocol
@dataclass(frozen=True)
class TicketResult:
tenant_id: str
ticket_id: str
label: str
score: float | None
abstained: bool
policy_version: str
input_tokens: int
output_tokens: int
class Tagger(Protocol):
def tag(self, ticket: str, labels: dict[str, str]) -> tuple[str, float | None, int, int]:
"""Return label, optional score, and token counts."""
def classify_ticket(
tagger: Tagger,
tenant_id: str,
ticket_id: str,
ticket: str,
labels: dict[str, str],
policy_version: str,
threshold: float = 0.72,
) -> TicketResult:
label, score, input_tokens, output_tokens = tagger.tag(ticket, labels)
allowed = label in labels
abstained = not allowed or score is None or score < threshold
return TicketResult(
tenant_id=tenant_id,
ticket_id=ticket_id,
label=label if allowed and not abstained else "abstain",
score=score,
abstained=abstained,
policy_version=policy_version,
input_tokens=input_tokens,
output_tokens=output_tokens,
)
def cost_by_tenant(results: list[TicketResult], input_rate: float, output_rate: float) -> dict[str, float]:
totals: dict[str, float] = {}
for result in results:
amount = result.input_tokens * input_rate + result.output_tokens * output_rate
totals[result.tenant_id] = totals.get(result.tenant_id, 0.0) + amount
return totals
The threshold is a configuration value, not a universal truth. Calibrate it on reviewed tickets and keep the calibration set separate from the final eval. A tenant with a small volume can still need a stricter review path if an incorrect tag triggers a sensitive workflow. A tenant with long conversations may need truncation rules, but those rules belong in the recorded policy version so a later result remains explainable.
A useful batch record has one row per attempt, including route name, model or embedding version, elapsed time, token counts, estimated cost, and whether a human accepted the label. Aggregate by tenant and policy version. Never hide retries inside a single total: a retry can consume tokens and add latency even when the final label looks normal.
Where each alternative fails in production
Fine-tuning is tempting because it promises a model shaped around the queue. It also creates a training-data lifecycle, a deployment artifact, and a new evaluation version to govern. If the underlying label policy is unstable, that work encodes yesterday’s disagreement. Prompted classification is a better diagnostic starting point because it exposes policy ambiguity before training turns it into a less visible artifact.
Embeddings can fail through semantic shortcuts. “Refund pending” and “refund requested” share vocabulary, yet their owners and service-level targets may differ. A nearest-neighbor classifier can also become overconfident as the example bank grows. Monitor score distributions by label and tenant, and sample near-threshold cases for review.
Zero-shot LLMs can fail through schema drift, instruction conflicts inside ticket text, or inconsistent handling of an unfamiliar label. Treat ticket text as untrusted input. The OWASP Top 10 for LLM Applications is a useful security checklist, especially when a generated label can trigger an action. Validate the output, allow only known labels, and keep abstention available.
Reranking can fail through candidate starvation and score confusion. Its score is a ranking signal, not automatically a calibrated class probability. Keep retrieval recall and final classification quality as separate measurements. I'm not sure a reranker is worth its extra moving part for a six-label queue; your mileage may vary when the catalog reaches thousands of operational labels.
The decision rule I would put in the runbook
Run a zero-shot baseline first, with deterministic parsing, an allowlist, and human review for abstentions. If stable labels have enough reviewed examples and a repeatable threshold, test embeddings plus lightweight classification on the same held-out set. Add reranking only when retrieval returns multiple credible candidates and the eval shows that ordering improves the intended per-label metric.
The catch is that embeddings are not suitable when the taxonomy changes every week, the examples are sparse, or labels are defined by workflow ownership rather than language. Stick with a zero-shot baseline when the policy is still being negotiated. Choose a conventional supervised classifier when the label set is stable, reviewed data is abundant, and the team wants a compact model with an explicit feature and calibration pipeline. Keep fine-tuning for the point after those baselines plateau and the training-data contract is owned.
The operational checklist should read as prose: pin the policy and prompt versions, split evaluation data by tenant and time, redact sensitive fields, validate labels, log abstentions, report per-tenant cost, sample errors, and compare the next revision against the same held-out set. Ship only when a reviewer can trace a label back to the input, policy, score, and billable work. That trace is the product.
Top comments (0)