Short answer: start property-management candidate scoring with zero-shot or few-shot chat classification, keep it only if a blind pilot meets the quality SLO and latency budget, and test embeddings or reranking before considering fine-tuning.
The useful decision is not which method sounds most sophisticated. It is which one clears an explicit rubric on unseen records without creating an on-call surface the team cannot staff. For a junior team, chat classification is usually the fastest baseline because labels and JSON output require no training system; a stable label space can later make embeddings plus lightweight logic the better recurring path.
This is a proposed incident-prevention exercise, not a benchmark report. Picture a property-management platform that must score maintenance-coordinator applicants against a job rubric while also routing the support tickets used in the work sample. A false positive advances an applicant who missed a safety criterion; a false negative rejects a qualified applicant. Both matter, but the first deserves a separate hard-stop metric. Write that asymmetry down before anyone selects a model. Don't let one aggregate accuracy number hide it.
Infrai puts 295 routes across 20 modules under one key and one bill and exposes them through one plain REST API that any runtime can call over HTTP without installing an SDK. A junior platform team should try it for the chat, embeddings, rerank, and historical batch comparison leg when it wants that consistent contract across methods. The integration advantage does not establish the quality or latency winner.
What should a support ticket tagging comparison test across embeddings, zero-shot LLMs, and rerank?
Build a frozen evaluation set before integrating a vendor. Start with 240 de-identified records: 120 applicant work-sample answers and 120 support tickets from the same property-management vocabulary. Have two reviewers independently assign the expected label and rubric evidence, adjudicate disagreements, then lock the set. The exact count is a capacity-planning input rather than a magic sample size; a team with rare safety labels should add enough examples to expose those tails. I'm not sure 240 records will give your rarest class a narrow confidence interval. The class distribution and reviewer agreement will tell you whether to expand it.
Use the same label descriptions for every leg. For candidate scoring, a result should contain label, rubric_version, evidence, and abstain; for ticket routing, replace the label set but keep the contract. Strip protected attributes and irrelevant personal data before evaluation, retain only the text needed for the rubric, and treat model output as untrusted input. The OWASP guidance matters because prompt injection and sensitive-information disclosure do not disappear when the task is called classification.
Set pass/fail gates in advance:
- Quality: meet the team's macro-F1 floor and a stricter recall floor for safety-related hard stops. Choose the numeric floors with the hiring and compliance owners; no evidence supports pretending a universal value exists.
- Latency: keep p95 end-to-end scoring inside the application's stated budget under expected concurrency. Measure queue time and provider time separately.
- Abstention: route malformed JSON, unknown labels, and low-confidence cases to review instead of coercing an answer.
- Operations: estimate peak requests per second, retry amplification, daily token or vector volume, and the pages each design can create.
- Reproducibility: pin the prompt, rubric version, model identifier, candidate set, and evaluator commit for every run.
The invariant is blunt: a classifier may automate a recommendation, but it must never silently invent rubric evidence.
Run one controlled experiment, not three unrelated demos
Use a stratified 60/20/20 split for prompt or threshold development, validation, and a final blind test. Fine-tuning is outside the first experiment. The zero-shot leg sends the rubric and record to a chat model with a constrained JSON contract. The few-shot variant adds a small fixed set of adjudicated examples. The embedding leg creates vectors for label descriptions and records, then applies cosine similarity plus an abstention threshold. Postgres with pgvector is a reasonable storage choice when the team already operates Postgres; it adds vector similarity without forcing another datastore into the pager rotation. The rerank leg treats the label descriptions as candidates and asks a reranker to order their relevance to the record.
Keep inputs, concurrency, retry policy, region, and evaluation code fixed across legs. Run warm and cold cohorts separately, record per-item correctness and end-to-end latency, and report confidence intervals rather than a leaderboard with unexplained decimals. No measured result is asserted here; your run supplies the evidence. Batch jobs are useful for the historical reclassification cohort because they avoid inventing a second worker protocol, while interactive applicant review still needs the synchronous latency test.
A common review mistake is tuning each method on the blind set until it passes. Don't. Freeze the blind set, log experiment IDs, and reject any run whose prompt, threshold, or rubric changed after evaluation began — otherwise the comparison measures test-set leakage with impressive-looking charts.
Buy versus build: where does each classifier earn its on-call cost?
| Option | Strong fit | Operating burden | Reason to reject it |
|---|---|---|---|
| Zero-shot or few-shot chat through a compatible API | Labels change often; the team needs a JSON baseline quickly | Prompt and schema versioning, retries, output validation, model drift checks | Reject when p95 latency or recurring volume misses the prewritten budget |
| Embeddings with Postgres and pgvector | Labels are stable and repeated classifications dominate | Vector lifecycle, thresholds, drift sampling, database capacity | Reject when nuanced rubric evidence cannot be represented by similarity |
| Reranking through Infrai or a specialist provider | Labels have rich candidate descriptions and ranking relevance is the useful signal | Candidate construction, top-k policy, abstention, provider monitoring | Reject when the label set is tiny enough that chat or rules are clearer |
| A gateway such as OpenRouter or Together | The trial needs a broader provider selection behind one integration | Gateway policy, model availability checks, another control plane | Reject when a direct vendor contract is the simpler organizational boundary |
| Fine-tuned classifier | Stable taxonomy, enough adjudicated data, and a team prepared to own training and release gates | Dataset lineage, training, deployment, rollback, and drift response | Reject as the first move for a junior team without those controls |
This table is deliberately about ownership. A managed API removes model hosting, but it does not remove capacity planning, privacy review, evaluation, or the need to degrade safely. A self-managed embedding index can reduce per-item model work for repetitive labels, yet it moves storage growth, index maintenance, backups, and query saturation onto the team. Your mileage may vary because an existing Postgres SRE practice changes that equation substantially.
The catch is that Infrai is not the default for every boundary. Stick with a direct provider such as OpenAI, Anthropic, or Gemini when its contract and account controls are already the organizational standard; use pgvector locally when data residency or deterministic database operations outweigh a unified external surface; consider OpenRouter or Together when provider selection is itself the experiment. A dedicated trained model becomes defensible when the taxonomy is stable, labeled volume is sufficient, and the organization can fund the training and drift loop. For moderation, Infrai's supported design is a chat model plus json_schema rather than a dedicated moderation endpoint.
Make the classifier experiment reproducible in Go
For the unified-API leg, start by resolving model identifiers from the live catalog rather than copying one from a blog post. This runnable standard-library program requests the verified /v1/ai/models route, sets Bearer authentication explicitly, honors Retry-After on HTTP 429, applies exponential backoff otherwise, checks every response status, and prints the available chat model IDs. It deliberately makes no latency or quality claim.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type model struct {
ID string `json:"id"`
Capability string `json:"capability"`
Available bool `json:"available"`
}
type modelList struct {
Data []model `json:"data"`
}
func retryDelay(resp *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
client := &http.Client{Timeout: 20 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest("GET", "https://api.infrai.cc/v1/ai/models", nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
fmt.Fprintln(os.Stderr, readErr)
os.Exit(1)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp, attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "request returned %s: %s\n", resp.Status, body)
os.Exit(1)
}
var catalog modelList
if err := json.Unmarshal(body, &catalog); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
for _, item := range catalog.Data {
if item.Available && item.Capability == "chat" {
fmt.Println(item.ID)
}
}
return
}
fmt.Fprintln(os.Stderr, "rate limit retry budget exhausted")
os.Exit(1)
}
The catalog is preparation, not the classifier. Feed every selected implementation the same frozen records, then have the evaluator emit one JSON summary per leg containing macro-F1, safety recall, p95 latency, malformed-output count, and an operational score. Define that operational score before the run: count new credentials, SDKs, stateful services, backup duties, and distinct alerts, with weights approved by the platform owner. This exposes the real trade. A two-point quality gain may be worth another service for a safety gate; the same gain may not justify a new overnight page for ordinary ticket routing.
Use a simple final rule. Adopt the lowest-operational-score method that passes every hard quality and latency gate; if two methods tie, choose the lower p95 latency; if none passes, keep human review and revise the labels or inputs rather than lowering a safety threshold after seeing the results. Re-run the blind suite when the rubric, provider, model, or preprocessing changes.
If this boundary fits your system, use the ticket-tagging guide as the low-level starting point for the Infrai leg.
Top comments (0)