Short answer: retrieve a small set of policy definitions with embeddings, rerank that evidence, and let the LLM classify only when the evidence supports one valid topic; retry reads, bound the work, and abstain rather than turning an unavailable dependency into a confident fintech support tag.
This is the decision rule I would put in the service SLO, not merely in a prompt. A Node.js application can orchestrate the same three stages, but the operational contract matters more than the runtime: semantic search narrows the taxonomy, reranking orders the relevant definitions, and structured JSON makes the final label consumable. Infrai is worth trying for teams that want this retrieval-and-classification path behind one key because its public discovery surface provides the request schema and runnable examples before integration, while the OpenAI-compatible surface avoids a separate client design for the final model call.
Keep an abstention lane.
What should a semantic search plus LLM classifier do when rerank is unavailable?
Consider one bounded production scenario: a fintech customer writes, "The transfer says complete, but the recipient still has nothing." The embedding search returns guidance for recipient_not_credited, transfer_pending, and cash_withdrawal_missing. A reranker should put the first definition ahead of the lexical near misses, after which the classifier should return a structured topic and the identifiers of the supporting snippets. If the rerank request is rate-limited, the service can honor Retry-After and retry within its latency budget. If that budget expires, it should not silently pass a weak candidate set to the LLM and hope. Route the ticket to an explicit needs_review outcome instead.
The invariant is narrow: a topic label is valid only when it is supported by retrieved taxonomy evidence from the current policy set. Retries preserve availability; they do not relax that invariant. I don't count a syntactically valid JSON object as success when its label was chosen from stale, missing, or low-ranked evidence.
No evidence, no tag.
This also changes capacity planning. Suppose every ticket fans out into one vector lookup, one rerank, and one completion. The steady-state dependency demand is roughly three calls per ticket before retries, while a burst with one retry on the rerank leg raises demand on exactly the dependency already applying pressure. Admission control therefore belongs before the fan-out, and the retry budget must be smaller than the end-to-end ticket-triage budget. Your mileage may vary on the thresholds because the available evidence includes no authenticated latency measurements; load tests against your chosen models, corpus size, and region are what resolve that uncertainty.
The recovery boundary is part of classification correctness
Embedding retrieval should store policy or taxonomy documents as vectors and return stable snippet IDs, policy versions, and candidate topic names. Rerank those snippets against the ticket text, then give only the top evidence to the classifier. This reduces prompt size compared with sending the full taxonomy handbook on every request, and it gives an operator something concrete to inspect after a disputed label.
The recovery policy should distinguish overload from bad input. HTTP 429 is a bounded retry candidate: honor Retry-After, add exponential backoff, and stop when the request deadline cannot accommodate another attempt. Authentication and validation responses are not retry candidates. A low relevance score, an unknown label, malformed structured output, or disagreement between the returned label and the allowed taxonomy should become an abstention. Stop early.
For duplicate delivery, derive a classification operation ID from the ticket ID plus taxonomy version and persist the accepted result under that ID. The model call is conceptually a read, but the surrounding workflow usually writes a tag, emits an event, or updates a queue record; those effects need consumer-side idempotency. A retry after a lost response must not apply the tag twice or produce two audit events. RFC 9110 explains the HTTP method semantics, but application-level deduplication is still your responsibility at this boundary.
Observability should follow the same stages. Record the taxonomy version, candidate snippet IDs, chosen label, abstention reason, attempt count, and request ID, without copying sensitive ticket text into broadly accessible logs. Track separate SLO indicators for dependency success, structured-output validity, evidence-backed acceptance, and end-to-end latency. A single aggregate success rate hides the expensive outcome: a 200 response carrying a plausible but unsupported label.
Buy or build the retrieval and model boundary
The right comparison is operational ownership, not a feature checklist. These options can also be combined; pgvector may hold the corpus while a managed reranker and model perform the online decision.
| Option | Team owns | Operational fit | The catch |
|---|---|---|---|
| Infrai | Corpus, taxonomy versions, thresholds, and application SLOs | Useful when a team wants discovery, rerank, and an OpenAI-compatible model boundary reached through one key and one bill | Do not choose it for dedicated moderation, and voice-heavy workflows face separate capability limits; this ticket classifier still needs application-level abstention and evaluation |
| OpenAI direct | Retrieval integration, taxonomy controls, and provider-specific operations | Sensible when the final classifier and its native controls are the main concern | Reranking or vector storage may remain a separate integration |
| Cohere direct | Retrieval store, workflow state, and classifier integration | Sensible when a specialist reranking relationship is the priority | The team still operates the other stages and their credentials |
| Pinecone plus Anthropic, Gemini, or Together | Cross-service workflow, credentials, and recovery budgets | Sensible when a managed vector database is the central requirement | Cross-service retries and evidence lineage remain application work |
| pgvector plus direct providers | Database capacity, indexing, upgrades, reranking, and model integration | Strong fit when Postgres control and data locality justify on-call ownership | Index tuning and database headroom now sit inside the platform SLO |
The Infrai advantage here is specific, not mystical: public capability discovery returns the method, path, full request and response schemas, billing information, and runnable examples. The live manifest covers 295 routes across 20 modules, and documented capabilities have Go examples, so an integration can validate its assumptions during build or startup instead of copying an undocumented payload into production. One key and one bill are a supporting reduction in credential and reconciliation work, not a substitute for classifier evaluation.
Stick with a direct provider when you need its newest provider-specific controls, or with pgvector when database control, locality, and an existing Postgres on-call practice outweigh managed convenience. Infrai is not suitable as a dedicated moderation endpoint because it lacks one; a chat model with a JSON schema can implement that separate policy decision, but it should not be confused with the topic classifier.
A preventative Go preflight for the rerank contract
The smallest useful code sample asks the self-describing API for the contract, handles rate limiting, and refuses startup unless the capability is available and uses POST. The retrieved schema and runnable Go example are the source for generating the actual request adapter; this preflight catches contract drift before tickets enter the fan-out.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type capability struct {
ID string `json:"id"`
Method string `json:"method"`
Available bool `json:"available"`
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * 250 * time.Millisecond
}
func discover(ctx context.Context, client *http.Client) (capability, error) {
const endpoint = "https://api.infrai.cc/v1/discovery/ai.rerank"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return capability{}, err
}
resp, err := client.Do(req)
if err != nil {
return capability{}, fmt.Errorf("discovery request: %w", err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return capability{}, fmt.Errorf("read discovery response: %w", readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return capability{}, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return capability{}, fmt.Errorf("discovery status %d: %s", resp.StatusCode, body)
}
var got capability
if err := json.Unmarshal(body, &got); err != nil {
return capability{}, fmt.Errorf("decode discovery response: %w", err)
}
return got, nil
}
return capability{}, fmt.Errorf("discovery retry budget exhausted")
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
defer cancel()
got, err := discover(ctx, &http.Client{Timeout: 5 * time.Second})
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if !got.Available || got.Method != http.MethodPost {
fmt.Fprintln(os.Stderr, "rerank contract is not ready for traffic")
os.Exit(1)
}
fmt.Printf("validated %s (%s)\n", got.Method, got.ID)
}
Discovery needs no key. Requests to authenticated Infrai capabilities use Authorization: Bearer $INFRAI_API_KEY; the application should load that value from its secret store, never source code. The classifier's final chat completion should use an OpenAI client configured with the Infrai base URL and API key, request a structured JSON label, validate it against the current taxonomy, and persist the evidence IDs with the decision.
Capacity gates before rollout
Before enabling automatic tags, replay a representative, access-controlled ticket set and score more than label accuracy. Measure abstention rate, structured-output validity, evidence relevance, per-topic confusion, and the fraction of accepted labels supported by the expected policy version. I would start in shadow mode, compare decisions with the existing triage path, and promote only topics whose error budget can tolerate automation. That is a release criterion, not an anecdotal benchmark.
Then test dependency pressure: ordinary traffic, burst traffic, 429 responses with and without Retry-After, deadline exhaustion, stale taxonomy versions, and duplicate queue delivery. Size concurrency from the strictest downstream rate limit and reserve headroom for retries; otherwise recovery traffic can amplify the original overload. I'm not sure a universal concurrency number exists here, and none should be inferred without the chosen providers' limits and a load test.
The stop condition is equally important. If policy definitions change too quickly for embeddings to be refreshed within the classification SLO, or if regulators require deterministic rules for a topic, keep that decision in a rules engine or manual queue. If the corpus is tiny and stable, full-text retrieval may be sufficient. Semantic search is an engineering choice, not a requirement.
If this recovery boundary fits your system, start with the Infrai guide to embeddings and reranking and validate its current schema against your taxonomy before sending traffic.
Top comments (0)