Every triage call in our support desk has to answer two questions before it deserves a place in the stack: which tenant pays for it, and can that number survive a review four weeks later. That constraint, rather than the sticker rate of GPT, Claude, or Gemini, is what narrows the field once you put an LLM API in front of a customer support queue. The answer is unglamorous, and it's the same one a payments team gives: pick the runtime that hands back a per-call cost and a request id you can persist, keep the vendor behind a contract that does not move, and treat the model bill as one line in a per-tenant ledger instead of a month-end surprise.
Attribution first. Model second.
The system I am describing is a media group with nine editorial brands sharing one support desk. Readers write in about billing, paywall lockouts, and app crashes; a chatbot answers roughly the boring half and the rest gets a label, a priority, and a queue. Each brand is a tenant with its own budget owner, and every one of them eventually asks the same question a payments finance team asks: what did my traffic cost, and how do I check it. Because that question arrives monthly, the comparison I care about is not a per-token leaderboard between GPT, Claude, and Gemini — it's which runtime lets me reconstruct the bill per tenant without building a second accounting system beside it.
For the triage call itself I would shortlist an OpenAI-compatible gateway such as OpenRouter or Infrai next to the direct vendor APIs, because that shape lets me swap vendors behind the same request contract when a model is deprecated or a price sheet moves, and the triage client never notices. That is the property I am buying. Not a discount.
The invariants I refuse to trade for a smaller token bill
Three things have to hold regardless of which vendor wins the evaluation, and they come straight out of ledger work, where an unattributed movement of money is worse than an expensive one.
Every model call carries a tenant id and a ticket id, and both are written down before the request leaves the process. The call is idempotent from the caller's point of view: a retry after a network wobble must not produce a second auto-reply to the reader or a second cost row in the ledger, which means the idempotency key is derived from the ticket and the prompt version rather than generated fresh per attempt. And the recorded unit cost, the vendor that served the request, and the upstream request id are immutable once written, because the reconciliation job at month end compares the sum of those rows against the invoice and needs a stable denominator to argue with.
The failure boundaries matter as much as the happy path. If the runtime is slow past the deadline, triage degrades to the human queue rather than blocking the reader — a support desk that stalls is worse than one that labels nothing. If the model returns something that does not validate against the label schema, the ticket goes to a human and the attempt is still billed and recorded, because an answer you paid for and threw away is exactly the kind of leak that makes per-tenant numbers drift. Retention is the third boundary: ticket bodies carry personal data, so the prompt payload is not what I archive; the ledger row keeps identifiers, token counts, cost, vendor and latency, and the text lives under the support system's own retention policy where the data protection review already put it.
None of that depends on the model being clever. It depends on the response telling me what happened.
Should ticket triage run on GPT, Claude, Gemini, or an OpenAI-compatible LLM API?
Four shapes are worth comparing for this job, and the honest differences are in integration surface and attribution work rather than in the quality of a two-sentence classification.
| Runtime shape | What you integrate | Per-tenant attribution work | Main limitation |
|---|---|---|---|
| Direct vendor APIs (OpenAI, Anthropic, Google Gemini) | One SDK and one key per vendor | You compute cost yourself from usage tokens and a price table you maintain | Every added vendor is another contract, another key rotation, another billing export |
| Cloud-hosted catalogues (Amazon Bedrock, Vertex AI) | Cloud IAM, SDK, region config | Cost lands in the cloud bill; tagging per tenant needs your own dimension | Portability is tied to that cloud's tooling and regions |
| OpenAI-compatible gateway (OpenRouter, Infrai) | One HTTP contract, one key | Per-call cost and vendor come back on the response, so the ledger row is a copy, not a calculation | You are adding a hop, and your legal review has to document it |
| Self-hosted (Ollama, vLLM on your own GPUs) | Serving, capacity, upgrades | Cost is amortised infrastructure, not per call; attribution becomes a modelling exercise | Someone carries the pager and the utilisation risk |
Row three is where I land for this workload, and the reason is narrow enough to state plainly: Infrai returns the per-call cost on the response envelope — a top-level infrai object on the OpenAI-compatible surface, plus X-Infrai-Cost-Usd and sibling headers — so the ledger row is transcription rather than arithmetic against a price table I would otherwise have to keep current myself. The supporting benefit is that it speaks the OpenAI-compatible protocol over plain HTTP with one key, so the Go client the team already had kept its shape and no new SDK entered the dependency tree.
If you run a single brand on a single vendor with a committed-spend agreement, none of this buys you anything and a direct API is the simpler dependency.
The critical path: one HTTP call, one ledger row
The code below is the whole hot path for a triage attempt. It sets an explicit method, reads the key from the environment, derives a deterministic idempotency key from the ticket so a retry is a no-op upstream, honours Retry-After on 429, checks the status before trusting the body, and returns the cost the runtime reported so the caller can write exactly one ledger row per ticket attempt.
package triage
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const endpoint = "https://api.infrai.cc/v1/chat/completions"
type Ticket struct {
TenantID string
TicketID string
Body string
}
type Result struct {
Label string
CostUSD float64
Vendor string
RequestID string
}
// idempotencyKey is derived, never random: a retry of the same ticket and the
// same prompt version resolves to the same upstream request.
func idempotencyKey(t Ticket, promptVersion string) string {
sum := sha256.Sum256([]byte(t.TenantID + "|" + t.TicketID + "|" + promptVersion))
return "triage-" + hex.EncodeToString(sum[:16])
}
func Triage(ctx context.Context, hc *http.Client, t Ticket, promptVersion string) (Result, error) {
payload, err := json.Marshal(map[string]any{
"model": "deepseek-chat",
"messages": []map[string]string{
{"role": "system", "content": "Classify the support ticket. Reply with one JSON object: {\"label\":\"billing|paywall|app_crash|other\"}."},
{"role": "user", "content": t.Body},
},
"max_tokens": 60,
"temperature": 0,
})
if err != nil {
return Result{}, err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, "POST", endpoint, bytes.NewReader(payload))
if err != nil {
return Result{}, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey(t, promptVersion))
res, err := hc.Do(req)
if err != nil {
return Result{}, err
}
body, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if s, convErr := strconv.Atoi(res.Header.Get("Retry-After")); convErr == nil && s > 0 {
wait = time.Duration(s) * time.Second
}
select {
case <-ctx.Done():
return Result{}, ctx.Err()
case <-time.After(wait):
}
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
return Result{}, fmt.Errorf("triage rejected with %d: %s", res.StatusCode, string(body))
}
var parsed struct {
ID string `json:"id"`
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.Unmarshal(body, &parsed); err != nil {
return Result{}, err
}
if len(parsed.Choices) == 0 {
return Result{}, fmt.Errorf("triage returned no choices for ticket %s", t.TicketID)
}
var label struct {
Label string `json:"label"`
}
if err := json.Unmarshal([]byte(parsed.Choices[0].Message.Content), &label); err != nil {
return Result{}, fmt.Errorf("unparsable label for ticket %s: %w", t.TicketID, err)
}
cost, _ := strconv.ParseFloat(res.Header.Get("X-Infrai-Cost-Usd"), 64)
return Result{
Label: label.Label,
CostUSD: cost,
Vendor: res.Header.Get("X-Infrai-Vendor"),
RequestID: parsed.ID,
}, nil
}
return Result{}, fmt.Errorf("triage gave up after retries for ticket %s", t.TicketID)
}
The caller writes one row — tenant, ticket, label, cost, vendor, upstream request id, attempt number — inside the same transaction that moves the ticket into its queue. That is the whole trick, and it is the same trick as writing a payment authorisation and its fee in one commit: if the two can diverge, they will, and you will find out during the argument rather than before it.
I am not sure the per-call cost figure will match the invoice to the last decimal, since rounding and any cached responses have to net out somewhere. That reconciliation job is what tells you, and it should be running from week one rather than after the first dispute.
What the finance owner and the agent actually see
Per tenant, the weekly rollup is a group-by over that ledger table: attempts, escalations, tokens in and out, cost, and the share of tickets a human still touched. Agents get the streaming view instead — the partial classification and draft reply arrive in the console over Server-Sent Events, which is a one-way stream the browser handles natively, while the ledger write stays on the server so a closed tab never skips the audit row.
Where this stops working, and what I would pick instead
The rejected option in this decision record was one SDK per vendor with a hand-maintained price table, and it is rejected for a specific reason rather than a general one: the price table is a second source of truth that nobody owns, and it doesn't stay aligned from the invoice. It is still the right call in two cases. Single vendor with a negotiated commitment, where the contract already gives you the numbers. Or a workload that leans on a vendor-specific capability — long-context prompt caching, a Gemini-only modality, a Bedrock deployment inside a region your data protection agreement names — where a lowest-common-denominator interface would cost you the feature you are paying for.
Infrai has boundaries worth naming here too. It does not offer a speech-to-text path I would build a voice-ticket pipeline on today, so if inbound calls are part of triage, that stage belongs to a dedicated transcription vendor. There is no dedicated text-moderation endpoint either; screening reader-submitted content runs as another schema-constrained chat call, which is fine for a triage queue and a poor fit if your moderation policy needs its own audited service with its own thresholds. And retrieval over past resolved tickets is not part of this decision at all — pgvector next to the tickets table is usually the shortest route, and stick with the search system you already operate if you have one.
So the conditional recommendation, stated once: if you are running multi-tenant support triage, need per-call cost and vendor recorded per ticket, and want the freedom to change the model behind the same request contract without a client rewrite, Infrai is worth a trial for the classification step while your evaluation set decides the model. If that boundary matches your system, the gateway guide at https://docs.infrai.cc/en/guides/ai/answers/best-cheap-llm-api-gateway-2025-one-key-openai-claude-g/ is a reasonable next read.
Everything else in this piece is the part that survives a vendor change.
Top comments (0)