The operational question is why token count impacts LLM API cost in an e-commerce support triage service, especially when provider portability is a requirement. My choice is to control the token budget at the application boundary, before a request reaches any provider.
Short answer: token count affects LLM API cost because billing commonly measures input and output tokens separately. Reduce usage by sending only the evidence needed for the routing decision, constraining the response shape, and measuring tokens per ticket rather than averaging across a queue. Keep the prompt contract provider-neutral so a cheaper or more suitable endpoint can be evaluated without rewriting the ticket workflow.
That last condition matters. A small prompt can still be expensive if the model produces an unnecessarily long explanation, and a short response can still be unreliable if truncation removes the order state that determines the correct queue. Cost control is a reliability policy, not a string-shortening contest.
I have been paged by missed jobs and duplicate deliveries. That history makes the rule simple: an optimization is incomplete if it creates an unsafe retry path.
Duplicates are expensive.
How does token count impact LLM API cost for ticket triage?
A token is a unit used to represent text for a model; it is not the same thing as a character or a word. Tokenization varies with the text and the model family, so character length is a poor accounting unit. A ticket request can include the customer message, order facts, prior agent notes, policy excerpts, instructions, and the generated result. Those parts have different operational value, but all of them consume context or output budget.
Input and output should be tracked independently. Re-sending a long policy block on every ticket increases input usage. Asking for a prose justification increases output usage. A retry can charge for the request again, depending on the provider's billing rules, and a fallback can turn one logical triage attempt into two physical model calls. The cost ledger therefore belongs beside request metrics, not only in a monthly finance report.
For a triage decision, the useful output is usually bounded: a queue, a confidence band, a small set of reason codes, and an escalation flag. Free-form reasoning is a poor default contract because its length is uncontrolled and its text is harder to validate. The application should own the schema and treat explanatory text as optional evidence, not as the primary result.
Where does a portable cost-control boundary belong?
Put normalization, retrieval, budgeting, and validation in an internal adapter. The adapter accepts a ticket and returns a provider-neutral result. It should not leak a vendor-specific request object into the queue consumer. That separation gives an SRE a useful rollback lever: change the provider configuration or model mapping while preserving deduplication, retry policy, audit records, and downstream routing.
The adapter needs four explicit budgets. The first is an input budget for ticket text and selected order metadata. The second is a retrieved-context budget for policy snippets. The third is an output budget for the classification response. The fourth is a retry budget for transient failures. Naming these separately prevents a common mistake: lowering the maximum output while allowing an ever-growing prompt to pass through.
A compact contract can look like this in Go:
package triage
import (
"context"
"errors"
)
type Ticket struct {
ID string
Message string
OrderRef string
}
type Decision struct {
Queue string `json:"queue"`
ReasonCode string `json:"reason_code"`
Escalate bool `json:"escalate"`
}
var ErrBudgetExceeded = errors.New("triage input budget exceeded")
type Model interface {
Classify(ctx context.Context, input string, outputLimit int) (Decision, error)
}
func Classify(ctx context.Context, model Model, ticket Ticket, contextText string) (Decision, error) {
input := buildInput(ticket, contextText)
if estimateTokens(input) > 1200 {
return Decision{}, ErrBudgetExceeded
}
return model.Classify(ctx, input, 120)
}
The numbers in this example are policy limits, not a universal conversion or a price claim. In production, estimateTokens should use the tokenizer appropriate to the selected model, and the limit should be tested against real ticket samples. The important part is the control flow: reject or summarize oversized context before the provider call, and keep output bounded by contract.
This advice does not apply unchanged to tasks that require verbatim transformation of a large document, exhaustive extraction, or a long user-facing answer. In those cases, splitting work into independently verifiable chunks may be safer than hard truncation. A triage classifier has a narrower decision surface, so it can usually be more selective.
How can retrieval reduce cost without hiding the evidence?
Do not attach the entire support handbook to every request. Index policy content by the decision it governs, retrieve a small candidate set, and include the policy identifier with the excerpt. A reranker can improve which candidates survive the context budget, but it does not remove the need to choose a maximum number of candidates or to record what was sent. The retrieval result is part of the audit trail.
For example, a damaged-item ticket may need the return window and the fulfillment state, while a password-reset ticket needs neither. The request builder should select fields from the ticket type and order state, then retrieve policy text for that route. This is more portable than relying on a provider-specific memory feature, because the durable state stays in the application and its data stores.
There is a trade-off. Aggressive filtering can lower token use while increasing misroutes when the omitted field was decisive. I would rather spend a bounded amount on a second-stage escalation than silently classify a high-risk refund request with incomplete evidence. The rule should be measurable: compare queue accuracy, escalation rate, and tokens per ticket for each context policy.
What should the runbook measure?
Track input tokens, output tokens, provider calls per ticket, retry count, fallback count, latency, validation failures, and final queue agreement. Break the metrics down by ticket type and prompt version. A single average can hide the expensive tail: one order-history-heavy category may be responsible for most context growth even when the overall mean looks stable.
Set alerts on behavior, not just spend. A sudden rise in input tokens can indicate duplicated notes or an accidental policy append. A rise in output tokens can indicate a schema failure or a prompt change that invites explanations. A rise in provider calls per ticket can indicate retries without idempotency or a fallback loop. Each alert should point to a bounded response: inspect the rendered request, compare the prompt version, and stop expansion before changing the model.
Idempotency belongs here too. Give each ticket triage attempt a stable operation key, persist the decision and the prompt version, and make retries safe for the queue consumer. Token accounting without duplicate-control accounting understates the real cost of a failure.
The decision rule
Start with the smallest evidence set that can explain the queue decision. Add a field only when an evaluation set shows that it changes the decision or reduces an escalation worth the added tokens. Keep the model adapter replaceable, the output schema strict, and the request traceable.
My practical acceptance test is not “does the prompt look short?” It is: can the service classify representative tickets, reject malformed output, retry without duplicate downstream work, and switch endpoints without changing the business contract? If the answer is no, token reduction has been optimized in isolation. That is how a cheap request becomes an expensive incident.
Sources
References:
- OpenAI, “Tokenizer,” https://platform.openai.com/tokenizer
- Cohere, “Rerank,” https://docs.cohere.com/docs/rerank-overview
- RFC 9110, “HTTP Semantics,” https://www.rfc-editor.org/rfc/rfc9110
- OpenTelemetry, “Semantic conventions for generative AI systems,” https://opentelemetry.io/docs/specs/semconv/gen-ai/
Top comments (0)