The cheapest production chatbot is rarely the one with the lowest advertised token rate. For a property-management support queue, the better decision is the runtime that produces a valid triage record at an acceptable total cost: input tokens, output tokens, retries, context retrieval, and the human review caused by malformed output.
TL;DR: compare models with the same ticket fixtures, schema, context window, and retry policy. Record input and output tokens separately. Choose the lowest total cost per accepted triage record, not the lowest cost per token.
The data flow is small enough to reason about. A ticket enters the system, a classifier extracts a fixed triage object, a validator checks it, and a queue worker sends failures to review or retry. The chatbot can still answer a resident later, but the first contract is structured correctness: category, urgency, property identifier, and a short rationale must land in predictable fields.
What does a fair token-cost comparison measure?
A fair comparison holds the workload constant. Use the same anonymized ticket set, the same system instructions, the same allowed labels, and the same output schema. Keep the message history policy fixed too. Otherwise a model that appears cheaper may only be seeing less context, or may be returning more prose that another runtime would reject.
The basic calculation is straightforward:
type Usage = {
inputTokens: number;
outputTokens: number;
};
type Rates = {
inputPerToken: number;
outputPerToken: number;
};
function tokenCost(usage: Usage, rates: Rates): number {
return usage.inputTokens * rates.inputPerToken
+ usage.outputTokens * rates.outputPerToken;
}
function costPerAcceptedRecord(
usage: Usage,
rates: Rates,
attempts: number,
accepted: number,
): number {
if (accepted <= 0) throw new Error('No accepted records');
return tokenCost(usage, rates) * attempts / accepted;
}
The rates belong in configuration, with an effective date and unit documented beside them. This keeps a comparison reproducible when a provider changes its price sheet. Do not mix a cached prompt, a fresh prompt, and a retry into one unlabeled token total.
I initially treated output as the obvious cost driver because support replies are visible to users. Triage changed that assumption. Repeated ticket history and retrieved lease or maintenance context can dominate input usage, while a short JSON result can still be expensive when validation failures trigger another attempt. The useful unit is therefore an accepted record, with retries charged to the record that needed them.
That is the trap.
For a fixture run, keep these fields beside the result rather than collapsing them into a single score:
| Measure | Why it belongs in the comparison |
|---|---|
| Input tokens | Long histories and retrieved policy text can grow this side of the bill. |
| Output tokens | Verbose explanations add cost even when the triage object is small. |
| Valid-record rate | A cheap response that fails validation is not an accepted result. |
| Retry and review rate | Rework turns nominal token cost into operational cost. |
No runtime wins every slice. A hosted endpoint can reduce operational work but adds dependence on network availability and service limits. A self-hosted server can offer deployment control, but capacity planning, patching, and accelerator utilization become the application's responsibility. A smaller model may lower token cost while increasing review work; a larger model may do the reverse. The right boundary is the one that satisfies the acceptance threshold for this queue, and that threshold should be written down before the comparison.
The limitation is practical: a token-only ranking cannot predict queue correctness, retry volume, or the cost of review. This trade-off makes a published per-token comparison useful for screening, but insufficient for the final production choice.
Why structured correctness changes the cost curve
A parser failure is not merely a quality problem. It is another request, a delayed queue item, or a human review. For property management, a false low-urgency label can delay a water leak; a missing property identifier can route a routine request to the wrong team. Those operational costs are outside the token invoice, but they belong in the selection decision.
Use a narrow schema and reject unknown values. Make the validator deterministic. If a field is optional, state that in the schema rather than asking the model to improvise. Keep the retry prompt short and include the validation error, not the entire conversation again, when the runtime allows that design.
A practical evaluation table has one row per candidate runtime and columns for input tokens, output tokens, valid-record rate, retry rate, latency, and cost per accepted record. Add a separate count for manual review. A small input-rate advantage is not meaningful if it comes with a large increase in rejected triage records.
Which architecture keeps the comparison honest?
Put a provider-neutral adapter between the queue worker and the model endpoint. The adapter should accept messages, requested output schema, and generation settings, then return normalized usage, parsed output, latency, and an error class. The application should not know whether a request went to a hosted API, a self-hosted server, or a fallback route.
That boundary also prevents a common lock-in mistake: storing provider-shaped responses as the business record. Store the validated triage object and the usage event instead. Keep the raw response under the retention policy that applies to support data, with personal information redacted before it enters evaluation logs.
Embeddings are a separate accounting line. If retrieval is used to find building policies or prior tickets, count embedding work and storage separately from the chat completion. The OpenAI embeddings guide describes embeddings as numerical representations used for relatedness and search; it should not be treated as evidence that retrieval is free or that a particular retrieval layout is correct.
A production decision rule
Run the same fixture set through each candidate, then inspect the rejected records rather than only the average. Segment results by short tickets, long histories, urgent maintenance language, and missing property details. Those slices expose whether a low average cost is hiding an unsafe failure mode.
Set a maximum retry count, a latency budget for the queue, and a manual-review path before launch. Record the prompt version, schema version, effective rate card, token usage, validation result, and final disposition. Re-run the fixture set after changing any of them. The comparison is stale when the workload changes.
The final choice should be the runtime that meets the triage correctness and latency thresholds at the lowest measured cost per accepted record. Token price is one input to that decision. It is not the decision itself.
Sources
References:
- OpenAI, “Embeddings guide”: https://platform.openai.com/docs/guides/embeddings
Top comments (0)