TL;DR: The least complex production design for the cheapest LLM chatbot cost per token is a small, schema-constrained triage call with a token ledger around it. Compare providers only after measuring prompt tokens, generated tokens, retries, and retained payloads on the same ticket sample. Then delete raw ticket text on a stated schedule. The cheapest token rate is irrelevant if a malformed decision causes a ticket to enter the wrong queue or forces a second call.
A fintech support ticket is a particularly unforgiving chatbot input. It can contain a disputed transfer, an account identifier, and an anxious customer asking for a human. The useful output is narrow: a queue, a priority, a reason code, and perhaps a redaction flag. The system should reject anything outside that contract.
What is the bill actually made of?
Start with the bill, not a provider comparison table. For each ticket, record input tokens, output tokens, the number of attempts, and the storage and retrieval work that your own system adds. A simple monthly estimate is:
def monthly_estimate(ticket_count, input_tokens, output_tokens,
input_rate, output_rate, retry_rate,
retained_bytes, storage_rate):
model_cost = ticket_count * (
input_tokens * input_rate + output_tokens * output_rate
)
retry_cost = model_cost * retry_rate
storage_cost = retained_bytes * storage_rate
return model_cost + retry_cost + storage_cost
This is a planning equation, not a price claim. input_rate and output_rate must come from the current published price sheet for the selected service, in the sheet's units. Keep input and output separate; providers do not necessarily price or count them in the same way. Keep retries separate too. A retry caused by invalid JSON is a quality cost masquerading as a token cost.
For an illustrative comparison, suppose one ticket has 800 input tokens and 120 output tokens. If validation rejects 8 percent of first attempts, the expected call volume is 1.08 attempts per ticket. Plug the live rates into that equation and the decision is inspectable. Change the prompt, output limit, or retry policy and the effect is visible. Do not turn the example into a claim about a real workload.
The dominant term is the one that changes the total. Measure it with a week of representative tickets, including long conversations and tickets that need escalation. In many systems the forgotten terms are more revealing than the advertised token number: a full conversation copied into every request, verbose explanations that the queue does not use, and a second model call used to repair an invalid object.
What should a fintech triage system retain?
Retention is part of the cost model because the raw ticket is usually the largest durable object in the workflow, while the final decision is small. Keep the decision record, its schema version, model-request metadata, validation result, and a content hash. Retain raw text only for the period required by the investigation and compliance policies that govern the business. The exact period belongs to those policies, not to an LLM vendor.
The explicit trade-off is retention versus replayability. Deleting raw text reduces exposure and storage, but it removes the easiest evidence for replaying a disputed classification. A hash and token counts cannot reconstruct the customer message. I would rather make that loss explicit in the runbook than pretend that an indefinite archive is free or that an opaque log is a sufficient audit trail.
Store sensitive fields separately from the model-facing text where the workflow allows it. Pass a stable reference instead of an account number, and make the redaction result part of the structured output. The queue worker needs a decision, not a second copy of every secret.
How do you compare production chatbot cost per token?
A fair comparison needs a common adapter and a common test corpus. OpenAI, Anthropic, and Gemini each expose model APIs with their own request shapes, output controls, token-counting tools, and failure responses. Those differences affect engineering work even when a spreadsheet puts all rates into the same currency unit. Use product names as labels for adapters, not as the architecture.
The adapter should return one internal result:
from dataclasses import dataclass
from typing import Any
@dataclass
class TriageResult:
queue: str
priority: str
reason_code: str
redaction_needed: bool
input_tokens: int
output_tokens: int
attempts: int
raw: Any
def accept_result(raw: dict, input_tokens: int,
output_tokens: int, attempts: int) -> TriageResult:
required = {"queue", "priority", "reason_code", "redaction_needed"}
if set(raw) != required:
raise ValueError("triage object has the wrong fields")
if not isinstance(raw["redaction_needed"], bool):
raise ValueError("redaction_needed must be boolean")
return TriageResult(
queue=raw["queue"],
priority=raw["priority"],
reason_code=raw["reason_code"],
redaction_needed=raw["redaction_needed"],
input_tokens=input_tokens,
output_tokens=output_tokens,
attempts=attempts,
raw=raw,
)
This boundary is where portability pays for itself. It also prevents a subtle accounting error: treating a provider's reported usage field as if every service reports the same categories. Save the original usage object for diagnostics, map known fields into the ledger, and mark unknown fields instead of silently calling them zero.
The ledger must be boring.
Do not compare only successful responses. Add malformed JSON, missing fields, refusal-like responses, long tickets, duplicate tickets, and tickets whose correct action is escalation. A provider that produces a slightly cheaper valid object but needs frequent repair calls is not cheaper in this workflow.
Can the cheapest call still be the wrong choice?
Yes, because structured correctness is the decision axis. A parser accepting syntactically valid JSON proves very little. The queue must be allowed, the priority must be in its enum, the reason code must match the ticket evidence, and a high-risk case must not be silently downgraded.
Use three gates: schema validation, domain validation, and policy validation. Schema validation checks types and required fields. Domain validation checks that a transfer-dispute reason is compatible with the selected queue. Policy validation checks that tickets containing a redaction flag or an escalation condition follow the human-review path. Each rejection should increment a labeled counter and record the token cost of the attempt.
A compact decision table makes the comparison honest:
| Measure | Why it matters | What to record |
|---|---|---|
| Valid structured output | Avoids repair calls and unsafe defaults | First-pass acceptance rate |
| Domain-correct output | Measures useful triage, not just parseability | Labeled test result |
| Token use | Connects prompt and response design to spend | Input and output counts |
| Retry rate | Exposes hidden cost from failed contracts | Attempts per ticket |
| Retention volume | Captures durable data exposure and storage work | Bytes and deletion age |
| Latency tail | Protects queue throughput during bursts | p95 and p99 by outcome |
These measures belong beside the price sheet. A single scalar cost per million tokens hides the behaviors that create operational work.
The deployment rule I would put in the runbook
Freeze a versioned ticket corpus before selecting a default. Include a human-labeled answer, a sensitivity label, and the expected JSON object. Run the same corpus through every adapter, then report cost per accepted decision, not cost per request. That denominator includes retries and excludes tickets that fell into manual review unless manual review is the intended correct outcome.
Cap output length, keep the prompt limited to fields used by the decision, and make an invalid result fail closed into a review queue. Alert on changes in token counts and first-pass acceptance, not just on provider availability. A prompt edit that adds two paragraphs to every request can be more expensive than a model change, and it is much easier to miss in a normal latency dashboard.
The practical stopping rule is simple: choose the smallest design that passes the domain-correctness threshold, then compare its accepted-decision cost under the same retention policy. Re-run the test when the prompt, schema, model configuration, or ticket mix changes.
The records to keep are the ones needed to explain and reproduce a decision. Everything else should have an owner, a retention date, and a reason to exist. That is how a token comparison becomes a production control instead of a misleading shopping list.
Top comments (0)