TL;DR: The least complex way to reduce token cost for repetitive marketplace questions is to answer stable questions from a tenant-scoped cache before calling a model, then retrieve only the invoice fields needed for the current turn. Keep a small, versioned prompt and measure input and output tokens per tenant. A cheaper model is a later decision, not the first control.
For a supplier-invoice chatbot, the bill is usually driven by repeated context: the same supplier rules, the same field definitions, and users asking the same questions about payment terms or tax totals. Sending that context on every turn multiplies the input-token count. The useful change is to stop resending material that has not changed.
That is the boundary.
Which chatbot API minimizes token cost for repetitive questions?
Start with an accounting identity rather than a vendor shortlist:
request_cost_units = (input_tokens * input_rate) + (output_tokens * output_rate)
monthly_units = sum(tenant.request_cost_units for tenant in usage_events)
The rates are provider-specific and can change, so the durable engineering work is measuring the two token counts and keeping the arithmetic outside the application prompt. For each request, record tenant ID, model identifier, input tokens, output tokens, cache hit status, retrieval count, and a prompt-schema version. Do not store the invoice text in a general-purpose cost table. Store a request ID and retention-safe metadata instead.
This accounting makes a common mistake visible: a short user question can still be expensive if the application appends twenty previously retrieved invoices and a long system instruction. Output tokens matter too, but invoice extraction normally benefits from a strict field schema and short explanations. The response contract should make that constraint explicit.
Can a cache answer the repeated question safely?
Yes, for the narrow class of questions whose answer is stable for a tenant and a document version. Examples include “What currency is this invoice in?” after extraction, or “Which fields are available?” A cache key can include tenant ID, invoice version, normalized question, locale, and prompt-schema version. That prevents a value extracted for one tenant or one revision from crossing a boundary.
A cache should not be a blind string lookup. Save the answer, the source field IDs, the extraction version, and an expiry or invalidation reason. If a supplier replaces an invoice, invalidate entries tied to the old document version. If the field schema changes, invalidate by schema version. The operational cost of keeping these links is small compared with debugging a plausible answer that came from yesterday’s document.
In email and OTP systems, I learned to treat a successful delivery as a state transition with evidence, not as a boolean. Use the same discipline here: a cache hit is useful only when it carries enough provenance for the UI and audit trail to explain why it was returned.
How much context should retrieval send?
Retrieve the smallest evidence set that can support the answer. For invoice extraction, that might be the supplier name plus the line-item, tax, total, or payment-term fields requested in the turn. A reranker can help order candidate passages, but ranking does not remove the need to cap the final context. Reranking documentation describes reranking as a relevance step; it is not a substitute for tenant isolation, field validation, or cost accounting.
A practical pipeline is: classify the question, check the exact-answer cache, retrieve candidate fields, apply a token budget, call the model only when needed, validate the structured response, and write a usage event. The token budget should be a policy by question type. “What is the invoice total?” needs less evidence than “Explain every discrepancy across these three invoices.”
Here is a deliberately plain interface for the decision point:
def answer_invoice_question(question, tenant_id, invoice_version):
key = make_cache_key(tenant_id, invoice_version, normalize(question), PROMPT_SCHEMA_VERSION)
cached = answer_cache.get(key)
if cached and cached["schema_version"] == PROMPT_SCHEMA_VERSION:
return {"answer": cached["answer"], "source_fields": cached["source_fields"], "cached": True}
fields = retrieve_fields(tenant_id, invoice_version, question, limit=6)
prompt = build_prompt(question=question, fields=fields, max_output_tokens=240)
result = model_gateway.complete(prompt)
answer = validate_invoice_response(result)
answer_cache.put(key, {
"answer": answer.text,
"source_fields": answer.source_fields,
"schema_version": PROMPT_SCHEMA_VERSION,
})
record_usage(tenant_id, result.input_tokens, result.output_tokens, cached=False)
return {"answer": answer.text, "source_fields": answer.source_fields, "cached": False}
The numbers in this example are policy placeholders, not a claim about a particular API. Set them from evaluation data. A hard limit of six retrieved fields and 240 output tokens is only sensible if the validation and fallback path handle cases that need more evidence. The tradeoff is deliberate: a smaller context lowers token use, but it can lower answer quality when the question spans several invoices.
What should each tenant be allowed to see?
Cost visibility is a tenancy problem as much as a billing problem. Aggregate usage by tenant, workspace, feature, model identifier, and cache outcome. Show both total cost units and the causes: cache misses, average input tokens, average output tokens, retrieval count, and validation retries. A single blended average hides the tenant whose invoices contain unusually long line items.
Put budgets and alerts beside the ledger, not inside the chat prompt. A soft alert can trigger review; a hard policy can route new requests to a smaller context or require an asynchronous workflow. Preserve enough event data to reproduce the calculation, then expire sensitive payloads on a defined schedule. What I would deliberately stop keeping is raw prompt text in the long-lived cost ledger. That makes incident investigation harder, but it reduces the blast radius of a billing table becoming an accidental document archive.
A decision rule that survives model changes
Before selecting an API, replay an anonymized set of representative questions and group results by tenant and question class. Compare answer correctness, cache-hit rate, input-token reduction, output length, validation retries, and observed cost units. The winning configuration is the one that meets the correctness and latency target with an explainable per-tenant bill. A lower advertised token rate can lose if it requires more context, more retries, or longer answers.
Keep the gateway interface provider-neutral. The application should ask for a structured invoice answer and receive usage metadata; it should not know how a specific service counts tokens. Tokenizers differ, so use the gateway’s reported counts for accounting and a tokenizer tool for local estimates. The public tokenizer tool is useful for inspecting how text is split, but production accounting still belongs to the response metadata from the service you actually call.
The simplest cost reduction remains the most durable one: do not ask a model to regenerate an answer that is already valid, do not attach fields unrelated to the question, and make every cache hit attributable to one tenant and one document version. That design keeps the selection decision measurable when models, prompts, and traffic patterns change.
Top comments (0)