If you just want the recommendation: choose the LLM API that produces the lowest cost per successfully resolved customer-support conversation while meeting your latency and correctness SLOs, then put a compatibility layer and an idempotent write boundary around it. A cheap token that triggers escalation, repeats a tool action, or forces a larger prompt isn't cheap.
I learned that distinction from an in-app chatbot incident, and it changed how I run model evaluations. I don't start with a public price sheet now. I start with a replay set, the support operations the bot may perform, and an error budget. GPT, Claude, Gemini, and OpenAI-compatible runtimes can all enter the test; their names do not get to decide the result.
The uncomfortable answer is that there is no permanent cheapest LLM API for every customer support chatbot. Traffic shape, context length, retry behavior, tool calls, cache policy, escalation rate, and on-call ownership move the total. Your mileage may vary.
Why did a cheap retry make our support bot more expensive?
One Friday, our client timed out after asking a model to summarize a case and append an internal note. The naive retry ran the whole operation again. We found 37 conversations with two note writes before the alert fired; the model response itself was acceptable both times, but our application had confused retrying generation with retrying a side effect. That was the only number I needed to stop treating token price as the primary selection metric. The invariant was sharper: a model request may be repeated, while a business operation must have one stable identity and an observable terminal state.
Short retries look harmless in a spreadsheet. In production they multiply input tokens, consume concurrency, stretch tail latency, and can repeat whatever sits after generation. If the bot can only answer read-only FAQ questions, this risk is smaller. If it can issue a refund, change an address, or write a case note, the application needs an idempotency boundary before model choice matters.
It gets expensive fast.
For capacity planning, I budget requests per conversation as a distribution rather than one average: initial answer, retrieval, tool selection, validation, and possible repair. I then reserve headroom for retries without allowing retry traffic to consume the whole concurrency pool — the boring bulkhead is what protects the customer-facing SLO. My evaluation records cost per resolved conversation, p50 and p95 time to first useful answer, successful tool-operation rate, escalation rate, and duplicate-effect count. Duplicate effects have a budget of zero. A provider that wins on nominal input price but loses on repair calls or escalations has not won the workload.
How should an in-app customer support chatbot compare LLM API compatibility?
Treat compatibility as tested behavior, not a label. An OpenAI-compatible API can reduce adapter work, but the support bot still depends on the fields and streaming events it actually consumes. I keep a small contract suite that sends the same sanitized conversation fixtures through every adapter, checks structured output against the same schema, cancels a stream midway, exercises tool selection, and records usage accounting. I won't accept a migration estimate based only on matching request JSON.
For browser delivery, Server-Sent Events are a sensible one-way streaming primitive: the browser receives an event stream over an EventSource connection, while the application server owns the upstream model connection. That separation gives me one place to normalize chunks, redact logs, enforce deadlines, and stop generation after the user leaves. It also keeps provider credentials out of the browser. MDN documents the event-stream mechanics and the low per-domain connection limit that can matter without HTTP/2, so multi-tab behavior belongs in the load test rather than an architecture diagram.
My buy-versus-build review looks like this:
| Runtime path | Team owns | Operational advantage | Catch |
|---|---|---|---|
| Direct model APIs | One adapter per API and its tests | Smallest middle layer | Switching cost grows with provider-specific behavior |
| Compatible gateway | Gateway policy, contract tests, and credentials | One application-facing interface | The gateway becomes part of the latency and failure budget |
| Self-hosted runtime | Serving, capacity, upgrades, and model evaluation | Maximum control over placement and scheduling | On-call and utilization risk move to the platform team |
I'm not sure why teams still count matching field names as portability; as far as I can tell, portability exists only when the same fixture produces an acceptable answer, the same tool intent, and the same cancellation behavior.
Put the retry boundary below the model call
The preventative path is application-level idempotency. The operation ID comes from the user action or workflow, not from an individual model attempt. Generation can be retried under a bounded policy, but committing the support action goes through a store that rejects a second commit for the same operation. In a real service, that store must be durable and shared across replicas; the in-memory store below only makes the ownership boundary visible.
package support
import (
"context"
"errors"
"sync"
)
var ErrAlreadyCommitted = errors.New("operation already committed")
type Action struct {
OperationID string
CaseID string
Note string
}
type Committer interface {
Commit(ctx context.Context, action Action) error
}
type MemoryCommitter struct {
mu sync.Mutex
seen map[string]struct{}
}
func NewMemoryCommitter() *MemoryCommitter {
return &MemoryCommitter{seen: make(map[string]struct{})}
}
func (m *MemoryCommitter) Commit(_ context.Context, action Action) error {
m.mu.Lock()
defer m.mu.Unlock()
if _, exists := m.seen[action.OperationID]; exists {
return ErrAlreadyCommitted
}
m.seen[action.OperationID] = struct{}{}
return nil
}
func ApplyOnce(ctx context.Context, c Committer, action Action) error {
err := c.Commit(ctx, action)
if errors.Is(err, ErrAlreadyCommitted) {
return nil
}
return err
}
The production version should make the uniqueness check and the business write one atomic transaction. I emit the operation ID into traces and structured logs, then alert on attempted duplicates separately from committed duplicates; the former tells me retry pressure is rising, while the latter is an SLO breach. Keep generated prose out of the deduplication key — two generations can differ while representing the same customer action.
This advice is not suitable when the model performs no durable action and every response is disposable. For a stateless internal drafting tool, a simpler bounded retry may be the right trade. Don't build a transaction coordinator for autocomplete.
Measure resolved conversations before choosing a runtime
I run a shadow evaluation with a fixed, versioned sample drawn from real intent categories after removing customer data. Each candidate gets the same retrieval results, system policy, tool schemas, maximum output, and retry cap. Human reviewers grade answer correctness and policy compliance without seeing the runtime name. The platform harness measures first useful token, total completion time, input and output volume, tool-call validity, escalation, and the number of attempts. This is where a supposedly inexpensive model can lose: it may need more context or more repair attempts to reach the same accepted outcome.
The denominator matters.
My worksheet computes total runtime spend plus the allocated cost of retrieval and platform operation, divided by conversations that meet the acceptance rubric without an avoidable escalation. I don't publish a universal dollar result because prices and workloads move, and the provided references do not establish comparable current prices for GPT, Claude, Gemini, or every compatible runtime. Measure current quotes directly, timestamp them, and keep them outside the architectural decision record so a price refresh doesn't rewrite the design rationale.
Retrieval gets its own capacity line. A Postgres deployment using pgvector can store vectors alongside support content and perform exact or approximate nearest-neighbor search, but that convenience does not erase index sizing, recall testing, vacuum behavior, or database SLOs. Stick with an external search system when the existing team already operates one well, or when independent scaling and retrieval features outweigh the appeal of fewer components. A compatible managed API is likewise a poor fit when data-placement rules demand infrastructure you control; self-hosting is a poor fit when the team cannot carry accelerator capacity and a new pager.
I make the final call only after a soak test covers peak arrival rate, long conversations, cancellation, and dependency slowdown. The winner is the workload result, not the brand. Re-run the evaluation when prompts, retrieval, model versions, or support policy change, because each can alter both quality and cost.
Top comments (0)