Transcript Replay: Compare Customer Support Chatbot LLM APIs by Accepted Answer

Short answer: the cheapest LLM API for a customer support chatbot is the one that passes a replay of your own tickets at the lowest cost per acceptable answer, not the one with the smallest advertised token rate. Keep GPT, Claude, Gemini, and OpenAI-compatible candidates behind one adapter, record the complete input and output for each turn, and price the resulting workload only after the quality gate passes.

This matters for an in-app bot because the invoice is downstream of architecture. Retrieval context, repeated history, retries, and verbose answers all change the bill. A cheap request that produces an unsupported answer is still waste; a capable request that receives the whole transcript on every turn may be waste too. The useful decision unit is a support outcome under a fixed test, with latency and usage attached.

Don't choose from a pricing table alone.

How should a customer support chatbot compare LLM API costs?

Start with a frozen set of representative support conversations. Remove secrets and personal data, preserve the details needed to answer each question, and write a compact rubric for what counts as acceptable: correct policy, grounded claim, appropriate refusal, valid tool request, and no invented account state. The same cases, prompts, retrieved passages, output limit, and tool definitions must reach every candidate. Otherwise the comparison measures configuration drift rather than an LLM API.

The primary metric should be cost per accepted case. Calculate it from recorded usage and the candidate's current billing rules after the run. Keep the raw counts as well, because a single blended dollar figure hides the mechanism: one candidate may use more output while another needs a longer prompt or more retries. Pricing changes. A durable harness stores units, not yesterday's conclusion. Quality comes first because low token usage can be misleading: a terse answer may omit the actual fix, while a fluent answer may contradict the retrieved article. Automatic checks can catch required citations, schema validity, forbidden actions, and exact policy statements, but a human should inspect borderline cases before a launch decision. I'm not sure any general-purpose judge can settle product-specific support quality on its own; agreement from reviewers using an explicit rubric is the evidence that would change that view. Latency belongs beside cost, not folded into it. Save time to first byte, total duration, retry count, and cancellation status for every case. A streaming response can improve perceived speed without changing total generation time, while a slow first useful sentence can push a user to submit the question again. That second submission is another request and, more importantly, a poor support experience. Finally, report slices instead of only an average: separate short FAQ answers from retrieval-heavy troubleshooting, tool calls, long multi-turn conversations, and cases that should escalate to a person. A candidate that looks economical in aggregate can still be the wrong fit for the one path that dominates real support work.

Build the replay harness before debating providers

The data flow can stay plain. A sanitized case enters the evaluator, relevant support passages come from a retrieval layer, a generic chat adapter sends the assembled messages, and an event record captures timing, usage, answer text, and evaluator results. pgvector is one option for vector similarity search inside Postgres; its project documents exact and approximate nearest-neighbor search. The adapter itself should depend on a small internal contract rather than assumptions about one remote API.

Here is the core of that harness. The endpoint, model name, authentication header, and response parser are injected because an OpenAI-compatible label doesn't guarantee identical behavior outside the common request shape.

type Role = "system" | "user" | "assistant";
type Message = { role: Role; content: string };

type Candidate = {
  id: string;
  endpoint: URL;
  model: string;
  headers: Record<string, string>;
  parse: (body: unknown) => {
    text: string;
    inputUnits?: number;
    outputUnits?: number;
  };
};

type ReplayResult = {
  caseId: string;
  candidateId: string;
  text: string;
  inputUnits?: number;
  outputUnits?: number;
  elapsedMs: number;
};

async function replayCase(
  candidate: Candidate,
  caseId: string,
  messages: Message[],
): Promise<ReplayResult> {
  const startedAt = performance.now();
  const response = await fetch(candidate.endpoint, {
    method: "POST",
    headers: {
      ...candidate.headers,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: candidate.model,
      messages,
      stream: false,
    }),
  });

  if (!response.ok) {
    throw new Error(`candidate request failed with ${response.status}`);
  }

  const parsed = candidate.parse(await response.json());
  return {
    caseId,
    candidateId: candidate.id,
    text: parsed.text,
    inputUnits: parsed.inputUnits,
    outputUnits: parsed.outputUnits,
    elapsedMs: performance.now() - startedAt,
  };
}

Keep pricing out of this function. Store the unpriced usage units with the candidate ID and run timestamp, then apply a versioned rate card in a separate report. If the response doesn't expose reliable usage data, mark the value missing; don't silently estimate it in the request path. The missing field is itself a comparison result because it affects how confidently a small team can control spend.

The harness also needs a stable prompt builder. Sort retrieved records deterministically, cap context by an explicit budget, and log document identifiers so a failed answer can be reproduced. For a support bot, retrieval evaluation is inseparable from model evaluation: changing the candidate and the retrieved evidence in the same run makes the result impossible to diagnose.

Test failure paths with synthetic cases. A rate-limit response such as 429 should exercise bounded backoff, but retries around write-capable tools need an idempotency key before they are enabled. Cancellation should stop local work and prevent a late answer from overwriting a newer turn. Bad structured output should fail validation rather than reaching a billing, refund, or account-change function. These aren't exotic edge cases. They are the boundary between a chat demo and a support runtime.

Compatibility is a test result, not a label

An OpenAI-compatible API can reduce adapter work, but request similarity alone doesn't prove interchangeability. The harness should probe the fields the application actually uses: streaming events, structured output, tool-call arguments, finish reasons, usage accounting, error bodies, cancellation, and ordering of concurrent turns. Avoid testing optional features that the product doesn't need; they add migration work without buying reliability.

Streaming deserves its own contract test. Server-Sent Events use the text/event-stream media type, messages are separated by a pair of newline characters, and comment lines can be used to keep a connection alive, as MDN documents. A browser-facing support service can normalize upstream chunks into its own small event vocabulary so the widget doesn't depend on a provider's wire details.

type ChatEvent =
  | { type: "delta"; text: string }
  | { type: "done" }
  | { type: "error"; retryable: boolean };

function encodeSse(event: ChatEvent): string {
  return `event: ${event.type}\ndata: ${JSON.stringify(event)}\n\n`;
}

function writeEvent(
  controller: ReadableStreamDefaultController<Uint8Array>,
  event: ChatEvent,
): void {
  controller.enqueue(new TextEncoder().encode(encodeSse(event)));
}

This internal boundary is deliberately smaller than any provider surface. It makes a change of backend possible without pretending every capability maps cleanly. The catch is that the abstraction can hide useful differences. If the product depends on a provider-specific feature, expose that requirement explicitly and test it; forcing it through a lowest-common-denominator interface usually produces confusing flags and weaker guarantees.

There are other limits. A hosted API may be unsuitable when policy requires inference inside a controlled network. Self-hosting may be unsuitable when demand is spiky and nobody can own capacity planning, model serving, and incident response. A multi-provider gateway may simplify routing, but it introduces another operational dependency and another place to reconcile usage. Stick with a direct integration when one backend meets the requirements and the team values fewer moving parts; keep an adapter when migration risk is more expensive than that extra code. No option wins everywhere.

Runtime approach	Integration burden	Best fit	Main limitation
Direct hosted API	One remote contract	A small team with one qualified backend	Migration requires deliberate adapter work
Multi-provider gateway	One internal entry point plus routing policy	Workloads that need tested backend mobility	Adds an operational dependency and usage reconciliation
Self-hosted inference	Serving, capacity, and model operations	Controlled-network requirements with steady load	The team owns capacity and incident response

Operate the decision, not just the benchmark

Before deployment, pin the replay cases and rubric in version control, record the prompt and retrieval configuration, and run the candidate set against identical inputs. Review rejected and borderline answers, then calculate cost only for candidates that clear the quality threshold. Check latency distributions and failure categories separately. Averages conceal the long tail.

During rollout, send a small controlled share of eligible conversations through the new path without changing the surrounding retrieval or tool code. Compare accepted-answer rate, escalations, input and output units, first-byte latency, total latency, cancellations, and retries. Keep the prior path available until the new one has seen the difficult slices represented in the offline set. Fast rollback matters more than a clever router.

After launch, keep the replay suite alive. Add a sanitized case when a new failure mode appears, rerun it when prompts, retrieval, tools, or model configuration change, and investigate shifts in units per accepted case before they become an invoice surprise. Revisit the choice when the workload changes, not on a calendar. A support bot that adds account actions has different risk and evaluation needs from the FAQ bot it replaced.

The final operational check is prose-simple even if the implementation isn't: verify that every answer is traceable to its prompt and retrieved records, every tool write is idempotent, every stream can be cancelled, every usage record retains its raw units, and every rollout has a tested return path. Then pick the candidate that meets those constraints with the lowest measured cost for this workload. That's a defensible definition of cheapest.

References

• MDN: Using server-sent events
• pgvector: Open-source vector similarity search for Postgres

Sources

The technical claims about browser-facing event streams and Postgres vector search are supported by the two primary references above.