Use a two-stage runtime for an in-app chatbot that summarizes marketplace sales calls: spend the latency budget on a structured draft, then let deterministic code decide which CRM actions are admissible. The model may propose an action; it must not commit one. That boundary is the useful answer to the question of which LLM API to use in 2026, because quality and latency are properties of the whole transaction, not of a model name.
TL;DR: choose an API only after measuring extraction quality on your call corpus, tail latency, structured-output behavior, retention controls, and replayability. Persist the transcript hash, prompt revision, model identifier, response, validation result, and action decision. Retry by idempotency key, and treat every CRM write as a separately authorized step.
There is no universal winner.
What should an in-app chatbot be allowed to do?
A sales-call assistant has two jobs that look similar in a demo and behave very differently in production. It must summarize what was said, and it must turn that summary into CRM work: a follow-up task, a stage suggestion, or a request for human review. The first is probabilistic text generation. The second changes business state.
I therefore define the output as an immutable proposal, not a command. A proposal contains a call identifier, evidence spans, confidence, a bounded action enum, and a schema version. A policy service checks the enum, verifies that the cited call exists, rejects missing evidence, and writes an outbox record. A worker performs the CRM mutation once, with an idempotency key derived from the call and proposal revision.
The distinction matters during retries. A timeout after a successful CRM write is indistinguishable from a timeout before the write unless the write has a durable key and a queryable result. “Exactly once” is a useful design mindset, but the practical guarantee comes from idempotent effects, an audit trail, and reconciliation. The trade-off is operational weight: a proposal store, outbox, worker, and reconciliation job take longer to build than a direct request, and they add queues that must be monitored. That cost is justified only when an incorrect CRM mutation is more expensive than delayed human review.
Which API decision survives a quality-versus-latency review?
The comparison should be an experiment with fixed prompts, the same redacted calls, and a declared acceptance rule. Provider documentation can establish available interfaces, but it cannot establish your transcript quality.
| Decision dimension | Measure | Failure boundary |
|---|---|---|
| Quality | Field-level accuracy, unsupported-claim rate, evidence coverage | Human review for low confidence or conflicting evidence |
| Latency | p50 and p95 from request receipt to validated proposal | Async completion when the tail exceeds the interaction budget |
| Contract | Schema validation and refusal behavior | Store a rejected proposal; never coerce it into a CRM command |
| Operations | Retry semantics, retention controls, regional requirements | Queue and reconcile rather than silently duplicate work |
For an initial bake-off, three commonly evaluated families are OpenAI, Anthropic, and Google Gemini. They expose different model catalogs, SDKs, context limits, structured-response facilities, and operational policies; those details change, so they belong in a dated test record rather than in a permanent ranking. The fair comparison is not “which model sounds best?” It is “which service produces an admissible proposal within the p95 budget on our 200 labeled calls?” A smaller or faster model may be adequate for short, clean calls, while a slower model may be necessary when speakers interrupt each other or the evidence is spread across a long transcript. That is a quality-latency trade-off, not a permanent product ranking.
A small example of the critical path makes the boundary concrete. The Go below is deliberately vendor-neutral: the adapter owns the provider request, while the application owns validation and authorization.
type Proposal struct {
CallID string `json:"call_id"`
Action string `json:"action"`
Evidence []string `json:"evidence"`
Confidence float64 `json:"confidence"`
Schema string `json:"schema_version"`
}
func prepareAction(p Proposal, call Call, revision string) (OutboxEvent, error) {
if p.CallID != call.ID || p.Schema != "sales-call-v1" {
return OutboxEvent{}, ErrInvalidProposal
}
if !allowedAction(p.Action) || len(p.Evidence) == 0 || p.Confidence < 0.80 {
return OutboxEvent{}, ErrNeedsReview
}
key := p.CallID + "/" + revision
return OutboxEvent{IdempotencyKey: key, Action: p.Action, CallID: p.CallID}, nil
}
The threshold is a policy example, not a universal benchmark. Tune it against false-positive CRM work, false negatives, and reviewer capacity. Log the decision, not only the generated text.
Why does a fast answer still need an audit trail?
Latency pressure often removes the records that make a system diagnosable. That is backwards for financial or marketplace workflows, where a later dispute can concern the source conversation, the transformation, or the resulting state. Keep a content hash and an access-controlled transcript reference; do not copy sensitive call content into every log line. Record the model request metadata, prompt revision, validator result, policy version, retry count, and downstream response.
A streaming UI can show partial text without treating partial text as a completed proposal. The final structured object should cross the same validator whether it arrived in 400 milliseconds or after an asynchronous job. A batch interface may be appropriate for backfills and nightly reconciliation, but it does not remove the need for per-record idempotency or result correlation; the Batch API documentation describes asynchronous processing and its associated result workflow. Batch processing is therefore a poor fit for a user waiting inside a chat window, even though it can be a sensible fit for replaying a quarter of archived calls.
Test the ugly paths. Replay a provider timeout, malformed JSON, a duplicate webhook, a CRM 409, a transcript with no actionable evidence, and a prompt revision that changes an enum. Compare the resulting ledger of proposals with the ledger of CRM effects. If those sets cannot be reconciled, the system is not ready for automation.
The rejected option and when it is valid
I reject direct model-to-CRM tool execution for this workflow. It is attractive because it compresses the demo into one request, but it mixes interpretation, authorization, and mutation in a boundary that is difficult to replay. A tool call can be proposed inside the model interaction; it should still become a validated outbox event before any external write.
Direct execution can be valid for a low-risk, reversible action such as drafting a private reply, provided the action is scoped, logged, rate-limited, and easy to undo. It is a poor default for changing opportunity stages, assigning ownership, or creating customer-visible commitments.
The final selection rule is therefore narrow: prefer the API whose measured quality clears the business acceptance threshold while its p95 and operational controls fit the interaction mode. Keep the adapter replaceable, but keep proposal validation, idempotent writes, and reconciliation in your own service. That architecture preserves choice without pretending that provider substitution solves workflow correctness.
Top comments (0)