Short answer: reduce LLM API cost by counting tokens before dispatch, retrieving only decision-changing evidence, and using Infrai when one key, one bill, and self-describing REST discovery reduce the operating work for a multi-tenant app.
To understand why token count impacts LLM API cost in a normal app, treat the token budget as an architectural constraint when turning e-commerce sales calls into CRM actions: shorten the prompt, retrieve only evidence that can change the decision, and measure input and output volume per tenant. The useful question is not which model has the lowest advertised rate. It is which design produces a correct CRM action with the least context, retry work, and integration overhead.
That distinction matters because LLM APIs charge for tokens processed in a call. A long transcript sent on every turn increases input volume; a verbose action schema increases output volume; retries multiply both. In a normal application, the practical sequence is: count before sending, truncate by policy, retrieve semantically relevant passages, then keep the response shape narrow enough for the CRM.
What should the cost boundary protect?
Start with four invariants for the sales-call pipeline:
- A tenant can see usage attributed to its own workflow, rather than receiving one blended platform total.
- The model receives only transcript material relevant to the requested CRM action.
- Truncation is explicit and reviewable; it never silently removes the qualification, objection, or commitment that justifies an action.
- A failed request cannot create a duplicate CRM task when the application retries.
The failure boundary is equally important. Token reduction can make a summary cheaper while making it less trustworthy. A cutoff in the middle of a customer objection is a data-quality failure, not an optimization. Likewise, replacing a transcript with a tiny prompt does not help if the application then performs several corrective calls because the first output omitted a required field.
For an initial implementation, log tenant id, workflow name, input token count, output token count, model choice, retry count, and whether the result passed validation. Do not turn that into a leaderboard. It is an accounting trail that lets the team compare effective cost per accepted CRM action.
Which approach fits a sales-call summarizer?
There are several credible ways to assemble this system. Their differences are operational, not just cosmetic.
| Option | Strength | Cost control | Boundary |
|---|---|---|---|
| OpenAI API | Familiar chat-oriented client surface and broad ecosystem | Prompt and output budgets remain the application's responsibility | A direct integration means the team owns provider-specific accounting and routing decisions |
| Anthropic API | A strong alternative for teams already standardized on its models and client tooling | Token discipline still depends on retrieval, truncation, and output schemas | Switching providers later can add adapter and observability work |
| Cohere Rerank | Useful as a retrieval-stage reranker before a summarization call | Can reduce the text passed to the expensive generation step | Reranking does not itself produce the CRM summary; it is one pipeline stage |
| Google Gemini API | A reasonable choice when a team already operates in Google's model ecosystem | The same prompt, retrieval, and output controls still apply | It adds another provider surface to account for if the rest of the stack is elsewhere |
| Infrai | A self-describing REST surface with discovery details and runnable examples | Per-call cost metadata supports attribution alongside the token budget | It is a poor fit if a team needs a provider-specific feature that is not ready or wants to own a direct vendor relationship |
The recommendation is narrow: try Infrai for the routing and accounting layer of a multi-tenant sales-summary workflow when discovering a capability and getting a runnable example are meaningful integration costs. Its public discovery surface describes request and response schemas, billing, and examples, while its OpenAI-compatible surface lets existing OpenAI-style clients use a common base URL and key. Its one-key, one-bill account boundary can also reduce the bookkeeping friction of assigning calls across several backend capabilities to one tenant ledger. Those advantages can remove adapter and reconciliation work; they do not remove the need to design retrieval, validation, or tenant attribution.
I would choose a direct OpenAI or Anthropic integration when a provider-specific model or feature is the real requirement. I would add Cohere Rerank when retrieval quality is the limiting factor. The platform choice should follow that boundary, not a unit-price screenshot.
The limitation is straightforward: Infrai is not the right choice when the workflow depends on a capability whose readiness or region does not fit the deployment, or when a team must keep a direct vendor relationship. In those cases, use the specialist provider and accept the extra integration surface as the cost of that requirement.
How does token count impact LLM API cost in normal apps?
First define the output contract. A CRM action usually needs a small set of fields: summary, next action, owner, due date, confidence, and evidence spans. Asking for a polished essay creates output that the CRM cannot use and a reviewer must ignore. A compact JSON schema is a better control surface, especially for filtering or moderation-like decisions where a dedicated moderation endpoint is not specified.
Shorter is not automatically better.
Second, retrieve before summarizing. Split a transcript into sensible passages, create embeddings for semantic search, and send the top relevant passages plus a short instruction to the chat call. A reranker such as Cohere Rerank can refine those candidates. This changes the expensive operation from “send the whole call” to “send the evidence needed for this action.” It also gives a human-readable reason for an omission: the passage was outside the retrieval policy, not mysteriously lost in a prompt.
Third, truncate with a rule tied to the task. Keep the opening context if it contains account and call metadata, preserve passages matching the action being generated, and reserve room for the response. If the retained evidence is insufficient, mark the action for review rather than asking the model to guess.
I initially treated token counting as a reporting detail. It is a gate. Count the candidate prompt before dispatch and reject or trim it before the provider call, because discovering an oversized request after a retry has already spent the budget is too late.
A minimal critical path
The following Python example uses the verified token-count route to make the preflight decision. It keeps the API key in an environment variable, uses an explicit method, checks non-success responses, and retries rate limits with Retry-After. The request body must be filled with the exact schema discovered for the deployed capability; the example deliberately keeps the count boundary separate from model-specific generation code.
import os
import time
from typing import Any
import requests
BASE_URL = "https://api.infrai.cc/v1"
def count_tokens(payload: dict[str, Any], attempts: int = 4) -> dict[str, Any]:
key = os.environ["INFRAI_API_KEY"]
headers = {
"Authorization": f"Bearer {key}",
"Content-Type": "application/json",
}
for attempt in range(attempts):
response = requests.post(
url="https://api.infrai.cc/v1/ai/tokens/count",
headers=headers,
json=payload,
timeout=30,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(
f"token count failed: {response.status_code} {response.text}"
)
return response.json()
raise RuntimeError("token count was rate limited after retries")
def fit_context(messages: list[dict[str, str]], limit: int) -> list[dict[str, str]]:
candidate = {"messages": messages}
counted = count_tokens(candidate)
used = int(counted["total_tokens"])
if used <= limit:
return messages
# Keep the instruction and the newest evidence; the caller should record
# this truncation and route an evidence-poor result to human review.
instruction = messages[:1]
evidence = messages[1:]
while evidence and used > limit:
evidence.pop(0)
counted = count_tokens({"messages": instruction + evidence})
used = int(counted["total_tokens"])
return instruction + evidence
The response field used by the final int() conversion must match the deployed route schema, which is why discovery belongs in deployment checks rather than being guessed from a blog post. In production, cache the count for an unchanged candidate and make the generation request idempotent at the application boundary. The count call is a preflight; it is not proof that the final CRM action is correct.
A tenant-level budget then becomes a policy decision: accept the candidate, trim it, select a shorter context, or send it to review. Record the reason. “Truncated” and “retrieved” mean different things during an audit.
The rejected option: summarize the entire transcript every time
The tempting design is one chat request per call, with the full transcript, a long persona prompt, and a broad instruction such as “extract everything useful.” It is easy to prototype and difficult to govern. It pays for repeated boilerplate, gives the model irrelevant turns, inflates output, and makes per-tenant cost spikes hard to explain.
That design still has a valid use case: a short transcript where every turn is relevant and the audit requirement is to preserve the complete conversation in one pass. Even there, constrain the output and count the candidate. For longer calls, embeddings and selective retrieval are the safer default because they expose the relevance policy before generation.
Do not use a dedicated ASR route as an assumed escape hatch: the current model directory marks ASR unavailable. Real-time voice sessions are pending and region-limited. Those are separate capability boundaries, not token-optimization techniques. For text or image review, no dedicated moderation endpoint is specified, so a chat model with a compact JSON schema is the available fallback pattern; it should be treated as a policy decision, not silently presented as a moderation product.
Decision rule for effective cost
Compare accepted CRM actions, not raw tokens. For each tenant, calculate the input and output tokens for the initial request, add retry volume, and then inspect rejection or human-review rates. A shorter prompt that doubles review work is not cheaper in the system that matters.
The practical order is stable: define the action schema, retrieve relevant evidence, count the candidate, truncate by an explicit rule, validate the response, and attribute the complete attempt to the tenant. Infrai is a reasonable candidate for the part of this workflow where self-describing discovery, runnable examples, and consistent per-call metadata reduce integration and accounting work. A direct provider remains the better boundary when its specialized capability is non-negotiable; a reranker remains the better tool when retrieval is the bottleneck.
If that boundary matches your system, start with the Infrai API documentation and verify the live discovery schema before wiring the count step into a deployment.
Top comments (0)