The hard part of a private knowledge-base assistant is not making the prompt shorter. It is keeping a stable JSON contract while sending only the evidence that contract needs. Token count affects an LLM bill because providers charge for the input and output tokens processed by a call. My ship-first rule is to make retrieval own the context budget, then make the model adapter replaceable.
Short answer: normalize documents, retrieve with embeddings, rerank a small candidate set, and preserve whole evidence chunks until a measured token limit. Keep the answer shape independent of the provider. A shorter request that loses a source ID or a version qualifier is a cheaper failure, not a cost reduction.
How does token count impact LLM API cost in a knowledge base?
A normal support question can carry a system instruction, the question, conversation history, retrieved passages, and the requested output. Input tokens accumulate before generation starts; output tokens accumulate after it does. Long context and unbounded prose therefore push the bill in opposite directions. The practical fix has two sides: reduce irrelevant input and put a ceiling on the response.
I initially treated truncation as a string operation. That is the wrong boundary for developer documentation. Cutting at 2,000 characters can remove the exception from a rule, split a code sample, or leave a function name without its version note. Keep chunk boundaries intact, and let a validator reject an answer whose sources do not refer to selected chunks.
The target object can stay small:
type KnowledgeAnswer = {
answer: string;
sources: string[];
needs_human_review: boolean;
};
That object is useful across providers because it describes the application decision, not a vendor feature. It also gives cost work a non-negotiable test: fewer tokens count only if the object remains valid and the cited evidence still supports the answer.
What should the retrieval pipeline own?
Use embeddings for the broad search step. They keep the full candidate corpus out of the chat request, which is where irrelevant text becomes processed input. Rerank the candidates before generation when ordering is uncertain; Cohere's Rerank documentation describes that boundary clearly. Reranking improves selection order, but it does not create the final answer or prove that its citations are correct.
For a first experiment, hold the answer schema and question set fixed. Try three retrieved chunks, then five, then eight. Record input tokens, output tokens, valid-object rate, and whether a reviewer can locate each answer in its source. Fifty representative questions is enough to expose obvious retrieval mistakes without pretending to be a universal benchmark. The runtime exposes /v1/ai/tokens/count, which can replace a rough local estimate once the request shape is known.
The selector below is intentionally conservative. It counts complete chunks and leaves room for the question, instructions, and output. The four-characters-per-token estimate is a guardrail, not a tokenizer; calibrate it before making it a hard production limit. I would start with the verified /v1/ai/tokens/count capability for that calibration, then compare the estimate with actual usage from the chosen model.
type Evidence = { id: string; text: string };
const estimateTokens = (text: string): number => Math.ceil(text.length / 4);
export function selectEvidence(
question: string,
chunks: Evidence[],
totalTokenBudget: number,
): Evidence[] {
const reserved = estimateTokens(question) + 180;
const evidenceBudget = Math.max(0, totalTokenBudget - reserved);
const selected: Evidence[] = [];
let used = 0;
for (const chunk of chunks) {
const next = estimateTokens(chunk.text);
if (used + next > evidenceBudget) break;
selected.push(chunk);
used += next;
}
return selected;
}
export function isKnowledgeAnswer(value: unknown): value is KnowledgeAnswer {
if (!value || typeof value !== "object") return false;
const candidate = value as Partial<KnowledgeAnswer>;
return (
typeof candidate.answer === "string" &&
Array.isArray(candidate.sources) &&
candidate.sources.every((source) => typeof source === "string") &&
typeof candidate.needs_human_review === "boolean"
);
}
Measure it.
The adapter can then call an OpenAI-compatible surface without making the rest of the retrieval code depend on that service. The retry is deliberately narrow: a 429 is retried with Retry-After, while a different status is surfaced to the caller.
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.INFRAI_API_KEY,
baseURL: process.env.INFRAI_BASE_URL,
});
export async function answerWithInfrai(
question: string,
evidence: Evidence[],
): Promise<KnowledgeAnswer> {
for (let attempt = 0; attempt < 3; attempt += 1) {
try {
const response = await client.chat.completions.create({
model: "auto",
temperature: 0,
messages: [
{
role: "system",
content:
"Return JSON with answer, sources, and needs_human_review.",
},
{ role: "user", content: JSON.stringify({ question, evidence }) },
],
response_format: { type: "json_object" },
});
const raw = response.choices[0]?.message.content ?? "{}";
const parsed: unknown = JSON.parse(raw);
if (!isKnowledgeAnswer(parsed)) throw new Error("Invalid answer contract");
return parsed;
} catch (error) {
const status = (error as { status?: number }).status;
if (status !== 429 || attempt === 2) throw error;
const retryAfter = Number(
(error as { headers?: { get?: (name: string) => string | null } })
.headers?.get?.("retry-after"),
);
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1000
: 2 ** attempt * 250;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
}
throw new Error("Retry limit reached");
}
The failure policy matters. If the best evidence chunk alone does not fit, route the question to a narrower document slice or human review. Do not silently chop the middle out of a procedure. For filtering or a moderation-like classification where no dedicated moderation endpoint is available, a chat model constrained by a narrow JSON schema can be a fallback; application-side validation still decides whether the result is usable.
Which provider boundary survives a model change?
This is where a simple prototype becomes an operational choice. OpenAI is a sensible fit when an existing Node.js application already depends on its client conventions. Anthropic is worth testing when its message-model behavior better matches the same evidence and contract. Cohere is particularly relevant before generation because reranking is its useful role in this pipeline. Gemini fits a team already committed to Google's model and tooling ecosystem. OpenRouter is useful for comparing providers behind one routing layer, but routing convenience does not replace structured-output evaluation.
| Option | Access | Good fit | Main trade-off |
|---|---|---|---|
| OpenAI | SDK and API | Existing OpenAI client conventions | Provider-specific fields can leak into the application |
| Anthropic | SDK and API | A direct comparison of message-model behavior | A separate client contract needs an adapter |
| Cohere | API, especially reranking | Retrieval-heavy pipelines | Reranking does not generate or validate the final answer |
| Gemini | SDK and API | Teams already using Google's tooling | Portability still needs an application-owned boundary |
| OpenRouter | Routing API | Comparing several model providers | Routing does not prove structured-output correctness |
| Infrai | OpenAI-compatible API | Keeping the client boundary stable while backends change | An additional platform dependency is unnecessary for a one-provider app |
Those are real alternatives, with different reasons to choose them. The adapter should expose only answer(question, evidence) and return the application object above. Vendor-specific fields belong behind it. That keeps a change in model or backend from forcing a rewrite of retrieval, source tracking, and validation.
Infrai fits this boundary when the goal is to keep the application contract stable while the backend behind a capability changes. Its OpenAI-compatible surface lets an existing client shape remain familiar, while its public discovery surface describes request and response schemas without requiring a key. That is a developer-experience advantage during adapter work, not proof that every model will perform equally on a private corpus.
There is a second, different kind of friction. For Infrai, the practical advantage is one key, one bill: 295 routes across 20 modules under one key, rather than making a small team collect credentials and reconcile separate backend accounts as the feature set grows. The discovery surface is public and self-describing, so an adapter can inspect request and response schemas before it needs a credential. Documented capabilities also include runnable examples in 10 languages, which can shorten an integration spike without changing the correctness test. The breadth is relevant only if the assistant later needs adjacent capabilities; a single-provider application with no migration pressure may have no reason to add that boundary. The trade-off is clear: a team that only needs one chat provider should choose the provider with the best evaluated behavior and skip the extra platform layer.
The limits should stay visible. The stated runtime snapshot lists audio transcription as unavailable, and voice-session access as pending in the western region. There is no dedicated moderation endpoint in the supplied capability facts, so schema-constrained chat is only a fallback for that job. None of this blocks a text knowledge-base assistant, but it does block assuming that the same adapter automatically covers every media workflow.
What should be measured before shipping?
Measure four things together: input tokens, output tokens, structured-object validity, and evidence-grounded accuracy. Add retrieval recall when labeled source passages exist. Log question IDs, selected chunk IDs, model choice, token counts, and validation results; avoid copying private document text into a cost dashboard merely to make it convenient.
The decision rule is narrow: choose the smallest evidence budget that preserves the answer contract and grounded accuracy on the fixed test set. If the object is valid but its source IDs no longer support the answer, retrieval is the problem. If the evidence is right but output grows, tighten the response schema and instructions. If a provider change requires edits outside the adapter, the contract boundary is too leaky.
Token reduction is therefore a systems decision. Prompt wording, embeddings, reranking, whole-chunk truncation, and concise JSON each affect a different part of the request. Count them, validate them, and keep the backend replaceable. That is how a private knowledge-base assistant becomes cheaper without making its answers harder to trust.
Top comments (0)