Token count impacts LLM API cost because every retrieved passage, instruction, and generated answer consumes part of the request. For a fintech assistant answering questions over a private knowledge base, the practical move is to retrieve fewer, better passages, reserve room for the answer, and measure input and output tokens separately. That keeps answers portable across model APIs because the application owns the context policy.
| Choice | What it optimizes | Cost risk | Portability risk |
|---|---|---|---|
| Send every matching chunk | Recall | High input usage | Low |
| Send a fixed top-k | Simple implementation | Unpredictable | Medium |
| Pack a token budget | Predictable context | Requires counting | Low |
| Summarize before answering | Short prompts | Loses qualifiers | Medium |
Recommendation: use a token budget with a relevance threshold, a hard document limit, and an explicit fallback when evidence does not fit. The budget should be part of your application contract, alongside the question, retrieved evidence, and answer schema.
How does token count impact LLM API cost?
An LLM request has at least two metered quantities: tokens sent to the model and tokens generated by it. The input includes the instruction, conversation history, tool or schema material, and retrieved knowledge-base text. The output includes the answer and any structured fields. A long retrieved passage can raise the input count even when the user asks a six-word question. A loose answer limit can raise the output count after the useful answer is already complete.
Tokens are pieces of text selected by a model-specific tokenizer. They are not the same as characters, words, or database rows. JSON punctuation, repeated headings, long identifiers, and copied tables all consume context. The exact count depends on the tokenizer used by the target model, so a character estimate is useful for an early guardrail, not for an invoice reconciliation.
A useful mental model is:
type Usage = {
inputTokens: number;
outputTokens: number;
};
function requestTokenTotal(usage: Usage): number {
return usage.inputTokens + usage.outputTokens;
}
function inputShare(usage: Usage): number {
const total = requestTokenTotal(usage);
return total === 0 ? 0 : usage.inputTokens / total;
}
Do not hide this usage in a provider-specific response object. Normalize it at the runtime boundary. Store the request class, tenant identifier, retrieval count, selected context tokens, and the returned usage. That lets you answer a better question than “which request was expensive?” You can ask whether the cost came from retrieval, history, instructions, or generation.
For a private financial policy article, one missing exception can be more damaging than ten irrelevant paragraphs. Cost reduction therefore starts with evidence quality. Cutting context blindly can make the system cheaper and less trustworthy at the same time.
What should a fintech retrieval budget contain?
Start with four separate ceilings: instruction tokens, conversation tokens, evidence tokens, and output tokens. Keep them independent in configuration. If one shared number controls all four, a longer chat can silently consume the room intended for evidence.
The decision rule is straightforward. Rank candidate passages, discard candidates below the relevance threshold, then add passages in rank order until the evidence budget is full. Keep a hard maximum number of passages as a second brake. A passage that is individually relevant can still be harmful when it repeats a definition already included by a higher-ranked passage. The tradeoff is deliberate: a little ranking and counting work buys a request shape that can be inspected before it reaches any model API.
For portable support, the ranker must be an interface. The application should not assume a particular request route, response property, or tokenizer. The adapter translates a provider response into this small internal shape.
type Candidate = {
id: string;
text: string;
score: number;
source: string;
};
type TokenCounter = (text: string) => number;
type ContextPolicy = {
maxEvidenceTokens: number;
minScore: number;
maxPassages: number;
};
function selectEvidence(
candidates: Candidate[],
countTokens: TokenCounter,
policy: ContextPolicy
): Candidate[] {
const selected: Candidate[] = [];
let used = 0;
for (const candidate of candidates) {
if (candidate.score < policy.minScore) continue;
if (selected.length >= policy.maxPassages) break;
const next = countTokens(candidate.text);
if (used + next > policy.maxEvidenceTokens) continue;
selected.push(candidate);
used += next;
}
return selected;
}
The score is not a truth score. It is a retrieval signal. Validate the selected text against source metadata, preserve the document title and effective date, and instruct the answerer to abstain when the evidence does not support a claim. In fintech, an outdated rate rule and a current rate rule can look semantically similar. Metadata filtering belongs before or alongside ranking, not after the model has written an answer.
How do you pack context without losing the useful exception?
The common failure is packing by row count: “send the top five chunks.” Rows have different lengths and different information density. A 900-token procedure and a 90-token exception should not have equal weight in the budget. Pack by measured tokens, then inspect the boundary cases.
Measure first.
I use a stable envelope so the model sees where evidence starts and ends. Each passage carries an identifier and source label. The labels cost tokens, but they make citations and audit logs possible. That is a good trade for a private knowledge base.
type Evidence = {
id: string;
source: string;
text: string;
};
function renderEvidence(items: Evidence[]): string {
return items
.map((item) =>
[
`<evidence id="${item.id}" source="${item.source}">`,
item.text.trim(),
'</evidence>',
].join('\n')
)
.join('\n\n');
}
function buildPrompt(question: string, evidence: Evidence[]): string {
return [
'Answer only from the evidence. If it is insufficient, say so.',
`Question: ${question.trim()}`,
'Evidence:',
renderEvidence(evidence),
].join('\n\n');
}
Do not use this renderer as a security boundary. Escape or encode metadata if document fields can contain markup-like text, and keep authorization checks outside the prompt. The model should receive only passages the requesting tenant is allowed to see. Token reduction never justifies weakening tenant isolation.
A good packing policy also protects the answer. Reserve output capacity before selecting evidence. If the total context is already near the model limit, the model may truncate or reject the request; if it is merely expensive, it may produce a longer answer than the user needs. Set an output limit appropriate to the answer schema, then ask for concise evidence-backed prose. The limit is a guardrail, not a substitute for a clear question.
The budget is a policy, not a constant.
Which savings survive normal application traffic?
The largest repeatable wins usually come from removing duplicate context. Do not resend a full conversation when a compact state object is enough. Do not paste the same compliance definitions into every turn when a versioned instruction can reference them through the retrieval pipeline. Do not include raw search metadata that the answerer cannot use.
Caching can help with repeated prefixes or repeated retrieval results, but it should be treated as an optimization layer. The correctness path must still work when the cache is cold or invalidated. Cache keys should include tenant, document version, authorization scope, query normalization, and policy version. A stale answer about a financial rule is a correctness incident, not a cache miss.
Summarization is more dangerous than it looks. It can reduce tokens while deleting a negation, an exception, or a date. Use it for conversational history that has been validated as disposable, not as a silent replacement for authoritative source passages. For source material, a smaller chunking strategy and better ranking are easier to inspect.
Track three ratios: selected evidence tokens per request, output tokens per successful answer, and unsupported-answer rate. A fourth metric is useful for a solo SaaS operator: review minutes per change. I would keep that last metric because the tradeoff is easy to miss: if a token optimization needs a bespoke evaluation harness every week, it may reduce API usage while increasing the real operating cost of shipping.
A weekly evaluation set can be small and still expose regressions. Include direct lookups, conflicting versions, no-answer questions, long policy exceptions, and questions that resemble a different tenant's documents. Record the expected source IDs, not only the final wording. That makes a cheaper context policy measurable without pretending that a single score captures answer quality.
When is the simpler runner-up better?
A fixed top-k policy is reasonable when documents are short, homogeneous, and already filtered by tenant and effective date. It is easier to explain and easier to debug. Start there if the retrieval corpus is small. Add token-aware packing once length variance or conversation history makes the request shape unpredictable.
A summary-first pipeline is reasonable for a long-lived chat where old turns are mostly conversational glue. It is a poor default for authoritative policy text unless the summary has a reviewable provenance link and a test that catches omitted exceptions.
A larger evidence budget is reasonable for genuinely ambiguous questions, but make ambiguity visible. Ask a clarifying question when two current sources disagree. Sending more text is not a resolution strategy.
More context is still a choice.
The portable design is the boring one: a tokenizer adapter, a ranker interface, a context policy, normalized usage records, and tests around empty or conflicting evidence. Provider adapters can change later. The application decision remains stable.
A token count matters because it measures the context and answer work you ask a model to perform. Reduce that work by improving what enters the request, reserving space for what must come out, and measuring the result at the application boundary. For a fintech knowledge assistant, the right optimization is the smallest evidence set that still preserves the relevant rule, exception, date, and source.
Top comments (0)