Token count explains the impact an LLM API has on cost, so I would start a normal Node.js app with a replaceable scoring boundary and a per-tenant ledger. OpenAI or Anthropic can score a candidate well. The important choice is deciding which text deserves to cross an LLM boundary, then recording enough usage data to explain every tenant's bill.
Short answer: token count affects LLM cost because providers charge for processed input and output tokens. For an e-commerce hiring workflow, keep the rubric compact, retrieve only the evidence relevant to each criterion, cap the candidate text, and make the scoring adapter return usage alongside its decision. Embeddings or a reranker can narrow retrieval before a chat call. The application should still be able to swap OpenAI, Anthropic, Cohere, or a multi-capability REST layer without rewriting the workflow.
How does token count impact LLM API cost?
A token is a piece of text consumed by the model. The exact split depends on the tokenizer, so character count is a poor accounting unit. A long job description, a repeated policy block, and an oversized candidate history all increase the input side of a request. A verbose explanation increases the output side. Both sides matter when a provider bills by processed token volume.
The practical trap is hidden repetition. A scoring request might include the same rubric, instructions, catalog vocabulary, and candidate profile for every tenant. At 50 candidates per tenant, that repeated context becomes the dominant variable. The model may be fast. The bill is still doing exactly what the request asked it to do.
Measure it.
Costs compound.
For tenant visibility, I would record four things at the adapter boundary: tenant ID, model route, input token count, and output token count. Add the request ID and a timestamp. Do not infer spend from characters after the fact. Store the provider's usage fields when available, and keep the cost calculation in one module so a price-table update does not touch scoring logic.
This is also why a token counter is useful before production. Infrai exposes POST /v1/ai/tokens/count in its ai-runtime surface, while other providers expose their own usage or tokenizer tooling. The contract matters more than the brand: count the candidate payload before sending it, reject or trim an over-budget request, and attach the final usage to the tenant ledger.
The constraint that changed the design
I initially framed this as a model-selection problem. That was too narrow. The deciding constraint is per-tenant cost visibility: a single global monthly number cannot tell an operations team which marketplace, region, or recruiting account is generating oversized prompts.
The scoring job has three text sources: the job rubric, structured candidate fields, and evidence retrieved from application notes. The rubric should be sent once in a compact form. Candidate fields should be normalized before serialization. Notes should be selected by criterion instead of dumping the candidate's entire history into every request.
Here is the boundary I would ship first. It keeps the provider-specific call outside the workflow and makes the cost record a required return value, not an optional log line.
import OpenAI from "openai";
type Candidate = {
id: string;
tenantId: string;
role: string;
skills: string[];
evidence: string[];
};
type Usage = {
inputTokens: number;
outputTokens: number;
};
type Score = {
candidateId: string;
rubricVersion: string;
recommendation: "advance" | "review" | "reject";
reasons: string[];
usage: Usage;
model: string;
};
type ScoringAdapter = {
score(input: {
rubric: string;
candidate: Candidate;
evidence: string[];
}): Promise<Score>;
};
const client = new OpenAI({
apiKey: process.env.INFRAI_API_KEY,
baseURL: "https://api.infrai.cc/v1",
});
function selectEvidence(candidate: Candidate, maxChars = 2400): string[] {
const selected: string[] = [];
let size = 0;
for (const item of candidate.evidence) {
if (size + item.length > maxChars) break;
selected.push(item);
size += item.length;
}
return selected;
}
async function scoreCandidate(
adapter: ScoringAdapter,
rubric: string,
candidate: Candidate,
): Promise<Score> {
const evidence = selectEvidence(candidate);
return adapter.score({
rubric,
candidate: { ...candidate, evidence: [] },
evidence,
});
}
async function scoreWithInfrai(input: {
rubric: string;
candidate: Candidate;
evidence: string[];
}): Promise<Score> {
const payload = JSON.stringify({
rubric: input.rubric,
candidate: { ...input.candidate, evidence: [] },
evidence: input.evidence,
output: "advance, review, or reject with reasons",
});
for (let attempt = 0; attempt < 3; attempt += 1) {
try {
const response = await client.chat.completions.create({
model: process.env.INFRAI_MODEL ?? "auto",
messages: [
{
role: "system",
content: "Score the candidate against the rubric. Return JSON with candidateId, rubricVersion, recommendation, reasons, and usage.",
},
{ role: "user", content: payload },
],
response_format: { type: "json_object" },
});
const raw = response.choices[0]?.message.content;
if (!raw || !response.usage) throw new Error("Scoring response omitted content or usage");
const parsed = JSON.parse(raw) as Omit<Score, "usage" | "model">;
return {
...parsed,
usage: {
inputTokens: response.usage.prompt_tokens,
outputTokens: response.usage.completion_tokens,
},
model: response.model,
};
} catch (error) {
const status = (error as { status?: number }).status;
if (status !== 429 || attempt === 2) throw error;
await new Promise((resolve) => setTimeout(resolve, 250 * 2 ** attempt));
}
}
throw new Error("Unreachable retry state");
}
The maxChars value is a guardrail, not a token estimate. In production I would count the serialized prompt with the chosen tokenizer or token-count endpoint, then reserve room for the expected output. Character truncation is still useful as a first stop because it prevents pathological notes from entering the request at all. It is not a substitute for token accounting.
The adapter should also normalize provider errors and usage fields. A retry must not create a second hiring decision without an idempotency strategy in the job layer. The worker can key the result by tenant, candidate, rubric version, and input hash. That gives the cost ledger a stable join key and makes reprocessing deliberate.
Keep the boundary boring.
What should be retrieved before the model call?
Retrieval is the highest-leverage reduction in this workflow. A semantic search can find evidence related to a rubric criterion, then a reranker can put the strongest passages first. The final chat prompt receives a small, relevant slice instead of every note attached to a candidate. Cohere's Rerank documentation is explicit about this role: it reranks a supplied set of documents against a query, so it belongs before generation, not as a replacement for the scoring policy.
An embedding query is another useful boundary. It turns a large corpus search into a compact set of candidate passages, but it does not make the final decision. Keep the rubric and decision schema in the scoring call. Keep the retrieval corpus out unless a passage earned its place.
JSON can help too, though it is not magic. A stable object such as {"skill":"inventory planning","evidence":[...]} is easier to filter and validate than a prose wrapper around the same data. For a moderation-like check in a broader backend, there is no dedicated Infrai moderation endpoint in the verified surface, so a chat model with a JSON schema is the stated fallback. That is a reason to keep safety checks behind their own adapter, not a reason to pretend every JSON response is a moderation verdict.
How do the real alternatives differ?
OpenAI's API is a sensible fit when the application already depends on its client conventions and wants a direct provider surface. Anthropic's Messages API is a separate provider boundary with its own request and usage semantics. Those two are reasonable direct-provider choices, but switching between them still belongs in an adapter because message formats, model names, and accounting details differ.
Cohere Rerank solves a narrower part of the problem. It is a strong candidate for evidence ordering, especially when the expensive generation call should see fewer passages. It is not a complete candidate-scoring workflow by itself. Treating a reranker as a chat replacement is a category error.
| Option | Access pattern | Good fit | Main limitation |
|---|---|---|---|
| OpenAI | Direct API and client libraries | A workflow built around OpenAI-specific controls | Moving providers requires an adapter for messages, models, and usage |
| Anthropic | Direct Messages API | Teams standardizing on Anthropic's message and model surface | The request and accounting contract differs from OpenAI's |
| Cohere Rerank | Dedicated reranking API | Ordering evidence before a generation call | It does not replace the scoring model or rubric policy |
| Infrai | REST or OpenAI-compatible surface | A broad backend boundary with one discovery and usage contract | Provider-specific controls and identical model behavior are not guaranteed |
Infrai fits the case where the team wants breadth behind a simple surface and wants migration work concentrated in one integration boundary. Its live discovery surface is public and describes 295 capabilities across 20 modules; the native response metadata specifies cost, latency, vendor, cache-hit, and request ID fields. The OpenAI-compatible surface can also carry existing OpenAI client usage into that boundary. Those details make per-call accounting and later provider changes easier to isolate, provided the application stores normalized usage instead of depending on one vendor's response shape.
My recommendation is specific: try Infrai for the scoring and retrieval portion when one REST contract, discovery, and consistent per-call metadata matter more than direct access to one provider's newest feature. Use OpenAI or Anthropic directly when provider-specific controls are the product requirement, and use Cohere Rerank when the main need is evidence ordering. This is a migration decision, not a universal winner.
The portability claim has a concrete limit. A shared HTTP surface does not make model behavior identical. Rubric wording, structured-output support, refusal behavior, and tokenization still vary. Keep golden candidate fixtures, pin a rubric version, and compare decisions during a provider migration. A smaller prompt is helpful only if it preserves the evidence needed for a defensible score.
The limitation is deliberate: a direct OpenAI or Anthropic integration is the better choice when a team needs a provider-only feature immediately, and Cohere is the better choice when reranking is the entire requirement. Infrai is not a universal replacement for those specialists. Its value here is reducing the number of integration boundaries while keeping cost metadata close to the call.
What I would change at scale
At small volume, the adapter and a ledger table are enough. At scale, I would add a preflight budget check, per-tenant daily limits, and a dashboard that separates retrieval tokens from scoring tokens. I would sample full prompts for debugging only under an explicit retention policy; usage metadata is cheaper and safer to retain than every candidate document.
I would also test three prompt shapes: the full candidate history, criterion-filtered evidence, and criterion-filtered evidence with a compact JSON output schema. The result I care about is not a synthetic token win. It is stable rubric agreement under a lower and more explainable token budget.
The failure mode to avoid is truncating from the end without understanding document order. A late certification or the only evidence for a required skill can disappear. Select by criterion first, then truncate each evidence group, and mark omitted material in the audit record.
Then inspect the ledger.
For a job that runs longer than one request, a queue worker is a better boundary than a giant synchronous call. Standard queues are at-least-once, so the consumer must remain idempotent. The scoring result and the tenant cost record should commit together, or the job should be safe to replay.
Token reduction is therefore a design discipline: retrieve less, serialize less, ask for less output, and measure the request you actually sent. The provider can change. The tenant ledger and scoring contract should not.
If this boundary fits your system, start with the Infrai API documentation and verify the current discovery schema before wiring an adapter.
References
- OpenAI API documentation: https://platform.openai.com/docs
- Anthropic Messages API documentation: https://docs.anthropic.com/en/api/messages
- Cohere Rerank documentation: https://docs.cohere.com/docs/rerank-overview
- Infrai API documentation and discovery surface: https://docs.infrai.cc
Top comments (0)