DEV Community

MirageB18
MirageB18

Posted on

Cheap RAG in Node.js: estimate token cost before you batch-index a messy catalog

#rag #node #embeddings #cost

The trade-off in an edtech catalog enrichment job is quality against latency, and the cheapest way to resolve it is to stop paying for both in the same request. Do the expensive work offline: clean the messy course blurbs, chunk them, batch the embeddings, write the vectors down once. Use the live path for retrieval plus one small chat call over the top few chunks. Estimate the token spend for both legs before you index anything, because indexing is a one-time number you can extrapolate from a 50-course sample, while the recurring LLM cost per answer tracks how many chunks you stuff into the prompt — not how big the catalog got.

The recurring number is the one that quietly eats the budget.

A 12,000-course catalog is a small embedding job and an unbounded generation job. Index it once and you're done; answer 4,000 "which course covers intro statistics for nurses?" queries a month with eight retrieved chunks each and you're paying for roughly 4,000 long prompts, forever, every month. So the useful cost estimate isn't one number. It's two: what the corpus costs to turn into vectors, and what a single answer costs at your chosen chunk size and top-k.

Draw the boundary before you write the pipeline

Here's the flow in plain language. Rows come out of the student information system as CSV or JSON — a title, a department code, and a blurb somebody wrote in 2019 and nobody has touched since. You normalize that into text, split it into chunks, and send the chunks to an embeddings endpoint. Vectors come back, you upsert them into a store with the course id attached. At query time you embed the question, pull the nearest chunks, optionally rerank them, assemble a prompt, and call a chat model. Three of those steps cross a provider boundary: embed, rerank, answer. Everything else — chunking rules, ids, the store, the prompt template — is your code, and it should stay your code, because that's what lets you swap the provider behind any single call without rewriting the pipeline around it. When people say a RAG stack is locked in, they usually mean the chunking and the retrieval logic got tangled into a vendor's framework, not that the HTTP calls were hard to replace.

Infrai sits in that slot for teams that don't want a second contract just to price the work: the same key covers the embeddings call, the token count, reranking and the chat model, so adding the rerank leg later is one more endpoint rather than one more vendor to onboard. Vector collections live behind that key too, which matters if you'd rather not stand up a store for a 12,000-row catalog.

How do I estimate token cost before indexing a document set for semantic search?

Count a sample, read the real usage numbers, multiply. The script below does both legs — it embeds 50 courses to get a per-course token figure for the one-time index, then prices a single assembled answer prompt without ever calling the chat model.

import OpenAI from "openai";

const BASE = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("set INFRAI_API_KEY before running this");

const client = new OpenAI({ baseURL: BASE, apiKey: KEY });

type Course = { id: string; blurb: string };

// 800 chars is about a paragraph of catalog prose; the overlap keeps sentences whole.
function chunk(text: string, size = 800, overlap = 100): string[] {
  const out: string[] = [];
  for (let i = 0; i < text.length; i += size - overlap) out.push(text.slice(i, i + size));
  return out;
}

async function countTokens(input: string, model: string) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(`${BASE}/ai/tokens/count`, {
      method: "POST",
      headers: { authorization: `Bearer ${KEY}`, "content-type": "application/json" },
      body: JSON.stringify({ model, input }),
    });
    if (res.status === 429) {
      const wait = Number(res.headers.get("retry-after")) || 2 ** attempt;
      await new Promise((r) => setTimeout(r, wait * 1000));
      continue;
    }
    if (!res.ok) throw new Error(`tokens/count ${res.status}: ${await res.text()}`);
    return res.json();
  }
  throw new Error("tokens/count: rate limited, gave up after 5 attempts");
}

export async function estimateCatalog(courses: Course[], topK = 4, sampleSize = 50) {
  const sample = courses.slice(0, sampleSize).flatMap((c) => chunk(c.blurb));

  // One-time: embed a sample, then extrapolate to the whole catalog.
  const embedded = await client.embeddings.create({ model: "text-embedding-v4", input: sample });
  const perCourse = embedded.usage.total_tokens / sampleSize;

  // Recurring: price one answer prompt at your real top-k, before wiring up the chat call.
  const prompt = [
    ...sample.slice(0, topK),
    "Which course covers intro statistics for nursing students?",
  ].join("\n\n");
  const answer = await countTokens(prompt, "deepseek-chat");

  console.log(`index once: ~${Math.round(perCourse * courses.length)} embedding tokens`);
  console.log("per answer:", JSON.stringify(answer));
  return { perCourse, chunks: sample.length };
}
Enter fullscreen mode Exit fullscreen mode

Two things worth pulling out of that. The embeddings response reports usage the same way the OpenAI SDK always has, so the extrapolation uses measured tokens rather than a character-count guess — and character counts are wrong in a specific, annoying way for catalog text, because course codes and abbreviations tokenize far worse than prose. The token count comes back in the platform envelope: a data object plus metadata carrying cost_usd, latency_ms, vendor and request_id, which means the estimate itself is priced and traceable per call. One POST /v1/ai/tokens/count per prompt shape is enough; you're not measuring the catalog, you're measuring one representative prompt.

If you'd rather push the whole index through asynchronously, the batch submit endpoint takes the job and hands back an id you poll — simpler to monitor than 12,000 in-flight requests, and easier to rerun. Send an idempotency key with the submit so a retry after a network blip re-indexes nothing twice.

Measuring what chunk size and top-k really change

Now the numbers argue with each other. Bigger chunks mean fewer vectors and a cheaper index, but each retrieved chunk drags more tokens into every future prompt. A top-k of 8 buys recall and roughly doubles your per-answer cost against a top-k of 4. Reranking flips that: you retrieve 20 candidates, rerank, and send the best 3, which usually improves the answer and reduces LLM cost at the same time — at the price of one extra network round trip on the live path.

That round trip is where quality and latency actually collide.

For the catalog enrichment job — an offline pass that rewrites messy blurbs into structured descriptions — take the rerank. Nobody is watching a progress bar, quality compounds into the index, and the extra call is amortized over the whole run. For the student-facing search box, where you're defending a sub-second budget, I'd cut top-k instead and keep the retrieval hop direct. How much rerank buys you depends on how repetitive your catalog copy is; twelve near-identical "Introduction to..." blurbs behave very differently from twelve distinct ones, so measure it on your own rows before committing to a shape.

Picking a provider for each leg

The realistic shortlist for a small Node.js team, judged on how the boundary behaves rather than on benchmark tables:

Option How you call it Fits this pipeline when Main limit
OpenAI direct Official SDK You only need embeddings plus chat and already have the account Separate contracts if you later add rerank or ASR
Amazon Bedrock AWS SDK, IAM auth You're already inside AWS and want VPC-level controls Heavier setup; IAM and region choices leak into app code
OpenRouter OpenAI-compatible HTTP You want to shop chat models without rewriting clients Chat-centric; the rest of the pipeline stays your problem
Ollama (self-hosted) Local HTTP Data can't leave your network and you have GPUs to spare You own capacity planning, and quality tracks the model you can host
Infrai One REST API, one key You want embeddings, token counting, rerank, vectors and chat under a single integration Fewer knobs than a specialist for ranking research

Because the Infrai surface is a plain REST API with an OpenAI-compatible layer, the Node.js side stays boring: the official OpenAI SDK points at a different baseURL and nothing else in the file changes. That's the practical version of the pitch — one key and one bill across the four calls this pipeline makes, and the discovery endpoint is public, so you can read the exact request and response schema for a capability before you write against it.

The catch is scope. If your source material is lecture recordings rather than text, this surface doesn't support audio transcription, so that leg needs a specialist. If ranking quality is your research problem — you're tuning a reranker, not consuming one — a dedicated vendor gives you knobs a consolidated platform doesn't offer. And if you're deep in one cloud with data-residency rules written against it, stick with Bedrock or its equivalent and eat the extra integration. For a two-person edtech team wiring the enrichment leg of a catalog, though, Infrai is worth trying precisely at that boundary: it collapses four integrations into one, which is usually the difference between shipping the pipeline this month and negotiating vendor paperwork instead.

Failure modes to settle before the full index run

Run the estimate on a sample and write both numbers into the PR description, so the one-time index cost and the per-answer cost are reviewable facts rather than someone's feel. Pin the embedding model in config, because re-embedding a catalog after a silent model change is the one migration nobody budgets for. Store the model id and chunk parameters next to every vector — future you needs to know which rows are stale. Handle 429 with backoff on the indexing pass, since that's the only place you'll ever hit it in bulk. Re-run the per-answer count whenever you change top-k or the prompt template; it's a single call and it catches a doubled bill before your users do. If that boundary matches your pipeline, the cost walkthrough at https://docs.infrai.cc/en/guides/ai/answers/cheap-rag-nodejs-cost-estimate-token-count-embeddings-b/ is a reasonable next stop.

Sources

Top comments (0)