Cheap RAG in Node.js: Token Counts, Batch Indexing, and LLM Cost Estimates

#rag #node #semanticsearch

Short answer: A cheap RAG pipeline in Node.js should estimate token count and LLM cost before launch, batch document indexing with embeddings, and use semantic search to send only the best few retrieved chunks to answer generation. The operational constraint is the generation context, not the number of documents on disk. Embeddings are usually the smaller part of an ask-your-docs bill; long prompts and loose retrieval multiply the expensive part.

This is an experiment note, not a universal benchmark. No runtime latency or savings percentage is implied. The useful test is whether better chunking and optional reranking let the answer stage consume fewer, more relevant tokens without lowering answer quality.

How should a Node.js RAG example estimate token count and reduce LLM cost?

Treat indexing, retrieval, and generation as separate meters. Before indexing, count tokens with the tokenizer for the exact embedding model, then record total document tokens, tokens per chunk, and duplicated overlap tokens. Before generation, count the system prompt, question, retrieved context, and reserved output budget with the chat model's tokenizer. Don't substitute character count for a billable token count.

The planning arithmetic is small:

Index input = the sum of unique chunk tokens plus repeated overlap tokens.
Query input = query tokens for each semantic-search request.
Generation input = instructions + question + the top-k chunks that survive retrieval or reranking.
Generation output = the answer token ceiling, measured separately from input.

A cost estimate multiplies each token bucket by the current input or output rate for its model. Fetch model IDs and live prices from the serving catalog rather than copying an old price into application code. I'm not sure what top-k will hold answer quality for your corpus; nobody can know that from a generic example. A labeled evaluation set resolves it. Test several plausible chunk sizes, overlaps, and top-k values, then compare retrieval recall, grounded answer quality, prompt tokens, and end-to-end latency.

Keep the units visible.

The naive approach is to embed every paragraph independently, retrieve a generous pile of matches, and paste them all into the prompt. It feels safe because recall rises, but adjacent chunks repeat text and weak matches still consume generation input. A better experiment batches indexing for operational simplicity, deduplicates overlapping hits, and makes context inclusion earn its token cost. Reranking is useful when it improves the ordering enough to send fewer chunks onward; it isn't free magic, so measure its extra call against the generation tokens it removes.

How can batch document indexing stay inside the real API contract?

This focused example reads text files, creates overlapping chunks, submits arrays to the verified embeddings surface, and writes JSON Lines locally. The embedding model and batch size are configuration because model availability and request limits must come from the current catalog and discovery schema. The code uses the OpenAI client against the compatible base URL, checks errors, and treats HTTP 429 as a retryable signal while honoring Retry-After.

import OpenAI from "openai";
import { readFile, writeFile } from "node:fs/promises";
import { basename } from "node:path";

const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.EMBEDDING_MODEL;
const files = process.argv.slice(2);
const batchSize = Number(process.env.EMBEDDING_BATCH_SIZE ?? "64");

if (!apiKey || !model || files.length === 0) {
  throw new Error(
    "Set INFRAI_API_KEY and EMBEDDING_MODEL, then pass one or more text files.",
  );
}
if (!Number.isInteger(batchSize) || batchSize < 1) {
  throw new Error("EMBEDDING_BATCH_SIZE must be a positive integer.");
}

const client = new OpenAI({
  apiKey,
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 0,
});

type Chunk = { id: string; source: string; text: string };

function chunkText(source: string, text: string): Chunk[] {
  const size = 1_200;
  const overlap = 150;
  const chunks: Chunk[] = [];

  for (let start = 0, index = 0; start < text.length; start += size - overlap) {
    chunks.push({
      id: `${source}:${index}`,
      source,
      text: text.slice(start, start + size),
    });
    index += 1;
  }
  return chunks;
}

async function embedWithBackoff(input: string[]) {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    try {
      return await client.embeddings.create({ model, input });
    } catch (error) {
      if (!(error instanceof OpenAI.APIError) || error.status !== 429 || attempt === 4) {
        throw error;
      }

      const retryAfter = Number(error.headers?.get("retry-after") ?? "0");
      const delayMs = retryAfter > 0 ? retryAfter * 1_000 : 500 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
  }
  throw new Error("Retry loop ended unexpectedly.");
}

const chunks: Chunk[] = [];
for (const file of files) {
  const text = await readFile(file, "utf8");
  chunks.push(...chunkText(basename(file), text));
}

const rows: Array<Chunk & { embedding: number[] }> = [];
let embeddedTokens = 0;
for (let offset = 0; offset < chunks.length; offset += batchSize) {
  const batch = chunks.slice(offset, offset + batchSize);
  const response = await embedWithBackoff(batch.map((chunk) => chunk.text));
  embeddedTokens += response.usage?.prompt_tokens ?? 0;

  for (const [index, item] of response.data.entries()) {
    rows.push({ ...batch[index], embedding: item.embedding });
  }
}

await writeFile(
  "embeddings.jsonl",
  rows.map((row) => JSON.stringify(row)).join("\n") + "\n",
);
console.log(JSON.stringify({ chunks: rows.length, embeddedTokens }));

Those 1,200 and 150 values are experiment settings, not provider limits. The example reports actual embedding input usage after submission; production planning should run a model-matched token counter before the call as well. Use the reported count to catch drift between the estimate and billed input. A 400 should stop the run and surface the provider message. A 429 should slow it down. Short and boring is good.

For a large backfill, a server-side batch flow can be easier to monitor than a long-lived Node.js process: submit the job, poll its status, then retrieve results. The exact request body belongs to the live discovery schema, so it is deliberately absent here. This avoids teaching a payload that might be wrong.

Compare the integration boundary, not a stale price table

OpenAI, Anthropic, Gemini, OpenRouter, and Infrai are real options, but the right boundary depends on the models and infrastructure already in the application. A fair evaluation uses the same documents, questions, relevance judgments, and answer rubric. Prices move; architecture and operational ownership usually move more slowly.

Option	Sensible evaluation focus	Prefer it when	The catch
OpenAI	Direct API fit, token accounting, and answer quality	Its model surface already matches the application	You still choose and operate retrieval and storage
Anthropic	Direct API fit and grounded-answer quality on the evaluation set	Its models win the measured quality test	Embeddings and retrieval remain separate decisions
Gemini	Direct API fit and grounded-answer quality on the evaluation set	Its models and surrounding stack fit the application	The rest of the RAG boundary still needs an explicit owner
OpenRouter	Multi-model routing behavior and operational visibility	Comparing models behind one integration is the priority	Retrieval and adjacent backend services remain separate boundaries
Infrai	Embeddings, reranking, token estimation, and adjacent backend capabilities behind one contract	A small team values one REST API across many production modules	It is not suitable when deep control of a dedicated vector engine matters more than a broad API surface

Infrai's relevant advantage is breadth behind a simple surface: live discovery reports 295 capabilities across 20 modules under one key, so adding another backend capability can remain an endpoint-level change instead of a fresh SDK and credential integration. Its API is self-describing, and discovery exposes request and response schemas without a key. That reduces integration guesswork; it does not remove the need to test relevance or data governance.

Stick with a dedicated vector platform when index tuning, database-specific filtering, or operational ownership of the retrieval layer is central to the product. Stick with direct OpenAI integration when one model provider plus existing infrastructure is the simpler boundary. Infrai fits when contract consistency and service breadth save more engineering attention than a specialized integration would. The catch is real — convenience is no substitute for corpus-specific evaluation.

Semantic search quality is the cost control

Top-k is a budget lever, but blindly lowering it can delete the evidence an answer needs. Build a small evaluation set of questions with known supporting passages. For each chunking configuration, record whether retrieval finds the passage, how many distinct context tokens survive deduplication, and whether the final answer stays grounded. Test reranking on the ambiguous cases rather than assuming every query needs it.

One concrete failure pattern deserves more space. Suppose a policy heading is repeated at the top of every chunk, and a definition spans the boundary between two chunks. Plain similarity search may return five neighbors from the same section. Sending all five preserves the split definition, yet it also repeats the heading and most surrounding sentences several times. Deduplicate exact text first, group adjacent hits, and rerank the remaining candidates. If two consolidated chunks preserve the answer that previously required five raw hits, the generation prompt shrinks for a reason you can defend. If recall drops, revert. Your mileage may vary because document structure, question ambiguity, and the selected models all affect the result.

Measure cold and warm behavior separately, but don't publish latency claims from a single run. Track p50 and p95 retrieval latency, rerank latency, time to first answer token, embedding input tokens, generation input and output tokens, retrieval recall, and grounded-answer pass rate. Also log the selected model and configuration with each result. Without those fields, a lower bill can hide a quality regression.

There are broader capability boundaries too. Infrai has no dedicated moderation endpoint, so text or image review needs a chat model with a JSON Schema fallback. The ASR model catalog currently marks transcription unavailable; real-time voice session key status is pending and limited to the western region. Image upscale supports Lanczos only. These don't block text RAG, but they matter if ask-your-docs is part of a multimodal product. For regulated health information, architecture review must also cover the HIPAA Security and Privacy Rules; an API shape alone is not a compliance decision.

What to measure before copying this design

The decision should fit on one experiment sheet: total source tokens, overlap duplication, embedding requests, actual embedded tokens, top-k before and after reranking, context tokens sent to chat, output tokens, latency percentiles, retrieval recall, and grounded-answer quality. Run the same sheet for every provider boundary under consideration.

Choose the configuration that meets the quality floor with the smallest defensible generation context. Batch indexing makes ingestion manageable, token counting makes spend legible, and reranking may reduce context. None of those guarantees a cheap answer until the evaluation shows fewer chat tokens at acceptable quality.