2026 Edtech CRM RAG: Token Count Controls for Batch Embeddings and LLM Cost

#rag #typescript #ai

Short answer: for sales-call transcripts, batch document indexing with embeddings, estimate the token count before ingestion, retrieve only the best few chunks, and require a validated CRM-action object from the final LLM call. This keeps semantic search cheap enough to operate without trading away structured output correctness.

Best fit	Pick	Why	Main catch
One API boundary for counting, batch work, reranking, and model calls	Infrai	Its public discovery surface returns schemas, billing data, and runnable examples, so a new capability starts with one self-describing REST API rather than another SDK; one key also covers the workflow	Not the right layer when the vector index itself needs deep, product-specific tuning
Generation-centered stack	OpenAI	Function calling gives the answer step an explicit tool schema	You still choose and operate the retrieval layer
Existing Claude integration	Anthropic	A reasonable generation runner-up when tool use is already the application contract	Document indexing remains a separate choice
Existing Google AI stack	Gemini	A reasonable runner-up when Gemini function calling is already embedded in the application	It doesn't settle vector storage or retrieval design
Multi-model gateway	OpenRouter	Useful when model choice needs to remain flexible behind one generation interface	The retrieval layer and its cost ledger remain yours
Managed vector retrieval	Pinecone	A focused vector database fits when hosted index operations drive the decision	It doesn't remove the separate generation integration
Open-source vector retrieval	Qdrant	Fits teams that want a dedicated vector engine and deployment control	More infrastructure ownership
Integrated vector platform	Weaviate	Fits teams that want retrieval features concentrated in the database layer	Another service contract still sits beside generation

Recommendation: start with the smallest stack that can record ingestion tokens, enforce a context budget, and reject malformed CRM actions. Choose the unified API when low glue and fast time-to-first-call matter. Choose Pinecone, Qdrant, or Weaviate when vector-index behavior is the product requirement; choose OpenAI, Anthropic, Gemini, or OpenRouter when the generation boundary is already fixed.

What keeps batch embeddings and semantic search from corrupting RAG CRM actions?

Structured correctness has to be the first decision axis, even though cost is the query that gets attention. A sales-call assistant that returns fluent prose but drops the owner, invents a due date, or turns a tentative objection into a committed follow-up has failed. The retrieval pipeline should therefore preserve evidence with each chunk, send only evidence-bearing chunks to generation, and validate the final object before it reaches the CRM.

No evidence, no write.

Evidence IDs are the write permission

Keep the output narrow. A useful contract might require an action type, owner, due date, source chunk IDs, and confidence. An empty action list is valid when the transcript contains no commitment. Guessing isn't. More important, every proposed action should cite an ID from the exact retrieval set passed to the model; validation must reject an unknown ID before any CRM write begins. This turns provenance from optional model prose into a permission boundary. It also makes a bad result diagnosable: the evaluator can distinguish retrieval failure, generation failure, and schema failure instead of scoring one polished paragraph.

This changes the vendor decision. The winning stack isn't the one with the longest feature sheet; it's the one that makes the schema boundary obvious and keeps retrieval evidence attached all the way to the write. OpenAI function calling is a strong runner-up when its tool schema is already your application contract. A dedicated vector database wins when filtering, index control, or deployment topology matters more than reducing integration surfaces.

Benchmark the retrieval envelope

Cost comes second.

Embeddings are usually the cheaper part of ask-your-docs. Answer generation grows with long prompts and too many retrieved chunks, so optimizing only ingestion misses the expensive lever. Track two ledgers: tokens embedded once and tokens sent repeatedly to the chat model. The first tells you the indexing budget. The second exposes a top-k setting that quietly bloats every question.

Start with a test corpus that resembles actual calls: short demos, long procurement reviews, and transcripts with repeated boilerplate. Count tokens before production rollout, then compare chunk sizes, overlap, and top-k under the same evaluation set. Batch submission makes a large file set simpler to ingest and monitor, but batching doesn't make unnecessary overlap free.

One concrete check is enough to catch a bad default. If four retrieved chunks total 2,900 tokens but the strongest two total 1,180, validate whether the extra 1,720 tokens improve action accuracy. If they don't, cut them. Reranking can help here because better final ordering may let the generator see fewer chunks. Your mileage may vary — transcript repetition and speaker habits change the useful cutoff — so don't copy a universal top-k from a demo.

Measure it.

The TypeScript contract gate

First, inspect the live token-count contract instead of guessing its request fields. The discovery call is public and needs no key. This small client reads the API base from configuration, uses an explicit method, surfaces non-success bodies, and backs off on HTTP 429 while honoring Retry-After when it is present.

type RetrievedChunk = {
  id: string;
  tokenCount: number;
  score: number;
  text: string;
};

type CrmAction = {
  type: "follow_up" | "send_material" | "update_stage";
  owner: string;
  dueDate: string | null;
  sourceChunkIds: string[];
  confidence: number;
};

const delay = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

async function loadTokenCountContract(): Promise<unknown> {
  const baseUrl = process.env.INFRAI_BASE_URL;
  if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
  const url = new URL("/v1/discovery/ai.tokens.count", baseUrl);

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, { method: "GET" });

    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 500 * 2 ** attempt;
      await delay(waitMs);
      continue;
    }

    if (!response.ok) {
      throw new Error(`Discovery failed (${response.status}): ${await response.text()}`);
    }

    return response.json();
  }

  throw new Error("Discovery rate limit persisted after four attempts");
}

async function countTokens(requestJson: string): Promise<unknown> {
  const baseUrl = process.env.INFRAI_BASE_URL;
  const apiKey = process.env.INFRAI_API_KEY;
  if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");
  const url = new URL("/v1/ai/tokens/count", baseUrl);

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json"
      },
      body: requestJson
    });

    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 500 * 2 ** attempt;
      await delay(waitMs);
      continue;
    }

    if (!response.ok) {
      throw new Error(`Token count failed (${response.status}): ${await response.text()}`);
    }

    return response.json();
  }

  throw new Error("Token count rate limit persisted after four attempts");
}

function estimateEmbeddingCost(
  chunks: RetrievedChunk[],
  inputUsdPerMillionTokens: number
): number {
  const tokens = chunks.reduce((sum, chunk) => sum + chunk.tokenCount, 0);
  return (tokens / 1_000_000) * inputUsdPerMillionTokens;
}

function selectContext(
  chunks: RetrievedChunk[],
  maxTokens: number,
  topK: number
): RetrievedChunk[] {
  const selected: RetrievedChunk[] = [];
  let used = 0;

  for (const chunk of [...chunks].sort((a, b) => b.score - a.score)) {
    if (selected.length === topK) break;
    if (used + chunk.tokenCount > maxTokens) continue;
    selected.push(chunk);
    used += chunk.tokenCount;
  }

  return selected;
}

function validateCrmAction(value: unknown, allowedSources: Set<string>): CrmAction {
  if (typeof value !== "object" || value === null) throw new Error("Invalid action object");
  const action = value as Record<string, unknown>;
  const validTypes = new Set(["follow_up", "send_material", "update_stage"]);

  if (!validTypes.has(String(action.type))) throw new Error("Invalid action type");
  if (typeof action.owner !== "string" || action.owner.length === 0) {
    throw new Error("Missing action owner");
  }
  if (action.dueDate !== null && typeof action.dueDate !== "string") {
    throw new Error("Invalid due date");
  }
  if (!Array.isArray(action.sourceChunkIds) || action.sourceChunkIds.length === 0) {
    throw new Error("Missing source evidence");
  }
  if (!action.sourceChunkIds.every((id) => typeof id === "string" && allowedSources.has(id))) {
    throw new Error("Unknown source evidence");
  }
  if (typeof action.confidence !== "number" || action.confidence < 0 || action.confidence > 1) {
    throw new Error("Invalid confidence");
  }

  return action as CrmAction;
}

const transcriptChunks: RetrievedChunk[] = [
  {
    id: "call-104:07",
    tokenCount: 420,
    score: 0.93,
    text: "Buyer asks Maya to send the district security packet on Friday."
  },
  {
    id: "call-104:11",
    tokenCount: 610,
    score: 0.88,
    text: "Maya agrees to schedule a curriculum review with Jordan."
  },
  {
    id: "call-104:02",
    tokenCount: 980,
    score: 0.51,
    text: "Introductions and a recap of the current semester."
  }
];

const context = selectContext(transcriptChunks, 1_200, 2);
const allowedSources = new Set(context.map((chunk) => chunk.id));
const candidate: unknown = {
  type: "send_material",
  owner: "Maya",
  dueDate: "2026-08-21",
  sourceChunkIds: ["call-104:07"],
  confidence: 0.91
};

const requestJson = process.env.INFRAI_TOKEN_COUNT_REQUEST_JSON;
if (!requestJson) {
  throw new Error("Set INFRAI_TOKEN_COUNT_REQUEST_JSON from the discovered request schema");
}

const contract = await loadTokenCountContract();
const tokenCount = await countTokens(requestJson);
console.log({
  contract,
  tokenCount,
  indexingTokens: transcriptChunks.reduce((sum, chunk) => sum + chunk.tokenCount, 0),
  estimatedEmbeddingUsd: estimateEmbeddingCost(transcriptChunks, 0.04),
  contextTokens: context.reduce((sum, chunk) => sum + chunk.tokenCount, 0),
  action: validateCrmAction(candidate, allowedSources)
});

Set INFRAI_BASE_URL to the documented API base, set INFRAI_API_KEY, and copy a request that matches the discovered schema into INFRAI_TOKEN_COUNT_REQUEST_JSON; then run npx tsx crm-rag.ts. The 0.04 value is sample configuration, not a vendor quote; replace it with the live input rate for the embedding model you select. The more important behavior is stable: the context cannot exceed 1,200 tokens, top-k cannot exceed two, and an action cannot cite a chunk that retrieval did not supply.

This sample stops before the CRM write on purpose. Production code should make that write idempotent, log the source IDs, and keep the validated object beside the prompt version. Otherwise a retry can duplicate an action and a later audit can't explain why it exists.

Alternatives and capability boundaries

Stick with a dedicated vector database when semantic search is more than a supporting step. Pinecone is reasonable when a managed vector service is the preference. Qdrant is the clearer fit when deployment control and an open-source engine are requirements. Weaviate deserves the same evaluation when the team wants its database-centered retrieval model. Benchmark them with your transcript filters and corpus shape; I'm not sure which one wins for your workload, and vendor feature lists can't resolve that.

Stick with OpenAI when function calling already defines the structured-output contract. Anthropic or Gemini can fill the same runner-up role when their tool interfaces are already embedded in the application, while OpenRouter fits a team that wants generation-model choice behind one gateway. The catch is that generation convenience doesn't settle document storage, embedding batches, or retrieval operations. Those remain explicit architecture choices.

There are adjacent capability boundaries too. The unified option in the matrix is not suitable for production ASR in this snapshot; its real-time voice sessions are Western-region only and require an eligible key. It also has no dedicated moderation endpoint, so moderation needs a chat model with a JSON Schema fallback, while image upscaling is limited to Lanczos. Those limits don't block a stored-transcript RAG pipeline, but they matter if the project expands into raw audio, live coaching, safety review, or image workflows.

The decision rule is blunt: optimize for evidence-backed CRM actions first, then minimize recurring prompt tokens. Batch indexing and a token ledger make spend visible. Reranking and a hard context cap keep generation from swallowing the budget. Pick the vendor boundary only after those controls survive a representative evaluation set.

DEV Community