Embeddings in TypeScript: Batching, Rate Limits, and the Retry That Corrupts Your Index

#typescript #node #rag #ai

Book: AI That Reads
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Embedding a large corpus is a job that looks like a for loop and
behaves like a distributed system. It runs for an hour, talks to a
rate-limited API, writes to a database, and fails partway through
in ways that a for loop has no answer for.

The bug worth the most attention is not the slow one or the
rate-limited one. It is a retry that succeeds and leaves your index
quietly wrong.

The naive version and why it is slow

for (const chunk of chunks) {
  const res = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: chunk.text,
  });
  await db.insert(chunk.id, res.data[0].embedding);
}

Fifty thousand chunks, one round trip each. At 150ms per call that is
over two hours, and almost all of it is waiting.

Embedding endpoints accept arrays. That is the first fix and it is
large:

const res = await openai.embeddings.create({
  model: "text-embedding-3-small",
  input: batch.map((c) => c.text),
});

One call, many vectors. Batch size is bounded by the request limit
and the per-request token limit, both of which you should read from
current docs rather than assume — and note the token limit binds
first when chunks are long. A batch of 512 short chunks may be fine
while a batch of 64 long ones is not.

Size batches by token estimate, not by count:

function batches(chunks: Chunk[], tokenBudget: number): Chunk[][] {
  const out: Chunk[][] = [];
  let cur: Chunk[] = [];
  let used = 0;
  for (const c of chunks) {
    const cost = estimateTokens(c.text);
    if (cur.length && used + cost > tokenBudget) {
      out.push(cur);
      cur = [];
      used = 0;
    }
    cur.push(c);
    used += cost;
  }
  if (cur.length) out.push(cur);
  return out;
}

Bounded concurrency

Batches in sequence still leaves the connection idle between round
trips. Batches with Promise.all over all of them at once gets you
rate-limited within seconds.

What you want is a fixed number in flight:

import pLimit from "p-limit";

const limit = pLimit(4);
await Promise.all(
  batches(chunks, 60_000).map((b) => limit(() => embedBatch(b))),
);

Four is a starting point, not a recommendation. The right number
depends on your account limits and on how much of the corpus you want
to re-run when something goes wrong. Higher concurrency finishes
sooner and loses more work on a crash.

Backoff that reads the response

Retrying on any error with a fixed delay is worse than not retrying,
because it turns one rate limit into sustained pressure.

async function withRetry<T>(
  fn: () => Promise<T>,
  attempts = 5,
): Promise<T> {
  let lastErr: unknown;
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      lastErr = err;
      if (!isRetryable(err)) throw err;

      const retryAfter = headerSeconds(err);
      const backoff = retryAfter ?? Math.min(2 ** i, 30);
      const jitter = Math.random() * 0.3 * backoff;
      await sleep((backoff + jitter) * 1000);
    }
  }
  throw lastErr;
}

Three details. isRetryable distinguishes 429 and 5xx from a 400 —
retrying a malformed request just spends your attempts. Retry-After
from the response beats your own backoff curve, because the server
knows when it will accept you. And jitter stops four workers that
were rate-limited together from retrying in lockstep forever.

The retry that corrupts the index

Now the one that matters.

Batching means embeddings come back as an array, and you map them
back onto your chunks by position:

const res = await embed(batch.map((c) => c.text));
for (let i = 0; i < batch.length; i++) {
  await db.insert(batch[i].id, res.data[i].embedding);
}

Positional mapping. It is correct as long as the response has the
same length and order as the request.

Here is where it breaks. A batch fails partway — a timeout, a
connection reset, one input over the token limit. Someone writes a
retry that drops the offending item and re-sends:

// the corrupting retry
const retryBatch = batch.filter((c) => c.id !== failedId);
const res2 = await embed(retryBatch.map((c) => c.text));
for (let i = 0; i < batch.length; i++) {          // ← stale length
  await db.insert(batch[i].id, res2.data[i].embedding);
}

The loop still walks the original batch while indexing into the
shorter response. Every chunk after the removed one gets the
embedding of its neighbour. res2.data[i] becomes undefined at the
end, which may throw — or may insert a null your column accepts.

Nothing surfaces. The insert succeeds. The index builds. Search
returns results, they are just subtly wrong: a query about billing
retrieves the chunk about shipping, because shipping is holding
billing's vector. Recall degrades by a few percent and no test
catches a few percent.

Two properties prevent it.

Never map by position across a retry boundary. Providers return
an index field precisely for this — use it, and pair it with your
own ids:

const byIndex = new Map(res.data.map((d) => [d.index, d.embedding]));
for (const [i, chunk] of batch.entries()) {
  const vec = byIndex.get(i);
  if (!vec) throw new MissingEmbedding(chunk.id, i);
  await db.insert(chunk.id, vec);
}

Make the write idempotent and keyed by content. Then a retry that
re-embeds something already stored is a no-op rather than a
different row:

INSERT INTO chunks (content_hash, embed_model, text, embedding)
VALUES ($1, $2, $3, $4)
ON CONFLICT (content_hash, embed_model)
DO UPDATE SET embedding = EXCLUDED.embedding;

content_hash is a SHA-256 of the normalised chunk text. It makes
the whole job restartable: rerun it from the top and unchanged
content is skipped, changed content is updated, and nothing is
duplicated. It is also what lets you resume after a crash without
tracking a cursor.

Failing loudly at the end

A long job that logs errors and continues will finish "successfully"
with gaps.

const failures: Array<{ id: string; err: string }> = [];

// ... inside the worker
catch (err) {
  failures.push({ id: chunk.id, err: String(err) });
}

if (failures.length) {
  logger.error("embedding run incomplete", {
    total: chunks.length,
    failed: failures.length,
    sample: failures.slice(0, 10),
  });
  process.exitCode = 1;
}

A non-zero exit is what makes your scheduler notice. Without it, an
ingest that embedded ninety-four percent of the corpus reports
success, and the missing six percent shows up as a user saying the
assistant "doesn't know about" a document that is right there.

The check that catches all of it

After a run, sample a few dozen rows and re-embed their text fresh.
Cosine similarity against the stored vector should be essentially 1.
Anything materially lower means a vector is attached to the wrong
text — which is exactly the failure that is otherwise invisible.

Cheap to run, and it is the only check that catches the positional
bug after the fact.

If this was useful

AI That Reads treats ingest
as the engineering problem it is — batching, concurrency, idempotent
writes, resumability, and verifying that what you stored is what you
meant to store.