Caching Embeddings in Node: A 60-Line Layer That Cuts the Bill

#typescript #node #rag #ai

Book: AI That Reads
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Your nightly ingest walks a docs repository, chunks every file, and
embeds every chunk. It has done this for months. On a normal night
maybe four files changed.

You are paying to re-embed a corpus that is, by volume, almost
entirely identical to last night's corpus. Embeddings are cheap per
call, which is precisely why nobody looks at this line item — it is
small, constant, and entirely avoidable.

Embeddings are also the easiest thing in an AI stack to cache,
because the function is genuinely pure. Same text, same model, same
vector. No temperature, no sampling, no context.

The key is where the thinking goes

The obvious key is a hash of the text. That is right and incomplete.

import { createHash } from "node:crypto";

export function cacheKey(text: string, model: string, dims?: number) {
  const norm = text.normalize("NFC").replace(/\s+/g, " ").trim();
  const h = createHash("sha256").update(norm).digest("hex");
  return `emb:${model}:${dims ?? "default"}:${h}`;
}

Three parts of that matter more than the hash itself.

Normalisation. Two chunks differing only in a trailing newline or
in Unicode composition are the same content and should hit the same
entry. NFC plus whitespace collapse handles the common cases. Do
not go further — lowercasing or stripping punctuation changes what
you are embedding, and the cache must key on exactly the bytes you
send.

The model in the key. This is the one people leave out, and it is
the one that corrupts the index. Switch from one embedding model to
another and a text-only key returns yesterday's vector from the old
model. Those vectors live in a different space. Comparing them
produces distances that are meaningless — and no error, ever. Your
search quality drops and nothing in the system explains why.

The dimension. Some models support variable output dimensions. A
1536-dim and a 768-dim vector of the same text under the same model
are different values and need different entries.

The interface

export interface VectorCache {
  getMany(keys: string[]): Promise<Map<string, number[]>>;
  setMany(entries: Map<string, number[]>): Promise<void>;
}

Batch-shaped, not single-key. Ingest works in batches, and a
per-item get against Redis turns one round trip into five hundred.

An in-memory implementation for tests and small jobs:

export class MemoryCache implements VectorCache {
  private store = new Map<string, number[]>();

  async getMany(keys: string[]) {
    const out = new Map<string, number[]>();
    for (const k of keys) {
      const v = this.store.get(k);
      if (v) out.set(k, v);
    }
    return out;
  }

  async setMany(entries: Map<string, number[]>) {
    for (const [k, v] of entries) this.store.set(k, v);
  }
}

And Redis for anything shared across processes:

export class RedisCache implements VectorCache {
  constructor(private redis: Redis, private ttlSec = 60 * 60 * 24 * 90) {}

  async getMany(keys: string[]) {
    if (!keys.length) return new Map();
    const raw = await this.redis.mgetBuffer(...keys);
    const out = new Map<string, number[]>();
    raw.forEach((buf, i) => {
      if (buf) out.set(keys[i], decode(buf));
    });
    return out;
  }

  async setMany(entries: Map<string, number[]>) {
    const pipe = this.redis.pipeline();
    for (const [k, v] of entries) {
      pipe.set(k, encode(v), "EX", this.ttlSec);
    }
    await pipe.exec();
  }
}

Store vectors as binary, not JSON. A 1536-dim float array is about
12 KB as a JSON string of decimals and 6 KB as Float32Array bytes,
and it parses without going through the JSON machinery:

const encode = (v: number[]) =>
  Buffer.from(new Float32Array(v).buffer);

const decode = (b: Buffer) =>
  Array.from(
    new Float32Array(b.buffer, b.byteOffset, b.length / 4),
  );

Float32Array loses a little precision against float64. For cosine
similarity over normalised embeddings that difference is far below
what affects ranking, and halving memory is worth it.

The layer itself

export function cached(
  embed: (texts: string[]) => Promise<number[][]>,
  cache: VectorCache,
  model: string,
) {
  return async function embedCached(texts: string[]) {
    const keys = texts.map((t) => cacheKey(t, model));
    const hits = await cache.getMany([...new Set(keys)]);

    const missIdx: number[] = [];
    texts.forEach((_, i) => {
      if (!hits.has(keys[i])) missIdx.push(i);
    });

    if (missIdx.length) {
      const fresh = await embed(missIdx.map((i) => texts[i]));
      const toStore = new Map<string, number[]>();
      missIdx.forEach((idx, j) => {
        hits.set(keys[idx], fresh[j]);
        toStore.set(keys[idx], fresh[j]);
      });
      await cache.setMany(toStore);
    }

    return keys.map((k) => hits.get(k)!);
  };
}

The final keys.map is the part to read twice. It rebuilds the
result in the original input order, which is what makes this a
drop-in replacement for the uncached function. A cache layer that
returns hits first and misses after is a correctness bug waiting for
a caller that maps by position.

Deduplicating with new Set before the lookup matters too — repeated
boilerplate across pages means the same text often appears several
times in one batch, and without the dedupe you fetch it more than
once.

Swapping it in is one line:

const embed = cached(rawEmbed, new RedisCache(redis), MODEL);

Invalidation is a non-problem here

Most caches are hard because the underlying value can change. This
one cannot: the same text under the same model gives the same vector
indefinitely.

So there is no invalidation logic. A model change produces a
different key, which is a miss, which is correct. A TTL exists only
to reclaim space for content that no longer exists, not for
freshness.

That is worth stating because teams sometimes build elaborate
versioning around embedding caches. The model in the key already
does it.

Where it also pays: queries

Ingest is the obvious win. Query-side is the quieter one.

Support corpora get the same questions repeatedly — "how do I reset
my password" arrives in a hundred phrasings, but a good fraction are
byte-identical. Wrapping the query embedder with the same layer
removes a round trip from the request path, which shows up as
latency rather than cost.

const embedQuery = cached(rawEmbed, queryCache, MODEL);

Use a shorter TTL there, and keep it in a separate keyspace so a
flush of query entries does not evict your corpus.

Measure the hit rate

A cache you cannot see is a cache you cannot trust.

metrics.increment("embed.cache.hit", keys.length - missIdx.length);
metrics.increment("embed.cache.miss", missIdx.length);

The number tells you real things. A nightly ingest hit rate that is
not overwhelmingly high means your chunker is not deterministic —
same input, different chunk boundaries, different hashes. That is a
chunking bug the cache just surfaced, and it is worth more than the
saved calls.

A hit rate that drops to zero overnight means someone changed the
model, the normalisation, or the chunker. All three are things you
want to know about.

The scope of the claim

This does not make embedding free. It makes you pay once per distinct
piece of content per model, which is the correct number of times.

Whether that is a large saving or a small one depends entirely on
your change rate. A corpus that turns over completely every night
saves nothing. A docs site where four files change saves nearly
everything.

If this was useful

AI That Reads covers the
ingest pipeline as a system — deterministic chunking, idempotent
writes, caching, and resumable runs over a corpus large enough that
restarting from zero is not an option.