I keep running into the same wall when I add embedding-based checks to a project: the moment a test needs a real embedding, it needs a network call and an API key. That makes the test slow, flaky, and impossible to run in a fork's CI where secrets are not available. In this tutorial I want to show a pattern that gets you both things at once: real embedding similarity in production, and a deterministic offline fallback so your tests never touch the network.
I will build it up in plain TypeScript. The technique is what matters, but I will ground it in a scorer I wrote for evalgate, an eval-in-CI tool, so you can see it in a real codebase.
Step 1: cosine similarity
Two pieces of text become two vectors. The standard way to compare them is cosine similarity, which measures the angle between the vectors and ignores their magnitude. It returns a value in the range -1 to 1, where 1 means the vectors point the same direction.
export function cosineSimilarity(a: number[], b: number[]): number {
const len = Math.min(a.length, b.length);
let dot = 0;
let na = 0;
let nb = 0;
for (let i = 0; i < len; i++) {
const x = a[i] ?? 0;
const y = b[i] ?? 0;
dot += x * y;
na += x * x;
nb += y * y;
}
const denom = Math.sqrt(na) * Math.sqrt(nb);
return denom === 0 ? 0 : dot / denom;
}
Two small details matter. I clamp the loop to the shorter vector so a length mismatch never throws, and I guard the zero-magnitude case so an empty vector returns 0 instead of NaN. Those two guards are the difference between a scorer that is robust and one that blows up on the first empty string.
Step 2: an embedding interface, not a hard dependency
Here is the key design move. The scorer never imports OpenAI or any specific client. It depends on an interface: something that can turn text into a vector. In production a provider supplies a real embed() method. When none exists, the scorer falls back to a local one.
const embed = ctx.provider.embed
? (t: string) => ctx.provider.embed!(t)
: (t: string) => Promise.resolve(localEmbedding(t));
const [outVec, expVec] = await Promise.all([embed(output), embed(expected)]);
const cos = cosineSimilarity(outVec, expVec);
// Map [-1, 1] to [0, 1] so a threshold reads naturally.
const score = (cos + 1) / 2;
const passed = score >= threshold;
Because I map the cosine result from its native -1 to 1 range into 0 to 1, a threshold like 0.8 reads the way people expect a similarity score to read. The whole scorer is provider-agnostic: swap in any embedding backend, or none, and the comparison logic is identical.
Step 3: the deterministic offline embedding
This is the part that makes tests trivial. Instead of a semantic model, I build a fixed-width bag-of-hashed-words vector. Each word is hashed into one of dims buckets, I count occurrences, then L2-normalize so cosine similarity behaves.
First I need a stable hash. FNV-1a is tiny and fast, and Math.imul keeps the multiply inside 32 bits:
export function fnv1a(input: string): number {
let hash = 0x811c9dc5;
for (let i = 0; i < input.length; i++) {
hash ^= input.charCodeAt(i);
hash = Math.imul(hash, 0x01000193);
}
return hash >>> 0;
}
Then the embedding itself:
export function localEmbedding(text: string, dims = 64): number[] {
const vec = new Array<number>(dims).fill(0);
const words = text
.toLowerCase()
.replace(/[^a-z0-9\s]/g, " ")
.split(/\s+/)
.filter(Boolean);
for (const word of words) {
const idx = fnv1a(word) % dims;
vec[idx] = (vec[idx] ?? 0) + 1;
}
const norm = Math.sqrt(vec.reduce((s, v) => s + v * v, 0)) || 1;
return vec.map((v) => v / norm);
}
The output depends only on the input string. The same text always produces the same vector on every machine, every CI run, forever. No network, no key, no randomness. Texts that share words land in overlapping buckets and score high; texts with no shared words score near zero. That is exactly enough signal to test that your wiring works: that the scorer runs, that a threshold gates correctly, that identical strings pass and unrelated ones fail.
The honest caveat
A hashing fallback is not semantic. It matches surface tokens, not meaning. "The cat sat on the mat" and "A feline rested on the rug" mean the same thing but share almost no words, so the local embedding scores them low. A real embedding model would score them high. That is the whole point of the split: the fallback exists to keep your tests deterministic and offline, not to judge meaning. Use the real provider when you actually care about semantic closeness, and let the fallback carry your test suite. Never ship the hashing embedding as your production similarity metric.
Why this shape is worth it
The design has one seam: an embed function that is either a real client or a pure local function. Everything downstream, the cosine math and the threshold check, is identical in both worlds. Your CI runs the exact same code path it will run in production, minus the network. That is what makes the tests trustworthy instead of merely green.
If you want to see the full scorer, the provider registry, and how it plugs into an eval-gate that runs on pull requests, it is all in the open at github.com/AgentPostmortem/evalgate. Borrow the pattern, wire in your own embedding backend, and keep your tests fast and offline.
Top comments (2)
I'd reject a dimension mismatch rather than truncate it. With the min-length loop, [1, 0] and [1, 0, 100] are treated as identical because the unmatched dimension contributes to neither norm. That produces a confident score for vectors that should not have been compared. Equal dimensions are only the first check; the embedding model/version should match too. An explicit invalid-input result would keep this failure separate from a genuine low similarity score.
Keeping the scorer behind a plain embed() seam is a neat move, since the local FNV-1a fallback keeps tests offline without changing anything downstream.