A chat completion that breaks throws. An embedding that breaks returns a perfectly well-formed array of floats that means something different from every vector already in your index, and nothing anywhere throws.
Why this failure is silent
Everything downstream of an embedding call is arithmetic on arrays. Cosine similarity between a 1,536-dimension query vector and a 1,536-dimension stored vector returns a number whether or not the two were produced by the same model, the same version, or the same normalisation. There is no type error to catch, no exception to log, and no assertion in your application that would notice.
What you observe instead is that retrieval quality got worse. Not zero — worse. The top result is still topically related, because two embedding models trained on similar data agree roughly about what is similar to what, and the degradation looks exactly like a prompt problem or a chunking problem. Teams lose weeks to that misattribution. The only cheap defence is a contract test that fails on the day the vector shape or the vector semantics change, rather than a quality metric that drifts three weeks later.
There are three distinct things to catch, and they need different assertions: the response is not the shape you parse (loud, caught by a schema), the vectors are the wrong length (silent until an insert fails, caught by a dimension assertion), and the vectors are the right length but from a different model (entirely silent, caught only by an invariant against a stored reference).
The base64 default that catches people
The single most reported compatibility failure on this endpoint is not about dimensions at all. The OpenAI API documents encoding_format as defaulting to float, but the official Python SDK sends base64 when you do not specify one — it is smaller on the wire and the SDK decodes it for you. This is recorded as openai-python issue 1490, and the OpenAPI specification itself does not describe the base64 response shape, which is openai-openapi issue 424.
Two consequences follow, and both are worth a test. First, a third-party endpoint that implements only float will receive a request asking for base64 that it does not understand, and depending on how it handles unknown enum values you get an error, an empty data array, or a string where your code expects a list. Second, a strict schema written from the published specification will reject a genuine OpenAI response — your contract test fails against the reference implementation, which is a confusing way to spend an afternoon.
The resolution is to be explicit. Send encoding_format deliberately on every call and assert on the branch you chose, then add one test that sends the other value and asserts it also works. Explicitness costs nothing and removes an entire class of surprise when you point the same code at a different base URL.
Pin the dimension to the index
The dimension is not a property of the response to be discovered; it is a constant your vector store was created with, and the test should assert the response matches that constant. Write it as a single exported value that the migration which created the index and the contract test both import. If those two numbers can disagree, they eventually will.
Two provider parameters make this less stable than it sounds. The dimensions parameter, supported on text-embedding-3 and later models, truncates the output to a requested size — and an endpoint that does not support it may ignore it silently and return its native size instead of erroring. And a model alias that resolves to a newer version can change the native size underneath you. Both are caught by the same one-line assertion, which is why it earns its place even though it looks trivial.
Assert on the batch as well as the single call. The response data array carries an index on each element precisely because order is not promised; code that zips the response array against the input array positionally is relying on an implementation detail. Send three distinguishable inputs, assert you get three elements, and assert that reordering by index is a no-op or, better, actually reorder by it and stop caring.
Invariants beyond the shape
Shape assertions catch the loud half. For the silent half — same length, different model — you need properties of the vectors themselves. None of these asserts a specific float, which would be meaningless across providers and versions.
- Determinism. Embed the same string twice in one run. The two vectors should be identical, or identical to within floating-point noise. A provider that returns materially different vectors for identical input is doing something you need to know about before you build a cache on top of it.
- Normalisation. Check the L2 norm. OpenAI’s embedding models return unit-length vectors, which is why dot product and cosine similarity are interchangeable against them. A provider that returns unnormalised vectors breaks any code that took that shortcut, and the breakage is a silent ranking change. Assert the norm is what you expect, and if it is not 1, assert that too so that a change is visible.
- A metamorphic ordering. Take three fixed strings: two paraphrases and one unrelated sentence. Assert that the similarity between the paraphrases exceeds the similarity between either and the unrelated sentence. This is a relation, not a value, so it survives model upgrades — and it is the only assertion here that fails when the endpoint quietly starts serving a different model at the same dimension.
- A stored reference vector. If you cannot tolerate a model change at all, commit one vector for one fixed string and assert the cosine similarity against a fresh embedding of the same string exceeds a threshold near 1. Note in the test that a failure here means the model changed, not that the code broke.
The suite
import { describe, it, expect } from "vitest";
import OpenAI from "openai";
import { z } from "zod";
// imported by the index migration too — one constant, not two
export const EMBEDDING_DIM = 1536;
const client = new OpenAI({ baseURL: process.env.TARGET_URL, apiKey: process.env.TARGET_KEY! });
const MODEL = process.env.TARGET_EMBED_MODEL!;
const Response = z.object({
object: z.literal("list"),
model: z.string().min(1),
data: z.array(z.object({
object: z.literal("embedding"),
index: z.number().int().nonnegative(),
embedding: z.array(z.number()),
})).min(1),
usage: z.object({ prompt_tokens: z.number().int(), total_tokens: z.number().int() }),
});
const dot = (a: number[], b: number[]) => a.reduce((s, x, i) => s + x * b[i], 0);
const norm = (a: number[]) => Math.sqrt(dot(a, a));
const cosine = (a: number[], b: number[]) => dot(a, b) / (norm(a) * norm(b));
describe("embeddings contract", () => {
it("returns the pinned dimension for every batch element, in index order", async () => {
const res = Response.parse(await client.embeddings.create({
model: MODEL,
input: ["alpha", "beta", "gamma"],
encoding_format: "float",
}));
expect(res.data).toHaveLength(3);
res.data.forEach((d, i) => {
expect(d.index).toBe(i);
expect(d.embedding).toHaveLength(EMBEDDING_DIM);
});
});
it("returns unit-length vectors", async () => {
const res = Response.parse(await client.embeddings.create({
model: MODEL, input: "a fixed sentence", encoding_format: "float",
}));
expect(norm(res.data[0].embedding)).toBeCloseTo(1, 3);
});
it("keeps paraphrases closer than unrelated text", async () => {
const res = Response.parse(await client.embeddings.create({
model: MODEL,
input: [
"The invoice was paid on Tuesday.",
"Payment for the invoice went through on Tuesday.",
"Otters groom their fur to keep it waterproof.",
],
encoding_format: "float",
}));
const [a, b, c] = res.data.map((d) => d.embedding);
expect(cosine(a, b)).toBeGreaterThan(cosine(a, c));
});
});
The last test is the one to keep if you keep only one. It costs a single API call, asserts nothing about any float, and is the only check in the file that notices a model substitution behind an unchanged model name.
Top comments (0)