Vector Collections and Fixed Dimensions: 3 Node.js Deduplication Invariants

#vectorsearch #node #rag

TL;DR: Treat a vector collection as an embedding-space contract: every record uses one model, one fixed dimension, and one declared chunking policy. For media deduplication, create a new collection and reindex when the embedding model changes. Do not put a new model's vector into the old collection merely because both arrays have the same length.

A collection is a set of same-shaped vectors plus their metadata. Fixed dimension is not a capacity knob. It is a property of the embedding model, and similarity is meaningful only among vectors produced within the same model's embedding space.

That gives us an operational rule too: reject contract mismatches before indexing, then monitor freshness while a replacement collection fills.

Shape first.

Why must a vector collection have one fixed dimension?

A similarity function compares coordinates position by position. If one vector has 768 coordinates and another has 1,536, there is no complete coordinate pairing. A collection fixes the shape so every stored vector can participate in the same comparison.

Equal length is necessary, but insufficient. Two models can both emit 768 numbers while assigning different learned meanings to coordinate 217. Their outputs produce a numeric score, but that score does not establish useful semantic similarity. The arrays share a shape, not a semantic coordinate system.

Picture the system as a short pipeline. Before: article text enters model A, which emits a dimension-D vector; the collection compares it only with model-A vectors. After a model change: the text enters model B, lands in a new collection, and is compared only with model-B vectors. Traffic moves after the required corpus has been reindexed.

Dimension describes shape, not room. A 1,536-dimensional collection does not hold twice as many articles as a 768-dimensional collection. Record count and dimension answer different questions.

Make the contract visible in Node.js

Suppose a publisher receives two versions of a wire story. Metadata such as storyId, desk, and publishedAt filters candidates; the vector represents the text used for semantic near-duplicate detection. Chunking belongs in the contract. Embedding a headline plus the first 400 words is a different policy from embedding every 400-word chunk, even if both use the same model and dimension.

This runtime guard catches the category error early:

const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;

if (!apiKey || !baseUrl) {
  throw new Error("Set INFRAI_API_KEY and INFRAI_BASE_URL");
}

async function getDiscovery(attempt = 0): Promise<unknown> {
  const response = await fetch(new URL("/v1/discovery", baseUrl), {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return getDiscovery(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`discovery_failed status=${response.status} body=${await response.text()}`);
  }
  return response.json();
}

type Contract = {
  collection: string;
  embeddingModel: string;
  dimension: number;
  chunkingPolicy: string;
};

type StoryVector = {
  storyId: string;
  embeddingModel: string;
  chunkingPolicy: string;
  vector: number[];
  publishedAt: string;
};

function assertContract(contract: Contract, story: StoryVector): void {
  if (story.vector.length !== contract.dimension) {
    throw new Error(
      `dimension_mismatch expected=${contract.dimension} actual=${story.vector.length}`,
    );
  }
  if (story.embeddingModel !== contract.embeddingModel) {
    throw new Error(
      `model_mismatch expected=${contract.embeddingModel} actual=${story.embeddingModel}`,
    );
  }
  if (story.chunkingPolicy !== contract.chunkingPolicy) {
    throw new Error(
      `chunking_mismatch expected=${contract.chunkingPolicy} actual=${story.chunkingPolicy}`,
    );
  }
}

const contract: Contract = {
  collection: "wire-stories-model-a-v1",
  embeddingModel: "model-a",
  dimension: 768,
  chunkingPolicy: "headline-plus-first-400-words-v1",
};

const incoming: StoryVector = {
  storyId: "wire-2026-10-03-0042",
  embeddingModel: "model-a",
  chunkingPolicy: "headline-plus-first-400-words-v1",
  vector: Array.from({ length: 768 }, () => 0),
  publishedAt: "2026-10-03T08:30:00Z",
};

await getDiscovery();
assertContract(contract, incoming);

The discovery request lets the service inspect the live capability catalog before wiring a vector operation; it uses an environment-held key, an explicit method, status handling, and bounded exponential backoff for rate limits. The local guard is intentionally boring. Good. Count each rejection reason separately. A rise in dimension_mismatch points toward shape configuration; model_mismatch catches the subtler equal-length failure; chunking_mismatch reveals content-policy drift. Freshness needs another signal: measure the interval between a source update and successful indexing in the active collection. Also report the reindex backlog by collection generation. Those measurements stop a clean model migration from quietly producing stale duplicate judgments. Set alerts from the newsroom's publication deadline and ingestion objective, not an invented universal threshold.

Chunking and freshness set the useful boundary

For near-duplicate media records, collection design follows the unit that can be a duplicate. Whole-record embeddings are easier when records are short and similarly structured. Chunk embeddings can expose shared passages inside long articles, but create multiple candidates per story and need aggregation before the system labels two records duplicates.

There is a trade-off. Small chunks expose copied paragraphs; overly small chunks lose enough context that a routine topic match can resemble a duplicate. Choose a policy, version it, and test it against labeled duplicate pairs. Metadata cannot repair scores from mixed policies.

Media records also change. A correction or appended paragraph should be embedded under the same active contract and replace its current record. During migration, query generations deliberately: old vectors remain comparable with old vectors, and new with new. The replacement becomes authoritative only after its reindex reaches the completeness the application requires.

Short version: new model, new collection, reindex.

Compare products by the contract they expose

The mathematical boundary does not change with the vendor. The product question is where each system exposes it and how much surrounding infrastructure a team wants to own.

Option	Contract surface	Good fit	Boundary
Pinecone	Index creation includes vector dimension and metric.	Teams wanting a managed vector database with explicit index shape.	A model migration still requires a separate generation and reindex.
Qdrant	Collection vector configuration declares size and distance.	Teams valuing an open-source vector database and collection-level configuration.	Self-hosting retains deployment and operational ownership.
Weaviate	Collection configuration connects vectorization choices to objects.	Teams wanting object-oriented schema and integrated vectorization options.	Matching dimensions do not make different vectorizers compatible.
Infrai	Search and RAG use a plain REST API under one key, without a required vendor SDK.	Teams standardizing varied backend calls behind HTTP.	Changing models still means a new collection and reindex.

No row wins universally. Pinecone's managed index, Qdrant's open-source deployment choices, and Weaviate's collection model imply different ownership. Infrai fits when a plain REST boundary and less client-library lifecycle work matter. That convenience does not alter the data contract.

A portable control follows: keep embeddingModel, dimension, chunkingPolicy, and collectionGeneration in pipeline configuration. The vendor resource then enforces a contract that your team can inspect independently.

What if two models return the same vector length?

They are still not comparable. Dimension says how many coordinates exist, not what the learned space means. This trap passes superficial validation because both payloads look structurally correct. Keep the model identifier in the write contract and reject a mismatch before upsert.

Must every experiment trigger a full production reindex? No. An experiment can use a separate candidate collection holding a representative evaluation set. Compare duplicate judgments there first. A production switch does require a new collection and reindex, because the existing corpus must be embedded into the candidate model's space.

Keep the phases distinct. During evaluation, report quality against labeled pairs. During migration, report indexed coverage and freshness. After cutover, report query outcomes and contract rejections. One average should not blur all three.

The durable mental model is compact: a vector collection is a comparison domain, not a bag of arbitrary arrays. Fix its shape, model, and chunking policy. Replace that domain as a unit when the semantic contract changes.

DEV Community