Exporting Vectors and Metadata From One Vector Database to Another

#ai #machinelearning #tutorial #llm

Embeddings you already paid for do not need to be paid for twice. The vectors are just arrays of floats, and moving them is a data job — with the specific hazards a data job has, none of which the vector part makes interesting.

Decide what you are actually moving

Before writing any code, settle four things, because each of them changes the plan rather than the implementation.

Dimension and metric must match. You are copying vectors, not re-deriving them, so the destination collection has to be created with the same dimension and the same distance metric as the source. A cosine index and an inner-product index over the same vectors do not rank the same way unless every vector is unit-length.
Normalisation state travels with the data. If the source normalised before storing, the exported floats are normalised. If it normalised internally and stores the raw vector, they are not. Do not guess — compute the L2 norm of a hundred exported vectors and look at the distribution.
Sparse and hybrid components may not be exportable.If the source holds a sparse vector alongside the dense one for hybrid search, check that you can read it back out. If you cannot, the sparse side has to be rebuilt at the destination and the migration is no longer purely a copy.
Deleted-but-not-compacted records. Some stores keep tombstones. Enumerate and count before you trust either side’s reported total.

Enumerating and paging the source

The read API you need is a cursor, not a query. A similarity search will not enumerate a corpus — it returns the top results for one vector, and running it with random probes gives you neither completeness nor termination.

Pinecone documents both an automatic paginator, index.list(), which yields batches of IDs for a namespace, and index.list_paginated(), which takes an optional prefix, a limit and a pagination_token and returns the next token alongside the results; you then fetch the vectors and metadata for each batch of IDs. The Qdrant Python client’s scroll returns a page of points together with the offset to pass to the next call, and takes flags for whether to include payloads and vectors — vectors in particular are not returned unless you ask.

# Pinecone: ids first, then fetch in batches
def export_pinecone(index, namespace, batch=200):
    buf = []
    for ids in index.list(namespace=namespace):
        for i in range(0, len(ids), batch):
            got = index.fetch(ids=ids[i:i + batch], namespace=namespace)
            for vid, rec in got.vectors.items():
                yield {"id": vid, "vector": rec.values, "meta": rec.metadata or {}}

# Qdrant: scroll returns (points, next_offset)
def export_qdrant(client, collection, batch=256):
    offset = None
    while True:
        points, offset = client.scroll(
            collection_name=collection, limit=batch, offset=offset,
            with_payload=True, with_vectors=True,
        )
        for p in points:
            yield {"id": str(p.id), "vector": p.vector, "meta": p.payload or {}}
        if offset is None:
            return

A cursor over a live index is not a snapshot. Records written after your cursor passed their position will be missed; records deleted behind it may or may not still appear. If the corpus is being written to during the export, you need a second pass: record the wall-clock time at which the export started, then after the bulk load, replay everything with an updated_at at or after that timestamp. This only works if your records carry a timestamp — if they do not, adding one is the first step of the migration, not an optimisation.

The interchange file

Write to a file, not directly from one client into the other. A streamed pipe means any failure restarts the whole job, and it gives you nothing to inspect when the destination rejects a record.

JSON Lines is the right default for anything up to tens of millions of vectors: one record per line, resumable by line number, greppable, and every language reads it. The one thing to be careful about is float precision. Embeddings are typically float32, and a float32 written as a decimal string and read back is only exact if enough significant digits are emitted. Python’s default json.dumps uses repr for floats, which round-trips a float64 exactly, so a float32 widened to float64 also round-trips — but a formatter with a fixed precision, or a language that trims trailing digits, will quietly truncate.

The cost of truncation is not catastrophic and that is what makes it dangerous: rounding to six decimal places perturbs cosine similarity in the fifth decimal, which reorders near-ties at the bottom of a top-k list and nothing else. It will not show up in a spot check. If you want the question closed rather than argued, write the vectors as base64-encoded little-endian float32 bytes and the metadata as JSON on the same line. Byte-exact, about a third smaller than the decimal form, and unambiguous.

import base64, json, numpy as np

def encode(rec):
    v = np.asarray(rec["vector"], dtype=np.float32)
    return json.dumps({
        "id": rec["id"],
        "dim": int(v.shape[0]),
        "v_b64": base64.b64encode(v.tobytes()).decode(),
        "meta": rec["meta"],
    })

def decode(line):
    r = json.loads(line)
    v = np.frombuffer(base64.b64decode(r["v_b64"]), dtype=np.float32)
    assert v.shape[0] == r["dim"]
    return r["id"], v, r["meta"]

Identifiers and metadata that do not survive

Two classes of transformation belong in the loader, and both should be applied while writing the file rather than at insert time, so that the file is a record of what you actually loaded.

Identifiers first. Where the destination constrains the ID type — a store that accepts only unsigned integers or UUIDs cannot take a compound string key — derive the new identifier deterministically from the old one with a fixed namespace UUID, and keep the original string in the metadata under a field like external_id. Every citation, log line and delete-by-document path in your application refers to the old identifier; if it stops existing anywhere, the migration has quietly broken features that have nothing to do with search.

Metadata second. Flatten nested objects if the destination requires flat metadata, drop nulls if nulls are unsupported, and coerce types consistently — a field that is sometimes the integer 2024 and sometimes the string "2024" will filter correctly for half your corpus. Do the coercion once, in the exporter, and log a count per field of how many records needed it. That count is usually the first real information anyone has had about the data quality of the index.

Check the destination’s per-record metadata size limit before the load rather than during it. Pinecone documents 40KB of metadata per record; if you have been storing chunk text in metadata, some records will exceed a limit like that and the load will fail partway with a half-populated index.

Loading into the destination

Create the collection explicitly, with the dimension and metric you confirmed, before loading anything. Auto-created collections take default parameters, and a default metric is exactly the sort of thing nobody checks until retrieval looks strange.

Create the destination collection with the source’s dimension and metric. If the destination lets you defer index building, defer it: bulk-loading into an unbuilt index and building once afterwards is substantially faster than maintaining a graph through every insert.
Load in batches sized by serialised bytes, not by record count, and make the loader idempotent — upsert by identifier, so a restart re-writes rather than duplicates.
Checkpoint the byte offset or line number of the interchange file after each successful batch. A three-hour load that cannot resume is a three-hour load you will run twice.
Retry on transport errors with backoff, but treat a validation error — a rejected identifier, an oversized payload — as fatal and stop. Skipping bad records means finishing with an index that is silently short, which is the failure this whole exercise exists to avoid.
Build the index, then run the verification below before pointing any traffic at it.

Proving the copy is complete

Counts are the weakest possible check and they are what everyone reaches for. Equal counts are consistent with a load that dropped a hundred records and duplicated a hundred others, which is a realistic outcome of a retried batch job.

Compare identifier sets instead. Both sides can be enumerated — that is how you got here — so hash each identifier to a 64-bit integer and combine them with an order-independent operation such as XOR, or sum them modulo a large prime. One number per side, comparable cheaply, and it disagrees if any identifier is missing, extra or changed. Then check a sample of vectors byte-for-byte: fetch fifty random identifiers from both, and assert the vectors are equal to within float32 tolerance rather than merely similar.

Finally, compare retrieval rather than storage: run a few hundred real queries against both and compare the top-k identifier lists. Perfect agreement is not expected — approximate indexes with different parameters disagree at the tail — but systematic disagreement points at a metric or normalisation mismatch you would otherwise ship. What changes in search quality after a migration covers how to read that comparison, and auditing completion builds the same check for the case where the vectors were recomputed rather than copied.