DEV Community: Distributed Data

Distributed Metadata Migration

Distributed Data — Tue, 21 Jul 2026 04:40:37 +0000

Distributed Metadata Migration

How to move large-scale metadata systems without losing correctness.

Migrating a metadata system sounds straightforward:

Copy existing records to the new system.
Replay writes that happened during the copy.
Validate the result.
Cut traffic over.

At small scale, that may work. At cloud scale, the migration itself becomes a distributed system.

The source may be sharded. The destination may use a different partitioning model. Objects may be small, large, versioned, or spread across many records. Writes may continue while the migration is running. Validation may take hours or days. A failed migration may leave partial state that must be cleaned up safely.

The hard part is not only copying data. The hard part is proving that the destination is complete, ordered, and safe to become authoritative.

Core Risks

A large metadata migration has four common risks:

Drift: source keeps changing while the destination is being copied.
Partial copy: only some records, pages, or chunks reach the destination.
Stale replay: old migration writes arrive after newer ones.
Unsafe cutover: traffic moves before validation is strong enough.

Any of these can create missing records, duplicate records, broken listing behavior, stale metadata, or cleanup bugs.

Testing Comes First

Before building the happy path, define the failures the migration must survive:

copy worker crashes mid-batch
replay worker sends the same batch twice
stale write arrives after a newer write
validation finds a missing object
cutover starts while writes are still arriving
cleanup sees a failed partial migration
large objects require many metadata pages
small buckets and large buckets need different strategies

These tests become the real design requirements.

A Safer Migration Pattern

snapshot -> copy -> replay -> validate -> cutover -> cleanup -> recover

The destination is built and validated before it becomes authoritative. The source remains authoritative until cutover is safe.

Source System       Migration State       Destination System       Clients
    |                     |                      |                    |
    |---- snapshot ------>|                      |                    |
    |                     |                      |                    |
    |---- copy batch ---->|---- write batch ---->|                    |
    |---- copy batch ---->|---- write batch ---->|                    |
    |                     |                      |                    |
    |---- change logs --->|---- replay logs ---->|                    |
    |                     |                      |                    |
    |                     |---- validate ------->|                    |
    |                     | count/list/checksum  |                    |
    |                     |                      |                    |
    |                     |==== CUTOVER ========>|---- active ------->|
    |                     |  authority boundary  |                    |
    |                     |                      |                    |
    |<--- cleanup old ----|                      |                    |
    |                     |                      |                    |
    |<--------------- recover / retry from migration state ---------->|

External authority should stay simple:

Before cutover: source = authoritative, destination = shadow
After cutover:  source = retired,       destination = authoritative

Step Summary

Snapshot: define the migration start point and the set of records to copy.
Copy: move existing metadata in batches, often partitioned by shard or virtual cell.
Replay: apply writes that happened after the snapshot using generation logs or change streams.
Validate: compare source and destination using list validation, counters, sizes, versions, or checksums.
Cutover: stop or narrow writes, run final validation, and make the destination authoritative.
Cleanup: remove partial, failed, or retired migration state only after it is safe.
Recover: make every batch, replay, validation, and cleanup step retry-safe.

Ordering Matters

During migration, the destination may receive repeated or delayed writes. Each partition should reject stale work.

One practical pattern is:

assign a monotonically increasing sequence number per destination partition
allow only one outstanding request per partition when ordering matters
accept only writes with a strictly newer sequence
treat stale sequence responses as safe duplicate/retry outcomes

This prevents old replay traffic from overwriting newer destination state.

Validation Matters

Validation is not a single check. Different buckets or datasets may require different validation levels.

Small datasets can sometimes be validated fully during a short write pause. Large datasets often need staged validation:

validate copied data
validate replayed logs
validate list output
validate per-partition counters
perform final validation during cutover

The goal is to fail early when possible and avoid discovering correctness bugs only after days of copying.

Where This Pattern Applies

This pattern applies to:

storage metadata migrations
database shard migrations
search-index rebuilds
tenant moves between cells
cache ownership migration
table-partition migration
control-plane metadata upgrades
object-store namespace migration

In each case, the system is not just moving records. It is changing which system is authoritative.

Production Test Cases

Validate the migration against failure and scale cases:

copy succeeds but validation fails
replay sends duplicate batches
stale sequence arrives after a newer sequence
worker crashes after partial batch write
final validation fails during cutover
cleanup sees partial destination state
small dataset cutover completes with full validation
large dataset requires staged validation
recovery runs twice for the same migration
destination rejects stale writes safely

Expected behavior is always the same: no missing metadata, no stale overwrite, no unsafe cutover, no cleanup of needed state, and safe retry from recorded migration state.

Core Lesson

Large-scale metadata migration is not a copy job. It is a correctness protocol.

Copy in batches -> replay changes -> validate continuously -> cut over once -> clean up safely

I wrote about why “move” is harder than copy + delete in distributed systems used for AI workflows. The key challenge is atomic visibility: no duplicates, no missing state, no partial destination.

Distributed Data — Tue, 21 Jul 2026 04:22:57 +0000

Distributed Data

Jul 21

Distributed Move Protocol

#distributedsystems #aidata

3 min read

Distributed Move Protocol

Distributed Data — Tue, 21 Jul 2026 04:02:00 +0000

Distributed Move Protocol

Atomic visibility for moving state across distributed systems.

Moving state sounds simple:

Copy state to a new place.
Delete state from the old place.

That works only when the source and destination are controlled by the same owner. In production systems, they may live on different partitions, services, databases, regions, or failure domains. At that point, a simple move becomes a distributed protocol.

The hard part is not copying data. The hard part is controlling visibility.

Why Copy and Delete Fails

A naive copy-and-delete approach creates three core risks:

Duplicate visibility: both source and destination appear live.
Missing visibility: source is gone before destination is visible.
Partial visibility: readers observe incomplete destination state.

Retries, crashes, concurrent operations, and cleanup jobs make these risks real. They are not edge cases; they are protocol requirements.

Testing Comes First

For production systems, the first step is defining the failures the system must survive. Those tests become the real requirements: when the destination becomes visible, when the source can be retired, and how retries, recovery, and concurrency behave.

Required Guarantees

A distributed move protocol needs two visibility guarantees:

If the source is gone, the destination must be visible.
If the destination is visible, the source must be gone.

So the system must never expose:

Source missing and destination hidden
Source visible and destination visible
Partial destination visible

Protocol Pattern

reserve -> prepare -> transfer -> verify -> finalize -> retire -> recover

The destination is prepared before it is visible. The source is retired only after the destination is finalized.

Source Owner        Protocol State        Destination Owner        Readers
    |                     |                      |                    |
    |<---- reserve ------>|<----- reserve ------>|                    |
    |                     |                      |                    |
    |                     | prepare hidden dest  |                    |
    |                     |--------------------->|                    |
    |                     |                      |                    |
    |---- read page ----->|                      |                    |
    |                     |---- write hidden --->|                    |
    |---- read page ----->|                      |                    |
    |                     |---- write hidden --->|                    |
    |                     |                      |                    |
    |                     |---- verify copied -->|                    |
    |                     |       state          |                    |
    |                     |                      |                    |
    |                     |==== FINALIZE =======>|---- visible ------>|
    |                     |  visibility boundary |                    |
    |                     |                      |                    |
    |<--- retire source --|                      |                    |
    |                     |                      |                    |
    |<---------------- recover / retry from recorded state ---------->|

External visibility should stay simple:

Before finalize: source = visible, destination = hidden
After retire:    source = gone,    destination = visible

Step Summary

Reserve: mark source and destination so conflicting operations see an in-progress move.
Prepare: create hidden destination state, such as IN_PROGRESS.
Transfer: copy required records, pages, versions, index entries, or chunk references.
Verify: confirm completeness using count, size, version, page, checksum, or Merkle-tree checks.
Finalize: make destination authoritative through one visibility transition.
Retire: delete, tombstone, redirect, or garbage-collect the source after finalize.
Recover: retry from recorded state; every step should be idempotent.

Example:

Source: staging/report-123
Destination: published/report-123
The system prepares the published destination invisibly, copies and verifies all state, finalizes visibility, then retires staging.

Where This Pattern Applies

This pattern applies beyond object storage rename:

Publishing data pipeline output
Promoting ML models from staging to production
Moving tenant metadata between cells
Rebalancing records across shards
Updating distributed indexes
Swapping analytics table partitions
Activating configuration versions
Migrating cache ownership

In each case, the system is changing which state is authoritative, not merely copying data.

Production Test Cases

Validate the protocol against failure and concurrency cases:

Copy succeeds, delete fails.
Delete is attempted before copy completes.
Destination is only partially copied.
Destination becomes visible too early.
Client retries after timeout.
Service crashes after reserve, prepare, transfer, or finalize.
Concurrent reads or writes touch source or destination.
Recovery runs more than once.
Garbage collection sees intermediate state.

Expected behavior is always the same: no duplicate visibility, no missing visibility, no partial visibility, and safe retry from recorded protocol state.

Core Lesson

Distributed moves need atomic visibility, not just data transfer.

Do the work invisibly -> verify it -> publish once -> clean up safely

Have you seen similar move, rename, publish, or promotion problems in distributed systems? I’d be interested in hearing which failure cases shaped your design.