Distributed Move Protocol

Atomic visibility for moving state across distributed systems.

Moving state sounds simple:

1. Copy state to a new place.
2. Delete state from the old place.

That works only when the source and destination are controlled by the same owner. In production systems, they may live on different partitions, services, databases, regions, or failure domains. At that point, a simple move becomes a distributed protocol.

The hard part is not copying data. The hard part is controlling visibility.

Why Copy and Delete Fails

A naive copy-and-delete approach creates three core risks:

• Duplicate visibility: both source and destination appear live.
• Missing visibility: source is gone before destination is visible.
• Partial visibility: readers observe incomplete destination state.

Retries, crashes, concurrent operations, and cleanup jobs make these risks real. They are not edge cases; they are protocol requirements.

Testing Comes First

For production systems, the first step is defining the failures the system must survive. Those tests become the real requirements: when the destination becomes visible, when the source can be retired, and how retries, recovery, and concurrency behave.

Required Guarantees

A distributed move protocol needs two visibility guarantees:

• If the source is gone, the destination must be visible.
• If the destination is visible, the source must be gone.

So the system must never expose:

• Source missing and destination hidden
• Source visible and destination visible
• Partial destination visible

Protocol Pattern

reserve -> prepare -> transfer -> verify -> finalize -> retire -> recover

The destination is prepared before it is visible. The source is retired only after the destination is finalized.

Source Owner        Protocol State        Destination Owner        Readers
    |                     |                      |                    |
    |<---- reserve ------>|<----- reserve ------>|                    |
    |                     |                      |                    |
    |                     | prepare hidden dest  |                    |
    |                     |--------------------->|                    |
    |                     |                      |                    |
    |---- read page ----->|                      |                    |
    |                     |---- write hidden --->|                    |
    |---- read page ----->|                      |                    |
    |                     |---- write hidden --->|                    |
    |                     |                      |                    |
    |                     |---- verify copied -->|                    |
    |                     |       state          |                    |
    |                     |                      |                    |
    |                     |==== FINALIZE =======>|---- visible ------>|
    |                     |  visibility boundary |                    |
    |                     |                      |                    |
    |<--- retire source --|                      |                    |
    |                     |                      |                    |
    |<---------------- recover / retry from recorded state ---------->|

External visibility should stay simple:

Before finalize: source = visible, destination = hidden
After retire:    source = gone,    destination = visible

Step Summary

• Reserve: mark source and destination so conflicting operations see an in-progress move.
• Prepare: create hidden destination state, such as IN_PROGRESS.
• Transfer: copy required records, pages, versions, index entries, or chunk references.
• Verify: confirm completeness using count, size, version, page, checksum, or Merkle-tree checks.
• Finalize: make destination authoritative through one visibility transition.
• Retire: delete, tombstone, redirect, or garbage-collect the source after finalize.
• Recover: retry from recorded state; every step should be idempotent.

Example:

• Source: staging/report-123
• Destination: published/report-123
• The system prepares the published destination invisibly, copies and verifies all state, finalizes visibility, then retires staging.

Where This Pattern Applies

This pattern applies beyond object storage rename:

1. Publishing data pipeline output
2. Promoting ML models from staging to production
3. Moving tenant metadata between cells
4. Rebalancing records across shards
5. Updating distributed indexes
6. Swapping analytics table partitions
7. Activating configuration versions
8. Migrating cache ownership

In each case, the system is changing which state is authoritative, not merely copying data.

Production Test Cases

Validate the protocol against failure and concurrency cases:

1. Copy succeeds, delete fails.
2. Delete is attempted before copy completes.
3. Destination is only partially copied.
4. Destination becomes visible too early.
5. Client retries after timeout.
6. Service crashes after reserve, prepare, transfer, or finalize.
7. Concurrent reads or writes touch source or destination.
8. Recovery runs more than once.
9. Garbage collection sees intermediate state.

Expected behavior is always the same: no duplicate visibility, no missing visibility, no partial visibility, and safe retry from recorded protocol state.

Core Lesson

Distributed moves need atomic visibility, not just data transfer.

Do the work invisibly -> verify it -> publish once -> clean up safely

Have you seen similar move, rename, publish, or promotion problems in distributed systems? I’d be interested in hearing which failure cases shaped your design.