Moved a table with about twenty million rows last month and learned three things the hard way.
Do the double-write before the backfill, not after. We did it the other way and spent two evenings reconciling.
Measure replication lag before you start, not during. Ours spiked and we had no baseline.
Write the rollback plan first. We had one, sort of, in someone's head. That is not a plan.
Top comments (2)
The double-write before backfill point matches what I'd expect, since otherwise rows written during the copy go missing and you end up reconciling by hand. A cheap addition is a verification step before cutover, such as comparing row counts and a checksum over key ranges on both tables. Did you throttle the backfill based on that replication lag, or run it at a fixed batch size?
The replication lag baseline is the detail that matters. Without it, a spike during the run tells you nothing.