On-Prem Cloud Data Migration: Lift-and-Shift vs Re-Architect, DataSync, DistCp & Cutover

cloud data migration is the multi-quarter, career-defining project that either lands your company on a cheaper, faster, more elastic platform — or strands it half-migrated with two systems to pay for, two sources of truth to reconcile, and a warehouse nobody trusts. Every byte of your on-premises estate — the NAS full of raw exports, the Hadoop cluster grinding through nightly ETL, the row-oriented data warehouse feeding every dashboard — has to reach cloud object storage, a cloud lakehouse, or a managed warehouse without losing rows, without corrupting a single Parquet file, and without a cutover outage that halts the business. The hard decision is never "should we move to the cloud"; it is how you move — whether you lift and shift the same engines unchanged or re-architect for cloud-native storage and compute — and which data transfer mechanism you trust to carry petabytes across a WAN that was never designed for it.

This guide is the senior-data-engineering walkthrough you wished existed the first time an interviewer said "walk me through migrating our on-prem Hadoop and warehouse to the cloud," or "how would you move 2 PB of HDFS when the WAN tops out at 10 Gbps," or "design a near-zero-downtime cutover and tell me how you'd roll back." It works through the four canonical decisions — the migration strategy split of lift and shift versus re-architect, managed file movement with AWS DataSync, distributed lake movement with DistCp, and the on-prem to cloud cutover itself — plus the four axes interviewers actually probe (data volume and bandwidth, downtime tolerance, transformation depth, and validation and rollback), the canonical setup for each tool, and the reconciliation gate that decides whether the switchover is safe. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse the modelling reps on the design practice library →, and stress-test the batch fundamentals on the data-processing practice library →.

---

On this page

• Why the migration approach determines everything downstream
• Lift-and-shift vs re-architect
• AWS DataSync — managed file & object transfer
• DistCp — distributed HDFS / data-lake copy at scale
• Cutover — backfill, dual-write, validate, switch over
• Cheat sheet — on-prem → cloud migration recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why the migration approach determines everything downstream

Four axes, one irreversible plan — the approach you pick binds the next two quarters

The one-sentence invariant: cloud data migration is a picking exercise between rehosting your engines unchanged (lift and shift) and redesigning them cloud-native (re-architect), carried out by a transfer mechanism sized to your data volume — a managed file service like DataSync, a distributed copy like DistCp, a database replicator like DMS, or an offline appliance like Snowball — and finished by a cutover that proves parity before it switches consumers over; and each choice trades transfer time against downtime, transformation depth, and the cost of validating and rolling back. The plan you commit to in month one dictates how long you run two platforms in parallel, how much re-engineering the warehouse consumers absorb, and whether the go-live is a five-minute switch or a weekend war room. Get the axes wrong and you discover — three months in — that the WAN cannot carry the delta, or that "lift and shift" quietly rehosted a nightly job that now costs 4× to run on cloud compute.

The four axes interviewers actually probe.

• Data volume & bandwidth. How many terabytes or petabytes, and how fat is the pipe? The governing equation is time-to-transfer = data volume ÷ effective bandwidth. A 100 TB dataset over a fully saturated 1 Gbps link is ~9 days; the same 100 TB over 10 Gbps is ~22 hours; 2 PB over 10 Gbps is ~19 days at 100% utilisation you will never get. When the network math exceeds your window, you go offline — physical appliances (AWS Snowball / Snowmobile, Azure Data Box, GCP Transfer Appliance). Interviewers open here because the arithmetic separates people who have shipped a migration from people who have only drawn one.
• Downtime tolerance. Can the business take a cutover outage, or do you need near-zero downtime? A big-bang cutover freezes writes, copies the final delta, validates, and switches — simple, but it requires a maintenance window. Near-zero downtime needs an incremental catch-up channel (CDC or dual-write) so the cloud stays within seconds of the source until you flip. The tolerance dictates the whole cutover design.
• Transformation depth. Are you moving bytes as-is (lift and shift — same file formats, same schema, same engine), or reshaping them (re-architect — CSV → Parquet, re-partition, change the query engine)? Depth drives risk and effort: byte-for-byte copies are verifiable by checksum; transformed copies must be reconciled semantically (row counts, aggregates, sampled diffs) because the bytes deliberately differ.
• Validation & rollback. How do you prove the migrated data equals the source, and can you back out? Every credible migration ships a reconciliation harness (row counts, per-partition checksums, aggregate diffs) as a hard gate before switchover, and keeps the source authoritative and writable until the cloud has soaked. Skipping this axis is how a silent 0.2% row loss becomes a finance-team incident during quarter close.

The 2026 reality — online for terabytes, offline for petabytes, incremental before cutover.

• Online transfer (DataSync, DistCp over Direct Connect / VPN, DMS) is the default for anything the network can carry inside the window. A dedicated interconnect (AWS Direct Connect, Azure ExpressRoute) is table stakes for a serious migration — the public internet is neither fast enough nor predictable enough.
• Offline transfer (Snowball, Data Box, Transfer Appliance) is the answer when volume ÷ bandwidth exceeds the window. You ship disks. A petabyte over a 10 Gbps link is weeks; a Snowmobile is a truck. The crossover is arithmetic, not preference.
• Incremental catch-up is what makes cutover safe regardless of tool. Bulk-copy the bulk of the data over days or weeks, then keep applying the changes (CDC log stream, DataSync scheduled sync, DistCp snapshot-diff) until the final delta is small enough to copy inside the cutover window.
• Re-architecture is usually incremental, not a big bang. The strangler pattern — lift-and-shift to land fast, then refactor workload by workload — is the dominant real-world approach because a simultaneous move-and-redesign multiplies risk.

What interviewers listen for.

• Do you reach for the time-to-transfer arithmetic (volume ÷ bandwidth) and name the offline threshold without prompting? — senior signal.
• Do you separate lift-and-shift (rehost, checksum-verifiable) from re-architect (reshape, semantically reconciled) rather than blurring them into "migrate"? — required framing.
• Do you describe cutover as a gated switch — bulk copy, incremental catch-up, validation gate, switchover, rollback — not a single copy? — required answer.
• Do you insist the source stays authoritative until the cloud soaks, with a concrete rollback path? — senior signal.
• Do you name the transfer tool per data type — DataSync for files/objects, DistCp for HDFS/lake, DMS for databases, Snowball for offline? — senior signal.

Worked example — the four-axis migration comparison table

Detailed explanation. The single most useful artifact for a migration interview is a memorised axis table for a concrete estate. Every senior migration discussion converges on it within the first ten minutes; having it in your head is what separates a fluent answer from a vague one. Walk through building the table for a hypothetical mixed estate moving to AWS.

• The estate. 50 TB on a NetApp NAS (raw file exports), a 2 PB on-prem Hadoop cluster (HDFS + Hive), and a 30 TB row-oriented on-prem warehouse feeding BI.
• The network. A 10 Gbps AWS Direct Connect link, shared with other traffic (realistically ~4 Gbps usable for migration).
• The window. Business wants the warehouse cut over inside a single weekend; the lake and NAS can trickle over weeks.
• The target. S3 as the landing zone; EMR/Athena for the lake; Redshift for the warehouse.

Question. Build the four-axis comparison and pick the transfer mechanism and cutover style for each of the three workloads.

Input.

Workload	Volume	Downtime tolerance	Transformation	Transfer candidate
NAS file exports	50 TB	weeks (low urgency)	none (lift-and-shift)	DataSync
Hadoop lake (HDFS/Hive)	2 PB	weeks	reformat to Parquet later	DistCp (+ Snowball for bulk)
Row warehouse	30 TB	one weekend	re-architect to Redshift	DMS + reconcile

Code.

Time-to-transfer arithmetic (effective, not nameplate)
======================================================
usable bandwidth  = 4 Gbps  = 0.5 GB/s  (10 Gbps DX, ~40% usable share)

NAS   50 TB  =  50,000 GB / 0.5 GB/s  = 100,000 s ≈ 1.16 days   -> ONLINE (DataSync)
WH    30 TB  =  30,000 GB / 0.5 GB/s  =  60,000 s ≈ 0.69 days   -> ONLINE (DMS bulk + CDC)
Lake   2 PB  = 2,000,000 GB / 0.5 GB/s = 4,000,000 s ≈ 46 days  -> exceeds window

Offline crossover:
  2 PB online ≈ 46 days at 4 Gbps  ->  ship it.
  Bulk 2 PB on Snowball appliances (parallel), then DistCp snapshot-diff
  for the delta accumulated during the appliance round-trip.

Step-by-step explanation.

1. The NAS is 50 TB with weeks of slack, no reshape, and file-shaped — this is textbook DataSync: an agent-based, verified, incremental online copy that finishes in ~1 day of transfer and can then run scheduled syncs until cutover.
2. The warehouse is only 30 TB but has a hard one-weekend window and a re-architecture (row-store → Redshift columnar). Volume is trivial online; the constraint is the reshape and the tight window, so the design is a DMS full-load + change-data-capture that keeps Redshift within seconds of the source, then a weekend cutover with reconciliation.
3. The lake is the hard one: 2 PB online at a realistic 4 Gbps is ~46 days, which blows any sane window. The arithmetic forces offline — bulk the 2 PB onto Snowball appliances, and use DistCp with snapshot-diff to copy only the delta that accrued while the appliances were in transit.
4. Notice the transformation axis is deferred for the lake: land it byte-for-byte first (fast, checksum-verifiable), then re-architect HDFS/Hive tables into Parquet-on-S3 with Athena after the data is safely in the cloud. Move-then-reshape beats move-and-reshape.
5. Every workload ends with a validation gate and a rollback story — the NAS by checksum, the warehouse by row-count and aggregate reconciliation, the lake by per-partition file counts and checksums. No switchover happens before its gate is green.

Output.

Workload	Transfer	Cutover style	Validation
NAS 50 TB	DataSync online + scheduled sync	trickle, low-risk	file checksums
Warehouse 30 TB	DMS full-load + CDC	big weekend, gated	row counts + aggregates
Lake 2 PB	Snowball bulk + DistCp -diff delta	phased	per-partition file count + checksum

Rule of thumb. Never pick a transfer mechanism from habit. Compute volume ÷ effective bandwidth first; if it exceeds your window, go offline. Then layer the four axes — volume, downtime, transformation, validation — and the tool and cutover style fall out of the constraints.

Worked example — what interviewers actually probe

Detailed explanation. The senior migration interview has a predictable structure: the interviewer opens with an ambiguous prompt ("how would you move our on-prem data platform to the cloud?"), then narrows to test whether you know the axes. Candidates who name the strategy and do the transfer arithmetic in the first two minutes score highest; candidates who say "we'd use a migration tool" score lowest. Walk through the grading rubric.

• Ambiguous opener. "How would you migrate our on-prem data platform to AWS?" — invites you to name lift-and-shift vs re-architect and size the transfer.
• Follow-up 1. "The lake is 2 PB and the WAN is 10 Gbps — now what?" — probes the volume/bandwidth axis and the offline threshold.
• Follow-up 2. "The business can't take an outage — how do you cut over?" — probes downtime tolerance and incremental catch-up.
• Follow-up 3. "How do you know the migrated data is correct?" — probes validation and reconciliation.
• Follow-up 4. "It's live on the cloud and wrong — what now?" — probes rollback.

Question. Draft a five-minute senior migration answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Strategy named	"we'd migrate everything"	"lift-and-shift to land fast, re-architect per workload after"
Transfer sizing	"copy it over the network"	"volume ÷ effective bandwidth; 2 PB at 4 Gbps ≈ 46 days → Snowball"
Downtime	"take a maintenance window"	"CDC/dual-write catch-up so the cutover delta is minutes, not days"
Validation	"spot-check some rows"	"row counts + per-partition checksums + aggregate diffs as a hard gate"
Rollback	"restore a backup"	"keep source authoritable and writable until the cloud soaks; flip DNS/consumers back"

Code.

Senior cloud-migration answer template (5 minutes)
==================================================

Minute 1 — strategy up front
  "I'd lift-and-shift to land in the cloud fast and de-risk, then
   re-architect workload by workload (strangler pattern). Move first,
   reshape second."

Minute 2 — size the transfer
  "Time-to-transfer = volume / effective bandwidth. Files -> DataSync;
   HDFS/lake -> DistCp; databases -> DMS. If volume/bandwidth exceeds
   the window (e.g. 2 PB at 4 Gbps ~= 46 days), go offline with
   Snowball and use snapshot-diff for the delta."

Minute 3 — cutover with catch-up
  "Bulk-copy the base, then keep the cloud within seconds via CDC or
   scheduled incremental sync. The cutover copies only the final small
   delta, so the outage is a switch, not a re-copy."

Minute 4 — validation gate
  "No switchover before reconciliation is green: row counts per table,
   per-partition file counts and checksums for the lake, and aggregate
   diffs (SUM/COUNT by day) for transformed data. It runs in CI."

Minute 5 — rollback + soak
  "The on-prem source stays authoritative and writable until the cloud
   has soaked under real load. Rollback is repoint consumers/DNS back to
   on-prem; the reverse-sync keeps the source current during soak."

Step-by-step explanation.

1. Minute 1 frames you as an architect: naming the strangler pattern ("move first, reshape second") signals you know that simultaneous move-and-redesign is the classic way migrations slip a year.
2. Minute 2 is the arithmetic. Doing volume ÷ bandwidth out loud and naming the offline threshold is the single strongest senior signal — it is the calculation weak candidates never perform.
3. Minute 3 addresses downtime by decoupling the bulk copy from the cutover. The insight is that catch-up shrinks the cutover delta to something copyable in minutes, which is what makes near-zero downtime possible.
4. Minute 4 makes validation a gate, not an afterthought. Naming the specific checks — row counts, per-partition checksums, aggregate diffs — and putting them in CI is what distinguishes "we tested it" from "we proved it."
5. Minute 5 is the reliability axis. Keeping the source authoritative and describing a concrete rollback (repoint consumers, reverse-sync during soak) is the answer that wins the offer; "restore a backup" is the answer that loses it.

Output.

Grading criterion	Weak score	Senior score
Names strategy in minute 1	rare	mandatory
Does transfer arithmetic	rare	senior signal
Decouples bulk copy from cutover	occasional	required
Names concrete validation checks	rare	mandatory
Names a real rollback + soak	rare	senior signal

Rule of thumb. The senior migration answer is a five-minute monologue covering all four axes — strategy, transfer sizing, gated cutover, validation and rollback — before the follow-ups arrive. Rehearse it once; deploy it every time.

Worked example — the "pick the approach" decision tree

Detailed explanation. Given a workload, the senior architect runs a short decision tree in their head. Codifying it makes the interview answer reproducible: hand me any workload and I can walk the tree out loud. Walk the tree with three canonical workloads — the NAS, the Hadoop lake, and the row warehouse.

• Q1. Does volume ÷ effective bandwidth fit the window? → yes = online transfer; no = go offline (Snowball) for the bulk, online for the delta.
• Q2. Is a schema/engine change required now? → no = lift-and-shift (checksum-verifiable); yes = re-architect (semantic reconciliation) — and prefer to defer it until after the byte-for-byte land.
• Q3. Can the workload take a cutover outage? → yes = big-bang cutover; no = incremental catch-up (CDC/dual-write) then a switch.
• Q4. What is the transfer tool for this data shape? → files/objects = DataSync; HDFS/lake = DistCp; relational DB = DMS.

Question. Walk the decision tree for the three workloads and record the approach each ends with.

Input.

Workload	Q1 (fits window online?)	Q2 (reshape now?)	Q3 (outage OK?)	Q4 (shape)
NAS 50 TB	yes	no	yes (low urgency)	files
Hadoop 2 PB	no	later	phased	HDFS
Warehouse 30 TB	yes	yes	no (one weekend)	relational

Code.

# Decision-tree helper (illustrative)
def pick_migration_approach(fits_window_online: bool,
                            reshape_now: bool,
                            outage_ok: bool,
                            shape: str) -> dict[str, str]:
    """Return the transfer + cutover + strategy for one workload."""
    transfer = {
        "files":      "DataSync",
        "hdfs":       "DistCp",
        "relational": "DMS",
    }[shape]

    if not fits_window_online:
        transfer = f"Snowball (bulk) + {transfer} (-diff / CDC delta)"

    strategy = "re-architect" if reshape_now else "lift-and-shift"
    cutover  = "big-bang" if outage_ok else "phased (CDC/dual-write catch-up)"

    return {"strategy": strategy, "transfer": transfer, "cutover": cutover}


print(pick_migration_approach(True,  False, True,  "files"))
# -> {'strategy': 'lift-and-shift', 'transfer': 'DataSync', 'cutover': 'big-bang'}

print(pick_migration_approach(False, False, False, "hdfs"))
# -> {'strategy': 'lift-and-shift', 'transfer': 'Snowball (bulk) + DistCp (-diff / CDC delta)', 'cutover': 'phased (CDC/dual-write catch-up)'}

print(pick_migration_approach(True,  True,  False, "relational"))
# -> {'strategy': 're-architect', 'transfer': 'DMS', 'cutover': 'phased (CDC/dual-write catch-up)'}

Step-by-step explanation.

1. The NAS short-circuits to the simplest path: online DataSync, lift-and-shift (files copy byte-for-byte), and a big-bang cutover is fine because nobody needs the NAS live to the second. This is the low-risk warm-up that builds team confidence.
2. The Hadoop lake fails Q1 (2 PB does not fit the window online), so the bulk goes on Snowball and DistCp snapshot-diff carries the delta. Reshape is deferred (later), so the migration is lift-and-shift even though a future refactor to Parquet-on-S3 is planned. Cutover is phased because the lake feeds many consumers you re-point gradually.
3. The warehouse passes Q1 (30 TB is easy online) but requires a reshape now (row-store → Redshift columnar) and forbids an outage. So it is a re-architect with DMS full-load + CDC and a phased, reconciled cutover inside the weekend window.
4. Notice Q2 and Q3 are orthogonal to Q1: transfer sizing (Q1) is a bandwidth question; strategy (Q2) is a schema question; cutover style (Q3) is a business-continuity question. Treating them as independent axes is what keeps the plan honest.
5. If a workload fails Q1 and demands zero downtime and needs a reshape — a 2 PB always-on transactional lake being replatformed — that is the genuinely hard migration, and the honest answer is "phase it: land raw, reshape incrementally, dual-write during a long soak." Refuse to promise a clean big-bang for that shape.

Output.

Workload	Strategy	Transfer	Cutover
NAS 50 TB	lift-and-shift	DataSync	big-bang
Hadoop 2 PB	lift-and-shift (refactor later)	Snowball + DistCp -diff	phased
Warehouse 30 TB	re-architect	DMS + CDC	phased, weekend

Rule of thumb. The four-question decision tree — fits-window, reshape-now, outage-OK, data-shape — is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any workload and get a strategy, a transfer tool, and a cutover style in under 60 seconds.

Senior interview question on migration approach

A senior interviewer often opens with: "You inherit an on-prem estate — a 2 PB Hadoop lake, a 30 TB row-oriented warehouse, and a 50 TB NAS — with a 10 Gbps Direct Connect link. Leadership wants everything on AWS within two quarters with no data loss and minimal downtime. Walk me through the sequencing, the transfer mechanism per workload, and the validation gate you'd enforce before any cutover."

Solution Using a strangler-pattern plan with per-workload transfer and a reconciliation gate

# 1. Sequencing — land fast, reshape later, cut over gated
Phase 0  Foundations: Direct Connect sized, landing S3 buckets, IAM,
         KMS, VPC endpoints, migration reconciliation harness in CI.
Phase 1  NAS 50 TB  -> DataSync online, scheduled incremental sync.
Phase 2  Warehouse 30 TB -> DMS full-load + CDC into Redshift (re-architect).
Phase 3  Lake 2 PB   -> Snowball bulk + DistCp snapshot-diff delta.
Phase 4  Per workload: validation gate -> cutover -> soak -> decommission.

# 2. Transfer arithmetic that drives the plan (effective ~4 Gbps = 0.5 GB/s)
python3 - <<'PY'
usable = 0.5  # GB/s effective on a shared 10 Gbps DX
for name, tb in [("NAS", 50), ("Warehouse", 30), ("Lake", 2000)]:
    seconds = tb * 1000 / usable
    days = seconds / 86400
    verdict = "ONLINE" if days < 14 else "OFFLINE (Snowball) + online delta"
    print(f"{name:10} {tb:>5} TB  ~{days:6.1f} days  -> {verdict}")
PY
# NAS          50 TB  ~   1.2 days  -> ONLINE
# Warehouse    30 TB  ~   0.7 days  -> ONLINE
# Lake       2000 TB  ~  46.3 days  -> OFFLINE (Snowball) + online delta

-- 3. The reconciliation gate every workload passes before cutover
--    (run against source and target; both must agree)
-- Row-count parity per table
SELECT 'orders' AS table_name, count(*) AS row_count FROM orders;

-- Aggregate parity (catches silent value corruption a count would miss)
SELECT date_trunc('day', created_at) AS d,
       count(*)                      AS rows,
       sum(total_cents)              AS revenue_cents
FROM   orders
GROUP  BY 1
ORDER  BY 1;

-- Per-partition file + checksum parity for the lake (shell, not SQL):
--   aws s3 ls --recursive s3://lake/orders/ | wc -l      # object count
--   hdfs dfs -count /warehouse/orders                    # dir/file/byte count
--   compare DistCp job counters: BYTESCOPIED, COPY vs SKIP

Step-by-step trace.

Step	Decision	Reasoning
Sequencing	NAS → warehouse → lake	ascending risk; easy win first builds confidence
NAS transfer	DataSync online + sync	50 TB fits the window; files are checksum-verifiable
Warehouse	DMS full-load + CDC	tight weekend window; re-architect row→columnar
Lake bulk	Snowball appliances	2 PB online ≈ 46 days blows the window
Lake delta	DistCp -diff snapshot	copies only what changed during appliance transit
Gate	row counts + aggregates + partition checksums	proves parity before any switchover
Rollback	source stays authoritative during soak	repoint consumers back if the gate or soak fails

After the plan runs, the NAS is byte-for-byte on S3 within days and kept fresh by scheduled DataSync; the warehouse lands in Redshift with CDC holding it within seconds of source until the weekend switch; the 2 PB lake arrives via Snowball with a DistCp snapshot-diff catching the transit delta. Each workload switches over only after its reconciliation gate is green, and the on-prem source stays writable through a soak period so rollback is a re-point, not a restore.

Output:

Workload	Landed via	Cutover delta at switch	Gate result
NAS 50 TB	DataSync + sync	last incremental (minutes)	file checksums match
Warehouse 30 TB	DMS + CDC	seconds of CDC lag	row + aggregate parity
Lake 2 PB	Snowball + DistCp -diff	snapshot delta since ship	partition count + checksum
Rollback readiness	source authoritative	n/a	re-point in < 15 min

Why this works — concept by concept:

• Strangler-pattern sequencing — landing lift-and-shift first and re-architecting per workload afterwards keeps each step independently verifiable and reversible. A simultaneous move-and-redesign couples two risks that should be de-risked separately.
• Transfer arithmetic drives the tool — computing volume ÷ effective bandwidth per workload is what decides online (DataSync/DMS) versus offline (Snowball). The 46-day figure for the lake is not a guess; it is the number that forces the appliance.
• Incremental catch-up shrinks the cutover — CDC for the warehouse and DistCp snapshot-diff for the lake keep the final switch delta tiny, which is the entire mechanism behind near-zero-downtime cutovers.
• Reconciliation as a hard gate — row counts catch missing rows, aggregate diffs catch silent value corruption a count cannot see, and per-partition checksums catch a dropped lake file. Green gate is a precondition for switchover, enforced in CI, not a hope.
• Cost — Direct Connect egress plus Snowball handling for the bulk, DMS replication instance for the warehouse CDC window, and DistCp cluster hours for the lake delta. The eliminated cost is a botched big-bang: the parallel-run overhead of a soak period is cheap insurance against an un-reconciled, un-rollbackable switch. Net O(delta) per cutover instead of O(estate) re-copy on every retry.

ETL
Topic — etl
ETL problems on migration and ingestion pipelines

Practice →

Design Topic — design Design problems on platform migration

Practice →

---

2. Lift-and-shift vs re-architect

lift and shift rehosts the same engines to land fast; re-architect reshapes for cloud-native cost and performance — most migrations do both, in that order

The mental model in one line: lift and shift (rehost) moves your workloads to the cloud with the least possible change — same engine, same file formats, same schema — trading a fast, low-risk landing against inherited technical debt and often-worse run-cost; re-architect (refactor) redesigns for cloud-native primitives — object storage, columnar formats, elastic/serverless compute — trading higher effort and risk against the real payoff of the migration; and the dominant real-world approach is the strangler pattern: lift-and-shift to get off the on-prem hardware quickly, then re-architect workload by workload once the data is safely in the cloud. Every senior migration is a portfolio decision across the classic 6 R's, not a single binary.

The four axes for the strategy choice.

• Speed to land. Lift-and-shift is fastest — you rehost the same Hadoop/warehouse onto cloud VMs or managed equivalents with minimal re-engineering. Re-architect is slowest per workload because you redesign storage layout, file format, partitioning, and sometimes the query engine.
• Run-cost after landing. Lift-and-shift frequently raises run-cost: a nightly Hadoop job that was "free" on owned hardware now bills for always-on cloud instances. Re-architect is where the savings live — separating storage (cheap S3) from compute (elastic EMR/Athena/serverless) and adopting columnar Parquet slashes both storage and scan cost.
• Risk & blast radius. Lift-and-shift is low-risk because behaviour is unchanged — same SQL, same outputs, byte-verifiable. Re-architect changes behaviour (new engine, new file format), so it needs semantic reconciliation and a longer soak.
• Tech debt. Lift-and-shift carries the on-prem debt (over-provisioned clusters, brittle cron ETL) into the cloud. Re-architect is the only axis that pays it down. This is why "lift-and-shift and stop" is a trap: you have paid the migration cost without collecting the migration benefit.

The 6 R's — the standard workload-disposition vocabulary.

• Rehost (lift-and-shift). Move as-is onto equivalent cloud infrastructure. Fastest; no code change.
• Replatform (lift-and-reshape). Small optimisations without redesign — e.g. self-managed Postgres → managed RDS, self-managed Hadoop → EMR. Minor changes, meaningful operational wins.
• Repurchase. Drop the workload for a SaaS/managed equivalent — e.g. on-prem BI → a cloud analytics service.
• Refactor / re-architect. Redesign cloud-native — HDFS/Hive → S3 + Parquet + Athena/Spark; row warehouse → columnar cloud warehouse.
• Retire. Turn it off — a surprising fraction of on-prem datasets and jobs are dead and should never be migrated.
• Retain. Leave it on-prem (regulatory, latency, or not-yet-worth-it) and integrate via hybrid connectivity.

Where each strategy wins.

• Lift-and-shift wins when the deadline is hardware-driven (a data-centre lease expiring), when the workload is stable and well-understood, and when you want a fast, low-risk first move that de-risks the rest.
• Re-architect wins when run-cost or performance is the reason for the migration, when the on-prem design does not map to cloud primitives (a monolithic HDFS + tightly-coupled compute), and when you can afford a longer per-workload timeline.
• The strangler hybrid wins almost always — rehost to escape the on-prem hardware on schedule, then refactor the highest-cost workloads first, measuring cloud spend as you go.

Common interview probes on strategy.

• "Lift-and-shift or re-architect?" — required answer: "usually both — strangler pattern; rehost to land, refactor per workload."
• "Why not lift-and-shift everything and stop?" — you pay the migration cost without collecting the cost/performance payoff, and you carry the tech debt.
• "What are the 6 R's?" — rehost, replatform, repurchase, refactor, retire, retain.
• "How do you validate a re-architected workload?" — semantic reconciliation (row counts + aggregates + sampled diffs), not byte checksums, because the bytes deliberately change.

Worked example — the 6 R's disposition grid for a workload inventory

Detailed explanation. Before any transfer, the senior architect inventories every workload and assigns each a disposition from the 6 R's. This grid is the migration's master plan; it decides what moves, how, and what never moves at all. Walk through building it for a representative estate.

• The inventory. Self-managed Hadoop ETL, a row-oriented warehouse, a self-hosted Postgres OLTP feeder, an on-prem Tableau server, a decade-old regulatory archive, and a pile of abandoned nightly reports.
• The dispositions. Each workload gets exactly one R plus a one-line justification and a rough effort estimate.

Question. Assign a disposition to each workload and justify it.

Input.

Workload	Cost/pain	Cloud-native fit
Hadoop ETL	high compute cost	poor (monolithic)
Row warehouse	slow, expensive scans	poor (row-store)
Postgres OLTP feeder	fine, just self-managed	good
On-prem Tableau	licensing + ops burden	replaceable by SaaS
Regulatory archive	must retain 7 yrs	low access
Abandoned reports	zero users	n/a

Code.

# Disposition assignment (illustrative — the grid as data)
inventory = [
    # (workload,            disposition,   why)
    ("hadoop_etl",          "refactor",    "HDFS/Hive -> S3+Parquet+Spark; kills always-on compute cost"),
    ("row_warehouse",       "refactor",    "row-store -> columnar cloud warehouse; scan cost + speed"),
    ("postgres_oltp_feed",  "replatform",  "self-managed -> managed RDS; same engine, less ops"),
    ("onprem_tableau",      "repurchase",  "swap for a managed cloud BI service"),
    ("regulatory_archive",  "retain",      "keep on-prem / cold object store; hybrid access, low churn"),
    ("abandoned_reports",   "retire",      "zero users -> never migrate; turn it off"),
]

for name, r, why in inventory:
    print(f"{name:22} {r:11} {why}")

Step-by-step explanation.

1. The two high-cost, poor-fit workloads (Hadoop ETL and the row warehouse) are the reason for the migration, so they get refactor — the strategy that actually collects the payoff. These are done later in the sequence (higher effort/risk) but land first in the business case.
2. The Postgres feeder works fine and maps cleanly to a managed service, so it gets replatform — same engine, managed operationally. Low effort, real ops win, no behaviour change to reconcile.
3. The Tableau server is a repurchase: rather than rehost a self-managed BI server, adopt a managed cloud analytics service. Migrating away from a workload is often cheaper than migrating it.
4. The regulatory archive is retain (or a cold cloud tier): it is rarely accessed, must be kept for compliance, and offers no payoff from re-architecting. Not everything should move; forcing it wastes effort.
5. The abandoned reports are retire — the single highest-ROI disposition. Every migration inventory finds dead jobs and datasets; deleting them shrinks scope, cost, and risk for free. Auditing usage before migrating is a senior habit.

Output.

Workload	Disposition	Effort	Migration payoff
Hadoop ETL	refactor	high	large (compute cost)
Row warehouse	refactor	high	large (scan cost/speed)
Postgres feeder	replatform	low	ops burden removed
Tableau	repurchase	medium	licensing/ops removed
Regulatory archive	retain	none	n/a
Abandoned reports	retire	none	scope removed

Rule of thumb. Inventory every workload and assign exactly one of the 6 R's before you move a byte. Audit usage first — retire the dead ones, retain the ones with no payoff — so you only migrate what earns its keep.

Worked example — rehost a Hadoop cluster to EMR (lift-and-shift)

Detailed explanation. The fastest way off on-prem Hadoop is to rehost the same Spark/Hive jobs onto EMR with data on S3 — a lift-and-shift that changes where the cluster runs without changing the jobs. It gets you off owned hardware quickly and sets up the later refactor. Walk through the rehost.

• Before. On-prem YARN cluster, HDFS storage, Spark + Hive jobs, always-on.
• After (rehost). EMR cluster reading/writing S3 via the s3a connector; the same Spark jobs, minimally repointed from hdfs:// to s3://.
• The point. Behaviour is unchanged and byte-verifiable; the win is escaping the hardware, not yet the cost model.

Question. Show the minimal changes to rehost a Spark job from on-prem HDFS to EMR + S3.

Input.

Component	On-prem	EMR rehost
Compute	self-managed YARN	EMR-managed YARN
Storage	HDFS	S3 (s3://)
Job code	Spark/Hive	same, repointed paths
Lifecycle	always-on	can stay on (lift-and-shift)

Code.

# On-prem job (before) — reads and writes HDFS
spark.read.parquet("hdfs:///warehouse/orders/") \
     .filter("status = 'shipped'") \
     .write.mode("overwrite") \
     .parquet("hdfs:///warehouse/shipped_orders/")

# EMR rehost (after) — identical logic, S3 paths only
spark.read.parquet("s3://prod-lake/warehouse/orders/") \
     .filter("status = 'shipped'") \
     .write.mode("overwrite") \
     .parquet("s3://prod-lake/warehouse/shipped_orders/")
# Nothing else changes: same transformations, same schema, same output.
# EMRFS handles s3:// natively; the job is a pure rehost.

# Submit the same job to an EMR cluster instead of on-prem YARN
aws emr add-steps --cluster-id j-XXXXXXXX \
  --steps Type=Spark,Name=shipped_orders,ActionOnFailure=CONTINUE,\
Args=[--deploy-mode,cluster,s3://prod-code/jobs/shipped_orders.py]

Step-by-step explanation.

1. The only change to the job is the storage scheme: hdfs:/// becomes s3://. The transformations, schema, and outputs are byte-for-byte identical, which is exactly what makes a rehost low-risk and checksum-verifiable.
2. EMR's EMRFS reads and writes S3 natively, so no application rewrite is needed — the same PySpark runs unchanged. This is the essence of lift-and-shift: change the substrate, not the code.
3. Submitting the job as an EMR step replaces the on-prem YARN scheduler with EMR's managed one. Operationally you are off your own hardware immediately; the jobs behave as before.
4. Crucially, this rehost does not yet capture the cloud cost model — if the EMR cluster runs always-on like the on-prem one, spend can be higher. The rehost is a landing, not the destination.
5. The rehost sets up the refactor: with data already on S3, the next step (next example) is to decouple compute, adopt columnar layout choices, and move to transient/serverless clusters — the changes that actually lower cost.

Output.

Aspect	On-prem	EMR rehost
Code changes	—	paths only (hdfs://→s3://)
Behaviour	baseline	identical (byte-verifiable)
Hardware	owned, fixed	managed, escaped
Run-cost	"free" (sunk)	possibly higher if always-on
Sets up	—	later refactor to transient/serverless

Rule of thumb. Rehost by changing the substrate, not the code — repoint hdfs:// to s3://, run the same jobs on EMR, and verify by checksum. Treat it as a fast landing that de-risks the estate, then refactor for cost; never mistake the rehost for the finished migration.

Worked example — re-architect HDFS + Hive to S3 + Parquet + Athena (refactor)

Detailed explanation. The refactor is where the migration pays off: decouple storage from compute, convert to columnar Parquet with sensible partitioning, and query with serverless Athena (or transient Spark) so you pay per scan instead of for an always-on cluster. Walk through the re-architecture and the cost logic.

• Before. HDFS files (often row-oriented or unpartitioned), Hive on an always-on cluster.
• After (refactor). Parquet on S3, partitioned by date, queried by Athena — storage and compute fully decoupled, pay-per-query.
• The reconciliation twist. Because the bytes deliberately change (format + layout), validation is semantic (row counts + aggregates), not checksum.

Question. Convert an unpartitioned Hive table to partitioned Parquet-on-S3 with Athena, and reconcile it semantically.

Input.

Change	Before	After
Format	text/row	Parquet (columnar)
Layout	unpartitioned	partitioned by dt
Compute	always-on Hive	serverless Athena
Billing	cluster-hours	per TB scanned
Validation	n/a	row counts + aggregates

Code.

-- 1. Land raw (rehost) already done: raw data sits at s3://prod-lake/raw/orders/
-- 2. Refactor: write partitioned columnar Parquet via a CTAS
CREATE TABLE analytics.orders_parquet
WITH (
    format            = 'PARQUET',
    parquet_compression = 'SNAPPY',
    partitioned_by    = ARRAY['dt'],
    external_location = 's3://prod-lake/curated/orders/'
) AS
SELECT
    id, customer_id, total_cents, status,
    date_format(created_at, '%Y-%m-%d') AS dt   -- partition key
FROM raw.orders;

-- 3. Semantic reconciliation — the bytes changed, so compare MEANING
-- 3a. Row-count parity
SELECT (SELECT count(*) FROM raw.orders)               AS src_rows,
       (SELECT count(*) FROM analytics.orders_parquet) AS tgt_rows;

-- 3b. Aggregate parity per day (catches value corruption a count misses)
SELECT dt, count(*) AS rows, sum(total_cents) AS revenue
FROM   analytics.orders_parquet
GROUP  BY dt
-- compare row-for-row against the same query on the source
ORDER  BY dt;

# 4. The payoff — Athena scans only the partitions and columns you touch
#    Query for one day reads ~1/365 of the data, columnar prunes columns:
#    "WHERE dt = '2026-08-17'" scans MBs, not the whole table -> cents, not
#    cluster-hours. Storage (S3) and compute (Athena) now bill independently.

Step-by-step explanation.

1. The refactor runs after the raw data has already been landed on S3 by the rehost, which keeps the two risks separate: the byte-for-byte move is verified first, then the reshape happens against data already safely in the cloud.
2. The CTAS rewrites the table as Snappy-compressed Parquet partitioned by day. Columnar format lets Athena read only the columns a query selects; date partitioning lets it skip every partition the WHERE excludes — the two changes that collapse scan cost.
3. Because the format and layout deliberately differ from the source, checksums are meaningless. Validation becomes semantic: identical row counts prove nothing was dropped, and identical per-day aggregates (COUNT and SUM) prove no values were corrupted in the rewrite.
4. Athena bills per terabyte scanned, and partition pruning plus columnar projection mean a single-day query reads megabytes instead of the whole table — the always-on Hive cluster's fixed cost becomes a per-query cost measured in cents.
5. Storage and compute are now decoupled: S3 holds the data cheaply and continuously, while compute is summoned only when a query runs. That decoupling — not the mere move to the cloud — is the source of the migration's cost and performance payoff.

Output.

Metric	Rehosted (Hive on EMR)	Refactored (Parquet + Athena)
File format	as-was	columnar Parquet
One-day query scan	whole table	one partition, projected columns
Compute billing	cluster-hours	per TB scanned
Storage/compute coupling	coupled	decoupled
Validation	checksum	row counts + aggregates

Rule of thumb. Re-architect after you land: convert to partitioned columnar Parquet, decouple storage from compute, and validate semantically (row counts + aggregates), because a refactor changes the bytes on purpose. The payoff is per-query cost and partition pruning, not the move itself.

Senior interview question on migration strategy

A senior interviewer might ask: "Leadership wants the on-prem Hadoop estate off owned hardware in six months, but also wants the cloud bill lower than today. Those two goals pull in opposite directions. Design a strategy that hits the hardware deadline without shipping a more expensive cloud platform, and tell me how you'd measure whether the re-architecture is actually paying off."

Solution Using a strangler-pattern rehost-then-refactor with cost instrumentation

# Strategy: rehost to hit the deadline, refactor to hit the cost target
Month 0-3  REHOST (lift-and-shift):
  - Land all HDFS data on S3 (DataSync/DistCp), repoint jobs hdfs:// -> s3://
  - Run jobs on EMR; hardware deadline met; behaviour byte-verifiable.
  - Tag every cloud resource with workload + cost-centre from day one.

Month 3-9  REFACTOR (per workload, highest-cost first):
  - Convert hottest tables to partitioned Parquet; move to Athena/transient EMR.
  - Retire dead jobs found in the inventory (free scope reduction).
  - Measure $/workload before vs after each refactor.

# Cost instrumentation — is the refactor paying off?
# Pull tagged spend per workload from Cost Explorer / billing export.
def refactor_payoff(spend_before: dict[str, float],
                    spend_after: dict[str, float]) -> None:
    print(f"{'workload':22} {'before$':>10} {'after$':>10} {'saved%':>8}")
    for wl in spend_before:
        b, a = spend_before[wl], spend_after.get(wl, b := spend_before[wl])
        saved = (b - a) / b * 100 if b else 0.0
        flag = "OK" if saved > 0 else "REGRESSION"
        print(f"{wl:22} {b:>10.0f} {a:>10.0f} {saved:>7.1f}%  {flag}")

refactor_payoff(
    spend_before={"hadoop_etl": 42000, "row_warehouse": 55000},
    spend_after ={"hadoop_etl": 18000, "row_warehouse": 21000},
)
# hadoop_etl              42000      18000    57.1%  OK
# row_warehouse           55000      21000    61.8%  OK

-- Correctness gate for each refactor (semantic, not checksum)
SELECT day, src_rows, tgt_rows, src_rev, tgt_rev,
       (src_rows = tgt_rows AND src_rev = tgt_rev) AS parity_ok
FROM   migration.reconciliation_orders
WHERE  parity_ok = FALSE;      -- must return zero rows to pass

Step-by-step trace.

Phase	Action	Goal served
Rehost 0-3mo	land on S3, run on EMR	hits hardware deadline, low risk
Tagging	cost-centre tag every resource	makes payoff measurable later
Refactor 3-9mo	Parquet + serverless, hottest first	hits the cost target
Retire	delete dead jobs	free scope + cost reduction
Cost gate	before/after $ per workload	proves refactor pays off
Correctness gate	row + aggregate parity	proves refactor is safe

After execution, the hardware deadline is met at month 3 by the rehost (the risky redesign is not on the critical path for the deadline), and the cost target is met incrementally as each high-spend workload is refactored to columnar-Parquet-plus-serverless. Per-workload tagging turns "is the cloud cheaper?" from an argument into a report: Hadoop ETL drops 57%, the warehouse 62%. Every refactor ships only after its semantic reconciliation gate returns zero mismatched rows.

Output:

Workload	Before $/mo	After $/mo	Saved	Correctness
Hadoop ETL	42,000	18,000	57%	parity_ok
Row warehouse	55,000	21,000	62%	parity_ok
Dead jobs	(some)	0	100%	retired
Deadline	—	met at mo 3	—	rehost verified

Why this works — concept by concept:

• Decoupling the two goals — the hardware deadline is met by the low-risk rehost, so the deadline never depends on the risky redesign. The cost goal is met by the refactor, which runs off the critical path. Trying to do both at once is what makes migrations miss both.
• Cost-centre tagging from day one — you cannot prove a refactor pays off without per-workload spend, and you cannot get per-workload spend retroactively. Tagging on landing is the cheap decision that makes the whole business case measurable.
• Highest-cost-first refactoring — sequencing the refactor by spend means the biggest savings land earliest, funding the rest of the migration and giving leadership an early, defensible win.
• Semantic reconciliation gate — a refactor changes the bytes on purpose, so correctness is proven by matching row counts and aggregates, not checksums. Zero mismatched rows is the switch condition.
• Cost — a parallel-run window (on-prem + rehosted cloud) during the rehost, plus engineering time for each refactor. The offset is the eliminated always-on compute and the retired dead workloads; measured, the platform is 55-62% cheaper per workload. Net: the deadline costs a short parallel run, the savings compound per refactor.

Design
Topic — design
Design problems on cloud re-architecture

Practice →

Data processing Topic — data-processing Data-processing problems on columnar formats

Practice →

---

3. AWS DataSync — managed file & object transfer

AWS DataSync moves files and objects between on-prem and cloud with a managed agent, built-in integrity verification, and incremental sync — the default for NAS and file-server migrations

The mental model in one line: AWS DataSync is a managed transfer service where a lightweight agent (a VM on-prem) reads from an NFS, SMB, HDFS, or object source and writes to an AWS target (S3, EFS, FSx) over an encrypted channel, automatically parallelising the copy, verifying data integrity with checksums, transferring only files that changed on subsequent runs, and throttling bandwidth so it never starves production — which makes it the right tool for migrating file servers and NAS estates and for keeping them in sync until cutover. Every file-shaped data transfer in a migration that fits online should start with DataSync rather than hand-rolled rsync-over-SSH scripts.

The four axes for DataSync.

• Data shape & source. Files and objects — NFS, SMB, self-managed HDFS, and object stores on the source side; S3, EFS, FSx for Windows/Lustre on the AWS side. It is not a database replicator (that is DMS) and not a MapReduce lake copier (that is DistCp); it is the file/object mover.
• Integrity & security. Every transfer is encrypted in transit (TLS) and DataSync verifies data integrity — by default it checksums transferred files and can verify the entire destination against the source. This is the feature that lets you trust the copy without writing your own validation for the file case.
• Incremental transfer. After the first full copy, DataSync compares source and destination metadata and transfers only files that were added or changed — so scheduled runs carry the delta, not the whole tree. This is the mechanism that keeps the destination fresh until cutover.
• Bandwidth & scheduling. A configurable bandwidth cap keeps DataSync from saturating a shared WAN, and tasks can be scheduled (e.g. hourly, or nightly during off-peak) so the incremental sync runs on a cadence without manual kicks.

The DataSync building blocks.

• Agent. A VM (deployed on VMware/Hyper-V/KVM/EC2) that reads the on-prem source and connects to AWS. It is the data-plane worker; you activate it against your account.
• Locations. Named endpoints — a source location (e.g. the NFS export) and a destination location (e.g. an S3 bucket + prefix). Locations are reusable across tasks.
• Task. The transfer definition binding a source location to a destination location, plus options: verification mode, overwrite behaviour, include/exclude filters, bandwidth limit, and schedule.
• Task execution. One run of a task. The first is a full copy; subsequent runs are incremental. Each execution reports bytes transferred, files verified, and any skips.

Transfer options that matter.

• Verification. POINT_IN_TIME_CONSISTENT (verify the whole dataset after transfer — safest), ONLY_FILES_TRANSFERRED (verify just what moved — faster), or NONE. Migrations should verify.
• Filters. includes and excludes glob patterns scope the transfer — e.g. exclude */tmp/* and *.lock, include only /exports/prod/*.
• Overwrite & preserve. Options to preserve POSIX metadata/ownership/timestamps and to control whether destination-only files are kept or deleted (a "mirror" vs "add-only" choice).
• Task reporting. Per-execution reports (to S3/CloudWatch) list exactly which files transferred, skipped, or failed verification — your evidence for the file-case reconciliation gate.

Common interview probes on DataSync.

• "Why DataSync over rsync scripts?" — managed agent, parallelism, built-in checksum verification, incremental transfer, bandwidth throttling, and reporting you would otherwise hand-build.
• "How do you keep the destination fresh until cutover?" — scheduled incremental task executions that carry only changed files.
• "How do you avoid saturating the WAN?" — the task bandwidth limit plus off-peak scheduling.
• "How do you prove the file copy is correct?" — verification mode plus the per-execution task report (files transferred, verified, failed).

Worked example — a DataSync agent + task for NFS → S3

Detailed explanation. The canonical DataSync setup: deploy and activate an agent near the NAS, define an NFS source location and an S3 destination location, and create a task that copies with verification. Build it end-to-end with the CLI.

• Agent. A VM on the on-prem network with a line of sight to the NFS server, activated against your account.
• Source location. The NFS export /exports/prod on the NAS.
• Destination location. s3://prod-lake/nas/ with an IAM role DataSync can assume.
• Task. Bind them with POINT_IN_TIME_CONSISTENT verification.

Question. Provide the CLI to activate the agent, create both locations, and create + start the task.

Input.

Object	Value
Agent	on-prem VM, activated
Source location	NFS /exports/prod
Destination location	s3://prod-lake/nas/
Verification	POINT_IN_TIME_CONSISTENT

Code.

# 1. Activate the agent (the VM is already deployed on-prem/EC2).
#    Browsing http://<agent-ip>/ returns an activation key.
aws datasync create-agent \
    --activation-key "AAAAA-1111-BBBBB-2222-CCCCC" \
    --agent-name "onprem-nas-agent"
# -> AgentArn: arn:aws:datasync:...:agent/agent-0abc...

# 2. Source location — the NFS export, reached via the agent
aws datasync create-location-nfs \
    --server-hostname "nas.corp.internal" \
    --subdirectory "/exports/prod" \
    --on-prem-config AgentArns="arn:aws:datasync:...:agent/agent-0abc..."
# -> LocationArn: arn:aws:datasync:...:location/loc-nfs-src...

# 3. Destination location — the S3 bucket + prefix + access role
aws datasync create-location-s3 \
    --s3-bucket-arn "arn:aws:s3:::prod-lake" \
    --subdirectory "/nas/" \
    --s3-config BucketAccessRoleArn="arn:aws:iam::123456789012:role/DataSyncS3Role"
# -> LocationArn: arn:aws:datasync:...:location/loc-s3-dst...

# 4. Task — bind source -> destination with verification
aws datasync create-task \
    --source-location-arn      "arn:aws:datasync:...:location/loc-nfs-src..." \
    --destination-location-arn "arn:aws:datasync:...:location/loc-s3-dst..." \
    --name "nas-prod-to-s3" \
    --options VerifyMode=POINT_IN_TIME_CONSISTENT,OverwriteMode=ALWAYS,PreserveDeletedFiles=PRESERVE
# -> TaskArn: arn:aws:datasync:...:task/task-0def...

# 5. Start the first (full) execution
aws datasync start-task-execution \
    --task-arn "arn:aws:datasync:...:task/task-0def..."
# -> TaskExecutionArn: .../execution/exec-0ghi...

Step-by-step explanation.

1. The agent is the data-plane worker: activated once, it reads the NFS export locally (fast LAN speed) and streams to AWS over the encrypted channel. Placing it near the NAS is what keeps the read side fast.
2. The NFS source location names the server, the subdirectory to copy, and which agent reaches it. The S3 destination location names the bucket, prefix, and an IAM role DataSync assumes to write — least-privilege scoped to that prefix.
3. Locations are reusable objects: the same S3 destination can back multiple tasks, and the same agent can serve several source locations. This is why they are defined separately from the task.
4. The task binds the two locations and sets behaviour. VerifyMode=POINT_IN_TIME_CONSISTENT verifies the whole dataset after transfer; PreserveDeletedFiles=PRESERVE means a file deleted on-prem is kept on S3 (add-only, safer for a migration than a destructive mirror).
5. The first execution is a full copy; DataSync parallelises it across many files automatically and reports bytes transferred and files verified when it completes — the start of your file-case evidence trail.

Output.

Step	Result
create-agent	agent activated, ARN returned
create-location-nfs	source location for /exports/prod
create-location-s3	destination s3://prod-lake/nas/
create-task	task binds src→dst with verification
start-task-execution	full copy runs, parallelised, verified

Rule of thumb. Deploy the agent close to the source for LAN-speed reads, define reusable source/destination locations, and always set a verify mode on a migration task. Prefer PreserveDeletedFiles=PRESERVE during migration so the destination is add-only until you deliberately switch to a mirror.

Worked example — incremental sync with scheduling, filters, and a bandwidth cap

Detailed explanation. After the first full copy, you keep the destination fresh with scheduled incremental executions that carry only changed files, while a bandwidth cap protects the WAN and filters exclude junk. This is what lets DataSync hold the destination within a cadence of the source until cutover. Configure it.

• Schedule. Run the task hourly (or nightly off-peak) so the delta stays small.
• Filters. Exclude */tmp/*, *.lock, *.part; include only production subtrees.
• Bandwidth. Cap at, say, 256 MB/s so migration never starves production traffic on the shared link.

Question. Reconfigure the task for scheduled incremental sync with excludes and a bandwidth limit.

Input.

Setting	Value
Schedule	hourly (rate(1 hour))
Excludes	*/tmp/*, *.lock, *.part
Bandwidth	256 MB/s (268435456 B/s)
Transfer mode	changed files only

Code.

# 1. Update the task: bandwidth cap + exclude filters + verify only what moved
aws datasync update-task \
    --task-arn "arn:aws:datasync:...:task/task-0def..." \
    --options VerifyMode=ONLY_FILES_TRANSFERRED,\
BytesPerSecond=268435456,\
TransferMode=CHANGED \
    --excludes FilterType=SIMPLE_PATTERN,Value="*/tmp/*|*.lock|*.part"

# 2. Attach an hourly schedule so incrementals run without manual kicks
aws datasync update-task \
    --task-arn "arn:aws:datasync:...:task/task-0def..." \
    --schedule ScheduleExpression="rate(1 hour)"

# 3. Inspect what an incremental execution actually moved
aws datasync describe-task-execution \
    --task-execution-arn ".../execution/exec-0xyz..." \
    --query '{files:FilesTransferred, bytes:BytesTransferred, status:Status, verified:FilesVerified}'
# {
#   "files": 1843,          <- only changed files, not the whole tree
#   "bytes": 5473921024,    <- ~5.1 GB delta this hour
#   "status": "SUCCESS",
#   "verified": 1843
# }

Step-by-step explanation.

1. TransferMode=CHANGED is the incremental switch: DataSync compares source and destination metadata and copies only files that were added or modified since the last run, so an hourly execution carries an hour of change, not the whole 50 TB.
2. BytesPerSecond=268435456 caps the transfer at 256 MB/s. On a shared WAN this is the guardrail that keeps the migration from starving production — you tune it against your usable-bandwidth budget.
3. The exclude filter drops transient junk (tmp, lock, part) so you neither waste bandwidth on it nor pollute the destination. Filters are evaluated per file, so the pattern list is the whole scoping mechanism.
4. Switching verification to ONLY_FILES_TRANSFERRED for the incremental runs verifies just the delta (fast) rather than re-verifying the entire dataset every hour — you reserve the full POINT_IN_TIME_CONSISTENT verify for the final pre-cutover run.
5. The hourly schedule turns the task into a self-driving sync: it keeps the destination within an hour of the source, so the eventual cutover copies only the last small delta. The per-execution report (files/bytes/verified) is your proof the sync is healthy.

Output.

Execution	Files moved	Bytes	Verify scope
1 (full)	all	50 TB	whole dataset
2 (hourly)	1,843	~5.1 GB	transferred only
3 (hourly)	902	~2.4 GB	transferred only
final (pre-cutover)	small delta	MBs	whole dataset

Rule of thumb. Run the first execution as a full, verified copy, then schedule TransferMode=CHANGED incrementals with a bandwidth cap and exclude filters to hold the destination fresh cheaply. Reserve the full point-in-time verification for the final pre-cutover run when correctness matters most.

Worked example — choosing a verification mode and reading the task report

Detailed explanation. DataSync's verification modes trade thoroughness against time, and the per-execution report is your reconciliation evidence for files. Getting the mode right per phase and reading the report correctly is what makes the file-case gate trustworthy. Walk through the trade-offs.

• Modes. POINT_IN_TIME_CONSISTENT (verify the whole destination against the source — slowest, safest), ONLY_FILES_TRANSFERRED (verify just the delta — fast), NONE (trust the transfer — only for throwaway).
• The report. A task report (to S3/CloudWatch) enumerates transferred, skipped, verified, and errored files.
• The gate. For the file case, "zero verification failures and expected file/byte counts" is the switch condition.

Question. Decide the verification mode per migration phase and interpret a task report to pass or fail the gate.

Input.

Phase	Recommended mode	Why
Initial full copy	POINT_IN_TIME_CONSISTENT	prove the base is byte-correct
Hourly incrementals	ONLY_FILES_TRANSFERRED	fast; only delta matters
Final pre-cutover	POINT_IN_TIME_CONSISTENT	last chance to catch drift

Code.

# 1. Enable a task report to S3 so every execution leaves an audit trail
aws datasync update-task \
    --task-arn "arn:aws:datasync:...:task/task-0def..." \
    --task-report-config '{
      "Destination": {"S3": {
        "S3BucketArn": "arn:aws:s3:::migration-reports",
        "BucketAccessRoleArn": "arn:aws:iam::123456789012:role/DataSyncReportRole",
        "Subdirectory": "nas-prod/"}},
      "OutputType": "STANDARD",
      "ReportLevel": "SUCCESSES_AND_ERRORS"
    }'

# 2. Gate check — parse an execution result; pass only if clean and complete
def datasync_gate(exec_result: dict, expected_files: int) -> bool:
    ok = (
        exec_result["Status"] == "SUCCESS"
        and exec_result["FilesVerified"] == exec_result["FilesTransferred"]
        and exec_result.get("FilesFailed", 0) == 0
        and exec_result["FilesTransferred"] >= expected_files  # nothing dropped
    )
    print("GATE", "PASS" if ok else "FAIL",
          f"(transferred={exec_result['FilesTransferred']}, "
          f"verified={exec_result['FilesVerified']}, "
          f"failed={exec_result.get('FilesFailed', 0)})")
    return ok

datasync_gate(
    {"Status": "SUCCESS", "FilesTransferred": 2_100_450,
     "FilesVerified": 2_100_450, "FilesFailed": 0},
    expected_files=2_100_450,
)
# GATE PASS (transferred=2100450, verified=2100450, failed=0)

Step-by-step explanation.

1. The full copy and the final pre-cutover run use POINT_IN_TIME_CONSISTENT because those are the moments correctness must be certain — the base must be provably byte-correct, and the final run is the last chance to catch any drift before the switch.
2. The hourly incrementals use ONLY_FILES_TRANSFERRED: verifying just the delta keeps each run fast, and re-verifying 50 TB every hour would waste the WAN for no gain since the untouched files were already verified.
3. Enabling a task report to S3 with SUCCESSES_AND_ERRORS gives you a durable per-file audit trail — exactly which files moved, were verified, or errored — which is the file-case equivalent of the row-count/checksum evidence you keep for databases.
4. The gate function encodes the switch condition as code: success status, verified count equal to transferred count, zero failures, and at least the expected file count (nothing silently dropped). Anything less fails the gate and blocks cutover.
5. Treating the gate as an automated check rather than a human eyeballing a console is the senior move — the file migration is only "done" when the report proves it, in CI, not when it "looks finished."

Output.

Signal	Value	Meaning
Status	SUCCESS	execution completed
FilesTransferred	2,100,450	delta/base moved
FilesVerified	2,100,450	equals transferred → clean
FilesFailed	0	no verification errors
Gate	PASS	safe to cut over the file estate

Rule of thumb. Verify the full dataset on the first and final runs and only the delta on the hourly ones, always emit a task report to S3, and encode the pass condition (verified == transferred, zero failures, expected count) as an automated gate. The file migration is done when the report proves it, not when it looks finished.

Senior interview question on AWS DataSync

A senior interviewer might ask: "You must migrate a 50 TB production NAS to S3 over a shared 10 Gbps link without impacting the application that reads it, keep S3 in sync for three weeks while consumers are re-pointed one by one, and prove at cutover that not a single file was lost. Design the DataSync setup — agent placement, task options, scheduling, bandwidth, verification — and the gate you'd enforce before switching each consumer."

Solution Using a throttled, scheduled, verified DataSync task with a report-driven gate

# 1. Agent near the NAS; source + destination locations (as before).
#    Then create the migration task with production-safe options.
aws datasync create-task \
  --source-location-arn      "$NFS_SRC" \
  --destination-location-arn "$S3_DST" \
  --name "nas-50tb-to-s3" \
  --options VerifyMode=POINT_IN_TIME_CONSISTENT,\
OverwriteMode=ALWAYS,\
PreserveDeletedFiles=PRESERVE,\
TransferMode=CHANGED,\
BytesPerSecond=536870912 \
  --excludes FilterType=SIMPLE_PATTERN,Value="*/tmp/*|*.snapshot/*|*.lock"

# 2. First run = full verified copy (off-peak, capped at 512 MB/s).
aws datasync start-task-execution --task-arn "$TASK"

# 3. Schedule hourly incrementals for the 3-week soak; switch delta runs
#    to ONLY_FILES_TRANSFERRED so they stay fast and cheap.
aws datasync update-task --task-arn "$TASK" \
  --schedule ScheduleExpression="rate(1 hour)" \
  --options VerifyMode=ONLY_FILES_TRANSFERRED,TransferMode=CHANGED,BytesPerSecond=536870912

# 4. Per-consumer cutover gate: final full-verify run must be clean.
def cutover_file_gate(final_exec: dict, source_file_count: int) -> bool:
    clean = (final_exec["Status"] == "SUCCESS"
             and final_exec["FilesFailed"] == 0
             and final_exec["FilesVerified"] == final_exec["FilesTransferred"])
    complete = final_exec["DestinationFileCount"] == source_file_count
    ok = clean and complete
    print("CUTOVER", "PASS" if ok else "HOLD",
          f"dest={final_exec['DestinationFileCount']} src={source_file_count} failed={final_exec['FilesFailed']}")
    return ok

cutover_file_gate(
    {"Status": "SUCCESS", "FilesFailed": 0,
     "FilesTransferred": 4200, "FilesVerified": 4200,
     "DestinationFileCount": 2_100_450},
    source_file_count=2_100_450,
)
# CUTOVER PASS dest=2100450 src=2100450 failed=0

Step-by-step trace.

Layer	Config	Effect
Agent	near the NAS	LAN-speed reads; no app impact
Bandwidth	512 MB/s cap	never starves the shared 10 Gbps link
First run	full + POINT_IN_TIME verify	provably correct 50 TB base
Soak	hourly CHANGED incrementals	S3 stays within an hour of source
Final run	full re-verify before each cutover	catches any late drift
Gate	dest count == src count, 0 failures	switch condition, per consumer

After setup, the NAS reads happen at LAN speed via the local agent so the production application never notices; the WAN is protected by the 512 MB/s cap; and S3 tracks the NAS within an hour for the whole three-week soak. Each consumer is re-pointed only after a final full-verify run confirms the destination file count equals the source and zero files failed verification. The task report in S3 is the durable evidence that nothing was lost.

Output:

Metric	Value
Production impact	none (agent-local reads)
WAN peak from migration	≤ 512 MB/s (capped)
Sync freshness during soak	≤ 1 hour
Final destination file count	equals source (2,100,450)
Verification failures	0
Gate per consumer	PASS before switch

Why this works — concept by concept:

• Agent-local reads — placing the agent on the NAS's LAN means the heavy read traffic stays local; only the compressed, deduplicated stream crosses the WAN, so the production application serving the same NAS is unaffected.
• Bandwidth cap on a shared link — BytesPerSecond is the guardrail that lets a migration coexist with production on one 10 Gbps pipe; without it the full copy would contend with real traffic.
• Full-then-incremental verification — the base is proven byte-correct once, hourly runs verify only their small delta cheaply, and a final full re-verify catches any drift right before the switch. Verification effort is spent where it matters.
• Report-driven per-consumer gate — the switch condition is machine-checked from the task report (destination count equals source, zero failures), so each consumer moves only on proof, not on a hunch, and the S3 report is the audit trail.
• Cost — the DataSync per-GB transfer charge for 50 TB plus three weeks of small hourly deltas, and the capped-bandwidth share of the WAN. The eliminated cost is a hand-rolled rsync fleet with no verification, no reporting, and no throttling — plus the incident cost of a silently-dropped file discovered after decommissioning the NAS. Net O(delta) per soak hour after the one-time O(50 TB) base.

ETL
Topic — etl
ETL problems on file ingestion and sync

Practice →

Data transformation Topic — data-transformation Data-transformation problems on landing-zone loads

Practice →

---

4. DistCp — distributed HDFS / data-lake copy at scale

DistCp parallelises a copy across a MapReduce/Spark cluster to move HDFS and data-lake data to object storage at petabyte scale — with snapshot-diff incrementals

The mental model in one line: DistCp (distributed copy) is a Hadoop tool that expands a copy into a MapReduce job — it builds the list of files to move, splits that list across many mapper tasks, and copies the chunks in parallel from hdfs:// to another HDFS or to an object store via the s3a:// connector, so throughput scales with cluster size rather than a single stream — and with snapshot-based -diff it copies only the files that changed between two HDFS snapshots, which is how you keep a petabyte-scale lake in sync until cutover. When the data lives in HDFS or a Hadoop-compatible file system, DistCp — not DataSync — is the data transfer engine.

The four axes for DistCp.

• Data shape & source. HDFS and Hadoop-compatible file systems (hdfs://, s3a://, wasb://, gs://). DistCp is the lake/HDFS mover; it runs on a Hadoop/Spark cluster and uses that cluster's parallelism as its transfer engine.
• Parallelism & throughput. The copy is a MapReduce job; the file list is partitioned across -m mapper tasks that copy concurrently. Throughput is a function of mapper count, per-mapper bandwidth, and object-store write throughput — you tune it, unlike a single-stream copy.
• Incremental via snapshots. With HDFS snapshots plus -diff snap1 snap2, DistCp copies only the files that changed between two point-in-time snapshots, which is the mechanism for keeping the target current after the bulk copy without re-scanning petabytes.
• Consistency & correctness. -update copies only files that differ (by size/checksum) and skips identical ones; DistCp compares source and target and reports copied/skipped/failed counts in the job counters — your reconciliation evidence for the lake case. Object-store semantics (once eventually-consistent, now strongly consistent on S3) matter for the write side.

The DistCp mechanics.

• List then copy. DistCp first builds the file list (the "copy-listing"), then the map phase copies files. There is no reduce phase for a plain copy — mappers do the work.
• Strategies. The default uniform-size strategy splits files so each mapper gets a roughly equal number of files; -strategy dynamic hands chunks to mappers as they finish, which balances load far better when file sizes are skewed (a few huge files among many small ones).
• Key flags. -m <n> (mapper count / parallelism), -bandwidth <MB> (per-mapper cap), -update (copy only differing files), -diff <s1> <s2> (snapshot incremental), -p (preserve attributes), -delete (mirror: remove target files absent from source).
• The s3a connector. The Hadoop AWS module writes to S3 as s3a://bucket/path, configured with credentials/role, multipart upload thresholds, and fast-upload buffering — the write side of an HDFS → S3 migration.

Common interview probes on DistCp.

• "Why DistCp instead of hdfs dfs -cp?" — it parallelises across the cluster (MapReduce) instead of a single client stream, so it scales to petabytes.
• "How do you do incremental copies?" — HDFS snapshots + -diff snap_old snap_new to move only the changed files.
• "How do you handle skewed file sizes / stragglers?" — -strategy dynamic so fast mappers pick up more work instead of waiting on a few huge files.
• "How do you verify the copy?" — -update (checksum/size compare) plus DistCp job counters (BYTESCOPIED, COPY, SKIP) and a follow-up count/checksum reconciliation.

Worked example — a basic DistCp copy from HDFS to S3

Detailed explanation. The canonical bulk copy: move an HDFS directory tree to S3 via s3a, parallelised across mappers. Configure the s3a write side and run the job. Walk through it.

• Source. hdfs:///warehouse/orders on the on-prem cluster.
• Target. s3a://prod-lake/warehouse/orders (the s3a connector).
• Parallelism. 100 mappers, per-mapper bandwidth cap to protect the WAN.

Question. Provide the s3a configuration and the hadoop distcp command for the bulk copy.

Input.

Parameter	Value
Source	hdfs:///warehouse/orders
Target	s3a://prod-lake/warehouse/orders
Mappers	100
Per-mapper bandwidth	50 MB/s

Code.

<!-- core-site.xml (or -D flags) — the s3a write side -->
<property><name>fs.s3a.aws.credentials.provider</name>
  <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value></property>
<property><name>fs.s3a.endpoint</name><value>s3.us-east-1.amazonaws.com</value></property>
<property><name>fs.s3a.fast.upload</name><value>true</value></property>
<property><name>fs.s3a.fast.upload.buffer</name><value>disk</value></property>
<property><name>fs.s3a.multipart.size</name><value>128M</value></property>
<property><name>fs.s3a.connection.maximum</name><value>200</value></property>

# Bulk parallel copy HDFS -> S3
hadoop distcp \
  -Dfs.s3a.fast.upload=true \
  -m 100 \                       # 100 parallel mapper tasks
  -bandwidth 50 \                # cap each mapper at 50 MB/s (WAN protection)
  -strategy dynamic \            # balance load across skewed file sizes
  -update \                      # copy only files that differ; skip identical
  -p \                           # preserve block size, replication, permissions
  hdfs:///warehouse/orders \
  s3a://prod-lake/warehouse/orders

# On completion, read the job counters for reconciliation evidence:
#   DistCp Counters: BYTESCOPIED, COPY (files copied), SKIP (already present)

Step-by-step explanation.

1. The s3a configuration is the write side: an instance-profile credentials provider (no static keys), fast-upload with disk buffering so large files stream without exhausting memory, a 128 MB multipart size, and a generous connection pool for the many concurrent mappers.
2. hadoop distcp launches a MapReduce job. It first builds the copy-listing of every file under hdfs:///warehouse/orders, then the map phase copies those files to the s3a target — the cluster's parallelism is the transfer engine.
3. -m 100 requests 100 mapper tasks, so up to 100 files copy concurrently; -bandwidth 50 caps each mapper at 50 MB/s so the aggregate (≤ 5 GB/s) stays within the WAN budget and does not saturate the shared link.
4. -strategy dynamic matters when file sizes are skewed: instead of pre-assigning an equal count of files per mapper (which lets one mapper stall on a huge file while others idle), dynamic hands out chunks as mappers finish, keeping all of them busy.
5. -update makes the job idempotent and restartable: it compares source and target by size/checksum and skips files already present, so re-running after a partial failure copies only what is missing. The COPY/SKIP/BYTESCOPIED counters are your reconciliation evidence.

Output.

Counter	Meaning	Example
COPY	files copied this run	812,004
SKIP	files already present (via -update)	0 (first run)
BYTESCOPIED	bytes moved	41.2 TB
Mappers	parallel copy tasks	100
Job status	SUCCEEDED	—

Rule of thumb. Configure s3a for fast multipart upload, run DistCp with -update -strategy dynamic and a per-mapper -bandwidth cap, and read the COPY/SKIP/BYTESCOPIED counters as your evidence. -update makes the job restartable — re-running copies only what is missing.

Worked example — snapshot-diff incremental copy

Detailed explanation. After the bulk copy, you keep the lake current by copying only what changed between two HDFS snapshots with -diff. This is the DistCp equivalent of an incremental sync and the mechanism that shrinks the cutover delta on a petabyte lake. Walk through the snapshot workflow.

• Enable snapshots. hdfs dfsadmin -allowSnapshot on the source directory.
• Snapshot before bulk. Take snap_bulk and DistCp from it.
• Snapshot before cutover. Take snap_cut; distcp -diff snap_bulk snap_cut moves only the delta.

Question. Show the snapshot lifecycle and the -diff incremental copy.

Input.

Step	Command
Allow snapshots	hdfs dfsadmin -allowSnapshot /warehouse/orders
Bulk snapshot	hdfs dfs -createSnapshot ... snap_bulk
Delta snapshot	hdfs dfs -createSnapshot ... snap_cut
Incremental	distcp -diff snap_bulk snap_cut

Code.

# 1. Enable HDFS snapshots on the source directory (once)
hdfs dfsadmin -allowSnapshot /warehouse/orders

# 2. Take the baseline snapshot and do the bulk copy FROM the snapshot
hdfs dfs -createSnapshot /warehouse/orders snap_bulk
hadoop distcp -m 100 -strategy dynamic -update \
  /warehouse/orders/.snapshot/snap_bulk \
  s3a://prod-lake/warehouse/orders

# 3. ...time passes, the on-prem lake keeps taking writes...

# 4. Just before cutover, take a second snapshot and copy ONLY the delta
hdfs dfs -createSnapshot /warehouse/orders snap_cut
hadoop distcp \
  -update -diff snap_bulk snap_cut \   # copy files changed between snapshots
  -m 60 -strategy dynamic \
  /warehouse/orders \
  s3a://prod-lake/warehouse/orders
# DistCp reads the snapshot diff report and moves only created/modified files,
# and applies renames/deletes on the target to match.

Step-by-step explanation.

1. -allowSnapshot turns the source directory into a snapshottable path. HDFS snapshots are cheap, copy-on-write point-in-time markers — creating one is O(1), not a data copy.
2. The bulk copy runs from the snap_bulk snapshot path (.snapshot/snap_bulk), so it copies a consistent point-in-time image even while the live directory keeps taking writes. This is what makes the bulk copy internally consistent.
3. Between the two snapshots the on-prem lake keeps changing — new partitions land, some files are rewritten. Those changes are not in snap_bulk, so the target is now stale by exactly the delta between the snapshots.
4. -diff snap_bulk snap_cut asks HDFS for the list of files created, modified, renamed, or deleted between the two snapshots and copies only those to the target, even applying renames and deletes so the target matches. On a petabyte lake this moves gigabytes, not petabytes.
5. Repeating steps 4 on a cadence (snapshot → -diff → copy) keeps the target within one diff of the source, so the final pre-cutover -diff copies a tiny delta — the DistCp mechanism behind a short cutover window for a huge lake.

Output.

Copy	Source	Files moved	Bytes
Bulk (snap_bulk)	full snapshot	812,004	41.2 TB
Incremental (-diff)	snap_bulk→snap_cut	3,120	190 GB
Final pre-cutover -diff	latest snapshots	240	9 GB

Rule of thumb. Enable HDFS snapshots, bulk-copy from a baseline snapshot for consistency, then use -diff old new to move only the delta between snapshots on a cadence. The final pre-cutover -diff copies a small delta, so even a petabyte lake cuts over in a short window.

Worked example — tuning mappers, dynamic strategy, and the small-files problem

Detailed explanation. DistCp throughput is a tuning exercise: too few mappers underuse the WAN, too many overwhelm the object store or the source NameNode, and millions of tiny files make per-file overhead dominate. Walk through diagnosing and tuning a slow DistCp job.

• Symptom. A DistCp job is far slower than the WAN should allow; a handful of mappers run for hours after the rest finish.
• Causes. Uniform strategy on skewed file sizes (stragglers), too few mappers, and a small-files tax (per-file open/close overhead).
• Fixes. -strategy dynamic, right-size -m, and compact small files (or copy them as aggregated blocks) before/after transfer.

Question. Tune a DistCp job suffering from stragglers and a small-files problem.

Input.

Symptom	Cause	Fix
Few mappers run for hours	size skew + uniform strategy	-strategy dynamic
WAN under-utilised	too few mappers	raise -m to match bandwidth
Per-file overhead dominates	millions of small files	compact / aggregate
Object store throttling (503s)	too many mappers/prefix hotspot	cap -m; spread key prefixes

Code.

# Before: uniform strategy, 20 mappers, skewed files -> long tail of stragglers
hadoop distcp -m 20 /warehouse/events s3a://prod-lake/warehouse/events
# ...18 mappers finish in 40 min; 2 mappers grind for 3 hours on huge files.

# After: dynamic strategy balances the tail; mappers sized to the WAN budget
#   usable WAN 4 Gbps = 500 MB/s ; per-mapper 25 MB/s -> ~20 useful mappers,
#   bump to 40 to overlap latency, dynamic hands work to whoever is free.
hadoop distcp \
  -strategy dynamic \
  -m 40 -bandwidth 25 \
  -update \
  /warehouse/events \
  s3a://prod-lake/warehouse/events

# Small-files tax: 8 million 4 KB files -> per-file open/close dominates.
# Compact into larger files on HDFS FIRST (one Spark/Hive job), then DistCp:
#   - coalesce to ~128-256 MB Parquet/Avro objects
#   - DistCp then moves thousands of big objects, not millions of tiny ones
# (If you cannot pre-compact, at least raise fs.s3a.connection.maximum and
#  use -strategy dynamic so idle mappers absorb the small-file backlog.)

Step-by-step explanation.

1. The straggler symptom — most mappers done, a couple grinding for hours — is the signature of the uniform strategy on skewed file sizes: each mapper got an equal count of files, but a few files are enormous, so those mappers run far longer while the rest sit idle.
2. -strategy dynamic fixes stragglers by handing file chunks to mappers as they finish rather than pre-assigning them, so a mapper that lands the huge files does not block completion — idle mappers pick up the remaining work.
3. Mapper count is sized to the WAN: usable 500 MB/s divided by a 25 MB/s per-mapper cap is ~20 fully-utilised mappers, and bumping to 40 overlaps network latency without exceeding the budget. Too few wastes bandwidth; too many throttles the object store.
4. The small-files tax is separate: eight million 4 KB files means per-file open/close/commit overhead dominates the actual byte movement. Compacting them into ~128-256 MB objects with a Spark/Hive job before DistCp turns millions of tiny transfers into thousands of efficient ones.
5. If the object store returns throttling errors (HTTP 503) under many mappers hitting one prefix, capping -m and spreading writes across key prefixes relieves the hotspot — the write side has limits too, not just the WAN.

Output.

Metric	Before	After tuning
Strategy	uniform	dynamic
Mappers	20 (2 stragglers)	40 (balanced)
Small files	8M × 4 KB	compacted to 128–256 MB
Wall-clock	~3.5 h	~45 min
WAN utilisation	~30%	~90%

Rule of thumb. Use -strategy dynamic to kill stragglers, size -m to usable bandwidth ÷ per-mapper cap, and compact millions of small files into ~128-256 MB objects before DistCp. If the object store throttles, cap mappers and spread key prefixes — the write side has limits too.

Senior interview question on DistCp

A senior interviewer might ask: "You must migrate a 2 PB on-prem HDFS lake to S3. The WAN can't carry it in the window, the lake keeps taking writes during the migration, and it's full of both huge Parquet files and millions of tiny log files. Walk me through the bulk transfer, the incremental catch-up, the tuning, and how you'd prove at cutover that every partition made it."

Solution Using Snowball bulk + DistCp snapshot-diff catch-up + counter-based reconciliation

# 1. Bulk 2 PB is offline (WAN would take ~46 days). Ship on Snowball,
#    but take an HDFS snapshot FIRST so the delta is well-defined.
hdfs dfsadmin -allowSnapshot /lake
hdfs dfs -createSnapshot /lake snap_ship        # baseline before appliances
# DistCp the snapshot to the Snowball's S3-compatible endpoint (on-prem),
# appliances travel to AWS, data imports into s3a://prod-lake/.

# 2. Compact tiny log files before/around the copy to dodge the small-files tax
#    (Spark job coalesces 4 KB logs into 256 MB objects on HDFS).

# 3. Online catch-up: while appliances are in transit, the lake keeps writing.
#    On import completion, snapshot again and -diff only the delta over the WAN.
hdfs dfs -createSnapshot /lake snap_cut
hadoop distcp \
  -update -diff snap_ship snap_cut \
  -strategy dynamic -m 80 -bandwidth 40 \
  /lake s3a://prod-lake/
# Only files changed since snap_ship cross the WAN -> GBs/TBs, not 2 PB.

# 4. Reconciliation gate — prove every partition made it, per partition.
# 4a. Directory/file/byte counts must match, per top-level partition
for p in $(hdfs dfs -ls /lake | awk '{print $NF}'); do
  src=$(hdfs dfs -count "$p" | awk '{print $2, $3}')          # files bytes
  dst=$(aws s3 ls --summarize --recursive "s3://prod-lake/${p#/lake/}" \
        | tail -2 | awk '{print $3}' | paste -sd, -)          # objects,bytes
  echo "$p  src=$src  dst=$dst"
done
# 4b. DistCp job counters: COPY + SKIP == source file count, 0 failures.

Step-by-step trace.

Stage	Mechanism	Purpose
Baseline snapshot	snap_ship before appliances	defines the delta boundary
Bulk 2 PB	Snowball offline import	WAN can't carry it in window
Small files	pre-compact to 256 MB	dodge per-file overhead
Catch-up	-diff snap_ship snap_cut over WAN	move only transit-window delta
Tuning	-strategy dynamic -m 80 -bandwidth 40	balance load, protect WAN
Gate	per-partition file/byte counts + counters	prove nothing dropped

After execution, the 2 PB bulk arrives via Snowball (offline, because the WAN math forbids online), the small-files tax is neutralised by pre-compaction, and only the writes that accumulated during the appliance round-trip cross the WAN via a snapshot -diff. The cutover gate compares per-partition file and byte counts between HDFS and S3 and checks the DistCp counters (COPY + SKIP equals source count, zero failures) — so "every partition made it" is proven per partition, not assumed.

Output:

Metric	Value
Bulk transfer	2 PB via Snowball (offline)
WAN delta (catch-up)	~tens of TB (-diff only)
Small-files handling	compacted to 256 MB objects
Per-partition parity	file + byte counts match
DistCp counters	COPY + SKIP == source, 0 failed
Cutover gate	PASS per partition

Why this works — concept by concept:

• Snapshot-bounded bulk + diff — taking snap_ship before the appliances leave defines a precise delta boundary, so the later -diff snap_ship snap_cut moves exactly the writes that happened during transit and nothing more. Without the baseline snapshot the catch-up would be a guess.
• Offline bulk by arithmetic — 2 PB over a usable WAN is ~46 days, which forbids online bulk; Snowball turns the bulk into a shipping problem and reserves the WAN for the small delta.
• Pre-compaction of small files — millions of 4 KB logs would make per-file overhead dominate both DistCp and the object store; coalescing to 256 MB objects turns the copy from IOPS-bound to bandwidth-bound.
• Dynamic strategy + capped mappers — -strategy dynamic kills stragglers on skewed file sizes and -bandwidth keeps the catch-up from saturating the shared link, so the delta copy is both fast and production-safe.
• Per-partition counter reconciliation — comparing HDFS -count to S3 object/byte counts per partition, plus DistCp's own COPY/SKIP/failed counters, proves completeness at partition granularity — the lake-case equivalent of row-count parity. Green per partition is the switch condition.
• Cost — Snowball handling and shipping for the 2 PB bulk, cluster hours for the compaction and catch-up DistCp jobs, and the WAN share for the delta. The eliminated cost is 46 days of saturated WAN (impossible) and the incident cost of a dropped partition discovered after decommissioning HDFS. Net O(delta) per catch-up after the one-time offline O(2 PB) bulk.

Data processing
Topic — data-processing
Data-processing problems on distributed copy at scale

Practice →

Optimization Topic — optimization Optimization problems on parallel throughput tuning

Practice →

---

5. Cutover — backfill, dual-write, validate, switch over

cutover is the gated switch from on-prem to cloud — bulk backfill, incremental catch-up, a reconciliation gate, the switchover, and a rollback path

The mental model in one line: cutover is the controlled moment you move consumers from the on-prem source to the cloud target, and the safe pattern is always the same shape — bulk-backfill the history, keep the cloud within seconds of the source via an incremental channel (CDC or dual-write), pass a hard reconciliation gate (row counts, checksums, aggregate diffs), switch consumers over in a defined window, and keep the source authoritative and reversible until the cloud has soaked — so that the choice between a big-bang outage cutover and a phased, near-zero-downtime cutover comes down to how much downtime the business tolerates, not to how you prove correctness. Every senior migration lives or dies at the cutover; the transfer tools only get the bytes there.

The four axes for cutover.

• Downtime tolerance. A big-bang cutover freezes writes on the source, copies the final delta, validates, and switches — simple, but it needs a maintenance window. A phased cutover uses an incremental channel so the cloud is always current, letting you switch with seconds of freeze — near-zero downtime at the cost of running dual pipelines.
• Catch-up channel. How the cloud stays current after the bulk copy: log-based CDC (for databases), scheduled incremental sync (DataSync), snapshot-diff (DistCp), or application dual-write. The channel determines how small the cutover delta gets.
• Validation depth. How you prove parity before the switch: row counts (catch missing rows), per-partition checksums/file counts (catch dropped lake files), and aggregate diffs (catch silent value corruption). For re-architected data the checks are semantic, not byte-level.
• Rollback. Can you go back? The source must stay authoritative and writable until the cloud soaks, with a reverse-sync so the source is never stale, so rollback is a consumer re-point — not a restore from backup.

Big-bang vs phased — the two cutover shapes.

• Big-bang. Announce a window; stop writes; final delta copy; reconcile; switch all consumers; resume writes on the cloud. Lowest complexity, requires downtime, and a single go/no-go decision. Right for small workloads and tolerant businesses.
• Phased / trickle. Dual-write or CDC keeps the cloud live; consumers move over in waves; each wave is reconciled before it flips; the source is decommissioned only after all consumers are on the cloud and soaked. Highest complexity, near-zero downtime, incremental risk. Right for always-on workloads.

The catch-up channel choices.

• Log-based CDC. For databases: the cloud target subscribes to the source's change log so it stays within seconds. Deletes and updates are captured natively. The gold standard for near-zero-downtime database cutover.
• Dual-write. The application writes to both source and target during the transition. Simple but risky — the two writes can diverge on partial failure, so it needs reconciliation and idempotency; often replaced by CDC where possible.
• Scheduled incremental / snapshot-diff. For files and lakes: DataSync CHANGED runs and DistCp -diff keep the target current on a cadence, shrinking the cutover delta.

The validation gate — what must be green before you switch.

• Row counts. Per table, source count equals target count. Catches lost rows.
• Checksums / file counts. Per partition for the lake; catches a dropped or corrupted file.
• Aggregate diffs. COUNT and SUM by a natural key (e.g. by day) on both sides; catches value corruption a count cannot see — essential for re-architected data.
• Sampled row diffs. Pull a random sample of keys and compare full rows on both sides; catches subtle column-level bugs.

Common interview probes on cutover.

• "Big-bang or phased?" — depends on downtime tolerance; name the incremental channel that enables phased.
• "How do you keep the cloud current during a long migration?" — CDC / dual-write / scheduled incremental sync.
• "How do you prove it's safe to switch?" — the reconciliation gate: counts + checksums + aggregates + sampled diffs, green in CI.
• "It's live and wrong — what now?" — rollback by re-pointing consumers to the still-authoritative source; the reverse-sync kept it current.

Worked example — big-bang vs phased cutover comparison

Detailed explanation. The first cutover decision is the shape: big-bang (a window, simple, downtime) or phased (dual-pipeline, complex, near-zero downtime). The choice is driven by downtime tolerance and workload criticality. Walk through comparing them for a specific workload.

• The workload. The row warehouse feeding BI — re-architected to Redshift, one-weekend window offered.
• Big-bang. Freeze the warehouse Friday night, copy the final delta, reconcile, switch BI to Redshift, done by Monday.
• Phased. CDC keeps Redshift live; move dashboards over in waves; decommission the old warehouse after all dashboards soak.

Question. Compare the two shapes for this workload and pick one.

Input.

Dimension	Big-bang	Phased
Downtime	a weekend window	near-zero
Complexity	low	high (CDC + waves)
Rollback	restore/window	re-point per wave
Risk shape	one big decision	many small decisions

Code.

Big-bang cutover (row warehouse -> Redshift)
============================================
Fri 20:00  Freeze writes on on-prem warehouse (read-only).
Fri 20:15  DMS final CDC drain; reconcile row counts + aggregates.
Sat 02:00  Gate GREEN -> repoint BI connection string to Redshift.
Sat 03:00  Smoke-test top 20 dashboards; resume writes on cloud.
Mon 09:00  Business opens on Redshift; on-prem kept read-only for rollback.

Phased cutover (same workload, if no window is allowed)
=======================================================
Week 1  DMS full-load + CDC; Redshift live and current.
Week 2  Move dashboards group A; reconcile; soak.
Week 3  Move groups B, C; reconcile each; soak.
Week 4  All dashboards on Redshift + soaked -> decommission on-prem.

Step-by-step explanation.

1. The big-bang plan is a single timeline with one go/no-go gate on Saturday morning: freeze, drain the last CDC delta, reconcile, and switch. Its virtue is simplicity — one decision, one window — and its cost is the weekend of read-only downtime.
2. The phased plan never freezes: CDC holds Redshift current while dashboards move in waves, each reconciled and soaked before the next. Its virtue is near-zero downtime; its cost is running both platforms for a month and making many smaller go/no-go decisions.
3. The rollback shapes differ: big-bang rolls back by re-pointing BI to the still-read-only on-prem warehouse within the window; phased rolls back a single wave to on-prem without affecting the waves already migrated — smaller blast radius per decision.
4. The choice is set by the offered window: because the business did offer a weekend, big-bang is the lower-complexity, lower-cost answer for this workload — you do not pay for a month of dual pipelines to avoid a downtime the business already accepted.
5. If the business had refused any window, the phased shape becomes mandatory despite its complexity — the incremental CDC channel is what buys the near-zero downtime, and the wave structure is what keeps each switch small and reversible.

Output.

Factor	Big-bang (chosen)	Phased (alternative)
Downtime	one weekend (accepted)	near-zero
Dual-run cost	~a weekend	~a month
Decisions	one gate	one gate per wave
Chosen because	window offered → simpler wins	would win only if no window

Rule of thumb. Let downtime tolerance pick the shape: if the business offers a window, big-bang is simpler and cheaper; if it refuses one, go phased with a CDC catch-up channel and wave-by-wave reconciliation. Do not pay for dual pipelines to avoid a downtime the business already accepted.

Worked example — dual-write + backfill + reconciliation for near-zero downtime

Detailed explanation. When no window is allowed and CDC is not available, the application dual-writes to both source and cloud during the transition, a backfill loads the history, and a reconciliation job continuously proves the two agree. This is the near-zero-downtime pattern; its danger is divergence, so reconciliation is mandatory. Build it.

• Dual-write. The app writes each mutation to on-prem and the cloud target.
• Backfill. A one-time bulk job loads pre-dual-write history into the cloud.
• Reconcile. A scheduled job compares source and target and flags drift.

Question. Implement the dual-write path and a reconciliation check that gates the cutover.

Input.

Component	Value
Source	on-prem warehouse
Target	cloud warehouse
Transition	dual-write + backfill
Gate	reconciliation drift = 0

Code.

# Dual-write during the transition window (app writes both; target is shadow)
def write_order(onprem, cloud, order: dict) -> None:
    onprem.upsert("orders", order)          # source of truth (authoritative)
    try:
        cloud.upsert("orders", order)       # shadow target (best-effort)
    except Exception as e:
        # Never fail the user request on the shadow write; record for repair.
        enqueue_repair("orders", order["id"], reason=str(e))
    # Divergence is expected and REPAIRED by reconciliation, not prevented.

-- Backfill history written before dual-write began (one-time, idempotent MERGE)
MERGE INTO cloud.orders AS t
USING onprem_export.orders AS s
ON t.id = s.id
WHEN MATCHED AND (t.updated_at < s.updated_at) THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT ...;

-- Continuous reconciliation — the cutover gate (must return zero drift)
WITH src AS (
    SELECT id, md5(concat_ws('|', status, total_cents, updated_at)) AS h
    FROM onprem.orders
),
tgt AS (
    SELECT id, md5(concat_ws('|', status, total_cents, updated_at)) AS h
    FROM cloud.orders
)
SELECT 'missing_in_cloud' AS kind, s.id FROM src s LEFT JOIN tgt t USING(id) WHERE t.id IS NULL
UNION ALL
SELECT 'extra_in_cloud',   t.id       FROM tgt t LEFT JOIN src s USING(id) WHERE s.id IS NULL
UNION ALL
SELECT 'value_mismatch',   s.id       FROM src s JOIN tgt t USING(id) WHERE s.h <> t.h;
-- Cutover gate: this query returns 0 rows for N consecutive runs.

Step-by-step explanation.

1. The dual-write path keeps on-prem authoritative and treats the cloud as a best-effort shadow: the user request never fails on a cloud write error, and any failed shadow write is queued for repair. This prevents the transition from harming production while accepting that divergence will happen.
2. The backfill MERGE loads history written before dual-write began, idempotently: matched rows update only if the source is newer, unmatched rows insert. Because it is a MERGE keyed on id, it is safe to re-run — restartable after any failure.
3. The reconciliation query is the heart of the pattern: it hashes the meaningful columns on both sides and reports rows missing in the cloud, extra in the cloud, or value-mismatched. This catches every failure mode of dual-write — dropped writes, duplicate writes, and silent value drift.
4. The repair queue closes the loop: rows flagged by reconciliation (or by the dual-write catch block) are re-applied to the cloud until the drift query returns zero. Divergence is repaired continuously, not assumed away.
5. The cutover gate is "the drift query returns zero rows for N consecutive runs" — sustained parity, not a single lucky pass. Only then do you switch reads to the cloud, and only after that soak do you stop dual-writing and decommission the source.

Output.

Reconciliation kind	Meaning	Gate requires
missing_in_cloud	dropped shadow write	0
extra_in_cloud	duplicate/rollback drift	0
value_mismatch	silent value corruption	0
consecutive clean runs	sustained parity	≥ N

Rule of thumb. Keep the source authoritative and the cloud a best-effort shadow during dual-write, backfill history with an idempotent MERGE, and run a hash-based reconciliation that repairs drift continuously. The cutover gate is sustained zero drift over N runs — never a single pass.

Worked example — the validation harness and rollback runbook

Detailed explanation. The switch is only as safe as the validation that precedes it and the rollback that backs it. A senior cutover ships an automated validation harness (the gate) and a written rollback runbook that anyone on call can execute. Walk through both.

• Harness. Row counts + per-partition checksums + aggregate diffs + sampled row diffs, run in CI, emitting a single PASS/FAIL.
• Runbook. The exact steps to reverse the switch if the cloud misbehaves during soak.

Question. Write the validation harness gate and the rollback runbook.

Input.

Check	Catches
row counts	missing/extra rows
partition checksums	dropped/corrupt lake files
aggregate diffs	silent value corruption
sampled row diffs	column-level bugs

Code.

# Validation harness — one gate over all four checks
def cutover_gate(checks: dict[str, int]) -> bool:
    # every value is a DRIFT count; all must be zero to switch
    failing = {k: v for k, v in checks.items() if v != 0}
    ok = not failing
    print("CUTOVER GATE", "PASS" if ok else f"FAIL {failing}")
    return ok

cutover_gate({
    "row_count_drift":        0,   # source vs target counts
    "partition_checksum_drift": 0, # per-partition file/byte checksums
    "aggregate_diff":         0,   # SUM/COUNT by day differ
    "sampled_row_diff":       0,   # random-key full-row compare
})
# CUTOVER GATE PASS

ROLLBACK RUNBOOK — cloud warehouse cutover
==========================================
Trigger: any of (dashboards wrong, error rate up, reconciliation drift > 0)
during the soak window.

1. FREEZE cloud writes (put target in read-only) to stop further divergence.
2. REPOINT consumers back to on-prem (BI connection string / DNS / feature
   flag). On-prem stayed AUTHORITATIVE and WRITABLE — it is fully current.
3. VERIFY on-prem serves reads correctly (smoke-test top dashboards).
4. RESUME on-prem writes (it never stopped being the source of truth).
5. CAPTURE the cloud drift for post-mortem; do NOT decommission on-prem.
6. ROOT-CAUSE, fix, re-run the validation gate before re-attempting cutover.

Rollback time objective: < 15 minutes (a re-point, not a restore).

Step-by-step explanation.

1. The harness collapses four independent checks into one PASS/FAIL gate: row counts, per-partition checksums, aggregate diffs, and sampled full-row diffs. Each targets a different failure mode, and all must be zero — a single non-zero drift blocks the switch.
2. Aggregate diffs and sampled row diffs are what make the gate trustworthy for re-architected data: counts alone pass even when every value is subtly wrong, so SUM/COUNT-by-day and random-key row comparison are non-negotiable when the bytes changed on purpose.
3. The rollback runbook exists because "it's live and wrong" is a when, not an if, for a large migration. It is written, versioned, and rehearsed so an on-call engineer executes it under pressure without improvising.
4. The runbook's safety rests entirely on keeping on-prem authoritative and writable through the soak: rollback is a re-point of consumers (connection string, DNS, or feature flag) back to a source that is fully current — a sub-15-minute operation, not a multi-hour restore from backup.
5. Rollback deliberately does not decommission on-prem: the source is retired only after the cloud has soaked cleanly for the agreed period. Decommissioning early is the mistake that turns a recoverable incident into a data-loss event.

Output.

Element	Value
Gate	single PASS/FAIL over 4 checks
Switch condition	all drift counts = 0
Rollback mechanism	re-point consumers to authoritative on-prem
Rollback time objective	< 15 min
Decommission on-prem	only after clean soak

Rule of thumb. Ship the validation harness as a single automated gate over counts, checksums, aggregates, and sampled diffs, and write a rehearsed rollback runbook whose safety depends on keeping on-prem authoritative and writable through the soak. Decommission the source only after a clean soak — never at the moment of switch.

Senior interview question on cutover

A senior interviewer might ask: "You're cutting a 24/7 order-processing warehouse over to the cloud with no allowed downtime. Design the cutover — the catch-up channel, the backfill, the reconciliation gate, the switchover, and the rollback — and tell me exactly what has to be true before you flip reads to the cloud and exactly how you'd back out if it goes wrong an hour after go-live."

Solution Using CDC catch-up + backfill + a sustained reconciliation gate + a re-point rollback

# Phased, near-zero-downtime cutover — order warehouse
Step 1  BACKFILL: DMS full-load of history into the cloud warehouse.
Step 2  CATCH-UP: DMS CDC streams every change; cloud stays seconds behind.
Step 3  RECONCILE: hash-based drift query runs every 5 min; must hit 0 and hold.
Step 4  SHADOW READS: mirror a % of read traffic to cloud; compare results.
Step 5  SWITCH: flip reads to cloud via feature flag (writes still to source).
Step 6  SOAK: source stays authoritative + CDC-current for the rollback window.
Step 7  PROMOTE: after clean soak, move writes to cloud; decommission source.

-- The switch condition (Step 3) — sustained zero drift
SELECT count(*) AS drift
FROM (
    SELECT id FROM onprem.orders
    EXCEPT
    SELECT id FROM cloud.orders
    UNION ALL
    SELECT id FROM cloud.orders
    EXCEPT
    SELECT id FROM onprem.orders
) d;
-- Flip reads only when drift = 0 for 12 consecutive 5-min runs (1 hour clean).

# Rollback (an hour after go-live, dashboards wrong)
1. Feature flag: reads back to on-prem (writes never left it).      # < 1 min
2. CDC still running -> on-prem is fully current; no data lost.
3. Smoke-test; capture cloud drift; root-cause offline.
4. Re-run the gate; only re-flip after 1 hour clean again.
Rollback time objective: < 5 minutes (flag flip; source never stopped).

Step-by-step trace.

Step	Mechanism	Guarantees
Backfill	DMS full-load	history present in cloud
Catch-up	DMS CDC	cloud within seconds of source
Reconcile	5-min hash drift query	parity proven, continuously
Shadow reads	mirror % of reads, compare	behaviour parity before switch
Switch	feature-flag reads to cloud	reversible in one flag flip
Soak	source authoritative + CDC-current	rollback stays a re-point
Promote	writes to cloud after clean soak	source retired only when safe

Because writes never leave on-prem until the final promote, the whole cutover is reversible with a single feature-flag flip: reads move to the cloud only after drift is zero for a sustained hour and shadow reads match, and if anything looks wrong an hour after go-live, reads flip back to a source that CDC kept fully current. There is no allowed-downtime violation — the switch and the rollback are both flag flips, not restores.

Output:

Metric	Value
Downtime at switch	~0 (feature-flag read flip)
CDC lag during soak	seconds
Switch condition	drift = 0 for 12 consecutive runs (1h)
Rollback time objective	< 5 min (flag flip)
Data loss on rollback	none (writes never left source)
Source decommission	only after clean soak + promote

Why this works — concept by concept:

• CDC catch-up channel — DMS change-data-capture holds the cloud within seconds of the source after the backfill, so the cutover delta is never more than a few seconds — the mechanism that makes zero-downtime possible.
• Sustained reconciliation gate — a hash/EXCEPT-based drift query that must read zero for a sustained hour, not one pass, proves parity is stable before the flip. Sampled shadow reads add behaviour parity on top of data parity.
• Writes-stay-on-source until promote — keeping writes on on-prem until the final promote is what makes the read switch reversible: rollback is a flag flip because the source never stopped being authoritative and CDC kept it current.
• Feature-flag switch and rollback — routing reads through a flag makes both the switch and the rollback sub-5-minute operations with no data movement, which is the only way to honour a no-downtime, always-reversible requirement.
• Cost — a DMS replication instance and CDC for the transition, a % of mirrored read traffic for shadow comparison, and a parallel-run period until promote. The eliminated cost is any downtime at all and the catastrophic cost of an irreversible switch with writes already on the cloud. Net O(seconds) of delta at switch and O(1) rollback, versus O(restore) if writes had moved early.

ETL
Topic — etl
ETL problems on backfill and reconciliation

Practice →

Design
Topic — design
Design problems on zero-downtime cutover

Practice →

---

Cheat sheet — on-prem → cloud migration recipes

• Which approach when. Lift-and-shift (rehost) to land fast and de-risk when a hardware deadline drives the schedule; re-architect (refactor) to columnar-object-storage-plus-serverless when run-cost or performance is the reason for the migration; and in practice do both via the strangler pattern — rehost first, refactor the highest-cost workloads afterwards. Inventory every workload against the 6 R's (rehost, replatform, repurchase, refactor, retire, retain) and retire the dead ones before moving anything.
• Time-to-transfer arithmetic. time = data_volume / effective_bandwidth, where effective bandwidth is the usable share of the link, not the nameplate (a shared 10 Gbps DX is realistically ~4 Gbps ≈ 0.5 GB/s for migration). If the result exceeds your window, go offline: Snowball / Data Box / Transfer Appliance for the bulk, online transfer for the delta. 2 PB at 4 Gbps ≈ 46 days → ship it.
• Transfer tool by data shape. Files / NAS / SMB / object → AWS DataSync (agent, verify, incremental, throttle). HDFS / Hadoop data lake → DistCp (MapReduce parallel copy to s3a://, snapshot -diff incrementals). Relational database → DMS (full-load + CDC). Bulk beyond WAN capacity → Snowball/appliance offline. Never hand-roll rsync-over-SSH for a serious migration.
• DataSync task template. Agent near the source for LAN-speed reads; reusable source/destination locations; task options VerifyMode=POINT_IN_TIME_CONSISTENT (first + final run), ONLY_FILES_TRANSFERRED (hourly incrementals), TransferMode=CHANGED, a BytesPerSecond cap to protect a shared WAN, exclude filters for junk (*/tmp/*|*.lock), an hourly schedule, and a task report to S3 as the file-case evidence trail. Gate: verified == transferred, failed == 0, destination count == source count.
• DistCp command + s3a tuning. hadoop distcp -update -strategy dynamic -m <mappers> -bandwidth <MB> -p hdfs:///path s3a://bucket/path. Size -m to usable bandwidth ÷ per-mapper cap; use -strategy dynamic to kill stragglers on skewed file sizes; enable HDFS snapshots and -diff snap_old snap_new for incrementals; pre-compact millions of small files into 128–256 MB objects; configure s3a for fast.upload, 128 MB multipart, and a large connection pool. Reconcile with the COPY/SKIP/BYTESCOPIED counters.
• Cutover shapes. Big-bang: freeze → final delta copy → reconcile → switch → resume on cloud (needs a window; one go/no-go). Phased: incremental catch-up (CDC/dual-write) keeps the cloud live → move consumers in reconciled waves → decommission after soak (near-zero downtime; many small decisions). Downtime tolerance picks the shape, not correctness.
• Catch-up channels. Database → log-based CDC (seconds behind, deletes captured). Application → dual-write (best-effort shadow + continuous reconciliation repair; risky, needs idempotency). Files → DataSync CHANGED. Lake → DistCp -diff. The channel shrinks the cutover delta to something copyable inside the window.
• Reconciliation gate (the switch condition). Row counts per table (missing/extra rows), per-partition file/byte counts + checksums (dropped lake files), aggregate diffs SUM/COUNT by day (silent value corruption — essential for re-architected data), and sampled random-key full-row diffs (column-level bugs). All drift = 0, sustained over N consecutive runs, green in CI — never a single pass, never an eyeball.
• Rollback runbook. Keep the on-prem source authoritative and writable through the whole soak, with a reverse/CDC sync so it never goes stale. Rollback = re-point consumers (connection string / DNS / feature flag) back to the source — a sub-15-minute operation, not a restore. Decommission the source only after the cloud soaks cleanly for the agreed window; early decommission turns a recoverable incident into data loss.
• Lift-and-shift validation vs re-architect validation. Rehosted (byte-identical) data is verifiable by checksum (DataSync verify, DistCp -update compare, file/byte counts). Re-architected (reshaped) data changed the bytes on purpose, so it must be verified semantically — row counts + aggregates + sampled diffs — because checksums are meaningless across a format/layout change.
• Bandwidth hygiene. Always cap migration bandwidth on a shared link (DataSync BytesPerSecond, DistCp -bandwidth) and schedule heavy runs off-peak; a Direct Connect / ExpressRoute interconnect is table stakes — the public internet is neither fast nor predictable enough for a serious migration.
• Migration sequencing. Order workloads by ascending risk: easy file/NAS lift-and-shift first (build confidence and tooling), then replatformed databases, then the high-value refactors and the always-on cutovers last. Tag every cloud resource with a cost-centre from day one so the re-architecture payoff is measurable, not argued.

Frequently asked questions

What is cloud data migration in one sentence?

Cloud data migration is the end-to-end process of moving an organisation's on-premises data estate — file servers and NAS, Hadoop / HDFS data lakes, and relational data warehouses — onto cloud storage and compute, either by rehosting the same engines unchanged (lift and shift) or by redesigning them cloud-native (re-architect), using a transfer mechanism sized to the data (AWS DataSync for files, DistCp for HDFS lakes, DMS for databases, Snowball appliances for petabytes too large for the network), and finishing with a gated cutover that proves parity before switching consumers over and keeps the source reversible until the cloud has soaked. The whole discipline lives in four axes — data volume and bandwidth, downtime tolerance, transformation depth, and validation and rollback — and senior interviews probe it because a botched migration strands a company paying for two platforms with a warehouse nobody trusts.

Lift-and-shift vs re-architect — when do I pick each?

Pick lift and shift (rehost the same engine onto cloud infrastructure with minimal change — e.g. Hadoop → EMR reading s3://) when a hardware deadline drives the schedule, the workload is stable, and you want a fast, low-risk landing you can verify by checksum; its downside is that it carries the on-prem technical debt into the cloud and can even raise run-cost if you rehost an always-on cluster unchanged. Pick re-architect (refactor to cloud-native primitives — columnar Parquet on object storage, elastic or serverless compute, decoupled storage and compute) when run-cost or performance is the reason for the migration, because that is the only strategy that actually collects the payoff; its downside is higher effort and risk per workload, and it must be validated semantically (row counts + aggregates) rather than by checksum because the bytes change on purpose. In practice most real migrations do both in sequence — the strangler pattern: lift-and-shift everything to escape the on-prem hardware on schedule, then re-architect the highest-cost workloads first, measuring cloud spend per workload as you go.

DataSync vs DistCp — which transfer tool do I use?

Use AWS DataSync when the data is files or objects — an NFS/SMB NAS, a file server, or an object store — moving to S3, EFS, or FSx: it deploys a managed agent, encrypts and verifies the transfer with checksums, copies only changed files on subsequent runs, throttles bandwidth to protect a shared WAN, and emits per-run reports you use as your file-case reconciliation evidence. Use DistCp when the data lives in HDFS or a Hadoop-compatible file system: it runs as a MapReduce job on your Spark/Hadoop cluster, splits the file list across many mapper tasks to parallelise the copy to s3a://, and supports HDFS snapshot -diff for incrementals — the right engine for petabyte-scale lake migrations where a single-stream copy would never finish. A rule of thumb: DataSync is the managed, file-shaped mover you configure; DistCp is the cluster-powered, lake-shaped mover you tune. And when the bulk is simply too big for the network in your window, neither runs online — you ship it on a Snowball appliance and use DataSync or DistCp only for the delta.

How do I migrate petabytes over a slow WAN?

Do the arithmetic first: time = data_volume / effective_bandwidth, using the usable share of the link (a shared 10 Gbps interconnect is realistically ~4 Gbps ≈ 0.5 GB/s for migration). At that rate 2 PB takes roughly 46 days, which blows any sane window — so the bulk goes offline on physical appliances (AWS Snowball / Snowmobile, Azure Data Box, GCP Transfer Appliance) that you load on-prem and ship to the provider, who imports them directly into object storage. The trick that keeps the migration consistent is to take a point-in-time marker before the appliances leave — an HDFS snapshot for a lake — so the writes that accumulate during the multi-week appliance round-trip are a well-defined delta. When the bulk import completes, you take a second snapshot and copy only that delta over the WAN with DistCp -diff (or DataSync CHANGED, or DMS CDC for a database), which is gigabytes-to-terabytes, not petabytes. So the pattern is: offline for the bulk, online for the delta, a snapshot to define the boundary between them, and a per-partition reconciliation gate before cutover.

How do I cut over with near-zero downtime?

Decouple the bulk copy from the switch. First bulk-backfill the history into the cloud, then keep the cloud continuously current with an incremental catch-up channel — log-based CDC for databases (the cloud stays seconds behind and captures deletes and updates natively), application dual-write where CDC is unavailable (writing to both source and a best-effort cloud shadow, with continuous reconciliation to repair drift), or scheduled incremental sync for files and lakes. Because the cloud is always within seconds of the source, the cutover no longer copies the whole dataset — it flips a switch. Keep writes on the authoritative on-prem source and move only reads to the cloud via a feature flag once the reconciliation gate has read zero drift for a sustained period (and, ideally, shadow reads confirm behaviour parity). The switch is a flag flip, and so is the rollback — because writes never left the source, backing out is a sub-five-minute re-point, not a restore. Promote writes to the cloud and decommission the source only after a clean soak.

How do I prove the migrated data is correct?

With a reconciliation gate that must be green — sustained, in CI — before any switchover, matched to whether the data was rehosted or re-architected. For a lift-and-shift (byte-identical) copy, verify by checksum: DataSync's built-in verification for files, DistCp's -update size/checksum compare and COPY/SKIP/BYTESCOPIED counters for the lake, and per-partition file and byte counts that must match between source and target. For a re-architected copy the bytes deliberately changed (new format, new layout), so checksums are meaningless and you validate semantically: row counts per table (catch missing or extra rows), aggregate diffs such as COUNT and SUM grouped by day (catch silent value corruption a count cannot see), and sampled random-key full-row comparisons (catch subtle column-level bugs). The gate is "all drift counts are zero for N consecutive runs," enforced automatically — never a single pass and never a human eyeballing a dashboard — and the source stays authoritative and reversible until the cloud has soaked, so a failed gate or a bad soak is a rollback, not a data-loss incident.

Practice on PipeCode

• Drill the ETL practice library → for the ingestion, backfill, incremental-load, and reconciliation problems senior migration interviews love.
• Rehearse on the design practice library → for the platform-migration, re-architecture, and zero-downtime cutover system-design scenarios.
• Sharpen the throughput axis on the data-processing practice library → for distributed-copy, columnar-format, and parallelism problems behind DistCp and the lake refactor.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis migration decision framework against real graded inputs.

Lock in cloud-migration muscle memory

Docs explain tools. PipeCode drills explain the decision — when lift-and-shift beats re-architect, when the WAN math forces a Snowball, when DistCp needs `-strategy dynamic`, when a cutover is safe to flip and how to roll it back. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL problems →
Practice design problems →