DEV Community

DataDriven
DataDriven

Posted on

Meta's Pipeline Migration Playbook for Data Engineer Interviews

The worst migration I ever ran finished at 3:40am with a Slack message that said "looks good." 2 weeks later, finance found out a join key had changed type in the new system, and about 6% of revenue rows had been quietly landing as nulls. We'd checked that the job went green and never compared the outputs, so a green job was all we got.

So when Meta published its write-up on migrating data ingestion systems at Meta scale in May, I read it twice. The system incrementally scrapes several petabytes of social graph data from MySQL into the warehouse every day. The post says they had "tens of thousands of ingestion jobs to migrate," and they finished with this line: "We've successfully transitioned 100% of the workload and fully deprecated the legacy system."

I think this post is the best primary-source answer you'll get to the most common pipeline question in a Data Engineer system design interview: "How would you migrate this safely?" It's also the template you should be using if you're staring down the Airflow 3.4 upgrade right now.

What Meta did with tens of thousands of data pipeline jobs

Every job went through 3 phases: shadow, reverse shadow, and cleanup. That's the whole skeleton. It's simple enough to draw on a whiteboard in 90 seconds, and that's a big part of why it works.

In the shadow phase, the new job reads the same source as the production job but writes to a separate shadow table, so production output stays untouched. Meta continuously compared row counts and checksums between the production and shadow outputs, and "any mismatches were logged to Scuba, Meta's data management system for real-time analysis." Mismatches got fixed before anything moved forward.

In reverse shadow, the roles swap. The new job's output now goes to the production table, and the old job keeps running and writes to the shadow table. Comparisons keep going the whole time. If the outputs diverge, the post says "the job would be rolled back immediately and data consumers would not be impacted."

Cleanup comes once there are no discrepancies. The old job (now the shadow) gets removed, and the new system delivers alone.

The reverse shadow phase is the part most people skip, and it's the clever bit. Rollback is a role swap: the old job is still alive and still writing, so switching back costs you a config change. You never rebuild anything at 3am.

Validation doesn't stop at cutover. Meta kept comparing old and new after the new system owned production. Most migrations I've seen stop checking the moment the switch flips.

There's a catch, and it's my inference rather than something the post states. Cheap rollback has a shelf life. Once cleanup removes the old job, rolling back means rebuilding the old path. So the reverse shadow window is your insurance policy, and you're paying for 2 systems while it runs.

The other detail I love is how Meta handles bad data that already landed. A partition with a quality issue gets marked bad in its metadata. For a delta partition, new data stops landing and an alert goes to a human. For a target partition, "the system would instead select an older partition and merge it with more deltas," and flagged partitions get fixed later with a backfill. Rollback is per partition. You don't nuke the whole pipeline because one Tuesday went sideways.

The promotion gates are what interviewers want to hear

Every job had to clear 3 gates before moving to the next phase. Verbatim from the post:

  • "There is no difference between the data delivered by the old system and the new system."
  • "No landing latency regression is observed."
  • "No resource utilization regression is observed."

For critical tables, they "defined and agreed on extra migration criteria" with the teams depending on them. The post doesn't list what those extra criteria were. And tooling promoted or demoted jobs between stages automatically, so no human was squinting at dashboards deciding whether a job "felt ready."

The bar is non-regression. The new system has to be no worse on latency and compute, and improvement is a bonus. I've watched migrations stall for months because somebody decided the new platform had to be faster on day one. Meta's standard is "match the old thing, then optimize." That's the right economics; engineer time spent chasing a 10% speedup mid-migration is time not spent finishing the migration.

The post also gives zero numeric thresholds. It never names a match percentage, a batch size or a phase duration, so "no difference" is the whole spec. A sharp interviewer will ask how you handle timing skew between 2 live writers, and the post doesn't say. You need to define your own quantitative exit criteria, and saying so out loud in an interview scores points.

Cost shaped the sequencing too. Creating a new ingestion job kicks off "a full dump, which is typically slow and expensive." So Meta held back shadow jobs that still had known bugs, since every retry meant another full dump. They also reused snapshot partitions from the old system as initial snapshots. The post admits that "migration capacity was limited, we could not run all shadow jobs at once," so jobs went in batches grouped by throughput, priority, and special cases.

Most migration advice treats this purely as a correctness problem. At Meta's scale it was a capacity planning problem too, and the most expensive thing in the room was the backfill.

Answering "how would you migrate this safely?" in a system design interview

I've sat on enough panels to know how this question usually goes. The candidate says "we'd run both in parallel, validate, then cut over." The interviewer says "validate how?" The candidate says "compare row counts." The interviewer writes "shallow" in their notes. Interview over, mentally.

I haven't found any official rubric for how interviewers score migration answers, so what follows is my structure, built from Meta's post and from what I've seen land on real panels.

Start by sequencing the work instead of migrating everything at once. Group jobs by risk and throughput, and begin with low-criticality, low-volume jobs so you find tooling bugs cheaply. Hold back anything that triggers an expensive backfill until the known bugs are fixed. Then quote the Meta line about limited capacity forcing batches; it shows you've thought about cost.

Next, define the gates before any job moves. Row counts and checksums cover correctness, landing latency and compute/storage cover performance, and critical tables get extra criteria negotiated with their downstream owners. Then put numbers on them, because Meta didn't publish any and your interviewer will want yours. "Zero checksum mismatches across 7 consecutive daily partitions" is an answer. "Make sure the data looks right" is fog.

Design rollback before anything else, because this is where you separate yourself. Stripe's online migrations post lays out the dual-write pattern: dual-write to old and new, move reads to new, move writes to new, remove the old data. They caught mismatches with GitHub's Scientist library. It's a great post, and it doesn't address rollback at all. Meta's reverse shadow does: the old job stays alive as the shadow, so rollback is a swap. Mention both and explain why the second one answers the question the first one leaves open.

Then cover bad data that already landed. In a CDC pipeline, switching back to the old system doesn't undo bad rows sitting in the warehouse. Mark the bad partitions, stop the deltas, alert, backfill. Interviewers rarely hear this, and it's closer to the actual job, which is mostly figuring out why 2M rows went missing last Tuesday and making sure it never happens again.

Close with cleanup: say when the old job dies and what condition triggers it. Most candidates never decommission anything in their answer, which tells me they've never been the person paying for 2 systems for 8 months because nobody would sign off on turning the old one off.

You'll notice none of this mentions a specific tool, because shadow tables, checksums, latency gates, and partition-level backfill work on any warehouse and any orchestrator. The candidate who explains this with Postgres and cron beats the candidate who lists 9 vendor products and can't explain a rollback.

If you want to practice saying this out loud, do it under pressure, with someone interrupting you with follow-ups. For the coding half of the loop we're biased, since it's ours, but for python practice problems, datadriven is great, and the pipeline architecture prompts on it (one has you moving 60 production pipelines to a cloud warehouse with 6am reports that can't miss) are the closest thing we've seen to the follow-up questions you'll actually get.

Running the same playbook on your Airflow 3.4 upgrade

Airflow 3.4.0 shipped on November 6, and the release notes have enough breaking changes that I'd treat it as a migration, with shadow runs to match. Highlights that will bite somebody:

  • Python 3.11+ is required. Python 3.10 support is gone across apache-airflow, the Task SDK, airflowctl, and providers.
  • Asset-triggered Dags now create one run per asset event by default. That covers single-asset schedules, asset OR expressions, and a couple of timetable types. You get batching back with batch_asset_events=True plus a reparse.
  • airflow db clean now deletes pending asset events for downstream Dags, so a Dag waiting on a multi-asset condition can lose events.
  • psycopg3 replaces psycopg2 as the default Postgres driver, the API server adds a second DB connection pool (plan PgBouncer and max_connections for up to 2x the pool), and the Task SDK swaps certifi for truststore, so workers need public CAs installed.
  • [workers] max_failed_heartbeats jumps from 3 to 30.
  • Downgrading below 3.4.0 can be slow, because XCom rows have to move back from xcom_v2.

Meta's checks compare outputs, but the biggest 3.4 change is about how many runs fire. A shadow environment can match production row for row and still trigger downstream Dags 40 times instead of once, so compare run counts and event consumption alongside row counts and checksums.

Meta wrote shadow output to separate tables for a reason, and your shadow Dags need the same isolation. If your shadow Dag writes the same asset URI as production, every Dag scheduled on that asset runs off the shadow write as if production had produced it, so give the shadow Dags their own asset URIs or pause those consumer Dags. Ask me how I know what happens when 2 environments both think they own the same table (actually, don't).

Meta's rollback was cheap because the old job kept running. Airflow's isn't: the downgrade path is slow, and db clean now destroys pending events. That makes shadowing against a cloned metadata DB your rollback window, because once you upgrade the real metadata DB, reverting gets expensive.

Meta's latency and resource gates line up neatly with the new connection pool and the heartbeat default. Neither changes a single output row. Both change DB load and how fast you detect a dead worker. Measure them during the shadow phase, the same way Meta measured "compute and storage quotas for the shadow jobs" before promoting.

Draining a Dag stops new scheduled and asset runs, then pauses the Dag once active runs finish, which gives you a ready-made version of Meta's cleanup step.

If you're a 3-person team with 40 Dags, you don't need Meta's full machinery. Running 2 of everything doubles compute for the length of the window, and Meta's post never quantifies that overhead. I'd cut the backfill first, because it's usually the expensive part. AWS documents a lighter option for its own migrations, sample-read validation, where scheduled jobs compare random subsets from both systems and it "doesn't require application code changes." For a small team, checksums on a sample of partitions plus a short reverse shadow on your 5 most important Dags covers most of the risk. That's my judgment, not a measured result; nobody I've found has published a cost comparison for small teams.

So: what's the scariest migration you've run without a rollback plan, and what would you have gated on if you'd done it again?

Top comments (0)