Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double the storage costs. They treat the migration like a switch-flip, and that is exactly how you cause a P0 incident.
Why I chose this topic: I’ve been burned by "big bang" migrations in healthcare environments where downtime is measured in lost compliance certifications. After three failed attempts at manual synchronization, I settled on a shadow-table strategy that treats the migration as a background process, not a deployment window.
Most engineers interact with the Hive metastore daily. They treat it like a source of truth, but it’s actually a glorified index of files that has no idea if a Parquet file is corrupted, moved, or partially deleted. We rely on it to tell us where our data lives, yet we treat it as an immutable oracle. When you move to Iceberg, you aren’t just changing a table format; you’re replacing a loose collection of files with a transactional state machine.
How it actually works
You don't migrate by moving data. You migrate by shadowing the write path.
The strategy is simple: keep your existing Parquet pipeline as the "Primary," and introduce a secondary "Shadow" sink that writes to an Iceberg table simultaneously.
First, you need a dual-write mechanism. If you’re using Spark, don't try to manage this in your application logic. Use a Kafka Connect sink or a structured streaming job that reads from your source (e.g., Kinesis or Kafka) and commits to both targets.
In your Spark job, keep your Parquet sink as-is. Add the Iceberg sink:
// The primary Parquet path
df.write.format("parquet").mode("append").save(parquetPath)
// The shadow Iceberg path
df.writeTo("iceberg_db.shadow_table")
.tableProperty("write.format.default", "parquet")
.tableProperty("format-version", "2")
.append()
The magic happens in the metadata. Iceberg creates a metadata/ folder inside your table directory. This is where the snapshot IDs live. Even if your source data is identical, the two tables will eventually diverge if your schema evolution isn't handled perfectly.
To ensure parity, you need an automated reconciliation job. Run a daily Spark job that performs a MINUS or EXCEPT operation between the two datasets:
-- Reconciliation check
SELECT * FROM parquet_table EXCEPT SELECT * FROM iceberg_table;
If that query returns rows, your shadow write failed or your schema translation is buggy. Do not proceed to the switch-over phase until that query returns zero rows for seven consecutive days.
Photo by Nicolas Arnold on Unsplash
The tradeoffs nobody mentions
This strategy is not free.
First, the storage cost. You are effectively doubling your storage footprint for the duration of the migration. In a healthcare context, this means keeping two copies of PII-encrypted data. Ensure your cloud lifecycle policies are set to expire the Parquet files only after you have fully decommissioned the old table.
Second, the "Ghost Latency." If your Spark job is writing to two different destinations, the overall job duration will be capped by the slower of the two. Iceberg’s commit process—which involves checking for conflicts in the catalog—can add 5-15 seconds to your job. If you have sub-second SLA requirements for your pipelines, this might be a non-starter.
Third, the metadata overhead. If you are migrating a table with 100,000+ partitions, the initial metadata load in Iceberg can cause your Driver OOM (Out of Memory) errors. You’ll need to set spark.driver.memory significantly higher than you think, often into the 16GB-32GB range, just to handle the snapshot state management during the initial transition.
Photo by Celine Nadon on Unsplash
When to reach for it (and when not to)
Reach for the shadow-table strategy if you have a "live" table—one that is being queried by BI tools or downstream microservices 24/7. This is non-negotiable in financial services where a missing row in a ledger report is a regulatory disaster.
Don't reach for this if your data is static. If you have a table that only gets updated once a month, just perform an INSERT OVERWRITE into a new Iceberg table during a maintenance window. The shadow strategy is complex because it’s meant for high-velocity, high-concurrency environments. If you don't have the write volume to justify the complexity, you are just inviting more failure points into your architecture.
Also, avoid this if your Parquet files are highly fragmented (millions of tiny 1KB files). Migrating "bad" data into Iceberg just gives you a "clean" format for "dirty" data. Use the migration window as an opportunity to perform a compaction job before you point your production queries to the new Iceberg table.
Conclusion
Migrating to Iceberg is less about the technical transition and more about the psychological shift from "file-based data" to "transactional data." By shadowing your writes, you buy yourself the time to catch the edge cases—like subtle schema mismatches or timezone shifts in timestamp columns—that would otherwise blow up in your face on a Monday morning.
The goal isn't to be fast. The goal is to reach a state where you can point your production traffic to the new table, delete the old Parquet files, and have your stakeholders remain completely unaware that anything happened under the hood.
Tags: #data #engineering #iceberg #migration
Cover photo by Valentin Lacoste on Unsplash.
Top comments (0)