Two years ago, I spent six weeks migrating a 400TB churn-prediction dataset from Delta Lake 2.4 to Apache Iceberg 1.3 just because our new executive hire insisted on using Trino for ad-hoc SQL. We broke three upstream Spark jobs, corrupted a partition manifest, and spent a weekend manually editing JSON metadata files in S3.
Today, that same pipeline is running on Delta 4.2 with UniForm enabled. My Trino users query the Iceberg-compatible metadata, my Spark jobs use the native Delta logs, and I haven't touched a manifest file in eight months. The "format war" is officially a waste of your time.
Why I chose this topic: I’m tired of seeing senior engineers treat table formats like sports teams. We are infrastructure providers, not fanatics, and the technical debt of format migration is a tax your balance sheet shouldn't be paying in 2026.
Stop acting like your choice of table format is a foundational architectural decision. It isn't. It’s a storage implementation detail that should be abstracted away from your compute engines. If you are currently sitting in a meeting debating whether to "standardize on Iceberg" while your primary ETL workloads are locked into Delta, you are prioritizing dogma over delivery.
Why the common approach falls short
The "pick a side" strategy ignores the reality of modern polyglot data stacks. In the financial services sector, I see teams force-feeding Iceberg into systems that don't need it because they want "openness," only to find that their Spark-based streaming ingestions struggle with Iceberg's commit overhead. Conversely, I see teams stick to Delta and effectively orphan themselves from the best-of-breed query engines that prioritize Iceberg’s mature REST catalog support.
The common failure mode is the "dual-write" pattern. Engineering teams try to maintain two copies of the same data—one as a Delta table, one as an Iceberg table. This is a disaster waiting to happen. You get drift. You get one table updated at 02:00 UTC and the other failing at 02:05 UTC because of a transient networking blip. Then your dashboard shows different numbers for the same metric, and suddenly you’re explaining to a VP of Finance why the company’s revenue shifted by $40k overnight.
Photo by Frames For Your Heart on Unsplash
The mechanics of the unified layer
UniForm (Universal Format) effectively makes the choice of format a logical abstraction rather than a physical one. By enabling delta.universalFormat.enabledFilesystems and delta.universalFormat.icebergCompatVersion in your table properties, you are telling the Delta log writer to perform the heavy lifting of generating Iceberg metadata on the fly.
When you issue a MERGE INTO or a standard INSERT in Spark 4.0, the engine writes the Parquet data files and the Delta _delta_log JSON/Checkpoint files. With UniForm, it simultaneously generates the Iceberg manifest lists and snapshot files. You aren't duplicating data; you are duplicating the metadata.
ALTER TABLE prod.customer_transactions SET TBLPROPERTIES (
'delta.universalFormat.enabledFilesystems' = 's3',
'delta.universalFormat.icebergCompatVersion' = '2'
);
Once that property is set, any engine that understands Iceberg—Trino, Flink, StarRocks—points to the same S3 prefix. The Delta log is the source of truth, and the Iceberg metadata acts as a read-optimized projection. You don't need to perform an msck repair or a full table migration. You just need to ensure your compute engine supports the Iceberg version you’ve negotiated.
The reality of failure modes
Do not think this is a "set it and forget it" panacea. The primary risk isn't the data; it’s the consistency of the metadata. If you are running an older version of a query engine that doesn't fully support Iceberg V2 features (like row-level deletes or complex partitioning), it will choke on the UniForm-generated metadata.
I once saw a Flink job crash because it expected the Iceberg metadata to reflect a snapshot state that hadn't been fully flushed to the Delta log yet. We solved this by adjusting the delta.checkpointInterval and monitoring the latency of the Iceberg metadata generation. If your delta-to-iceberg-conversion lag exceeds your business SLA for fresh data, you haven't fixed the problem—you've just moved it into a background process.
You must treat the Iceberg metadata as a materialized view of your Delta table. If the view is stale, your query engines will be wrong. Monitor the file modification times in your metadata/ folder. If they aren't updating in lockstep with your main Delta logs, your conversion job is failing silently.
Photo by Daniel Lloyd Blunk-Fernández on Unsplash
The objections (and my answers)
"But what about vendor lock-in?"
This is the most common objection I hear. The argument is that by using Delta's UniForm, you are still "locked in" to the Delta protocol. My response is simple: define "locked in." If you can point Trino, Flink, and DuckDB at your data and query it without a specialized connector, you aren't locked in. The data is in Parquet. The metadata is standards-compliant Iceberg. If Delta disappeared tomorrow, you have a perfectly valid Iceberg table sitting in your bucket. The "lock-in" is purely in the writer engine, and let's be honest—you're going to keep using Spark or Flink anyway.
"Won't this increase storage costs?"
It does, slightly. Generating Iceberg metadata files takes up space. But let's look at the math. If you have 400TB of data, the metadata files account for maybe a few gigabytes. If you are worried about the storage cost of an extra 0.001% of your footprint, you have much bigger problems in your cloud billing department. The cost of manual migration, inconsistent data, and engineering time wasted on format parity is orders of magnitude higher than the storage cost of a few extra JSON and Avro files.
"Is this production-ready for mission-critical healthcare data?"
I’ve been running this for 14 months in a HIPAA-compliant environment. The key is to keep your Delta version current. Using delta-spark 4.2+ is non-negotiable. The stability of the UniForm implementation is tied directly to the version of the Delta kernel. Don't try to run this on legacy Databricks runtimes from 2022 and expect it to behave. If your platform team isn't willing to keep the stack updated, then yes, stay away from it. But if you’re maintaining your infrastructure, it’s arguably safer than maintaining two separate, competing table formats.
Conclusion
The era of choosing sides is dead. We are moving toward a world where the file format is an internal implementation detail of the storage layer, and the table format is just a view. By leveraging UniForm, you stop being a caretaker of format-specific metadata and start being an architect of data flow.
Stop the migrations. Stop the dual-writing. Stop the "Iceberg vs. Delta" religious wars. Enable the compatibility layers, point your engines at the unified metadata, and spend your time building features that actually generate value for your users. In 2026, the best engineer in the room isn't the one who knows every obscure nuance of the Iceberg manifest spec; it’s the one who makes the entire storage layer invisible to the rest of the company.
Top comments (0)