It was 3:00 AM on a Tuesday when the PagerDuty alert for our primary billing pipeline hit. We had a job failing on a Spark 3.5 cluster because our upstream vendor decided to switch their export format to Iceberg, while our entire analytical stack was locked into Delta Lake 3.0. The migration cost us six hours of downtime and roughly $45,000 in SLA penalties. The issue wasn’t the data quality; it was a religious war between two metadata layers that refused to speak the same language.
We treat table formats like sports teams, but they’re just protocols. You don’t need to pledge allegiance to Databricks or the Apache Software Foundation. You need your data to be readable by the specific query engine that actually does the job, whether that’s Trino, DuckDB, or Spark.
The illusion of engine-agnosticism
Most engineers assume that because they use an open format, they are portable. That’s a lie. If you write a Delta table, you are tethered to the Delta Standalone reader or a Spark implementation that supports the Delta protocol. If you use Iceberg, you’re at the mercy of the Iceberg Catalog and its specific manifest file structures.
The thing we rarely look at is the metadata root. In Delta, it’s a directory of JSON logs (_delta_log/). In Iceberg, it’s a snapshot-based tree structure starting from a metadata file that points to manifest lists. They are fundamentally different ways of tracking state, yet they both aim to solve the same problem: atomic ACID transactions on top of a pile of Parquet files.
Photo by Paolo Chiabrando on Unsplash
How it actually works
UniForm (Universal Format) is the bridge. Specifically, Delta Lake UniForm allows you to write in Delta format while the engine automatically generates the Iceberg metadata in the background.
When you enable delta.universalFormat.enabledIceberg in your table properties, you aren't just tagging metadata; you are triggering an asynchronous background process that translates the Delta log into Iceberg manifests.
Here is what that looks like in a Spark session:
ALTER TABLE my_production_table SET TBLPROPERTIES (
'delta.universalFormat.enabledIceberg' = 'true',
'delta.iceberg.catalogName' = 'my_hive_catalog'
);
Behind the scenes, Delta is essentially running a translation layer. Every time a commit happens in the _delta_log/, a background Spark job (or the writer itself, depending on configuration) maps those file additions and removals into the Iceberg snapshot format.
For the end user, this is magic. You point your Trino or Starburst cluster at the same S3 prefix you use for Spark, and Trino sees a perfectly valid Iceberg table. You aren't duplicating the data files—the Parquet files remain identical. You are only duplicating the metadata pointers.
The tradeoffs nobody mentions
If this sounds too good to be true, it’s because it involves operational "hidden" costs.
First, the background translation job is not free. If you are doing high-frequency streaming writes (e.g., every 30 seconds), the overhead of keeping the Iceberg metadata in sync can cause write latency spikes. I’ve seen commit latencies jump from 200ms to over 2 seconds because the cluster had to lock and update both the Delta log and the Iceberg snapshot history.
Second, version skew is a real failure mode. If your Delta version is 3.2, but the Iceberg translation logic is lagging behind in the current library version, you might end up in a state where the table is readable, but "time travel" queries fail. I once debugged a case where SELECT * FROM table AS OF VERSION AS OF '2026-05-01' worked in Spark but returned a TableNotSupported error in Trino because the Iceberg manifest was missing the specific partition evolution metadata that Delta had handled natively.
Finally, you are doubling your storage metadata footprint. In a multi-petabyte environment, the _delta_log and the metadata/ directory for Iceberg will grow to millions of files. If your object store has high latency on LIST operations, your catalog discovery will eventually become the bottleneck, not the data retrieval itself.
Photo by Pankaj Patel on Unsplash
When to reach for it (and when not to)
Use UniForm if your organization has a split-brain architecture. If you have a legacy Spark-based heavy processing pipeline but your analysts insist on using Trino or Snowflake for ad-hoc exploration, this is the only way to avoid the maintenance nightmare of double-writing data.
Do not use UniForm if your primary goal is "future proofing" without a clear current need. If your entire stack is already Spark-native, UniForm is just adding complexity and potential points of failure. Stick to pure Delta. If your stack is fully integrated with Iceberg-native tools like Tabular or Nessie, don’t introduce Delta just to "bridge" things.
The decision comes down to your query engine requirements. If your data must survive a lift-and-shift from a proprietary Databricks environment to a self-managed Trino cluster, UniForm is a lifesaver. If you are a startup with a single query engine, you’re just paying for extra compute cycles to translate metadata that nobody is reading.
Conclusion
We are moving into an era where format-lock is becoming a legacy burden. By using Delta as your primary writer and UniForm to project Iceberg metadata, you get the robust ecosystem support of Delta and the engine interoperability of Iceberg.
Stop worrying about which company’s "standard" wins the market. Focus on the metadata translation layer that keeps your data accessible. The goal isn't to pick a side; the goal is to ensure that when your primary query engine goes down, you can pivot to another one without having to rewrite your entire data lake. In 2026, the only real technical debt is a siloed format.
Tags: #data #engineering #delta #iceberg
Cover photo by Taylor Vick on Unsplash.
Top comments (0)