Every vendor blog post I've read this year about Apache Iceberg v3 says the same thing: faster deletes, upgrade now. Snowflake moved v3 to GA on May 7. Databricks shipped support in Runtime 18.0. Apache Iceberg 1.11.0 landed May 19 and declared the spec production-stable. AWS followed over the summer with Glue 6.0, Redshift, and S3 Tables all claiming v3 readiness.
The marketing is loud. The substance is real but narrower than the press releases suggest. I've been digging through the actual spec changes, testing engine support matrices, and cataloging the gotchas nobody puts in the blog title. Here's what working data engineering teams actually get, and where the gaps are.
Deletion Vectors Solve a File Problem
The headline feature of Iceberg v3 is deletion vectors, and the way vendors frame them has been consistently misleading. Every blog leads with "10x faster deletes." The real win is architectural, and understanding the architecture is what separates a surface-level take from a staff-level one.
In v2, every delete operation produced a separate positional delete file. Delete 500 rows across 1,000 data files and you've spawned 1,000 new delete files in your metadata layer. Run CDC pipelines that touch millions of rows daily, and those delete files multiply fast. Read queries then had to join each data file against its corresponding delete files to figure out which rows were still alive. That join overhead is where read performance died.
v3 replaces positional delete files with Roaring bitmaps stored in Puffin sidecar files. Each data file pairs 1:1 with a single deletion vector. A transaction deleting rows across 1,000 data files packs all 1,000 deletion vectors into one Puffin file. At read time the engine checks a bitmap, O(1) lookup per row, instead of joining against a pile of delete files.
Dremio's testing showed 50 to 80 percent read improvement when deletion vectors replaced v2 positional deletes. Merge-on-Read workloads under high churn saw 5 to 10x speedups. Real numbers. But they measure something specific: the elimination of join overhead between data files and delete files. A table that rarely sees deletes barely notices the upgrade.
The performance gain from deletion vectors comes from killing file sprawl. If your v2 tables accumulate hundreds of positional delete files per day from CDC, v3 is an immediate quality-of-life upgrade. If they don't, the benefit is real but modest.
The economics reinforce this. Every v2 delete file was an S3 object. Every read that referenced those files issued GET requests. Multiply that across a CDC-heavy lakehouse and the API charges add up fast. v3 collapses all of it into bitmaps. Storage is 2 cents a GB; the S3 GET bill at scale is where the real money hides.
Compaction is still mandatory. Deletion vectors shrink the metadata cost of tracking dead rows, but the underlying data files those rows live in don't clean themselves up. You still need periodic compaction to reclaim space. The operational discipline hasn't changed; the file-level mechanics got cleaner.
Row Lineage: Native CDC at the Format Level
The other big v3 addition is row lineage, and this matters for anyone building change data capture on an open table format.
v3 tables carry 2 implicit metadata fields on every row: _row_id (a unique 64-bit identifier assigned at first write, surviving updates, compactions, and partition changes) and _last_updated_sequence_number (the snapshot sequence when the row was last modified). The AWS prescriptive guidance describes it as "engine-agnostic and interoperable, built into the Iceberg V3 specification, alleviating the need for custom, engine-specific change tracking implementations."
Before v3, CDC on Iceberg meant full-table diffs or bolting on external tooling to detect changes. I've built those pipelines. They work, in the same way duct tape on a leaking pipe works. Row lineage makes change detection a filter predicate: query rows where _last_updated_sequence_number exceeds your last checkpoint and you have your delta. No full scans. No external watermark tables.
The theory is clean. Production is messier.
After upgrading from v2 to v3, existing rows carry _row_id = null until they're rewritten via compaction or update. Your CDC pipeline needs null handling during the transition, or you schedule a full compaction pass post-upgrade to backfill lineage across the table. Multiple migration guides call this out explicitly: compaction after a v2-to-v3 upgrade is required for homogeneous lineage coverage.
The other catch: _row_id and _last_updated_sequence_number aren't returned by SELECT * on all engines. StarRocks and Presto have open GitHub issues where these columns require explicit SELECT or silently drop rows with null lineage values. ClickHouse has a known bug where filtering on lineage columns omits pre-upgrade rows entirely. If your read path runs through anything outside the Spark/Databricks/Snowflake core, test before you depend on it.
VARIANT Replaces the JSON String Hack
Every data engineering team I've worked on has jammed semi-structured data into a VARCHAR column as a JSON string at some point. It works. The query performance is awful, the storage efficiency is bad, and whoever has to parse that column downstream curses your name. But it ships.
The VARIANT type in v3 stores semi-structured data in binary encoding that preserves native types. A date field inside a VARIANT is typed as a date, not a string that looks like one. Snowflake's benchmarks on 500-million-row tables showed 31.4% space savings over JSON strings and a 2.3x total query speedup.
The mechanism behind the performance is shredding: at write time, frequently occurring fields get extracted into typed Parquet columns alongside the binary VARIANT blob. The query engine pushes predicates to those shredded columns, applies row-group and page-level skipping, and only decodes the full VARIANT binary for rows that pass filters. You get the flexibility of schema-on-read with the performance you'd expect from typed columnar storage.
Shredding carries roughly 35% write overhead. Streaming-heavy or write-dominated workloads might still prefer VARCHAR(JSON) and eat the query cost. The optimization pays off on write-once, read-many patterns. Know your access patterns before flipping the switch.
Rounding out the v3 spec: default column values, geometry and geography types, nanosecond timestamps, and multi-argument partition transforms. Useful additions. But deletion vectors, row lineage, and VARIANT are the 3 features that change pipeline design.
The Platform Write Gap Nobody Advertises
Here's where the distance between announcement and reality gets uncomfortable.
Snowflake's v3 is GA, but new Snowflake-managed Iceberg tables still default to v2. Opt-in required. In-place v2-to-v3 upgrades? Not supported on Snowflake. You're looking at full table recreation through export and reimport. External engine write support didn't ship until May 26, 19 days after the v3 GA announcement, and it only covers Snowflake-managed tables. That's a narrower story than the press release tells.
Databricks has the most complete v3 support today. Runtime 18.0 enables deletion vectors and row lineage by default on new v3 tables in Unity Catalog. If you're Databricks-native, the path is clear.
AWS is fragmented. Athena can't read v3 tables at all; the engine throws "Cannot read unsupported version 3." If Athena is your primary analytics engine, v3 blocks 100% of your analytical queries on any table you upgrade. Redshift supports v3 but silently changes timestamp mappings and drops support for VARIANT, STRUCT, LIST, MAP, and GEOMETRY. Glue 6.0 shipped v3 in August, but enabling VARIANT disables Lake Formation fine-grained access control and managed compaction, and it's only available in 15 regions.
The Python ecosystem is the quiet bottleneck. PyIceberg reads v3 tables but can't write them. ML and data science teams that ingest through Python are stuck on v2 for writes. Open-source Trino doesn't support v3 either; only the proprietary Starburst Galaxy fork does. The spec is converging. The implementations are not. That gap defines the operational reality for the next 12 months.
Interviews, Migration, and What to Do About It
v3 internals are already landing in hiring loops. If you're interviewing at companies running lakehouse architectures, expect questions on how deletion vectors differ from positional deletes, how row lineage enables CDC without full scans, and when merge-on-read beats copy-on-write. These are concepts questions, and concepts transfer across engines. Knowing the Databricks-specific API for enabling deletion vectors doesn't tell an interviewer much; understanding why bitmaps replaced positional deletes does. If you're prepping for these conversations, that's exactly why we made sure data engineering practice problems are covered on datadriven; v3 architecture is the kind of topic where understanding the mechanics matters more than memorizing syntax.
My actual advice on migration: don't rush. If v2 is stable and your workloads are healthy, v3 is worth planning for on your own schedule. Map your engine dependencies first. If Athena or PyIceberg sits in your critical path, you're blocked until those engines catch up. If you're Databricks-native or running Spark on EMR, start migrating non-critical tables and see how compaction, lineage coverage, and downstream consumers behave before you touch anything finance depends on.
The open table format convergence story looks great in keynotes. The engine support matrix tells a messier story. Apache Iceberg v3 delivers real improvements to how lakehouse architectures handle deletes, CDC, and semi-structured data. The spec earned its GA label. Whether your entire stack can use it today is a separate question, and the answer depends more on your engine mix than on the spec itself.
What's your team's v3 timeline? Blocked on engine support, running it in production, or still on v2 and not feeling any pressure to move?
Top comments (0)