Many contributors were still flying home from Community Over Code in Glasgow, and the lists picked up where the hallway talks ended. Iceberg wrote unstructured files into its v4 spec. Polaris argued about who owns those files once they sit behind a catalog. Parquet spent the week deciding what a vector is and what a format version means. The common thread was AI data: embeddings, files and semantic models all moved toward being first-class citizens of the open lakehouse.
Apache Iceberg
The biggest outcome was the passing vote to add a File type to the v4 spec. Talat Uyarer opened the vote for the FILE type working group. It closed with 8 binding +1s, from Steven Wu, Péter Váry, Daniel Weeks, Gang Wu, Sung Yun, Amogh Jahagirdar, Yufei Gu and Russell Spitzer, plus 19 non-binding +1s. The new type stores a range of bytes either inline or as a reference to an external object, with fixed sub-fields for URI, offset, size, content type and checksum. It mirrors Parquet's FILE logical type, so images, PDFs and audio can sit in a table column with real metadata and stats. Kurtis Wright voted +0. He argued that maintenance jobs need a signal for when a referenced object is safe to delete, or tables sharing a media bucket will step on each other. Daehong Jeon and Orest Tokovenko have already volunteered to implement the type in iceberg-rust and PyIceberg.
That vote followed the result of the baseline v4 AMT spec vote, which passed with 9 binding and 9 non-binding +1s. With the Adaptive Metadata Tree in the main spec, the follow-ups started at once. Anoop Johnson proposed extending "tight" stats beyond min and max to null, NaN and value counts, so an engine could answer count(*) WHERE col IS NULL from metadata alone. Kevin Liu asked whether v4 should be Parquet-only for data files as well as manifests. He pointed to V3 feature gaps in ORC and Avro and thin support outside Java. Péter Váry and Junwang Zhao pushed back hard on narrowing the spec, and Junwang named Vortex, Lance and Nimble as formats worth keeping the door open for. Kevin will move it to a dedicated thread.
AI-oriented types kept coming. In the File and Vector sync thread, Péter Váry proposed int8 and float16 primitives for vector indexes. Philipp Fischbeck wanted them only as vector element types, and Russell Spitzer suggested voting the Vector type in first. Péter also revived derived columns, and Szehon Ho suggested landing them through the secondary index spec. On the stats side, Yuya Ebihara proposed an HLL sketch blob type for Puffin. Theta sketches cost Trino 10-20% on inserts, and without a standard HLL type every engine invents its own. Yong Zheng also flagged that the REST spec is not valid OpenAPI 3.1, which he found while upgrading Polaris.
Apache Polaris
JB Onofré put Polaris Directories up for review. A Directory is a table-like entity over an object-store prefix, with its contents tracked in a regular Iceberg inventory table. Ankur Kotwal found a bug where the scanner can delete a file a commit may have already landed, and he wants it fixed before 1.9.0. Tornike Gurgenidze argued the inventory should be the source of truth, like Iceberg manifests, so Polaris could offer snapshot listings and time travel over files. Yufei Gu suggested phasing the feature rather than rushing it into the release JB has targeted for the end of October.
Qiegang Long shared a proof of concept for multi-table, multi-statement transactions, with Spark and Trino running BEGIN ... COMMIT across tables. JB welcomed it but set a process: design doc first, then small PRs, plus a persistence SPI so it does not depend on PostgreSQL. JB also asked how Polaris should support custom entities without patching core enums, since semantic models, tags, lineage and directories each touched about ten classes.
Process got attention too. Yufei Gu proposed an opt-in AI code review bot. JB backed it with labelling rules drawn from ASF guidance. Robert Stupp asked who pays, what data it keeps and why a second bot is needed next to Copilot. vignesh a reported that the bot missed context agreed on the list. Separately, the IRC Labels design settled on labels as a read-only projection of Polaris tags.
Apache Arrow
Raúl Cumplido announced Arrow 26.0.0, with 337 resolved issues. Arrow Rust 59.3.1 passed its vote with four binding +1s. Andrew Lamb then announced patch releases on four arrow-rs branches at once, a rare step for a project that normally ships forward. JB gave a heads up on Arrow Java 20.0.0, with a vote expected soon.
On the format side, Rusty Conover posted draft PRs across five languages so dictionary-encoded extension types survive IPC. The trigger was DuckDB failing to export a dictionary-encoded UUID column, and he plans a format vote in about a week. Tornike Gurgenidze proposed letting Flight SQL servers advertise their Substrait dialect, which ADBC would inherit. The PMC also welcomed Richard Baah as a committer.
Apache Parquet
The vector debate dominated. Jan Finis argued a finite vector type adds rules without new capability. Daniel Weeks and Gang Wu countered that logical types carry meaning, just as STRING promises valid UTF-8. Ryan Blue proposed two types: a finite vector for embeddings and a fixed-length list that allows NaN. Russell Spitzer backed the two-type path.
Alkis Evlogimenos laid out a concrete versioning proposal after auditing eight readers. Unknown encodings fail safely, but unknown page types and codecs can silently drop or corrupt rows in some readers. So encodings like ALP could ship in PAR1 files, while new page types or codecs would require a new compatibility level. Andrew Lamb and Divjot Arora were supportive. Daniel Weeks also rejected redefining distinct_count as an estimate, and Gunnar Morling contributed FILE type fixtures that parquet-java and arrow-rs both read correctly.
Apache Ossie
The incubating semantic interchange project was the busiest list of the week. Julian Hyde asked whether the project is truly open, calling Slack and working groups a walled garden. Chris Eubank agreed the point was fair and made it explicit that decisions happen on dev@. A run of SQL dialect PRs led Chris to propose making dialect a free-text field. The converters moved to their own repo, and JB is now preparing the first core release.
Cross-Project Themes
Unstructured data arrived at three layers at once. Parquet has a FILE logical type, Iceberg voted in a matching File column, and Polaris is designing Directories over raw prefixes. All three hit the same open question: who decides when a referenced object can be deleted, and which layer holds the truth. Kurtis Wright's +0 in Iceberg and Tornike's inventory argument in Polaris are the same concern from two angles. Expect a shared answer to emerge rather than three separate ones.
The vector debate also crosses projects. Parquet is defining the physical type while Iceberg debates element types like int8 and float16, and Will Edwards suggested Iceberg might be the better home for the finiteness contract. Meanwhile, two projects openly wrestled with AI in their own process, from Polaris's review bot to contributors disclosing LLM-assisted reports on Polaris and Parquet.
Looking Ahead
Watch for a Polaris 1.9.0 release candidate and whether Directories make the cut. Arrow Java 20.0.0 and ADBC 25 votes should land, along with Rusty Conover's dictionary extension format vote. In Iceberg, look for Kevin Liu's format thread and a Vector type vote. In Parquet, watch whether the two-type vector proposal and the versioning design move toward a vote.
Resources & Further Learning
Get Started with Dremio
- Try Dremio Free: Build your lakehouse on Iceberg with a free trial
- Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow: Learn how Dremio brings the open lakehouse stack together
Free Downloads
- Apache Iceberg: The Definitive Guide: O'Reilly book, free download
- Apache Polaris: The Definitive Guide: O'Reilly book, free download
Books by Alex Merced
Top comments (0)