Release managers had a humbling week. Iceberg Rust 0.11.0 failed two release votes, Polaris 1.8.0 drew a -1 on its first day, and the Polaris Catalog Migrator needed a second candidate. None of those failures came from broken code paths in the usual sense. They came from license notices, a crate that failed to publish, and an encryption check that one engine skipped and another engine enforced. The lakehouse projects are now mature enough that the hard problems live at the seams: between languages, between engines, between a spec and the many readers that implement it.
That theme runs through every project this week. Iceberg and DataFusion voted to move the Rust DataFusion integration into the DataFusion project, which redraws a boundary that had been drawn in the wrong place. Parquet spent the week arguing about what a reader should do when it meets a file from the future. Polaris debated how to add Cloudflare R2 without bending its S3 configuration into a shape it was never meant to hold. Apache Ossie (incubating) proposed that one document should hold one semantic model, which is a seam question too. And Arrow shipped a Rust release that puts the new ALP floating point encoding into a production Parquet implementation for the first time.
This issue covers the dev lists for Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, Apache DataFusion, and Apache Ossie (incubating), using the Pony Mail archives at lists.apache.org. Every thread link below points to the public archive so you can read the full discussion yourself.
Apache Iceberg
The DataFusion integration finds a new home
The biggest structural change of the week was a clean two-project vote. Kevin Liu opened the Iceberg-side vote to move the iceberg-datafusion crate out of apache/iceberg-rust and into a new apache/datafusion-iceberg repository under the Apache DataFusion project. The move grew out of a discussion thread where Andrew Lamb and Kevin agreed to hold one vote in each community. Gabriel Musat had already done the heavy lifting by porting the code and its full history into a pull request on the new repository.
The Iceberg vote passed with 17 +1 votes, five of them binding, from Kevin, Fokko Driesprong, Renjie Liu, Szehon Ho, and Russell Spitzer. There were no -1 or +0 votes. Xuanwo summed up the reasoning in his +1: DataFusion contributors get a natural place to maintain the integration, and the code stays under Apache governance. On September 16, Andrew confirmed on the same thread that both votes had passed unanimously.
Why does this matter to people who never read Rust? The integration is the glue that lets a DataFusion query plan read and write Iceberg tables. When it lived inside iceberg-rust, every DataFusion upgrade forced a change in the Iceberg repository, and the people who understood DataFusion's planner internals were not the people reviewing the pull requests. Putting the crate next to the engine it plugs into puts the reviewers next to the code. It also sets a pattern. Integration code belongs with the project whose APIs churn fastest, and the table format library can stay focused on the format.
Iceberg Rust 0.11.0 fails twice, and the failures teach something
Danny Jones and Shawn Chang ran the 0.11.0 RC1 vote, and early verifiers reported clean results. L. C. Hsieh ran 2,209 tests on an arm64 Mac with no failures. Anoop Johnson ran the full suite on Ubuntu 24.04, including Docker-based integration tests against MinIO, the REST catalog, Hive Metastore, and Spark. He flagged one flaky test that failed under parallel load with a SQLite "database is locked" error but passed in isolation.
Then Kevin Liu voted -1 for a reason no unit test catches. The new iceberg-property-macro crate depends on nothing unusual, but it lists iceberg itself as a versioned dev-dependency, and iceberg depends on the macro crate. That circle means cargo publish cannot package either crate first. Kevin pointed out that the project hit the same trap with 0.5.0, when the vote passed, the publish failed, and the team had to cut 0.5.1. He put up a fix and added a cargo publish --dry-run check to CI so the problem surfaces on every pull request. He also noted that make test from the tarball failed because the Hive test image's apt repository had expired. Danny agreed and cancelled the vote.
The RC2 vote opened on September 15 and lasted less than a day. Alexander Bailey voted -1 after testing and realizing that the Rust encryption work skipped the tamper-proofing checks that Java requires. Files written by the Rust RC were unreadable in Java. He posted a fix in iceberg-rust PR #3236. Danny closed RC2 and asked reviewers to prioritize the fix so RC3 can follow.
This is the multi-language Iceberg story in miniature. Encryption is one of the headline features in 0.11.0, and the only reliable test for it is a cross-engine round trip. A Rust writer that passes every Rust test can still produce files that a Java reader rejects. The community caught it during the vote, which is the process working. But it also makes the case for the next item.
A shared verification repository goes live
Neelesh Salian announced that the apache/iceberg-verification repository now exists, following an earlier discussion and a successful vote. The repository will hold shared conformance fixtures for every Iceberg implementation. Early work covers type fixtures validated with JSON Schema and golden reference tables. Neelesh asked for reviewers on the open pull requests and for new issues from anyone with ideas about the fixture format or contribution model.
Read that announcement next to the two failed Rust votes. A golden table written by Java and read by Rust, Go, C++, and Python, plus the reverse, is exactly the test that catches an encryption gap before a release candidate exists. The verification repository is still young, but it points at the right problem.
Third-party notices in compiled wheels
Danny Jones raised a licensing gap in pyiceberg-core, the Python package built from iceberg-rust. The wheels ship compiled third-party Rust code, but they do not reproduce the copyright notices for those dependencies. Danny opened iceberg-rust issue #3239 with a proposed fix that generates a THIRD-PARTY-LICENSES file at build time, and he plans to confirm the approach on the legal-discuss list.
Ryan Blue agreed that anything distributed in compiled artifacts needs matching LICENSE and NOTICE content. He was less sure about the THIRD-PARTY-LICENSES file name, but said a pointer at the bottom of LICENSE is reasonable, and a NOTICE file with the legally required notices is mandatory. Shawn Chang pointed to Apache Paimon's Rust bindings as a working example. Their workflow scopes dependencies from the Python bindings' Cargo.toml, generates a report per target wheel, stages it in the artifact, and verifies it. Danny said this must be fixed before new releases of pyiceberg-core, and by extension iceberg-rust, since the two ship together.
Iceberg Java 1.12.0 is ready to cut
Neelesh Salian gave the 1.12.0 release thread two updates. On September 12 he reported that all prior correctness fixes had merged, with one exception: PR #17984, which commits manifest list encryption keys together with the snapshot that uses them. Gábor Kaszab asked whether the missing cleanup mechanism for unused encryption keys in TableMetadata (PR #16353) belongs in this release. On September 16 Neelesh said #17984 had merged, that the key cleanup interface needs more debate and does not need to be rushed, and that the release candidate was ready to cut.
Alex Reid replied on September 17 asking for two more items. One is a Kafka Connect fix (PR #17713) that prevents re-committing already committed files during certain rebalances. The other is the REST client implementation of referenced-by, which the spec added last year. As of this writing, 1.12.0 has not been released. Watch the list for the RC vote.
Defining what gc.enabled means
Alexander Bailey started a careful thread about the gc.enabled table property. His PR #17791 relaxes the check so that snapshot expiry can proceed in a metadata-only mode when garbage collection is disabled. Reviewers pushed back, and Alexander concluded that the real issue is that no one has defined the property. It is not in the spec. He catalogued at least three meanings inside the Java repository alone. CatalogUtil.dropTableData skips data files but deletes metadata. Several Spark actions refuse to run at all. The Hive helper maps the flag onto Hive's external table purge setting. He then showed that iceberg-rust, iceberg-go, and iceberg-cpp each copied Java's strictest check, one of them word for word, while PyIceberg has no check at all.
Russell Spitzer explained the history. The flag arrived when Iceberg started snapshotting existing data, where Iceberg owns new metadata but another system still owns the data files. The invariant is simple: do not remove this table's data files. He was happy to write that down in a spec appendix, and he did not see the Java behavior as ambiguous, since every operation that fails or skips work is one that can delete a data file. Daniel Weeks took a different position. He sees gc.enabled as convention rather than specification and prefers to rely on catalogs to enforce deletion rules than on clients to behave.
Alexander asked Dan whether that means catalogs should vend credentials without delete permission and leave deletion to catalog-managed maintenance. He then narrowed his PR. When gc.enabled=false, the PR allows expiry only with CleanupLevel.NONE, which removes snapshot entries without touching any file. The default mode and METADATA_ONLY stay blocked. That is a sensible scoping move. It keeps the data-file invariant intact and leaves the bigger enforcement question for a separate discussion.
Faster position delete checks in the vectorized reader
A contributor writing as 전대홍 posted a performance proposal after Eduard Tudenhöfner suggested taking it to the list. Spark's vectorized reader asks the position delete index whether each row is deleted, one row at a time, while building the row-id mapping for a batch. Positions in a batch form a contiguous range, so every call repeats the same key extraction, bounds check, and binary search through the Roaring bitmap containers.
PR #18027 adds a forEachInRange default method to PositionDeleteIndex. The default keeps the current loop, so external implementations keep working. BitmapPositionDeleteIndex overrides it to resolve the range to at most two underlying bitmaps and walk each once. The contributor measured 2.6x to 9.3x less delete-check CPU, with end-to-end gains of 13 to 19 percent on narrow integer projections and noise-level gains at ten or more columns. They noted that the largest win lands just below 6.25 percent delete density, where the Roaring container is still a sorted array.
Péter Váry suggested trying a batch iterator in the style of RoaringBitmap's batch iteration. The contributor built it and benchmarked six variants on a 5,000-row batch. The PR's approach came in at 0.45 microseconds per batch on sparse deletes, against 64.99 for the current code and 0.63 for the batch iterator. They kept the minimal API change and offered to test a persistent iterator if the community prefers that design.
Row-level concurrency for V3 deletion vectors
EJ Song and Huaxin Gao continued a thread on row-granular conflict detection for deletion vectors. Today, two concurrent UPDATE or MERGE commits that touch the same data file conflict even when they delete different rows. Huaxin supported the direction but warned that the extra check adds driver-side reads during commit, and suggested gating it behind a table property. She pointed to prior art, including Iceberg PR #17754, which already merges concurrent deletion vectors for pure delete commits.
EJ laid out a layered design. An operation gate comes first, then a metadata check on partition and column bounds, and only then a single small bitmap read when two commits really overlap on one file. Commits on disjoint files read nothing. EJ argued that the bitmap tier does not need a table property because its cost is already bounded. A later tier that compares actual values to resolve overlaps a bitmap cannot decide reads data files, and that tier is where a property makes sense. This is the kind of change that makes V3 tables far friendlier to high-concurrency streaming and CDC workloads.
Access delegation for the FILE type
Sung Yun opened a discussion on access delegation for the proposed FILE type, which lets an Iceberg column reference external objects such as images or documents. Remote signing and vended credentials work when a storage client reads the object. Sung's concern is the multimodal case, where an inference service outside the query engine needs to fetch the object. He proposed letting clients ask the catalog to pre-sign FILE reference URLs, with a spec PR (#18080) and a client proof of concept (#18110). He had earlier posted a related implementation that handles pre-signed URLs returned in place of native paths in a plan response.
Prashant Singh noted that he and William Hyun opened a pre-signed URL proposal three months ago. Their proofs of concept, including one on Azure, showed limits in remote signing across clouds. He asked to build on those tracks rather than start parallel ones, and Sung agreed. Daniel Weeks drew a useful line. Pre-signed URLs from plan tasks arrive ready to use, so file IO only needs to detect and follow them. Remote pre-signing is an earlier step, where a native s3:// path goes to the catalog for signing before a stream opens. Dan wants remote pre-signing to become a real alternative to remote signing, largely because of Azure's limits, and does not think it needs a new /presign endpoint. He also said he disagrees with the direction of the current bulk-signing proof of concept and wants the FILE sync to align the designs.
Smaller threads worth a look
Shangqing Yang called a vote to add the optional key-id field to the Snapshot schema in the REST OpenAPI spec, which aligns the spec with what SnapshotParser already supports. Sreesh Maheshwar revived the question of replacing MinIO in tests after MinIO images disappeared from Docker Hub and broke CI. Kevin Liu, Xuanwo, and Szehon Ho backed moving to RustFS, which PyIceberg and Polaris already use. Neelesh Salian asked for long-term support and prompt CVE fixes, citing an earlier security concern.
Renjie Liu gave a +1 to a proposal for a standard User-Agent format for REST clients, and Kurtis Wright asked how the format should order vendor, engine, integration, library language, and runtime. A contributor asked for review of an ObjectStore-based S3 backend for iceberg-rust in PR #3165. Péter Váry reported from the index support sync that the group will prioritize scalar indexes before other index types, that index data lives in region files tracked by a tracking file, and that the scalar index spec PR #16961 is ready for review. Huaxin Gao scheduled one more constraint sync for September 17 and asked for reviews of the CHECK constraint spec PR #17822.
Ryan Blue shared the September board report with a new format that summarizes shipped releases and spec changes instead of sync highlights. It lists 40 committers and 25 PMC members, and releases including Terraform Provider 0.1.0, PyIceberg 0.12.0, Rust 0.10.0 and 0.10.1, and C++ 0.3.0. Ryan asked release managers to write real highlights in their vote and announce emails, and Danny Jones committed to doing that for iceberg-rust. On September 17, Fokko Driesprong announced that Gang Wu has joined the Iceberg PMC, crediting Gang's work on the inception and growth of Iceberg C++.
On the community side, Kevin Liu announced the first Apache Iceberg Virtual Meetup, covering GSoC projects and commutative compaction on Friday, September 18. Alex Stephen announced a Seattle community meetup on October 21, with session ideas due October 6. Sung Yun, Walaa Eldin Moustafa, and others volunteered for the Iceberg Summit 2027 selection committee.
Apache Polaris
Polaris 1.8.0 reaches a vote, and a NOTICE question stops it
Jean-Baptiste Onofré opened the vote for Apache Polaris 1.8.0 RC0 on September 17, after flagging on the proposal thread that he was preparing it. The candidate ships source tarballs, Helm charts, a Python CLI wheel on Test PyPI, and staged Maven artifacts. Within hours, Dmitri Bourlatchkov voted -1. Bundled Iceberg jars such as iceberg-api-1.11.0.jar carry a NOTICE that references Kite, and commons-math3-3.6.1.jar carries one that references Orekit. The Polaris bundle NOTICE does not propagate either. Dmitri asked whether those notices belong in the Polaris bundle, and the vote is open as of this writing.
The same pattern hit the Polaris tools repository. Ajantha Bhat's Iceberg Catalog Migrator 1.1.0 RC1 drew a -1 from Dmitri because the CLI jar bundles a taglib.tld file with an Oracle copyright header under GPL v2 with the Classpath Exception, and the bundled LICENSE does not mention it. Ajantha posted RC2 on September 17.
If you are keeping score, that is four release candidates across Iceberg Rust and Polaris this week that stopped on licensing or packaging, not on logic. Fat jars and compiled wheels pull in other projects' legal text, and the ASF requires it to travel with the artifact. The reviewers doing this checking are doing unglamorous, important work.
Cloudflare R2 support and the shape of S3 configuration
The most instructive design thread on the Polaris list started with a first-time contributor. Austen Tomek from Chicago Trading Company proposed R2 support with scoped credential vending. R2 speaks the S3 API but has no AWS STS. Cloudflare instead lets a server holding a parent API token sign short-lived, bucket-scoped credentials locally. Austen's prototype does that inside Polaris, maps Polaris location grants to prefix and access scopes, and reuses the existing credential cache.
Yufei Gu confirmed the model: Polaris issues the credentials without calling Cloudflare, the client signs its own S3 requests, and R2 enforces scope and expiry. His first instinct was to treat R2 as S3-compatible storage without a new config type, and he asked whether unmodified clients work. Austen reported that they do. PyIceberg 0.11.1 and 0.12.0, DuckDB 1.5.5, and Iceberg Java 1.11.0 through the stock RESTCatalog and S3FileIO all read and wrote without any R2-specific client code. Spark, Trino, and Flink are not tested yet.
The debate then split along a clear line. Sushant Raikar laid out two options, a first-class R2 type with its own FileIO or R2 as S3-compatible storage, and pushed for a lowest common denominator across S3-compatible backends. Jean-Baptiste argued for building under the existing S3 config and noted that AwsStorageConfigurationInfo already carries endpoint, stsEndpoint, pathStyleAccess, and an stsUnavailable flag. He framed the real design question as a pluggable vending path for S3-compatible stores without STS. Dmitri leaned toward a separate R2StorageConfigInfo, since mixing Cloudflare account IDs into an AWS-shaped class blends unrelated concepts.
Prithvi S offered the cleanest synthesis. Keep the client side as plain S3, with s3:// locations and S3FileIO. On the server side, avoid overloading stsUnavailable, because that flag means "vend nothing" today, and R2 needs the opposite: vend, just not through STS. Reusing it silently changes behavior for existing MinIO and NetApp catalogs.
Austen closed the loop on September 14 and again on September 16. R2 has no IAM roles or ARNs, so he left those fields empty. He added one new S3 field, now named credentialVendingMechanism, with a server allowlist, SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, that defaults to STS only. Following Dmitri's suggestion, mechanisms are CDI beans looked up by identifier, so downstream builds can add their own without changing the API spec. Stage one is PR #5513, and the R2 bean follows in a second PR. Yufei noted the underlying mismatch: the S3 storage type mixes shared S3 settings with AWS-specific ones, and a future refactor can split them. This is a good template for how to add a backend to a catalog without special-casing it.
Pagination moves from optional to default
Pagination came up in four threads. Yong Zheng opened separate discussions for the generic table API and the policy API. In both cases the REST spec advertises page-token and page-size, but the server calls PageToken.readEverything() and ignores them. For generic tables, Prithvi and Jean-Baptiste steered toward a default method on the GenericTableCatalog SPI, since federated Hive, Hadoop, and BigQuery catalogs implement that interface and a concrete-class overload is unreachable from the handler. Yong posted PR #5533. For policies, the group agreed to paginate listPolicies now, including when filtering by policy type, and to handle applicable-policies separately.
Ayush Saxena raised a spec tension around a configurable server-side maximum page size. The Iceberg REST spec says a server must return everything in one response when the client sends no page token. A server cap truncates that response, and an older client that ignores next-page-token reports a partial list as complete. Dmitri called server-side truncation a basic overload control and a reasonable deviation, since administrators can choose the setting per deployment. He proposed merging.
Then Yufei proposed turning LIST_PAGINATION_ENABLED on by default. Jean-Baptiste noted that the maximum page size defaults to unlimited, so clients that send no pagination parameters see no change. Ayush, Yong, Nándor Kollár, and Dmitri all backed keeping the flag as an escape hatch for a couple of releases, and Yong opened PR #5534. EJ Wang raised the same question for the new Tag API, which is covered below.
Read replicas, and the consistency they break
Yong Zheng started a thread on offloading read-only requests to a database replica when a client sends a custom header. His motivation is connection math on the JDBC backend: at 50 connections per pod and a 5,000-connection database limit, a deployment tops out near 100 pods.
The pushback was about correctness. Prithvi pointed out that the load test Yong cited ran at roughly 400 requests per second sustained with mostly idle connections, so connection limits were not the bottleneck. He also reminded the group that Iceberg clients expect to read their own writes, and asynchronous replicas break that. Jean-Baptiste argued against header-based routing, since Spark and Flink will never send a Polaris-specific header, and against verb-based routing, since some reads in JdbcBasePersistenceImpl have side effects such as idempotency-key bookkeeping. He suggested routing at the persistence method level. Dmitri was more open to an explicit opt-in header, perhaps renamed to make the staleness risk obvious, and warned that the shared entity cache will misbehave with two connection pools. Yufei asked for measurements first: connection wait times, database query time, and metadata file IO, which often dominates loadTable.
Location overlap checks
Dmitri Bourlatchkov opened a discussion on hasOverlappingSiblings(). With the OPTIMIZED_SIBLING_CHECK flag on, persistent implementations search the whole catalog. With it off, LocalIcebergCatalog searches only immediate siblings. Dmitri argued that a speed-up flag should not change what a validation checks, and proposed narrowing the implementations. Prithvi and Eric Maynard disagreed. Eric recalled that the catalog-wide search was an intentional fix for a security gap under certain configurations. Jean-Baptiste identified the concrete case: with ALLOW_UNSTRUCTURED_TABLE_LOCATION, a table under one namespace can point inside another namespace's tree, and only the catalog-wide check sees it. He prefers a confusingly named flag over a correctly named, weaker check. Dmitri agreed with catalog-wide checks and asked why that coverage is not the default.
Tags, semantic model privileges, and the CLI
EJ Wang posted progress on the Tag spec. PR1 covers the API contract, PR2 covers definition CRUD, and a new PR3 adds assign and unassign for catalogs, namespaces, tables, and top-level columns, with atomic detach-all on JDBC and in-memory backends. Version tokens are opaque, and stale updates return 409. Dmitri asked for persistence SPI details and a dedicated section on new authorizer operations. EJ later asked the community to pick between keeping Tags under the existing catalog URL root or giving them an independent /api/tags/v1 root, and between opt-in and always-on pagination.
Yufei Gu asked for feedback on dedicated semantic model privileges in PR #5492, splitting the single CATALOG_MANAGE_CONTENT privilege into list, create, read, write, drop, full metadata, and grant management. Prithvi supported it and caught that NAMESPACE_FULL_METADATA and CATALOG_FULL_METADATA should not act as umbrellas, and Yufei fixed that. Jean-Baptiste strongly backed dedicated privileges that mirror other entity types. Semantic models in a catalog, with their own grants, connect directly to the Ossie work later in this issue.
A GitHub discussion asking for table and view support in polaris setup export/apply became a CLI thread. Yong proposed adding register and create. Prithvi argued that register, which takes a name and a metadata.json location and preserves history, is the primitive operators need. Jean-Baptiste found that the generated REST client already exposes register, so the change is mostly wiring. Dmitri and Yufei agreed that create is outside the CLI's scope for now. The sequence the group settled on is register first, then export and apply with table and view grants, and file-based create later if anyone needs it.
Rounding out the week, Dmitri put consistent multi-object persistence changes on the September 17 community sync agenda. Arun Suri agreed to defer his ambiguous JDBC commit fix until PR #5263 merges, then add a reconciliation step that reloads the entity and treats a matching metadata location as success. And Yufei, Ayush, Robert Stupp, and Jean-Baptiste discussed GitHub Actions queue delays caused by ASF-wide runner saturation. Ayush tested which short jobs can move to ubuntu-slim, and the group decided to hold off unless delays return.
Apache Arrow
Arrow Rust 60.0.0 ships the first ALP-enabled Parquet implementation
Andrew Lamb called the vote for Arrow Rust 60.0.0 on September 10, and it passed on September 15 with five +1 votes, three binding, from L. C. Hsieh, Jeffrey Vo, and others. Andrew did not undersell it. He called the release "epic" and wrote that arrow-rs is, as far as he knows, the first Parquet implementation to include the new ALP encoding. He credited Kosta Tarasov and Devan for leading that work, Ed Seidl for the new Parquet PageIndex structures, and Richard Baah and others for a long list of performance improvements. The crates, including arrow and parquet 60.0.0, are on crates.io.
ALP, short for Adaptive Lossless floating-Point compression, targets float and double columns, which general-purpose encodings handle poorly. Sensor readings, prices, and model features all live in those columns. Having a shipped implementation in Rust matters because DataFusion, iceberg-rust, and a long list of Rust-native engines consume the parquet crate directly.
Xuanwo's review of the release is worth reading for anyone who runs release votes. He first voted +0 because arrow-array/src/delta.rs contains MIT-licensed code derived from chronoutil, and the root LICENSE.txt does not identify it. After another look he decided a NOTICE mention is enough and changed to +1. Andrew filed issue #11098 to track the fix for the next release.
Object Store 0.14.2 needs a second candidate
The Rust object_store crate had a similar week. Andrew's 0.14.2 RC1 drew a -1 from Xuanwo because src/client/s3.rs was missing the first line of its ASF license header. Andrew traced the defect back more than three years and asked whether it needed a new candidate, since it was not a regression. Xuanwo moved to +0 but noted the vote had just started and a wrong license header is a real risk for downstream users. Andrew cut RC2 with the header restored, and it passed on September 15 with three binding votes. Version 0.14.2 is on crates.io. object_store sits under DataFusion, iceberg-rust, and many other Rust data tools, so its releases propagate quickly.
A canonical range type and Lakehouse Day EU
Rok Mihevc revived the proposal for an arrow.range canonical extension type for bounded ranges. Florian replied that he still wants to finish it and addressed Felipe Oliveira Carvalho's earlier suggestion to use separate types for open and closed bounds. After reading how Arrow C++ resolves compute kernels, Florian concluded that separate types help less than they seem. Every extension type reports Type::EXTENSION, so a kernel matcher needs a custom check on the extension name either way. A single arrow.range with a closed flag is the same amount of work, and the flag is read once when the type is reconstructed, not on every compute call.
Danica Fine invited the Arrow community to Lakehouse Day EU 2026 on Saturday, October 10 in Glasgow, co-located with Community Over Code. The full-day, multi-track event covers Iceberg, Flink, Spark, Arrow, Parquet, Polaris, Gravitino, Paimon, Hudi, XTable, Fluss, and more. The schedule includes a talk revisiting an Arrow-based client protocol redesign and a closing panel on the future of the open lakehouse.
Apache Parquet
Format 2.14.0 is released
Fokko Driesprong announced on September 11 that the Parquet Format 2.14.0 vote passed, with binding +1 votes from Gang Wu, Gábor Szádovszky, Andrew Lamb, and Fokko, plus non-binding votes from Divjot Arora and Vinoo Ganesh. Fokko thanked László for correcting links that had a copy-paste error. Andrew followed up with website pull requests that document the 2.14 changes, including ALP and the FILE logical type, and asked for reviews.
On the same day, Divjot Arora closed the vote on Extended Precision Nanosecond Timestamps with seven +1 votes, five binding, from Daniel Weeks, Micah Kornfield, Ryan Blue, Gábor Szádovszky, and Fokko. The change will merge as parquet-format PR 601. Nanosecond timestamps with a wider range matter for anyone storing high-frequency trading or telemetry data that also needs to represent dates far from the epoch.
ALP implementations keep moving across languages. The September 9 sync notes from Andrew Lamb report that the Rust implementation is merged, C++ is waiting on approval after several review rounds, and Java is under review. Vinoo Ganesh posted an update on the Java side: he split parquet-java PR #3791 out of the main ALP PR #3397 based on Russell Spitzer's feedback, and plans to merge the smaller PR first. The sync notes also recorded progress on the vector logical type. The group roughly agreed that the finite-element requirement is a MUST, that statistics help but readers must not depend on them, and that a LIST-based layout is the more popular option, though without full consensus.
What should a reader do with a file from the future?
The week's defining Parquet debate began when Ryan Blue split a question out of the versioning proposal thread. Daniel Weeks had summarized that thread's community sync discussion, noting that the real disagreement is less about recording a version in the file and more about what guarantees readers get. Ryan framed three options. Option 1: a reader must fail on a file written with an unsupported format version. Option 2: a reader should attempt to read it. Option 3: leave it to implementations.
Ryan was careful to scope the question. He is not talking about "preview" features like new encodings or forward-compatible logical types, which already fail only when a reader projects the affected column. The question covers changes to metadata semantics, such as making path_in_schema optional, adding offset and size fields to page headers so page data can move, or fixing statistics written with the wrong sort order. In a separate reply, Ryan said he strongly supports option 1. Option 2 requires a guarantee that every future change will either fail or read correctly in every existing reader, including custom Thrift parsers built for speed whose handling of missing required fields nobody knows.
Andrew Lamb wrote a clear summary of the trade-off. Option 1 makes the spec easier to change but forces readers to reject files they know how to read. Option 2 lets readers handle more files but makes every future spec change carry a compatibility burden that is hard to define. Andrew's view is that Parquet is implicitly option 2 today, and that changing it now creates confusion. Ryan disagreed: the current state is that Parquet avoids breaking changes altogether, using tricks like a second set of min and max fields when the string sort bug surfaced, and he wants a real mechanism for making breaking changes.
Antoine Pitrou questioned the "preview" label and asked whether Thrift parsers actually reject payloads missing a required field. Will Edwards argued for what he called option 0: keep Thrift-based evolution, bump the parquet-format package version as a hint to implementers, and do not gate files on a version. He noted that one widely used reader does not even check the trailing PAR1 magic bytes. Ryan replied that he voted against making path_in_schema optional precisely because it breaks existing guarantees.
Later in the week, the momentum shifted toward option 1. Kurtis Wright said a clear failure with an actionable error is the better user experience, and that implementations choosing to skip version checks accept the consequences. Xiening Dai voted for option 1, opposed changing the magic bytes because many tools use them to detect file type, and proposed shipping a reader patch that fails fast on newer versions before any breaking change lands. Fokko Driesprong also backed option 1 and suggested folding the path_in_schema footer trim into the proposed new footer, so all the breaking changes arrive together in one version.
This debate matters well beyond Parquet. Every table format, every query engine, and every lakehouse catalog depends on Parquet readers behaving predictably. A strict version gate makes the format easier to improve, which is how Parquet gets smaller footers and faster metadata. It also means operators will need to track reader versions across every engine in their stack before they turn on new writer features.
New type proposals and forward compatibility
Neelesh Salian proposed WIRE, a logical type for serialized wire-format messages such as Protocol Buffers and Thrift compact. Today, teams either store those messages as opaque blobs, explode every field into columns with a full decode on write, or convert to Variant and carry a field-name dictionary that protobuf never needed. WIRE stores the original message verbatim in a value column, so it round-trips byte for byte, and shreds the queried fields into native columns using the Variant shredding layout. Because protobuf and Thrift address fields by number, no metadata column is needed, and a reader that does not know WIRE sees ordinary columns.
Micah Kornfield pushed the decimal floating-point proposal toward a formal design document. He listed two open issues: how to represent normalization, possibly through statistics, and how to represent the value itself, which needs benchmarked alternatives to the IEEE 754 layout. Jiayi Wang sent a reminder about the footer sync and shared the updated Modular Footer proposal. Micah asked her to start a DISCUSS thread for visibility and to either broaden issue #530 or open a new proposal.
Gábor Szádovszky and Jörn Horstmann continued a thread on rewriting files with unknown metadata. Gábor argued that a rewriter that does not understand a logical type such as a shredded VARIANT will produce a useless tangle of structs and lists, so it should fail. Jörn pointed out that unknown column orders affect every column in a file, with no way to mark just one column, and that arrow-rs uses an internal unknown variant it cannot serialize. Gábor said he is open to an UNKNOWN column order that makes min and max unusable. Michael Chavinda also asked for a reviewer to add DataHaskell's Parquet reader to the implementation support matrix.
Apache DataFusion
The iceberg integration arrives
On the DataFusion side, Andrew Lamb opened the vote to accept apache/datafusion-iceberg, and it passed on September 16 with 14 +1 votes. Andrew called it a first step toward better Iceberg integration. The GitHub traffic on the dev list showed the work already underway, with Gabriel Musat's PR porting the iceberg-rust history and a second PR adapting the project to compile and pass tests in its new home.
For DataFusion users, this means Iceberg table support will evolve on DataFusion's release cadence, reviewed by people who know DataFusion's physical planning and pushdown APIs. For Iceberg users who embed DataFusion, it means one fewer version lock between two fast-moving Rust projects.
Releases, LTS, and CI
Three release votes closed. DataFusion 55.1.0 passed with nine +1 votes, six binding, and Tim Saucer ran the release. Xuanwo's verification ran 11,100 tests and 502 SQL logic test files, and he flagged several .slt files missing license headers, then fixed them himself in PR #25182. The sqlparser-rs 0.63.0 vote drew a -1 from Xuanwo for a missing NOTICE file and a bundled flamegraph SVG containing CDDL-licensed JavaScript. Andrew showed both issues were years old, filed fixes, and asked Xuanwo to reconsider. Xuanwo moved to +0, and the release passed on September 13 with five binding votes.
Andrew also asked whether DataFusion needs LTS versions, motivated by making third-party integrations easier. That question lands at the right moment, given that DataFusion now hosts the Iceberg integration and has a proposal to host Variant support too. Kosta Tarasov started that Variant discussion around PR #22908, which brings datafusion-variant into the main repository as an extension crate. Andrew suggested keeping it external with better cookbook documentation and moved the conversation to issue #21301. Kosta's open questions include whether Spark semantics should be the naming baseline.
Andy Grove enabled a GitHub merge queue for DataFusion Comet to cut CI load. Pull requests now run a smaller tier, while Spark 3.4, 3.5, and 4.0 SQL tests and older Iceberg suites run once in the queue on the exact merged tree. He also proposed Comet 1.1.0 before the end of September, about six weeks after 1.0.0, and mentioned that he used Claude to triage open high-priority bugs for the release. Tim Saucer announced that Dewey Dunnington is a new DataFusion committer. Andrew's sync notes covered subquery decorrelation, a new blocked aggregation proposal in issue #24704 that asks aggregate authors to review the API, and automating backports.
Apache Ossie (incubating)
The first release is a plumbing exercise
Jean-Baptiste Onofré set expectations for the first Ossie release. It will be 0.3.0, not a 1.0 spec, and its main purpose is to prove the release machinery and complete a full legal check, with a source distribution only. Russell Spitzer agreed that the team should treat it as practice for license checking and release management. Markus Weimer offered help. Kurt Stirewalt asked that the ontology working group's PR #332, which adds two optional fields to the ontology spec, be included, and asked what the formal process is for proposing features for a release.
Microsoft's Power BI group plans to announce its participation
Markus Weimer asked for guidance because Microsoft's Power BI group wants to announce its commitment to Ossie at the upcoming Fabric Community Conference. The group plans to build a converter between Power BI semantic models and Ossie and contribute it. Russell said stating an intent to contribute and support the project is fine as long as no one claims control. Josh Klahr suggested adding Microsoft to the Ossie ecosystem page. Jean-Baptiste pointed to the incubator's branding and publicity guides and recommended "co-creator" or "co-author" language that puts the community first.
A BI vendor committing to a bidirectional converter is exactly the kind of adoption signal a semantic model spec needs. Semantic layers only become portable when the tools on both ends read and write the same format.
One model per document, and richer modeling
Yufei Gu proposed that each Ossie JSON or YAML document hold exactly one semantic model, with its fields at the document root instead of inside a semantic_model array. Several converters already warn and convert only the first model in a document. Jean-Baptiste supported it and asked for a sequencing plan so converter fixtures do not break in the gap. Khushboo Bhatia agreed, and Yufei opened converter PR #396 to merge right after the spec change.
The modeling discussions went deep. On hierarchy support, Yufei shared a Mondrian-style example separating attributes from hierarchies. Julian Hyde credited the attribute-based hierarchy model to Analysis Services 2005, noted that each level must be functionally dependent on its parent, and said he now sees hierarchies mostly as presentation hints. Josh Klahr asked whether hierarchies should be navigation aids or real containment objects. Yufei argued that drill-down needs to know which states belong to which country, which is more than a hint.
On metrics trees, a contributor proposed an optional depends_on array for metric lineage. Julian Hyde replied that dependency information is derivable, that redundant data causes its own headaches, and that Ossie should invest in being a real language with a formal specification, compliance tests, and a compiler that produces the dependency tree. Chris Eubank proposed adding the SQL-standard FILTER (WHERE ...) aggregate modifier to the expression language in PR #382. Mario De Felipe raised, in the cardinality discussion, how to declare source values that mean "missing," such as SAP's type-specific initial values, and filed issue #401.
Agents as spec users
Marco Ciavarella from Exmergo proposed a dedicated place for "field reports" written by coding agents that build with Ossie. His first report, drafted by Claude Code, mirrored a production dbt MetricFlow layer into one Ossie document with 11 datasets, 77 fields, 15 relationships, and 27 metrics. Marco's argument is that agents generate fragmented issues, and a structured report captures what worked, what confused the agent, and what workarounds it needed. Jean-Baptiste liked the separation between blocking and costly issues and the explicit "didn't test" section, and suggested a GitHub discussion category.
Contributions kept flowing too. Poorva Barve introduced herself and sent fixes for the Snowflake converter, which invented a fake table when a query source began with a comment, and the dbt converter, which mangled compound aggregate arguments like SUM(orders.gross - orders.tax). Other threads proposed a bidirectional OpenMetadata converter and LinkML converters that open a path to RDFS and OWL. Ankit Tandon shared notes from the Ontology working group.
Cross-Project Themes
Release hygiene is now the bottleneck. Count the stopped votes: Iceberg Rust RC1 on a publish cycle, Iceberg Rust RC2 on a cross-engine encryption check, Polaris 1.8.0 on bundled NOTICE files, the Polaris Migrator on a bundled GPL file, Arrow Object Store RC1 on a license header, and sqlparser-rs on a NOTICE file and CDDL JavaScript. Add Danny Jones' pyiceberg-core wheel notices and Xuanwo's MIT attribution note on Arrow Rust 60. The pattern is clear. Rust crates, Python wheels, and fat jars all bundle third-party code, and every one of them has to carry the right legal text. Xuanwo, Dmitri Bourlatchkov, and Danny Jones are doing verification work that protects every downstream user. Projects that automate license reports in CI, the way Shawn Chang described for Paimon, will spend fewer weeks restarting votes.
Cross-implementation testing is the missing layer. The Rust encryption gap, the gc.enabled inconsistencies across five clients, and Parquet's reader version debate all describe one problem. A spec is only as reliable as the least careful implementation of it. The new iceberg-verification repository is Iceberg's answer. Parquet is choosing between a strict version gate and a softer contract. Ossie's converter bug fixes and agent field reports are the same problem appearing early, while the spec is still small enough to fix cheaply.
Integration code is moving to the engine. The iceberg-datafusion move, the DataFusion Variant discussion, and Andrew Lamb's LTS question all ask where integration code should live and how stable the host needs to be. The answer this week was to move integrations next to the APIs they depend on and give third parties a stable release line to build against.
Semantics are becoming catalog objects. Polaris is adding dedicated semantic model privileges. Ossie is simplifying its document shape while a major BI vendor plans a converter. Iceberg is designing FILE-type access delegation for multimodal inference services. The lakehouse catalog is growing from a table registry into the place where business meaning, access policy, and non-tabular data references all live together. That is good news for anyone building agents, which need exactly that combination of context and control.
Looking Ahead
Watch for Iceberg Java 1.12.0 RC1, which Neelesh Salian said is ready to cut, and for Iceberg Rust 0.11.0 RC3 once the encryption fix in PR #3236 merges. The Polaris 1.8.0 vote will turn on how the community resolves the NOTICE question, and the Catalog Migrator RC2 vote runs through the weekend. On the Parquet side, the reader versioning thread appears to be converging on option 1, and a formal decision will shape how the new modular footer ships.
The Iceberg virtual meetup on GSoC projects and commutative compaction runs Friday, September 18. The Iceberg constraint sync meets September 17, and the Polaris community sync the same day takes up consistent multi-object persistence. Further out, Lakehouse Day EU lands in Glasgow on October 10, and the Seattle Iceberg meetup follows on October 21.
Keep Learning
If you want to go deeper on Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, and the agentic lakehouse, I have written books that cover the architecture, the catalogs, and the practical engineering behind all of it. You can find every one of them at books.alexmerced.com.
Top comments (0)