Release managers ran the show this week, and license files kept tripping them. Iceberg 1.12.0 went to a second release candidate after reviewers found bundled web assets without matching license entries. Polaris 1.8.0 failed two votes over a NOTICE gap and a wrong file header. The Polaris catalog migrator reached its fourth candidate for the same reason. Behind that release grind sat the design work that shapes next year. Parquet opened review on a Modular Footer and a vector type. Iceberg moved its materialized view spec toward a vote. DataFusion took ownership of the Iceberg Rust integration. Apache Ossie pushed toward its first source release and proposed a REST API that catalogs like Polaris can implement.
The thread that ties the week together is the contract between client and server. Polaris, Iceberg, and Parquet each spent real time on one question: what should a server or reader do when it cannot deliver what the client asked for? Every project landed on the same answer. Fail loudly, and write the behavior down.
Apache Iceberg
1.12.0 heads into RC1 after a license cleanup
Neelesh Salian opened the 1.12.0 release discussion with the release branch in good shape and a key fix already merged. Alex Reid asked to include a Kafka Connect fix that stops the connector from committing already committed files during certain rebalances. That PR had already merged, so it ships in 1.12. Reid also asked about the REST client side of "referenced-by," a spec field added last year. Salian deferred that work to the 1.13.0 milestone because it does not block the release.
The RC0 vote drew one of the most thorough verification rounds the project has seen in a while. Yuya Ebihara confirmed Trino's CI passed against the staged artifacts. His tests cover HMS, Glue, JDBC on PostgreSQL, and REST catalogs including Polaris, Unity, S3 Tables, and Tabular, plus Nessie and Snowflake. He also flagged a caveat he raised months ago. Version 1.12.0 removes the old helper logic for reading and writing partition statistics. The new logic depends on the Hadoop-based Parquet client, and some downstream projects forbid that dependency.
Manu Zhang reported 10,991 tests passing across the api, core, and parquet suites with zero failures. Anoop Johnson rebuilt the source tarball from the tag and confirmed it matched, apart from two generated version files. Xin Huang compared all 7,083 common files between the tarball and the commit and found them identical.
Two reviewers found the problem that sank RC0. Xuanwo noticed a Bootstrap stylesheet in the docs site assets. It kept its copyright line, but the package lacked the MIT permission notice and the root LICENSE did not account for it. Gianluca Graziadei found a Lottie JavaScript file in the same folder with the same gap. Neither called it a blocker, since both files have shipped since 1.9.0. Salian filed an issue, the fix merged, and he cut a new candidate.
A geospatial behavior change stays in
Steven Wu used the vote thread to flag a behavior change most users will never notice until a string comparison breaks. A PR changed GeometryType and GeographyType so toString() always writes resolved defaults. A plain geometry now prints as geometry(OGC:CRS84), and geography prints as geography(OGC:CRS84, spherical).
Xin Huang, who wrote the original change, explained the goal. The PR persists resolved CRS and algorithm values in metadata even when they match the defaults. The toString() change was a side effect he had not recognized as a behavior change. He had a fallback PR ready to revert it.
Szehon Ho argued for keeping the new output. Two equal geography types printed differently depending on whether a user set the default explicitly, which looked like a bug. The new form also matches the canonical serialization in Appendix C of the spec. Huaxin Gao added a practical reason from her UDF work. Function definition IDs are built from parameter type strings, so two equal types that print differently look like two different overloads. Old metadata still loads, because geometry and geometry(OGC:CRS84) parse to the same type. The community agreed to keep the change and list it under "Behavior Change" in the release notes. The fallback PR closed.
RC1 and a Caffeine surprise
The RC1 vote opened on September 22. Renjie Liu cast a binding +1 after checksum, signature, license, and test runs. Xuanwo reported 10,470 passing tests across the common, API, core, and data modules. Szehon Ho reported 8,400 core tests passing with zero failures and a Spark test on Spark 4.2 built from verified source.
Graziadei raised a pre-existing bug. Reading variant columns throws NoSuchMethodError when commons-lang3 older than 3.13 sits on the classpath, because iceberg-parquet uses a method without declaring the dependency. The bug exists in 1.11.0 too, so Salian plans to patch it on the release branches instead of blocking. Szehon Ho added another pre-existing fix to the wish list if a new candidate appears.
Then Kurtis Wright cast a non-binding -1. He found that iceberg-gcp-bundle-1.12.0.jar bundles the Caffeine cache library with no entry in its LICENSE file. He disclosed that he found it by prompting Claude to validate every license in the candidate. That disclosure matters. It shows AI-assisted review catching the class of issue that has failed releases across three projects this month. Watch this thread for whether RC1 survives or a third candidate follows.
Iceberg C++ 0.4.0 lands a big v3 update
Manu Zhang proposed Iceberg C++ 0.4.0 after 174 commits since 0.3.0 shipped in June. The list is long. It covers Iceberg v3 types including timestamp_ns, timestamptz_ns, unknown, geometry, and geography, plus column defaults, binary deletion vectors, and row lineage. It adds table operations such as MergeAppend, DeleteFiles, RowDelta, OverwriteFiles, RewriteFiles, and ReplacePartitions. It brings parallel manifest reads and writes, lazy scan planning, an LRU cache, and commit and scan metrics. On the catalog side it adds REST SigV4, OAuth2 token exchange, vended credentials, an early Hive Metastore client, and ResolvingFileIO.
Gang Wu backed the release and noted that variant support waits on an Arrow C++ PR, so it moves to 0.5.0. Junwang Zhao asked to hold for late October to include CherryPickOperation. Zhang pushed back with a clear release philosophy. The project will not reach "feature complete" soon, so smaller and faster releases get users feedback earlier. Zhao agreed after updating the v3 roadmap issue, which showed most planned tasks done.
The 0.4.0 RC0 vote moved fast. Gang Wu and Renjie Liu cast binding +1 votes. Liu built on Ubuntu 26.04 with GCC 15.2 and reported 4,152 passing tests across 19 release test executables. Raúl Cumplido verified on Debian 14. Xin Huang verified on macOS arm64 and disclosed he used GPT 5.6 to help.
Fittingly, the week also brought a PMC seat for Gang Wu. Fokko Driesprong announced it and credited Wu as instrumental in the inception and growth of Iceberg C++. More than 35 community members replied with congratulations.
Iceberg Rust 0.11.0 fails RC2 over encryption
The Rust 0.11.0 RC2 vote failed on a real bug. Alexander Bailey tested the candidate and found he had not implemented the tamper-proofing checks for the new encryption work. Files written by Rust were unreadable in Java. He posted a fix and voted -1. Danny Jones agreed to fail the vote because encryption is one of the headline features of the release, and asked for review on the fix before RC3.
A second Rust issue surfaced in a license thread on pyiceberg-core. Jones noticed that the Python wheels compile Rust dependencies into the extension but do not reproduce their copyright notices. Ryan Blue confirmed that distributing third-party code in compiled artifacts means LICENSE and NOTICE must reflect it. Kevin Liu explained the root cause. The iceberg-rust source release bundles nothing, but the wheels do. Shawn Chang pointed to the Apache Paimon Rust bindings, which generate a third-party license report per wheel, as a model. The group agreed to fix this before the next pyiceberg-core release.
The DataFusion integration moves out
Andrew Lamb reported that both vote threads passed unanimously. The DataFusion integration for Iceberg Rust now lives in a new apache/datafusion-iceberg repository owned by the DataFusion project. Kevin Liu thanked Matt for starting the conversation and said he hopes the move improves integration for both projects. The DataFusion section below covers the first week in the new home.
Materialized views approach a vote
Jan Kaul posted an update on the materialized view spec. Marius Grama and Walaa Eldin Moustafa built proof-of-concept implementations in Spark and Trino. Kaul says the POCs prove the spec is sound. Moustafa laid out the plan. The community sync on October 1 hosts a final discussion, then the spec moves to a vote.
One open question remains. Should materialized views get their own format version field? Steven Wu argued no. A materialized view combines the view and table specs, so a breaking change bumps one or both of those versions. A separate version inside the view spec looks odd.
Moustafa followed with a thread on Spark routing for materialized views. The implementation splits in two. Iceberg core owns the metadata, records source states, and runs basic freshness checks. The Spark side reads the storage table when fresh and evaluates the view query when stale. He chose DSv2 catalog-based routing and updated it for the new loadRelation API in Spark 4.2.
This matters for anyone building semantic layers or acceleration on Iceberg. A standard materialized view definition means one engine can create a precomputed result and another engine can safely use it. Today each engine invents its own approach.
Access delegation for the FILE type
Sung Yun's thread on access delegation for the FILE type connects two efforts. Prashant Singh explained that he and William opened a spec proposal three months ago for file-level access through pre-signed URLs. An Azure proof of concept showed the limits of remote signing across cloud providers.
Daniel Weeks drew a useful line between two problems. Pre-signed URLs returned by the plan and tasks endpoints arrive already signed, so file IO only detects and executes them. Remote pre-signing routes a storage-native path through the catalog for signing before a stream opens. Weeks wants remote pre-signing to become a viable alternative to remote signing, mostly to address Azure's limits.
Singh raised a hard operational problem. Scan planning and execution happen in different phases. A Spark driver plans, and executors read later. A URL can expire before an executor uses it. So how do clients refresh vended URLs? Yun published two documents to frame the use case: a FILE type use case analysis for governed tables, and a batch signing proposal. The FILE type connects to Parquet's 2.14 release and a new Arrow extension type, covered below.
Encryption keys get a rethink for V4
Gábor Kaszab opened a discussion on reusable encryption keys. Today the encryption-keys list in table metadata holds both data encryption keys for manifest lists and the key encryption keys that wrap them. Kaszab wants to extend encryption to statistics files and V4 root manifests, and he questions some assumptions in the current design.
Gidon Gershinsky agreed that in V4, data key metadata belongs in per-file structures, such as snapshots for manifest lists. That makes cleanup easier and retires the key-id field. Key encryption keys stay in the shared table list. Gershinsky warned that reusing a data key across files requires careful handling to avoid breaking AES-GCM, so the practical answer is a fresh random key per file. Xander Bailey wants that rule written into the spec. He also asked the group to keep one common key metadata structure and one wrap and unwrap model, so implementations share code across V3 and V4.
Smaller threads worth your time
Neelesh Salian announced the iceberg-verification repository. It holds shared conformance fixtures for all Iceberg implementations. Early work covers type fixtures with JSON Schema validation, golden reference tables, and read conformance. Andrei Tserakhau confirmed writer conformance comes later and opened an epic issue to track the full scope. With implementations in Java, Python, Rust, Go, and C++, shared fixtures are how the community keeps them honest.
Andrei Tserakhau also split a decision out of the collation work into its own thread on collation versions. Collated columns store collation-aware min and max bounds so they stay prunable. ICU ordering changes across versions, so each bound carries the version it was computed under. A reader uses a bound only when it can reproduce that version. Otherwise it scans the file. The open question is whether one file carries bounds for several collation versions. The answer fixes the on-disk metadata layout, so Tserakhau wants it locked.
Alexander Bailey continued the gc.enabled semantics thread. He asked Dan Weeks whether catalog enforcement means vending credentials without delete permissions. Rejecting a snapshot update alone does not stop a client that holds storage delete rights. His narrower PR allows snapshot expiration with gc.enabled=false only when cleanup is set to NONE.
Kurtis Wright backed a proposal to standardize the User-Agent header for REST clients and asked where vendor information belongs in the string. A user named dzeri96 raised a Spark defaultDatabase issue about a hard-coded default database name in the SessionCatalog.
Péter Váry shared notes from the index support sync. Puffin reduces file-open overhead. Bloom filter scaling problems look like rate limiting. The group decided to prioritize scalar indexes before other index types. Váry also posted three AGENTS.md PRs that cover testing requirements, comment and javadoc rules, and AI disclosure for agent-written contributions.
Ryan Blue shared the September board report with a new format. It summarizes shipped releases and spec changes instead of sync highlights, so it reports what users can use today. Kevin Liu suggested adding the new blog site and the LICENSE and NOTICE revamp. Neelesh Salian suggested Gang Wu's PMC seat and the verification repository.
On the community calendar, Kevin Liu announced the first virtual meetup on GSoC projects and commutative compaction. In-person meetups are set for the Bay Area on September 23, Seattle on October 21, Copenhagen on November 5, and Berlin on November 18. And a high school student in India asked for Iceberg swag because he tells his classmates about the table format. He sent the same note to Arrow. That is the kind of reach no marketing budget buys.
Apache Polaris
1.8.0 fails twice on license details
Jean-Baptiste Onofré put Polaris 1.8.0 RC0 up for a vote on September 17. Dmitri Bourlatchkov voted -1 the same day. Iceberg jars carry a NOTICE that references Kite, and commons-math3 references Orekit. The Polaris bundle NOTICE did not propagate either one. Onofré cancelled RC0 and suspected the gap appeared during an earlier Iceberg version bump.
RC1 passed the automated checks, the license report, and smoke tests on the binary distribution, source distribution, and Helm chart. Alex Dutra still voted -1. A test file added in a recent PR carried a Dremio license header, but Dremio did not contribute that file, so it needs the default ASF header. Onofré cancelled RC1 and promised another full pass over LICENSE and NOTICE before RC2.
The Polaris Iceberg Catalog Migrator 1.1.0 followed the same path. Robert Stupp gave RC2 a binding +1 and noted the rebuilt CLI jar was byte-identical to the staged one. Bourlatchkov then found bundled files under the Mozilla Public License that the bundle LICENSE did not mention. Ajantha Bhat noted the same gaps existed in 1.0.0. He also ran Codex against the findings from each failed candidate to catch similar problems. RC3 opened on September 23.
Pagination: one flag flips, one fails closed
Two pagination threads resolved in a way that respects the Iceberg REST catalog spec. Yufei Gu proposed enabling list pagination by default. Today LIST_PAGINATION_ENABLED defaults to false, so clients cannot use pagination without server setup. Onofré pointed out that the max page size defaults to unlimited, so flipping the flag breaks no client that omits page parameters. Ayush Saxena, Yong Zheng, Nándor Kollár, and Bourlatchkov all agreed. Zheng opened the PR. The flag stays as an escape hatch for a couple of releases.
The harder thread, on separating page-size limits from forced pagination, started with a spec problem Gu caught. When the max page size is set, the server returns a partial list plus a continuation token to a client that never asked for pagination. That violates the REST spec. Saxena explained the original intent. Large catalogs face memory pressure and denial-of-service risk from unbounded list calls, and a cap that exempts unpaged requests bounds nothing.
Gu asked for evidence and posted payload math. A namespace with 100,000 tables of 100-character names produces about 13 MiB of uncompressed JSON. That is not an obvious crisis. Prithvi S named the real danger: a 200 response with a continuation token on a request that never asked for paging is a silently incomplete list. He cited PyIceberg 0.10 and 0.11, which make one unparameterized list call and return it.
Bourlatchkov proposed the fix everyone accepted. When a maximum is configured, paginated requests get capped. Unpaged requests that fit get the full list. Unpaged requests that overflow the cap fail with an error. Vignesh A added two details. Probe at max plus one so the server does not load everything into memory before throwing, and make the error message clear. Saxena opened the PR. Gu signed off on September 21.
Zheng also added pagination to the generic table API and the policy API, and proposed table and view management in the CLI. Bourlatchkov closed out the page size maximum discussion with a merge plan.
When access delegation cannot be satisfied
Youngrae Kim brought a spec alignment question to the list. When a client sends X-Iceberg-Access-Delegation and Polaris cannot satisfy it, Polaris fails with HTTP 400. Examples include vended credentials against an S3-compatible store without STS, or remote signing on a server that does not implement it. The Iceberg REST spec reads as if the server can instead return the table without delegated access.
Saxena corrected one premise. The spec does include a signal: an empty storage-credentials field on a 200 response, which clients must check. His objection is that "must check" is not "does check." Bourlatchkov made the architectural case. Whether a client uses local credentials or vended ones is a deployment decision, not a runtime choice. A client that already holds credentials has no reason to also request delegation.
Alex Dutra added context. The "must first check" language exists for backward compatibility, which is why Polaris vends credentials in both places. Iceberg 1.12 adds a remote-signing-config field that follows the same dual-location pattern. That is one more reason to write down the "delegation not provided" semantics now.
The group chose fail-fast and documentation. Kim then took the question upstream and opened an Iceberg thread asking whether "any or none" in the spec permits rejecting the request. This is exactly how cross-project clarity gets built: a catalog finds an ambiguity, picks a safe behavior, and asks the spec owners to settle it for everyone.
Location overlap checks get honest names
Bourlatchkov opened a thread on hasOverlappingSiblings() behavior. The persistence implementations search the whole catalog, but the caller uses them only when the OPTIMIZED_SIBLING_CHECK flag is on. With the flag off, Polaris checks only immediate siblings. He called it a logical inconsistency, since a speed switch should not change what a validation covers.
Eric Maynard explained the history. The original sibling check was limited by the metastore and exposed catalogs to security issues in some configurations. The flagged check was really a corrected, thorough check, and it deserved a name like "v2." Onofré and Prithvi S narrowed the scenario. The gap bites only when ALLOW_UNSTRUCTURED_TABLE_LOCATION lets a table escape its namespace's location tree. The group agreed to keep the catalog-wide check, rename or better describe the flag, and document when catalog-wide checks are essential. Bourlatchkov floated one more idea: make the flag settable per catalog and enable it by default for new catalogs.
Soft delete, audit trails, and consistent writes
Prithvi S proposed opt-in table soft-delete with a hold period and later undrop. Today a Polaris DROP is final from the catalog's view. Operators who need recoverable drops for legal holds or Nessie migrations build workarounds that REST clients bypass. The feature is off by default. Onofré supported reserving the table name during the hold, with DROP PURGE as the explicit way to get the name back. Bourlatchkov went further. The name-based location cannot be reused until data is purged, or a new table can read an old one's files. Prithvi S explained that expiration runs lazily on namespace access in phase one, not as a scheduled sweep.
Prithvi S also opened an issue for a supported audit trail. Polaris has built an event pipeline since 1.0, with JDBC and CloudWatch in 1.2, multiple listeners in 1.5, an async executor in 1.6, and Kafka and OpenTelemetry listeners in 1.7. That is still not an operator-facing audit story. Gu suggested documenting current delivery semantics first and expanding authentication and authorization events. Adnan Hemani argued the events table is one listener among several and operators choose where events land.
Bourlatchkov posted a recap of the sync on consistent multi-object changes. Entity version numbers work for RBAC grants but not for cases like location overlaps between concurrent table creations. He found a PR that shows a failed request leaving persisted side effects. EJ Wang asked what backend assumptions the design targets. Bourlatchkov said the real goal is an SPI with clear contracts that both JDBC and NoSQL persistence can implement. Serializable JDBC transactions solve the problems raised, but they rule out in-memory caching for reads inside a transaction.
Arun Suri described a related fix for ambiguous JDBC commits. When a commit outcome is unclear, his PR reloads the entity and checks whether the metadata location already matches. If it does, the commit counts as a success. Only an unconfirmable outcome raises CommitStateUnknownException, which prevents unsafe metadata cleanup.
Cloudflare R2, tags, sharing, and federation
Austen Tomek pushed forward Cloudflare R2 support with scoped credential vending. His first PR adds a credentialVendingMechanism setting, with STS as the default. Gu asked whether the existing endpoint field is enough to detect R2. Tomek said no. Self-hosted S3-compatible stores such as MinIO use arbitrary hostnames, a failed match silently falls back to STS, and Polaris has two endpoint fields that can disagree.
EJ Wang updated the Polaris Tag spec. Bourlatchkov prefers a separate /api/tags/v1 root, since the existing catalog API was not designed for extension. Wang asked to proceed under the current root for now and revisit before release.
Bourlatchkov also asked the Open Sharing API thread to build share access on the existing authentication and authorization framework instead of a new mechanism, with external identity provider support. David Chaava asked how Polaris should handle vendor-specific validation for federated REST catalogs after reviewers pushed BigLake-specific checks out of the shared admin path. Prithvi S detailed how user-defined principal properties forward to OPA and Ranger through one shared helper. And Onofré argued for dedicated semantic model privileges that mirror existing entities. That last point connects directly to Ossie, below.
The team also settled schema upgrade docs. A single DROP TABLE IF EXISTS beats listing dependent indexes, because it works on both v3 and v4 schemas. Gu wants to drop the upgrade SQL section entirely. Dutra wants to keep it and add migration tests per backend.
Apache Arrow
Rust 60.0.0 ships the first ALP Parquet support
Andrew Lamb announced Arrow Rust 60.0.0 with five +1 votes, three binding. He called the release "epic." Arrow Rust is the first Parquet implementation to ship the new ALP encoding for floating-point data. Lamb credited Kosta and Devan for leading that work. The release also includes new Parquet PageIndex structures from Ed and many performance improvements from Richard Baah and others. The object_store 0.14.2 release also passed with three binding votes and went to crates.io.
Dropping macOS Intel
Raúl Cumplido proposed dropping macOS Intel support. Homebrew stopped building Intel bottles, and Apple's latest macOS release dropped Intel. The project cannot test some setups in CI anymore.
The community brought data. Antoine Pitrou cited the Steam survey, which shows about 11% of Mac users still on Intel CPUs. Joris Van den Bossche checked PyPI and found x86_64 at 6.6% of PyArrow macOS downloads over 30 days: 57,121 against 807,414 for arm64. Sutou Kouhei ran a ClickHouse query over PyPI data. The Intel Mac share of all PyArrow downloads is tiny.
Ruoxi Sun suggested best-effort source builds with no CI guarantee. Jacob Wujciak argued that dropping CI means dropping Intel binaries too, since cross-compiling adds the maintenance cost the change aims to remove. Wes McKinney offered his 2019 Intel Mac Pro for testing commits to main. Kurtis Wright asked whether Arrow has a policy for how many releases a platform gets before deprecation. The thread has broad support. Watch for a formal vote.
Guidelines for AI-generated contributions
Pedro Matias revived the AI contributions thread with a focus on reviewers. Arrow has AI guidelines for contributors but none for reviewers facing low-quality PRs with weak author engagement. He wants an agreed way to handle authors who ignore the rules.
Rok Mihevc proposed a GitHub label, needs-author-engagement. It flags PRs where the author needs to respond substantially before review continues. It focuses on the author's observable understanding and engagement, not on whether they used AI. Nic Crane and Dmitry Chirkov supported it. Onofré pointed to the ASF's generative tooling guidance, which encourages disclosure. He also made a governance point. At the ASF, people contribute as individuals, so quality discussions happen in public even between colleagues at the same company.
Format work: FILE, JSON schemas, and ranges
Mandukhai Alimaa opened a format PR for a canonical FILE extension type named arrow.parquet.file, following the precedent of arrow.parquet.variant. Kent Wu updated his proposal for a JSON representation of Arrow schemas. Each type now has exactly one allowed form, either a simple string or an object. Metadata uses the object form because many implementations expose metadata as a map, which cannot hold duplicate keys.
A contributor pinged the thread on an arrow.range canonical extension type for bounded ranges. The proposal uses one range type with a closed parameter instead of separate types per closedness, since Arrow's kernel matching already sees extension type parameters. Ian Cook posted the biweekly community meeting for September 23.
Apache Parquet
The Modular Footer goes up for review
Jiayi Wang introduced the Parquet Modular Footer for broad review after months of biweekly working group meetings. The proposal covers motivation, format design, compatibility, and benchmarks, with a parquet-format PR and tracking issue. Wang expects a lot of feedback because the proposal introduces several large conceptual changes. Micah Kornfield had asked Wang to move the discussion to the list for visibility and to tie up the old footer tracking issue.
The footer matters for anyone running wide tables in a lakehouse. Today a reader parses the entire Thrift footer before reading any column, and that cost grows with column count. A modular footer lets readers load only what they need.
How readers handle unknown versions
The footer work drives a related question: how should readers behave with unsupported format versions? Xiening Dai, Fokko Driesprong, and Steve Loughran all voted for option one: fail on an unknown version. Driesprong argued that breaking changes like the new footer will break old readers anyway. The spec is the only thing the project controls, and quiet partial reads erode trust. He suggested bundling smaller speedups such as path-in-schema into the new footer version instead of cutting corners.
Loughran made the counterweight point. Because a reader must fail fast on an unknown version, moving to a new version needs a very good reason. The work to carry FlatBuffers alongside Thrift exists to deliver speed while keeping compatibility. He sketched a future "2.0" library that reads and writes both classic and new layouts but writes classic files by default.
ALP is live, and FastLanes gets a hard look
With the 2.14 format release done, Andrew Lamb published the ALP blog on the Parquet site on September 22. ALP, short for Adaptive Lossless floating-Point encoding, first appeared in a SIGMOD 2024 paper from CWI. Lamb also opened website PRs for the 2.14 changes, including ALP and FILE.
Prateek Gaur posted a performance study of FastLanes to separate its ideas from one another. For plain PFOR, the layout gain depends heavily on hardware, because the Parquet C++ reader is already vectorized. Antoine Pitrou asked whether the same speedup applies to DELTA_BINARY_PACKED by breaking the sequential dependency with interleaved partial sums. Gaur had already done it. Three draft PRs against Parquet C++ implement a log-step SIMD scan and batched miniblock unpacking, for about a 2.15x speedup on narrow 32-bit pages on Graviton4.
Gaur reports that the FastLanes transposed layout is still roughly 2x faster on top of that. Alkis Evlogimenos argued a new layout doubles the writer's decision space and suggested shelving FastLanes until Parquet has more data-dependent encodings. Gaur disagreed and asked to decouple FastLanes from the PFOR encoding discussion. There, Evlogimenos asked for a PFOR-delta versus DBP comparison and whether DBP blocks can live between PFOR blocks. Lamb asked what DBP stands for, which is a good reminder to define acronyms on public lists. It means delta binary packed.
A vector type for embeddings
Rok Mihevc opened a PR for a LIST-based VECTOR logical type after a focused community call. Every non-null vector has exactly num_elements values. Elements must be non-null, finite, and numeric or boolean primitives, with no nesting. Readers that do not know VECTOR read it as a plain LIST. Physical layout changes, vector indexes, and quantization are out of scope.
Pitrou objected to the narrow rules. Parquet is a general-purpose format. Why forbid NaN values when NumPy tensor data contains them? He argued for a general fixed-size list type and leaving specialist types to domain software, the way Arrow stores its schema in Parquet metadata. Mihevc defended the scope. Vector stores reject non-finite and nested values, and statistics are sometimes not written or read. He then asked the key question: what is the gate for Parquet to define a logical type? He also asked whether removing the finiteness rule makes the proposal acceptable, and whether it is time to revisit the Parquet extension type discussion.
For lakehouse teams storing embeddings next to business data, this thread decides how vectors land in Parquet files for years.
Types and statistics
Thomas Kissinger returned to the decimal floating-point proposal. Kornfield asked for a formal design doc and named two open issues: how to represent normalization and how to store values. Kissinger argued that statistics describe what was written but do not define a writer's obligation, so a logical-type parameter should state whether a column preserves quantum or canonicalizes.
Adrian Garcia Badaracco proposed classifying distinct_count as inexact. The spec never says whether the field is exact or an estimate such as a HyperLogLog sketch. An exact definition raises its own questions about nulls, negative zero, and NaN bit patterns. Query planners use this field for join ordering, so a clear definition matters. Michael Chavinda asked to add DataHaskell's Parquet reader to the implementation support matrix. Julien Le Dem posted the Parquet sync.
Apache DataFusion
The Iceberg integration finds a new home
Andrew Lamb announced the vote result to accept the apache/datafusion-iceberg repository into the DataFusion project, with 14 +1 votes. He called it a first step toward better Iceberg integration. The cross-list discussion confirmed both projects approved.
The new repository got to work fast. Gabriel (gabotechs) merged a PR that ported iceberg-rust history into the new repo, then a PR that made the project compile with passing tests after review from Kevin Liu. Lamb fixed a misnamed .asf.yaml file that kept GitHub issues disabled. A contributor resubmitted a PR to make IcebergTableProvider::try_new public, since the only provider with write support needs a public constructor. The original PR was lost when the integration moved.
This move is a smart division of labor. Iceberg Rust focuses on the table format library. DataFusion owns the query engine glue, where its maintainers know the planner and execution internals best.
Sync notes and a new committer
Lamb started posting notes from the DataFusion sync call because about ten people attend regularly and the topics interest the whole community. Adam GS covered subquery decorrelation, with many issues filed and ideas for testing query planner passes in isolation. Raz shared a prototype for blocked aggregation and asked aggregate authors whether the new API works for them. Haresh proposed CI automation to identify and backport PRs. Kurtis Wright asked for a calendar invite. The Parquet, Arrow, and Iceberg communities publish iCal links, and DataFusion does not yet.
The PMC also welcomed Dewey Dunnington as a committer. Dunnington is well known across the Arrow ecosystem, so this strengthens the link between DataFusion and Arrow.
Apache Ossie (incubating)
Converters leave the main repo
Yufei Gu opened the week's biggest Ossie thread on the scope and maintenance of converters. Converters translate Ossie semantic models to and from vendor formats. Keeping them all in the main repo creates three problems. Converter dependencies complicate licensing. Each converter has its own structure and CI, which complicates binary releases. Each needs platform knowledge and an ongoing maintainer. Gu suggested keeping a small set as reference implementations.
Julian Hyde, drawing on his experience as a founder and mentor of past incubator projects, agreed with much of it. He warned that moving converters to vendor repos means contributors are not seen as ASF contributors, and that licensing limits apply wherever the code lives. His advice: focus on the first release, because every incubating project underestimates it.
Onofré shared the history. Ossie started as a specification, which alone did not justify an incubator project, so converters went into the main repo. The plan from day one was to split them out. He proposed freezing converters, then creating a repo per converter, such as ossie-dbt and ossie-databricks, and keeping only the spec and docs in the main repo.
Piotr Sowiński played devil's advocate with lessons from LinkML. There, all converters live in one repo and ship as one package, because they share parsing code and users get one install that works. Russell Spitzer favored separate repos and compared it to Parquet, with parquet-format separate from its implementations. Markus Weimer did not care about repo count but insisted every repo stay inside the ASF project, because ASF provenance makes code easier for downstream users to adopt.
Toward the first source release
Kyoung Min Kim found 22 files missing ASF license headers ahead of the 0.3.0 source release, across the CLI plugin code, the Sigma converter, and NVIDIA test fixtures. He opened a fix and asked for a header-only CI check. Chris Eubank and Onofré both backed it. Given how the other projects spent their week, this is good timing.
Semantics, compliance, and a REST API
Chris Eubank pulled several threads into one plan for semantics and compliance suites. His diagnosis is sharp. Given the same model shape, two vendors can return different answers to the same question, which weakens Ossie as a standard. The plan layers an expression language, two query interfaces, and compliance suites. Onofré supported the layered framing and asked for proof that Layer 3 SQL reduces to Layer 2 primitives, ideally through shared fixtures where the same model and question produce the same answer both ways. Eubank reported that the OSSIE_SQL_2026 dialect definition merged, and proposed a fixture format that pins a Layer 3 query, its Layer 2 form, and the base SQL answer to one value.
Onofré then proposed an Ossie REST API spec, an OpenAPI file for producing, consuming, and orchestrating Ossie models. He named catalogs such as Apache Polaris, Unity, Collibra, and DataGalaxy as implementers, with query engines as clients. Sebastien Gandon said Qlik has started building an Ossie-native REST API to import and export its semantic models, and proposed a dedicated media type, application/vnd.ossie+yaml.
Modeling details
Markus Cozowicz proposed making relationship cardinality explicit. Today "from" means the many side and "to" means the one side, which cannot describe one-to-one or many-to-many. Onofré pointed to evidence that clears the usual bar for promoting a vendor extension into the core spec. The Microsoft, Databricks, and OrionBelt converters each invented a private stash for cardinality. Cozowicz revised the proposal to reuse the ontology term multiplicity with PascalCase values, default to ManyToOne when absent, and skip OneToMany since it is the same relationship with endpoints swapped.
Shao Xie proposed decoupling ontologies, semantic models, and mappings into three document kinds joined by external references. Gu questioned whether three document types are worth the extra catalog complexity. Ankit Tandon shared Ontology working group notes. Victor Morgante reported that the Python ontology parser rejects reverse verbalizations of a relationship, which fact-based modeling allows.
On GitHub Discussions, Jochen Christ explained the split between Ossie and the Open Data Contract Standard. Ossie now has a core spec for BI semantic models and an ontology spec at the conceptual layer. Data contracts cover more than schema and semantics, including terms of use, ownership, quality, and SLAs. A user asked how Ossie handles unstructured data. Gu answered that the community focuses on structured data for now and asked for concrete use cases.
Cross-Project Themes
License hygiene is now the release bottleneck. Iceberg 1.12.0, Polaris 1.8.0, the Polaris catalog migrator, pyiceberg-core, and Ossie 0.3.0 all hit LICENSE, NOTICE, or header problems this week. The pattern is clear. As projects ship more binary bundles, wheels, and Helm charts, the list of third-party code inside each artifact grows faster than anyone tracks by hand. Ossie's push for a header check in CI and Paimon's per-wheel license report point to the fix: automate it. Notice also who found the problems. Kurtis Wright used Claude, Ajantha Bhat used Codex, and Xin Huang used GPT 5.6 to verify candidates, and all three disclosed it. AI review is now part of the ASF release process, and the disclosure norm is forming in real time.
Fail loudly, then write it down. Polaris chose fail-fast for unsatisfiable access delegation and for oversized unpaged list calls. Parquet chose fail-fast for unknown format versions. Iceberg keeps a visible behavior change for geospatial types and documents it instead of hiding it. The common thread is respect for the client. A silent partial answer is worse than an error, because the client cannot tell it is wrong. That principle is what makes open standards safe to build on. It is also why the Polaris team took its access delegation question back to the Iceberg list instead of patching around it.
Specs are getting conformance suites. Iceberg launched iceberg-verification with golden tables and read conformance. Ossie is designing compliance fixtures that pin one answer across query layers. Parquet's distinct_count thread asks the spec to say what a statistic guarantees. When five languages implement one table format, and many vendors implement one semantic model format, the spec text alone is not enough. Shared fixtures turn "compatible" from a claim into a test result.
The FILE type ties three projects together. Parquet 2.14 shipped FILE support. Arrow is defining arrow.parquet.file as a canonical extension. Iceberg is working out how a governed REST catalog delegates access to those files. This is the plumbing for storing references to images, PDFs, and other unstructured objects next to structured rows, with the catalog in charge of who reads them. It is the base layer for multimodal AI on lakehouse tables.
Semantics move into the catalog. Onofré wants dedicated semantic model privileges in Polaris, and he proposed an Ossie REST API that names Polaris as an implementer. Put those together and the direction is plain. The open catalog that governs Iceberg tables is lining up to govern the semantic models that describe them. For AI agents that need consistent business definitions, that pairing matters as much as the table format.
Looking Ahead
Watch the Iceberg 1.12.0 RC1 vote and whether the Caffeine license gap forces RC2. Iceberg C++ 0.4.0 needs one more binding vote to pass. Iceberg Rust 0.11.0 RC3 waits on the encryption tamper-proofing fix. Polaris 1.8.0 RC2 and the catalog migrator RC3 both need clean license passes.
On design, the October 1 Iceberg sync hosts the last materialized view discussion before a vote. Parquet's Modular Footer review opens in earnest, and the VECTOR type debate needs an answer on what earns a logical type. Arrow's macOS Intel thread is ready for a formal vote. Ossie heads toward its first source release with converters moving out of the main repo.
Keep Learning
If this newsletter helps you follow the open lakehouse, my books go deeper on Apache Iceberg, Apache Polaris, agentic analytics, and the rest of the stack. Find the full catalog at books.alexmerced.com.
Top comments (0)