This was a week of decisions. Iceberg closed two spec votes, Parquet reopened its versioning question with a cleaner ballot, Polaris shipped 1.7.0 and filed its first quarterly board report as a top-level project, and Ossie started arguing about how a young project should govern its own repository. Underneath all of it runs a single thread: the open lakehouse stack is now big enough that changing anything requires a process, and the communities are spending real energy building those processes in public.
Apache Iceberg
The headline result came from Neelesh Salian, who closed the vote to add the variant type to the REST catalog spec with 7 binding and 15 non-binding +1 votes and no dissent. Twenty-two votes on a single spec change is a large number by any standard, and the binding list reads like a roll call of the people who maintain the format: Russell Spitzer, Szehon Ho, Renjie Liu, Kevin Liu, Daniel Weeks, Yufei Gu, and Amogh Jahagirdar. The original vote thread ran for the full 72 hours and drew participation from contributors across the Java, Rust, Python, and C++ implementations.
The change itself sounds small. Variant columns can now be represented in table schemas exchanged over REST. The practical effect is larger. Variant is the type that lets Iceberg store semi-structured data without flattening it into strings or forcing a schema on write. Until this vote, a catalog and an engine had no agreed way to describe a variant column to each other over the wire. Teams that wanted variant had to keep the type inside a single engine's world. Now the catalog protocol carries it, and any REST catalog implementation can hand a variant schema to any client that understands the type.
The second vote to close belonged to Alexandre Dutra, who announced the passage of the remote signing configuration proposal with 5 binding +1 votes from Russell Spitzer, Prashant Singh, Daniel Weeks, Eduard Tudenhöfner, and Yufei Gu. Remote signing is how a client asks a catalog to sign a storage request rather than holding cloud credentials itself. Plenty of deployments already use it. What the spec lacked was a formal description of how the configuration should be expressed, which meant every catalog and client pair had to agree bilaterally. Formalizing it removes a class of integration work that nobody enjoys and nobody gets credit for.
Both of these votes point at the same shift. The Iceberg REST spec used to trail the table spec, filling in gaps as engines discovered them. This year it has become a first-class surface with its own vote cadence, its own reviewers, and its own backlog. That matters for anyone building a catalog, because the surface area you have to implement is growing on a schedule you can now watch.
Streaming appends and the metadata bloat problem
Amogh Jahagirdar opened one of the more consequential design threads of the week by proposing a change to the default append mode for Spark Streaming in Iceberg Java 1.13. Spark Streaming currently uses fast append, which writes a single new manifest pointing at all the new files and skips manifest binpacking. In a streaming workload that produces small commits every few seconds, that behavior piles up tiny manifests fast. Reads degrade, and operators have to run aggressive manifest rewrites just to stay level.
Jahagirdar pointed out that Flink and Kafka Connect already use merging append on their write paths, and for good reason. Merging manifests on write costs something, but not much, and it saves a bigger cost at read time. He drew a nice parallel to deletion vectors: merging DVs on write beats paying for a pile of position deletes on every scan. Manifests follow the same logic.
The team added a Spark write configuration in pull request 17403 that lets users pick the append mode, but left the default at fast append so that upgrading to 1.12 would not surprise anyone relying on the one-manifest-per-commit assumption. The proposal is to flip that default in 1.13. Jahagirdar also asked other client libraries defaulting to fast append to reconsider.
The thread drew responses from Steve Zhang, Russell Spitzer, Gianluca Graziadei, Daniel Weeks, Hongyue Zhang, and Kevin Liu. Nobody argued for keeping fast append as the default. The discussion turned instead to how V4 changes the picture, since the new format aims to deliver true low-latency small commits without metadata bloat in the first place. If V4 lands the way its authors intend, the append mode question becomes a transition-period concern rather than a permanent tradeoff.
The Spark version tax
Anurag Mantripragada raised a maintenance problem that has been building for two years. Iceberg supports each Spark minor version by copying the entire spark/ tree. There is a spark/v3.5, a spark/v4.0, a spark/v4.1, and soon a spark/v4.2. Adding a version means a pull request touching several hundred files and more than 100,000 lines. His thread on a shared-source layout makes the case that this stops being sustainable when Spark moves to quarterly minor releases, which is what SPARK-54633 proposes. Four of those pull requests a year is not a maintenance plan.
Sebastian Baunsgaard suggested the alternative in a community sync: one shared source tree with small per-version shim directories, the approach Delta Lake uses, starting at Spark 4.3 and moving forward. Mantripragada built a proof of concept against 4.1 and 4.2 as stand-ins, since 4.3 does not exist yet. The result was 555 shared files against 73 or 74 per-version shim files, which puts 88 percent of the code in one place.
Xin Huang responded with questions about the mechanics. The interesting part of the proposal is what it does not try to do. It does not retroactively collapse the existing version trees. It draws a line at a future Spark version and changes the pattern from that point on, which lets the current supported versions age out on their own schedule.
Read restrictions and the nested type problem
Prashant Singh brought the last open question on the Read Restrictions spec to the list before calling a vote. His thread on overlapping column projections walks through a specific and genuinely hard case.
Read restrictions bind an action to a field ID. A nested type has a field ID for the container and separate field IDs for everything underneath it. Take a struct address with subfields street and city. A catalog could return one projection that masks the whole struct to a fixed value and a second projection that replaces city with null. Outer-most-wins gives you both fields masked. Inner-most-wins gives you a null street and a masked city, even though the catalog asked for the entire struct to be masked. Two reasonable rules, two different answers, and no obvious winner.
Singh laid out two options. Option A forbids the overlap outright, so a reader that receives one fails the query under the existing fail-closed rule. Option B defines precedence and allows it. The current spec pull request takes option A, and Singh explained why by looking at what other systems do. BigQuery does not allow policy tags on struct columns at all. Redshift only applies masking policies to scalar values on a SUPER path, and where both can be expressed, it calls the pair a conflict and makes an administrator resolve it. There is no industry consensus to borrow, so the spec declines to invent one.
Daniel Weeks had asked that the alternative be explored properly before the group settled, which is why the thread exists. This is a good example of a community resisting the urge to define semantics just because it can. Leaving a case undefined and failing loudly beats defining it wrong and failing quietly three years later.
Rust, manifests, and delete performance
Shawn Chang ran the vote for Iceberg Rust 0.10.1 RC1, which collected +1 votes from Kevin Liu, Xin Huang, L. C. Hsieh, Alexander Bailey, Anoop Johnson, Matt Butrovich, Danny Jones, Renjie Liu, and Sung Yun before passing and shipping as 0.10.1. The Rust implementation keeps a release pace that most Apache projects would envy, and the breadth of that voter list says something about how many organizations now depend on it.
That dependency shows up in the bug reports. Stephan Berger from Hansetag filed an issue about equality delete file application scaling as O(data rows times applicable delete keys) in iceberg-rust. His post is a good read for anyone who has inherited an open source patch. An earlier pull request aimed at the same problem exists, but the authors' organization moved away from the project. Berger took the approach as inspiration, rebuilt it on current main using a RowFilter with ArrowPredicateFn, and ended up with a diff of 781 added and 172 removed lines, much of it tests. He asked the list how to proceed given the contributing guide's preference for smaller pull requests, and noted two approved fixes he would rather rebase on top of first. Shawn Chang replied. This is the unglamorous work that keeps a young implementation honest.
Varun Lakhyani continued pushing on integrating EagerInputFile into the manifest reader, a thread that drew nine replies from Russell Spitzer, Kevin Liu, vaquar khan, and Daniel Weeks. Lakhyani proposed a property to enable the behavior by default and ran benchmarks, arguing it helps the V4 Parquet manifest path in particular. Manifest reading is one of those areas where a few percent compounds across every query in a warehouse, so the scrutiny is warranted.
Gianluca Graziadei kept working through review feedback on Hilbert curve clustering for rewrite_data_files, pull request 16827. Tanmay Rauth had asked whether scanned bytes would be a better metric than file count. Graziadei explained that his benchmark writes each file with a single row group at roughly 16 MB, so scanned bytes would only rescale the file count, and measuring at the byte level properly would pull in filesystem, compression, and object store variance that deserves its own thread. He also asked for review from whoever built the bit interleaving in the Z-order implementation, since the Hilbert path reuses that byte encoding.
Metadata visibility and Puffin
Shangqing Yang opened a discussion on Puffin file reference metadata tables that is worth reading for its restraint. Iceberg stores several kinds of auxiliary metadata in Puffin files, including registered table statistics and deletion vectors. Those references are discoverable today through different metadata paths, which makes simple questions hard. Which Puffin files does a snapshot reference? Which references come from statistics and which from deletion vectors? Which retained snapshots share the same physical Puffin file?
The proposal adds two metadata tables, puffin_files for one selected snapshot and all_puffin_files for all retained snapshots. Each row covers one tuple of snapshot ID, metadata source, and physical Puffin file path. The initial sources are statistics and deletion vectors.
What the proposal deliberately excludes is the interesting part. It does not open Puffin files, read footers, parse blob payloads, expose unreferenced blobs, or identify orphans. Yang wants a follow-up table for footer-level blob metadata but is keeping it out of the current pull requests so the first step only establishes snapshot-level reference discovery. The open design question is naming and scope, specifically whether this should be statistics-specific or a general Puffin reference table. Yang argues for the general model with a source column, which keeps the table useful as more Puffin-backed metadata arrives.
Manu Zhang also revived the question of whether incremental append scan semantics belong in the spec, and Péter Váry and Flavio Junqueira continued organizing a dedicated sync for Iceberg index support.
The file type proposals need to converge
Russell Spitzer bumped the discussion on adding a file data type to Iceberg in V4, noting that Talat and others have a competing proposal and that consensus would be better than parallel efforts. Daniel Weeks agreed and pointed at the original thread, arguing the community should consolidate around updating that proposal rather than starting fresh. Both noted that August vacations are slowing things down.
A file data type matters more than it sounds. It is the piece that lets a table reference an external blob, an image, a PDF, or a video as a first-class column value rather than a string path with conventions bolted on. Parquet is working the same problem from the other end with its FILE type, which makes convergence across the two specs worth the wait.
Process, tooling, and the community itself
Kevin Liu proposed changing the GitHub squash-merge settings so that commit titles and bodies come from the pull request rather than the branch. The current settings sometimes produce commits on main titled "Initial commit," which makes the history on main diverge from what reviewers actually approved. The thread drew eleven replies including John Zhuge, Neelesh Salian, Yufei Gu, Szehon Ho, Hongyue Zhang, Daniel Weeks, and Maximilian Michels. Small change, real payoff for anyone who has ever bisected an Iceberg regression.
Danny Jones from Amazon asked the iceberg-cpp contributors for feedback on the two-week experiment with GitHub Copilot pull request review. Manu Zhang replied. Kevin Liu opened a parallel issue for iceberg-rust. The Iceberg community voted to experiment with automated review on its language implementations rather than adopt it wholesale, which is the right shape for a change like this. Reporting back on what actually happened is the part that most organizations skip.
Scott Haines proposed a virtual community meetup and showcase series, modeled on what the DataFusion community runs through GitHub issues. Elizabeth Garrett Christensen and Viktor Kessler responded with interest. With conferences and in-person meetups eating most of the calendar, a virtual series gives contributors who cannot travel a way to show their work.
Kevin Liu also asked a question that got eleven replies in a day: are dev list emails going to Gmail spam? Russell Spitzer, Eduard Tudenhöfner, Neelesh Salian, Alex Stephen, Matt Butrovich, Yufei Gu, Gianluca Graziadei, Scott Haines, Maximilian Michels, and Renjie Liu all confirmed some version of the problem. Spitzer had already flagged it on the file type thread, where he opened with a note about bumping the discussion out of spam. A project whose governance runs on a mailing list has a real problem when the mailing list stops arriving, and this one is worth watching.
The community also welcomed Maximilian Michels as a new committer, with congratulations from Yu Guo, Daniel Weeks, Fokko Driesprong, and Rui Fan on the announcement thread. Michels spent part of his first week handling Slack invite requests, which is a fair summary of what committership actually involves.
Apache Polaris
Polaris shipped 1.7.0 this week, and the path there says something about how the project runs releases. Jean-Baptiste Onofré opened the vote on release candidate 0, Yufei Gu found a problem, and Onofré canceled it within a day rather than pushing through. Release candidate 1 drew twelve replies and votes from Alexandre Dutra, Yufei Gu, Russell Spitzer, Robert Stupp, Yong Zheng, Francois Papon, and Ayush Saxena before passing.
The announcement covers a release with a clear theme: control over where data lives and who can prove they touched it. A Kafka event listener publishes Polaris events to a topic. GCS principal attribution arrived as the Google counterpart to AWS STS session tags, chaining a catalog-signed JWT through a Workload Identity Federation token exchange and service account impersonation so the Polaris principal shows up in GCS Data Access audit logs. Two new feature flags handle table locations. DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED gives managed locations an unpredictable suffix so no two tables share a path prefix. ALLOW_CLIENT_SPECIFIED_TABLE_LOCATION, on by default, can be set to false to reject caller-specified locations entirely and force Polaris to manage every path.
That last pair deserves a moment. Client-specified table locations are a quiet source of security problems in multi-tenant catalogs, because a caller who picks the path can sometimes point at data they should not reach. Making the strict behavior available as a flag, on an opt-in basis, is the responsible way to ship a change like that.
The first quarterly board report
Onofré drafted the August board report and put it out for community review, drawing comments from Robert Stupp, Francois Papon, Alexandre Dutra, and Yufei Gu. This is Polaris's first quarterly report after the monthly reports that followed its graduation to top-level project on February 18, 2026.
The numbers are worth reading if you track catalog projects. Polaris has 32 committers and 19 PMC members, a ratio of roughly eight to five. Nándor Kollár joined as committer on June 30. The last PMC addition was Sung Yun on April 27. The project kept its monthly release cadence through the quarter with two feature releases and a patch, and the report notes no issues requiring board attention.
The most interesting line describes how the character of the work changed. Earlier reports centered on the graduation transition and post-graduation stabilization. This quarter the community moved toward expanding what the catalog does, with major design proposals for Data Sharing, OpenLineage, and a Polaris Directory all under active discussion. A project that has stopped worrying about whether it works and started arguing about what it should become is a healthy project.
Where the console lives
Robert Stupp revived the question of whether the Polaris Console belongs in the main repository, and his argument is a good one. Back in June he was fine with keeping the first Console release in polaris-tools while leaving the main-repository and server-bundling question open. Proxy authentication work in polaris-tools pull request 260 changed the calculus. The Console already implements a browser-side PKCE flow. That pull request starts adding a second, proxy-specific model for user information, session failures, logout, and reauthentication.
Stupp's position: that is too much security-sensitive behavior for a mostly static UI. He would rather serve the Console's static assets from polaris-server and let Quarkus own the browser-facing OAuth and OIDC flow and the session, with the Console consuming only the authenticated identity and the APIs. He was careful to separate this from whether the Console ships enabled or exposed by default. Onofré, Yufei Gu, and Sayantan Samajpati joined the thread.
The general principle applies well beyond Polaris. When a UI starts growing its own auth stack, that is usually a signal the auth belongs somewhere else.
Encryption gets a catalog-side plan
Two threads this week pushed on Iceberg table encryption from the catalog side. ITing Lee proposed adding decrypt-only access for legacy AWS KMS keys. Today every explicitly configured KMS key gets encrypt permissions when Polaris vends write-capable credentials. During key rotation, that means an older key kept around for reading existing data can also encrypt new writes, which defeats the point of rotating.
The proposal adds a legacyKmsKeys list. Keys in allowedKmsKeys get encrypt, decrypt, and data key generation. Keys in legacyKmsKeys get only kms:DescribeKey and kms:Decrypt. The existing currentKmsKey is deprecated in favor of allowedKmsKeys while keeping its behavior. The configuration rejects keys that appear in both lists, because AWS combines permissions from applicable Allow statements and an overlapping entry would hand encryption rights back to the legacy key. A rotation moves the old key from one list to the other after new writes switch over. Yufei Gu responded.
Hiroaki Kawai went broader with a post on catalog security prerequisites for Iceberg table encryption. His framing is that the discussion should start from Iceberg's own catalog security requirements rather than from whether Polaris performs encrypted file I/O. A client can already use a KMS and encryption-aware FileIO with Polaris as its REST catalog. In that arrangement the client writes encrypted data files, delete files, manifests, and manifest lists while metadata.json stays plaintext, and Polaris stores the metadata pointer without touching the KMS. The catalog requirements existed before any Polaris-side consumer of encrypted metadata.
Kawai prepared two draft patches. Pull request 5185 pins the expected table key ID, including its absence. Pull request 5186 pins and verifies the current encrypted metadata revision and stacks on the first. He also mapped how these relate to pull request 5060, which lets asynchronous server-side purge read encrypted manifests, and 5127, which proposes a catalog-level representation and persistence model for KMS configuration. He continued the thread on the current state of Iceberg table encryption in Polaris later in the week.
Encryption in a catalog is one of those features where the hard part is not cryptography. It is deciding what the catalog is allowed to know and what it must refuse to trust.
Persistence, exceptions, and an API in question
Romain Manni-Bucau drove nine replies on the Polaris-managed JDBC datasource thread, with Yufei Gu and Robert Stupp weighing in. He extracted the work into a pull request against Quarkus and a standalone repository, and floated Quarkiverse as a home so the community maintains it with Quarkus support behind it. He also flagged that he would have limited time in the coming week and asked for someone to pick it up. Alexandre Dutra and Yufei Gu continued the related thread on making the relational JDBC schema name configurable.
Harshita Joshi opened a discussion on HTTP status codes in the PolarisException hierarchy tied to pull request 5206, and Alexandre Dutra replied. Yufei Gu picked up the related question of the right HTTP status for CommitStateUnknownException from federated catalogs. These sound like bikeshedding until you remember that a client library decides whether to retry based on the status code it gets back, and a commit whose state is unknown is exactly the case where a wrong retry corrupts data.
Robert Stupp opened the week's most open-ended Polaris question with a thread on the future of the notification API. Pull request 5222 prompted him to look at the endpoint more broadly. His observation is that despite the name, the endpoint is effectively an inbound catalog synchronization API. A remote catalog or sync agent uses it to create, update, validate, or drop externally managed table state in Polaris. He was explicit that fixing the concurrency behavior in that pull request seems reasonable and should not wait on the larger conversation. Stupp also continued the thread on consistent multi-object changes in Polaris persistence.
EJ Wang scheduled a review session for the Polaris Tag Spec design proposal and set up a recurring Polaris Tag sync, with Onofré and Robert Stupp joining the design discussion. Adnan Hemani and Onofré continued the OpenLineage proposal follow-up, and Srinivas Rishindra picked up multiple StorageConfigurationInfos per catalog.
Apache Arrow
Arrow had a quiet week by volume and a good one by substance. Dewey Dunnington ran the vote for nanoarrow 0.9.0, a release covering 38 resolved issues from five contributors. Bryce Mecum, David Li, Gang Wu, Sutou Kouhei, and Raúl Cumplido voted, and Dunnington closed it successfully. nanoarrow is the small C library that lets projects produce and consume Arrow data without pulling in the full C++ implementation, and it has quietly become the integration path of choice for database engines and language bindings that want Arrow interop without the dependency weight.
Andrew Lamb opened the vote for Arrow Rust 59.2.0 RC1, which drew participation from Jeffrey Vo, Raúl Cumplido, Ed Seidl, L. C. Hsieh, and Krisztián Szűcs. The arrow-rs cadence continues to be one of the most reliable clocks in the ecosystem, which matters because so much of the Rust data stack, including iceberg-rust and DataFusion, sits directly on top of it.
The week's biggest Arrow thread was social. Lamb announced that Jeffrey Vo has joined the Arrow PMC, and fifteen people replied to say so: Kosta Tarasov, Curt Hagenlocher, Raúl Cumplido, Gang Wu, Ed Seidl, Ruoxi Sun, Dewey Dunnington, Rok Mihevc, Kevin Gurney, wish maple, Kevin Liu, Weston Pace, Ian Cook, Matt Topol, and Xuanwo. A congratulations thread with that many names from that many sub-projects is a reasonable proxy for how connected the Arrow community still is after ten years.
Rich Bowen sent a note that every project on this list should read. The Community over Code hackathon in Glasgow is ten weeks out, running October 11 to 14, and only 2 of 25 listed projects have posted any task information. His ask is simple: write up focus areas, curated issues, and good first issues, submit a pull request to the comdev-events-site repository, and tell the dev and users lists. Attendees decide whether to register and book travel around now. A hackathon with no task list gets no attendees.
Ian Cook also ran the Arrow community meeting on July 29.
Apache Parquet
Parquet had the busiest technical week of any project on this list, and the center of it was a vote about how the format evolves at all.
Julien Le Dem opened a second vote on using major version numbers to release forward-incompatible changes. The first vote generated questions, so rather than clarify a proposal mid-ballot, the group spent two weeks discussing and started fresh. That is unusually disciplined behavior for an open source vote.
The proposal sets four goals: a clear definition of what is forward compatible, a clear definition of what supporting a specific Parquet version means for readers and writers, version numbers the ecosystem can coordinate around, and a path that gets new features into mainstream use in a reasonable time. Mechanically, forward-incompatible changes accumulate against the next major version, for example Version 3, and are marked "in preview." The existing bar still applies, which means two implementations and cross-testing before anything enters the spec even in preview. While in preview, features can be written behind a feature flag so integration testing can happen, and all implementations are encouraged to add read support as early as possible.
Neelesh Salian, Andrew Lamb, Divjot Arora, Alkis Evlogimenos, Gang Wu, Daniel Weeks, Ed Seidl, and Prateek Gaur all participated. Le Dem also tied the ballot back to the ongoing versioning discussion thread.
Why versioning matters right now
The sort order thread makes the abstract versioning question concrete. Divjot Arora opened a discussion on forward compatibility for new sort orders after a change added the IEEE_754_TOTAL_ORDER sort order and a nan_count field to row group and page statistics. NaN values invalidate min and max statistics, so writers leave them out and set nan_count to signal their presence.
Here is the disconnect. The merged parquet-java implementation emits nan_count but not the new sort order, which the spec permits. The arrow-rs implementation emits both, but it is not merged and carries "api-release" and "next-major-release" labels. The format change landed in parquet-format 2.13.0 and was treated as forward compatible, while both reference implementations are treating the new sort order as forward incompatible. Worse, adopting nan_count without IEEE_754_TOTAL_ORDER can produce incorrect results.
Arora laid out two options. Treat new sort orders as forward compatible and update implementations to adopt the new order, relying on the spec rule that readers ignore min and max statistics for unrecognized sort orders. Java, C++, and Python have been verified to handle unrecognized union values gracefully, and older arrow-rs versions that failed have been fixed. The alternative is to treat new sort orders as forward incompatible, revert both IEEE_754_TOTAL_ORDER and the newly merged INT96_TIMESTAMP_ORDER before parquet-format 2.14.0, and revert the Java implementation before 1.18 ships. Gang Wu, Ed Seidl, and Jan Finis all weighed in.
This is exactly the case the versioning vote exists to prevent. A change that looked compatible on paper turned out to be incompatible in practice because the implementations disagreed about what compatible means. Divjot Arora and Fokko Driesprong also continued the 1.18.0 RC1 release vote alongside this.
Compression, FILE, and the blob question
Alkis Evlogimenos reopened a point that got dropped before the FILE type merged. His argument is that the bytes of a self-reference should use the same compression codec as the column's inline field. As merged, a self-reference can only be stored uncompressed. For images and video that is fine. For text blobs like HTML, JSON, and logs it makes self-references close to useless, because those are precisely the values large enough to spill out of line but small enough that PLAIN encoding wrecks the storage bill.
His reasoning on the objections is worth quoting in substance. On applying the rule to external references too, he says the asymmetry is the point: an s3:// reference can be read by many systems that know nothing about Parquet, and its encoding is decided above Parquet, sometimes outside any engine. A self-reference lives inside Parquet and is written by the Parquet writer, so Parquet owns those bytes. On letting the engine own compression, he notes the engine already controls the blob's own compression through content_type and can pass pre-compressed bytes. The inline codec is a different thing, the storage compression Parquet applies to the column. On the objection that force-compressing images wastes CPU, he points out the writer picks the codec per column chunk and per page in v2, so a chunk written uncompressed stays uncompressed. Inheritance just propagates the existing choice. On compaction, he notes it only affects external references, since compaction rewrites the whole file anyway.
The thread ran twelve replies, with Daniel Weeks, Russell Spitzer, and Antoine Pitrou all engaging. Evlogimenos framed the timing well: FILE has not shipped in a release yet, so this is cheap to fix now and expensive to fix later.
Encodings, and a lot of them
Andrew Lamb merged the parquet-format change for ALP, the adaptive lossless floating-point encoding, crediting Prateek Gaur and many other contributors. The remaining work is merging the examples from parquet-testing and then the implementations. Lamb followed up with a thread on an ALP example file for implementations, drawing responses from Curt Hagenlocher and Prateek Gaur. Floating point columns are everywhere in time series and scientific data, and they compress badly under general purpose codecs, so a purpose-built encoding here is real money for a lot of workloads.
Prateek Gaur opened a discussion on OnPair as a string encoding after benchmarking it against FSST, DELTA_LENGTH_BYTE_ARRAY, dictionary encoding, and zstd, lz4, and snappy page compression across 30 string corpora. His summary is honest about the tradeoff. OnPair decodes faster than every compressed alternative he measured and wins on ratio for most text-heavy columns, but its training pass makes encoding substantially slower. Andrew Lamb and Arnav Balyan replied. Encode-once, read-many is the standard lakehouse pattern, so an encoding that trades write time for read speed and ratio deserves a serious look.
Serge Rielau proposed an extensible decimal floating-point type that follows the pattern set by the recent TIMESTAMP(unit) proposal, parameterizing the number of significant digits and basing the layout on IEEE 754. Thomas Kissinger replied. Adam Reeve continued the discussion on adding a VECTOR repetition level for fixed-size-list serialization, which matters for embedding columns.
Parquet as a point-query store
Will Edwards from Spotify brought the most surprising thread of the week. His team has been exploring how to use the data lake for fast point queries, not just batch scans. The example he gives is an AI agent answering a question about what you did last summer, which is a single-row lookup against a store designed for full-column scans.
Their finding: extract metadata into a fast key-value store and you know exactly which byte ranges of which files to read, skipping footer loading and search entirely. That changes the performance and cost profile substantially, and there are tricks on the write side that make the pattern work better. Edwards described it as the same idea as the metadata store that speeds up analytic workloads, indexed by key instead. He linked Spotify's engineering post and offered to share performance details.
Andrew Lamb, Alkis Evlogimenos, Haocheng Liu, and Julien Le Dem all engaged. This thread connects to the index work happening in Iceberg and to the modular footer effort in Parquet, and it is one more piece of evidence that the lakehouse is being asked to serve workloads nobody designed it for.
Community and sync notes
Antoine Pitrou announced Rok Mihevc as a new Parquet committer, and twenty-two people replied. Burak Yavuz, Matt Topol, Neelesh Salian, Jiayi Wang, Prateek Gaur, Raúl Cumplido, Russell Spitzer, Ian Cook, Arnav Balyan, Divjot Arora, Ed Seidl, Krisztián Szűcs, Fokko Driesprong, Andrew Lamb, Adam Reeve, Julien Le Dem, Alkis Evlogimenos, Corwin Joy, Gunnar Morling, Gang Wu, and Gidon Gershinsky all sent congratulations. Mihevc has been working on the vector type proposal and the new footer design.
Julien Le Dem posted notes from the July 29 community sync, with action items and review requests in bold. The attendee list is a useful snapshot of who is investing in Parquet right now: Datadog, Apple, Databricks, Snowflake, Spiral, and G-Research all had people in the room, working on versioning, sort orders, the modular footer, the FILE amendment, the vector type, encodings, and timestamp nanos. Jiayi Wang canceled the August 4 footer sync.
Apache DataFusion
DataFusion's list was quiet, but the one thread that ran matters. Andy Grove opened the vote for Apache DataFusion Comet 1.0.0 RC1, with votes from Oleks V., L. C. Hsieh, Marko Milenković, and Andrew Lamb.
A 1.0.0 is a promise. Comet is the accelerator that swaps DataFusion's native execution in underneath Spark, so a Spark job runs the same SQL and gets vectorized native operators without a rewrite. That has been an experiment for two years. Calling it 1.0.0 says the project believes the API surface is stable enough for people to build on, which changes the risk calculation for teams considering it in production.
The wider point is that Comet, iceberg-rust, arrow-rs, and DataFusion now form a coherent native stack for lakehouse work. Every one of them shipped or voted on a release in the same week.
Apache Ossie (incubating)
Ossie, the semantic layer and ontology project in the incubator, spent the week on the questions every young project has to answer before it can do anything else.
Jean-Baptiste Onofré reported on the first Apache Ossie releases discussion, noting that the first community meeting settled on starting at version 0.3.0 rather than 1.0.0. He had already done a large pass renaming "OSI" to "Ossie" and has follow-up pull requests coming, including one for Dependabot. Yong Zheng had raised open issues, and Onofré's response was that there is time to fix them before the release. Will Pugh joined the thread. Starting at 0.3.0 is the right call for a project this early, because a 1.0.0 sets expectations that an incubating project cannot yet meet.
Will Pugh drove a discussion on development guidelines that ran five replies with Yong Zheng, Quigley Malcolm, and Kunal Bhattacharya. He had circulated a document, gathered feedback, and narrowed the disagreement to two points: Make versus Just as the task runner, and a single Python environment versus many. He added tabs to the document laying out the tradeoffs for each and asked people to confirm he had the tradeoffs right before stating a preference. The goal is a proposal the project can vote on. This is how you turn a style argument into a decision.
Justin Talbot opened a request for extended review on pull requests 246 and 237, covering foundational semantics and the compliance suite that builds on it, before a vote is called. When the compliance suite depends on the semantics document, getting the semantics wrong means getting the tests wrong, and tests are much harder to change once implementations depend on them.
Elsewhere on the list, Markus Weimer and Sahil W continued the canonical file suffix discussion, Sahil W and Quigley Malcolm worked through unified Python linting and formatting, Quigley Malcolm and Onofré discussed the process for a new organization to join a working group, and Onofré and Yong Zheng covered the upcoming Java 17 end of life.
Joshua Klahr proposed extended metadata fields for fields and metrics, and Markus Weimer opened a thread with the best subject line of the week, PowerBI goes down under and needs a map, which drew Klahr and Markus Cozowicz. Richard SG Kim introduced XSOLCORP Korea's interest in the ontology and catalog integration working groups. A steady stream of GitHub-bridged discussions from djwaldo and MarioDeFelipe covered relationship cardinality, semantic filters, display names, universal calendar support, and how the community expects the semantic interchange format to be used in practice.
Ossie is worth tracking even if you have no plans to use it. Every catalog in this stack is being asked to hold semantics that no table format defines, and a shared vocabulary for metrics, dimensions, and relationships is the missing piece between a catalog and a BI tool.
Cross-Project Themes
Three patterns ran across all six lists this week.
The first is that compatibility became an explicit, versioned contract rather than an assumption. Parquet ran a formal vote to define what forward incompatible means and how to release it. Iceberg voted two additions into the REST spec with the same rigor it applies to the table spec. Polaris shipped feature flags that let operators pick strict behavior on their own timeline instead of taking a breaking change on the maintainers' schedule. Amogh Jahagirdar proposed flipping a default in 1.13 rather than 1.12 specifically so that upgrading does not surprise anyone. Every one of these is the same instinct: the ecosystem is now large enough that changes need an announced path, a flag, or a version number attached.
The second is that metadata visibility keeps surfacing as a first-class need. Shangqing Yang wants metadata tables that answer which Puffin files a snapshot references. Will Edwards wants metadata extracted into a key-value store so a point query never reads a footer. Robert Stupp wants to know what the Polaris notification API actually is before deciding its future. Prashant Singh wants a catalog to be able to state read restrictions precisely enough that no client has to guess. In every case, the thing being asked for is not new capability. It is the ability to see what is already there. That is what happens when a stack matures past the point where any one person can hold it in their head.
The third is that automation and AI are now inside the development process itself, and the communities are being careful about it. Danny Jones asked for a candid report on what two weeks of Copilot review actually did for iceberg-cpp rather than assuming it helped. Kevin Liu opened the parallel question for iceberg-rust. Anurag Mantripragada mentioned using Claude to build his shared-source proof of concept, which is the sort of disclosure that should be normal and mostly is not. Kevin Liu's squash-commit proposal exists because generated and branch-derived commit messages were degrading the history. None of these are grand statements about AI. They are practical decisions about tooling made by people who will have to live with the results.
There is a fourth thread, quieter and more concerning. Iceberg contributors spent a day comparing notes on dev list mail landing in Gmail spam, and Russell Spitzer had to bump a design thread specifically because it had been filtered. Apache governance runs on mailing lists. When delivery becomes unreliable, participation becomes uneven in ways that are hard to see and harder to correct. Worth watching whether other projects report the same.
Looking Ahead
The Parquet versioning vote is the one to follow. If it passes, expect the sort order question to resolve quickly, because the framework will finally exist to say which bucket a change belongs in. Watch for whether IEEE_754_TOTAL_ORDER and INT96_TIMESTAMP_ORDER get reverted ahead of parquet-format 2.14.0 and what that means for the parquet-java 1.18 timeline.
On the Iceberg side, the file data type proposals need to converge, and both Russell Spitzer and Daniel Weeks said as much. August vacations are slowing that down, so expect movement later in the month. The Read Restrictions vote should follow soon now that the nested type question has been aired. And the Spark shared-source proposal deserves more eyes, because the decision made there sets the maintenance cost of Iceberg's Spark integration for years.
Polaris has three large proposals in flight, Data Sharing, OpenLineage, and the Polaris Directory, plus an unresolved architectural question about where the Console lives. Any one of those could produce a vote in the next month.
For Arrow and every other project on this list, the Community over Code hackathon task lists are due sooner than they feel. Ten weeks out is when people book travel.
Keep Going Deeper
If this newsletter is useful to you, the books go further. I write about Apache Iceberg, Apache Polaris, lakehouse architecture, catalogs, and the AI workloads now landing on top of all of it. Every title I have written, across O'Reilly, Manning, and self-published work, lives in one place.
Top comments (0)