Last week the projects decided what new things the lakehouse should hold: files, vectors and semantic models. This week they worked on whether anyone can trust what those things say about themselves. Iceberg passed votes on tight stats, table constraints and a new REST header value, and opened another on stats-only fields. Polaris worked out who is allowed to skip credential vending, how storage configuration should live in a catalog, and how a transaction should span tables. Parquet kept arguing about what a vector promises and what a null count actually counts. Ossie, the incubating semantic interchange project, started preparing its first release.
Underneath the spec work sat a harder problem that showed up on almost every list. Maintainers on Iceberg, Arrow and Polaris all said, in different words, that AI-generated pull requests and shared CI capacity are now growing faster than the number of people who can review and run them. That theme gets its own section near the end, because it may shape these projects more over the next year than any single spec change.
Apache Iceberg
The v4 spec learns to make promises
The biggest outcome of the week was Anoop Johnson's vote to replace per-column tight_bounds with a file-level tight_stats flag. It passed on October 27 with 9 binding +1s from Amogh Jahagirdar, Daniel Weeks, Eduard Tudenhöfner, Fokko Driesprong, Gang Wu, Russell Spitzer, Ryan Blue, Steven Wu and Szehon Ho. Andrei Tserakhau, Anurag Mantripragada, Hongyue Zhang and Huaxin Gao added non-binding +1s, and nobody voted against it.
The change sounds small but it matters a great deal for query planning. In the earlier design, each column carried its own flag saying whether its lower and upper bounds were exact. The new design puts one tight_stats flag on the data file, and that flag covers null counts, NaN counts and value counts as well as bounds. If the flag is missing, readers must assume the stats are not tight. This is the vote that came out of Anoop's discussion last week about extending "tight" beyond min and max. With it in place, an engine can answer count(*) WHERE col IS NULL from metadata alone and know the answer is exact.
Two details in the spec are worth calling out for anyone who runs table maintenance. Deletion vectors no longer widen stats automatically, so applying a delete does not quietly downgrade a file's metadata. And engines may re-tighten stats without rewriting data files. That opens the door to a cheap maintenance job that recomputes stats on files whose rows were partly deleted, instead of a full compaction. The spec PR is apache/iceberg#18585.
Table constraints arrive, and primary keys replace identifier fields
Huaxin Gao's vote to add table constraints to the v4 spec also passed on October 27, with 8 +1s, 5 of them binding by Huaxin's count. Ryan Blue, Russell Spitzer, Steven Wu and Eduard Tudenhöfner marked their votes binding, and Szehon Ho also voted +1. Andrei Tserakhau, Anoop Johnson and Junwang Zhao voted non-binding.
The spec adds three constraint kinds to table metadata: check, unique and primary-key. Each one carries an enforced flag, so a table can declare a constraint that the writer promises to honor without the catalog trying to police every commit. More precisely, the flag controls whether writers must verify the rows they add or modify. An unenforced constraint is still useful. It tells the optimizer and downstream tools what the data is supposed to look like, which is how many warehouses have long treated declared keys. Each snapshot also records a status for every constraint: validated, valid, invalid or unvalidated. That last part is the clever bit. It means a reader can tell whether a constraint was actually checked for the data it is about to read, rather than trusting a declaration that may have gone stale three commits ago.
In v4 the primary key replaces identifier field IDs, the older mechanism used mostly for upserts and CDC. Upgraded tables get an unenforced primary key built from their identifier fields, so nothing breaks on migration. During review, Ryan raised a question about type promotion for columns referenced by a check constraint. Huaxin's reply says his comments were addressed, so check PR #17822 for the final rule before relying on it. For practitioners, this is the change that will matter most in a year's time. Optimizers in many engines already use declared keys for join elimination and better cardinality estimates, and Iceberg tables will finally have a standard place to declare them.
Stats-only fields head to a vote
Ryan Blue opened a discussion on stats-only fields on October 22 and moved it to a formal vote on October 27. Earlier drafts called the idea non-materialized columns. A new stats-fields list assigns table field IDs to things that are not columns, such as the output of a partition transform or an expression like to_lower_case(name). Their stats can then live in content_stats and drive pruning without ever appearing in the schema.
Ryan explained the name change in his opening mail. "Stats-only fields" says what the feature does, while "non-materialized columns" described what it is not. He also considered unifying the idea with generated columns, such as default-value expressions, and decided against it for the same reason the group kept expressions inside check constraints. Splitting them would mean adding an expression and then a column that depends on it, and that indirection buys nothing. The spec PR is apache/iceberg#18405.
Péter Váry initially wanted to tie the feature to derived or generated columns, which he revived last week. Ryan argued that keeping it out of the schema is simpler and avoids committing the spec to generated-column semantics too early. Péter came around and gave his +1. In the vote thread, binding +1s from Daniel Weeks, Amogh Jahagirdar and Steven Wu arrived quickly. Anoop Johnson and Maximilian Michels added non-binding support. The vote runs 72 hours and should close around October 30.
The proposal already has a follow-up. Andrei Tserakhau started a thread on collation bounds, which would let an engine keep separate min and max values for a case-insensitive or locale-aware ordering of a string column. Ryan prefers a new type with standardized collation metadata over overloading stats-only fields. Either way, the pattern is clear: v4 is turning stats from a best-effort hint into a typed, declared contract.
The File type creates its first follow-up work
Last week the community voted a File type into v4, letting a column hold a reference to an external object such as an image or PDF. This week the follow-ups began. Sung Yun proposed a batch remote signing endpoint for the REST spec. His reasoning is simple. With remote signing, a client asks the catalog to sign each object request. When a table holds one Parquet file per few million rows, that is manageable. When each row can point at its own object, the number of signatures grows with rows instead of files. The design is at s.apache.org/irc-batch-signing and the PR is #18254.
A related REST change passed outright. youngrae kim's vote to allow "none" as an X-Iceberg-Access-Delegation value passed with 7 +1s, including binding votes from Yufei Gu, Sung Yun and Eduard Tudenhöfner. A client can now tell the server it does not want any delegated credentials, for example because it already has its own storage access. This closes a long-running awkward case where a server could not satisfy a delegation request and had no clean way to say so. As the Polaris section shows, it landed just in time for Polaris to use it.
Auth, CDC and correctness
Alex Dutra brought back an idea from a Community Over Code Glasgow session on REST client auth. He proposed a new BearerAuthManager that reads a token from a local file and simply sends it. The work on a second-generation OAuth2Manager stalled on scope creep, so the plan is to shrink that effort and separate who produces a token from who uses it. A sidecar or platform agent refreshes the file, and the Iceberg client stays out of the identity business. The Glasgow group agreed on two tracks. The first is a downscoped OAuth2Manager v2 with minimal functionality, done right and standards-compliant. The second is the BearerAuthManager, which is opinionated, lightweight and has no dependencies. Users who need in-process authentication would keep OAuth2Manager, and everyone else could switch to the file-based approach. Alex is starting with the bearer manager because it is simpler, and he promised a separate thread for the downscoped OAuth2 work. It addresses an issue that has been open for about a year (#13550).
On streaming ingestion, koodin 9 proposed Kafka Connect CDC writes without equality deletes. The sink coordinator would resolve deletes against the base snapshot into deletion vectors and commit the data and DVs together in one RowDelta. Since v4 already forbids writing new equality deletes, the sink needs some answer here. The background matters. An earlier delta writer for the sink (#14797) was never merged, mainly because equality deletes committed by a streaming sink slow down every reader until compaction runs. Flink takes a different route, converting equality deletes into deletion vectors after commit with a maintenance task that keeps a primary key index in Flink state. The new proposal does the lookup before commit with a stateless scan that prunes candidate files by partition and deleted keys. It reads only the key columns and row positions, and hides the lookup behind an interface so a real index can replace it later. For concurrency, the RowDelta validates from the base snapshot, uses the deleted keys as the conflict filter, and checks that the files the new DVs point to still exist, which catches a concurrent compaction. Andrei Tserakhau asked whether the sink should do CDC at all, since append-only ingestion plus a Spark MERGE already works. Péter Váry asked for performance testing first and warned that a slow resolve step could starve concurrent commits.
Two correctness threads deserve attention from anyone maintaining an Iceberg implementation. Marius Grama reported a silent correctness bug, first seen in Trino, where stats on non-key columns in equality delete files wrongly prune delete manifests. The fix in PR #18338 ignores those stats, and Péter's view is that equality deletes are being phased out anyway. Andrei also found that NaN and Infinity do not round-trip in JSON single-value serialization. Java writes the strings "NaN" and "Infinity" but cannot read them back. iceberg-rust, PyIceberg and iceberg-cpp write null and lose the value. iceberg-go refuses to write them at all. Raghvendra Yadav backed standardizing on the strings Java already writes. It is a small bug, but five implementations disagreeing on one value is exactly the kind of drift a multi-language spec has to catch early.
Yuya Ebihara's HLL sketch proposal for Puffin, covered last week, picked up +1s from Denys Kuzmenko, Junwang Zhao, Raghvendra Yadav and Jian Chen. It would add apache-datasketches-hll-v1 next to Theta for distinct-value estimates. The REST OpenAPI thread from Yong Zheng also drew replies from Alex Dutra, Péter Váry and Yufei Gu about making the spec valid OpenAPI 3.1.
Releases, subprojects and security
The only release vote on the list was the Iceberg Terraform Provider. Matt Topol's 0.2.0 RC0 got a full verification and +1 from Xuanwo, then a -1 from Alex Stephen because LICENSE and NOTICE files were out of date. Matt cut RC1 on October 26. Version 0.2.0 matters because it is the release that lets the provider be listed in the Terraform Registry, which makes managing Iceberg catalogs as infrastructure code much easier.
The Rust sync notes, posted by Danny Jones, say the group agreed to ship 0.11 from the current branch "ASAP" without recutting. The DataFusion Iceberg integration has moved out of the repo and will see its first release under DataFusion governance. Matt opened a v4 tracking issue, iceberg-rust#3513, and the group discussed a File Format API like the one in Java, with a Vortex proposal doc linked. The group also agreed to start a roughly three-month community roadmap modeled on DataFusion's, partly to give contributors a clear signal about which PRs are wanted. Kevin Liu uploaded the recording to YouTube.
For Python users, Orest Tokovenko proposed dropping Python 3.10 from PyIceberg. Python 3.10 reached end of life on October 1, and PyArrow 26 ships no 3.10 wheels. The proposal sets requires-python >=3.11 and raises the PyArrow floor from 18 to 26 in one PR. If you pin old PyArrow versions in production, now is the time to plan an upgrade.
Russell Spitzer also posted a Confluent connector vulnerability update. The vulnerable 1.9.2 Kafka Connect zip on Confluent Hub was uploaded by Confluent, not built by the project, so the project will not patch that artifact. PR #15212 adds an official runtime zip instead. The only remaining scanner findings are Jackson CVEs inside Parquet's bundled copy, which ties neatly into the parquet-java 1.18.2 release discussed below.
Community health
The thread most worth reading in full is Russell Spitzer's Iceberg Community Health Stats. Russell built a dashboard at russellspitzer.com/iceberg-community-health and flagged a drop in contributors who come back for a second PR after their first, especially in the Java repo. He took the idea from Patrick McFadin's Cassandra project health dashboard, which he saw at Community Over Code. Russell said he has no prescriptions yet, only worry about the decline in "First to Second" contributors across the ecosystem. He was also open about how he built it, describing the site as "pretty viciously vibe-coded." The data and code are on GitHub, and he invited people to dig in. Jared Yu said AI-generated PRs are burying a small group of maintainers, which slows reviews for everyone, including genuine first-timers. Several people said Russell's original mail landed in spam, and Kevin Liu is chasing ASF Infra about delivery. Hongyue Zhang asked to put the dashboard on the community sync agenda. Yufei Gu and Ryan Blue also joined the discussion.
On the event side, Talat Uyarer finalized the Bay Area meetup agenda for November 2 at Google in Sunnyvale. Anurag Mantripragada announced that the Iceberg and Spark sync goes monthly at 9 AM Pacific, starting November 11.
Apache Polaris
1.9.0: scope settles, the date holds for now
Polaris 1.9.0 is still in the scoping thread JB Onofré opened on October 4. This week Yufei Gu asked to include PR #5263 because it fixes a potential table corruption issue. Dmitri Bourlatchkov added #5856 to the milestone. JB agreed to both and still wants Directories in the release as a beta feature. He wrote that he wants to put the release to a vote before Halloween, even after losing power and internet in a storm midweek. As of October 28 there was no release candidate.
If Directories ship as beta, 1.9.0 will be the first Polaris release where the catalog governs raw object-store prefixes next to tables. Last week's review raised real concerns, including Ankur Kotwal's scanner bug and Tornike Gurgenidze's argument that the inventory table should be the source of truth. So the beta label will matter. Treat it as a preview to test, not a feature to put production pipelines on.
Storage access gets more flexible
The busiest design area was how Polaris talks to object storage. Dmitri Bourlatchkov proposed typed storage config options to skip credential vending on Azure and GCS. They are AzureStorageConfigInfo.skipSasTokenGeneration and GcpStorageConfigInfo.skipCredentialDownscoping, matching the existing AWS stsUnavailable option. Ayush Saxena weighed in, vignesh a implemented it in PR #5884, and Dmitri linked it directly to the Iceberg "none" vote. Together they give a clean path for engines that already hold their own storage credentials, such as workloads running inside a trusted network.
At the opposite end, Tommy Bollhofer described a real deployment that needs more than credential vending can offer. His team runs Polaris against an on-prem FlashBlade whose STS supports only AssumeRoleWithWebIdentity. As a result they hand static storage keys to every engine, which defeats the point of a governed catalog. The keys go to every service in every environment: Polaris itself, Trino, Spark for table maintenance, and PyIceberg and DuckDB clients. Each new environment adds rotation work and widens the blast radius of a leaked key. Tommy also noted his client versions are mixed. Trino runs on Iceberg 1.12, Spark on the 1.10 runtime, and PyIceberg and DuckDB use their own Iceberg implementations, so client-side support for remote signing will vary. His longest jobs are Spark maintenance runs that take minutes, so he thinks remote signing could ship without token refresh support at first. His team offered to test early builds against FlashBlade and report which engines work. Alex Dutra judged that enough evidence to restart remote signing and is preparing an experimental PR on top of the earlier work. Combined with Sung Yun's batch signing proposal in Iceberg, remote signing looks set to become a first-class option rather than a niche one.
Yufei Gu also changed direction on multiple storage configurations per catalog. After Srinivas Rishindra's input, the group agreed to skip a transitional phase and make storage configuration its own catalog-scoped entity with REST endpoints. The open question is how a table binds to one. Yufei prefers a table property set at create time over a "pending binding" model. Separately, Yufei proposed blocking a risky config change: flipping ALLOW_UNSTRUCTURED_TABLE_LOCATION from true to false can strand renamed tables behind 403 errors.
Transactions, sharing and discovery
Qiegang Long's multi-table transaction work moved from proof of concept to design. He opened umbrella issue #5833 and posted the design doc. Engines would get BEGIN ... COMMIT across statements and tables. Writes are staged on Iceberg table branches and published with the existing multi-table commit API. The proof of concept runs Spark and Trino at READ COMMITTED and REPEATABLE READ isolation. Ankur Kotwal argued for a server-side publish step, citing the gaps he hit while adding commit_transactions to PyIceberg. His examples were concrete. PyIceberg can fast-forward a branch but cannot cherry-pick, so a Python client could not re-apply appends onto a main branch that has moved. He also found that a delete matching no rows staged nothing, so the transaction went out without its "main has not moved" check. Gaps like that are easier to close once on the server than in every client. On snapshot expiry, Ankur suggested rejecting snapshot removal while a registered read point still covers it, while keeping a 410 response for unregistered handles so routine expire jobs do not start failing. An Iceberg cross-post is coming, since any real answer has to work for every REST catalog, not just Polaris.
The long-running Open Sharing design also moved. Dmitri Bourlatchkov pushed for a Shares API that does not assume any particular authentication scheme. Dennis Huo split the work into two API specs, one for shares and membership and one for consumers, listings and credentials. Madhan Neethiraj, who works on Apache Ranger, asked how authorization would work. Dennis answered that every layer becomes pluggable: what is shared, who can read it, how it is authorized and how consumers authenticate.
Dennis also proposed a catalog discovery endpoint, GET /api/catalog/polaris/v1/catalogs. Today a client with only a catalog role needs to know the catalog name before it can configure itself. Polaris already has a listCatalogs call, but it is a management API. It needs a root-level privilege and returns the full catalog model, including storage configuration. The new endpoint is for data-plane readers. At first it would return only catalog names, the value a client passes as the warehouse to the REST /config endpoint. Dennis also tied it to the in-progress Shares API, where a consumer needs some way to discover what has been shared with it. JB supported the idea, suggested the name listAccessibleCatalogs, and set a simple rule: a catalog is listed if and only if getConfig would succeed for that caller. He also asked for paging from day one. This is a small feature that would make life easier for tools and agents that connect to Polaris without being told where to look.
Labels, semantic models and other threads
The IRC Labels design settled on option B, in which Iceberg REST labels are a read-only view over Polaris tags. Dmitri clarified that importing labels from federated catalogs happens only when sub-catalog RBAC for federated catalogs is enabled, and Andrei Tserakhau agreed.
Arnab Karmakar asked for review of PR #5205, which validates semantic models against a bundled Apache Ossie JSON Schema, version 0.2.0.dev0, with a 1 MiB size cap. It is an early sign of Ossie definitions being consumed inside a catalog, which is the whole point of a semantic interchange standard.
Not every thread was about features. vignesh a proposed dropping metrics reports instead of returning HTTP 500 when the reporter's database write fails. Yufei objected that the Iceberg spec never says metrics are best effort, and that some deployments feed metrics into billing or auditing. He wants the question raised with Iceberg first. Harshita Joshi asked for review of a fix for staged creates, used by DuckDB, which were dropping a catalog's default table properties. Prithvi S explored opt-in table soft-delete, and Dmitri moved to soft-roll authorization on GET /v1/config.
The AI review bot thread from last week continued. Robert Stupp asked which tool runs the reviews and who pays. Yufei answered that the tool is his own project, can use more than one model provider, and is funded by his employer, and that ASF Infra sponsorship is being explored. Every comment will be labeled as AI-generated. vignesh a worried that the bot misses decisions made on the list and in syncs, and that newer contributors will treat its comments as required.
Apache Arrow
A busy release week
ADBC Libraries 25 passed its vote with 6 binding +1s from Bryce Mecum, Curt Hagenlocher, Gang Wu, Matt Topol, Raúl Cumplido and Sutou Kouhei, plus a non-binding +1 from Xuanwo. David Li announced it on October 26. The headline is the C# core reaching 1.0.0, with C# drivers at 0.25.0. The C, C++, GLib, Go, Python and Ruby packages are at 1.13.0, and Java, R and Rust at 0.25.0. Bryce worked with Dewey to get three R packages, adbcdrivermanager, adbcpostgresql and adbcsqlite, accepted on CRAN. npm publishing still needs rework.
On the Rust side, Jeffrey Vo ran votes that delivered patch releases on three older lines: 57.3.2, 58.4.1 and 59.3.2, each passing with 3 binding +1s. That completes the multi-branch backport plan Andrew Lamb announced last week. Jeffrey also ran a vote on 60.1.0, a minor release on the newest line, which passed on October 28 with binding votes from Jeffrey, Andrew Lamb and L. C. Hsieh. Xuanwo found that the release audit script fails on a missing license header in .config/nextest.toml, which Jeffrey will fix next release. Matt Butrovich tested arrow-pyarrow on Python 3.15 using PyO3's forward compatibility flag.
Arrow Java 20.0.0, which JB flagged as imminent last week, saw no vote. No list message explains the delay. On GitHub, a release-candidate CI job for macOS on ARM has been failing since a bundled protobuf upgrade, and a fix PR is open. That is the likely cause, but it is an inference, not a statement from the project.
Funding for PyArrow
Alenka Frim announced that the Sovereign Tech Fund is investing in PyArrow. Germany's Sovereign Tech Agency will fund about 1.9 full-time roles for the rest of 2026 and all of 2027. The applicants were Alenka Frim, Nic Crane, Rok Mihevc and Tadeja Kadunc. The money covers PyArrow type annotations, the long-standing bug backlog, documentation, extension type support and tooling to help maintainers handle agentic PRs. Alenka said the contract funds work that has stalled and is unlikely to attract regular corporate sponsors. Alenka, Rok and Tadeja have already started on type annotations, building on work Rok has invested in for a long time. Nic is working through bug reports that have sat open for years. In 2027 Alenka plans to research building the full documentation without PyArrow installed, then improve the Python User Guide, API examples and Cookbook. Tadeja will focus on extension types and related compute kernels. Open Source Collective is fiscal host, on behalf of Arctos Alliance. Curt Hagenlocher, Antoine Pitrou and Raúl Cumplido replied with congratulations. For a library that sits under pandas and much of the Python data stack, paid maintenance time is good news for everyone downstream.
AI contributions and CI pressure
The AI contributions thread picked up again. Raúl Cumplido opened a PR to write down reviewer guidance. Nic Crane posted an impact report that found about a third of recent PRs skip AI disclosure. His seven recommendations include moving the PR cap into .asf.yaml before November, mandatory disclosure, and a bot that converts non-disclosing PRs to drafts. He also suggested giving active non-committer reviewers the collaborator role so they can help triage. Raúl pointed to CPython's new AI triage labels as a model.
Nic's report is worth reading in full because its diagnosis applies well beyond Arrow. He named three problems. Maintainers spend time reviewing PRs that are later abandoned, or that need more rounds of feedback because they were mostly AI-generated. Maintainers also spend time enforcing AI rules, while non-committer reviewers have little power to flag problem PRs. And the volume itself keeps growing. His other recommendations include adopting a rule against AI-written review replies and auto-drafting PRs that sit idle for 14 days with conflicts, failing CI or requested changes. He also suggested gating PR opening on triage state, as pandas does, and tracking monthly how many new-contributor PRs get no human response. He singled out growing the merge group from people already triaging as the only measure that adds capacity rather than cutting inflow. Nic noted, with some irony, that he used AI to help analyze the data.
Jarek Potiuk opened the most heated thread of the week on making Arrow's GitHub Actions CI lighter. Using an Apache Magpie workflow-optimization skill, he estimated Arrow used 110,000 to 125,000 runner minutes the previous week, with Linux packaging about 40 percent of that. He suggested 30 to 40 percent savings, partly by running extra workflows nightly instead of per PR. He also noted that pull_request_target is being disabled for ASF repos in November. That change affects three Arrow workflows, including the PR limit bot. Jarek suggested replacing the bot with the .asf.yaml creation_cap setting, which enforces the same limit of three open PRs per contributor with no workflow at all. He also spotted that nightly failure reports had been silent since October 21. The report job installs the latest Python 3, now 3.15, and a dependency has no 3.15 wheel yet, so the job fails before sending email. Pinning Python 3.13 fixes it. Benson Muite suggested Fedora's Packit for RPM builds. Raúl disputed one of Jarek's findings, saying nightly reports do work. Andrew Lamb reported 3 to 7 hour CI queue waits for arrow-rs and DataFusion contributors, and his numbers showed Airflow using far more shared runner time than the Arrow repos. The exchange got sharp before Jarek apologized on October 27. The underlying issue is real: the ASF's shared runner pool is now a bottleneck that contributors feel directly.
Flight SQL and community
Tornike Gurgenidze's Flight SQL proposal to advertise a server's Substrait dialect continued. Curt Hagenlocher asked whether clients could negotiate a dialect version. Tornike replied that GetSqlInfo cannot take parameters, so negotiation would need a new RPC. He argued the existing minimum and maximum Substrait version codes are enough for now. Ian Cook posted logistics for the October 21 community meeting.
One housekeeping note: mail-archive.com is missing several Arrow messages from this week, including the 60.1.0 vote opener and some of Andrew Lamb's replies. JB reported on the Ossie list that some ASF list mail was not being delivered and that Infra was working on it, which likely explains the gaps.
Apache Parquet
parquet-java 1.18.2 becomes the release to watch
Manu Zhang proposed a parquet-java 1.18.2 patch release on October 26. The 1.18.x branch already carries several fixes. They stop writers from flushing incomplete files after a failure and fix a dictionary lookup collision between dotted field names and nested paths. They also correct Bloom filter column path handling, release row-group buffers properly and fix protobuf handling of empty and recursive messages. The release also bumps Jackson to 2.22.3 to clear CVEs. Steve Loughran asked to include master's Thrift and Snappy upgrades, Fokko Driesprong backed the release, and Manu opened backport PRs. Divjot Arora also replied in support.
This matters beyond Parquet. The Jackson CVEs inside Parquet's bundled copy are the last scanner findings blocking a clean Iceberg Kafka Connect runtime, as Russell Spitzer noted on the Iceberg list. Meanwhile the 1.19.0 release candidates have sat unreleased since September, so 1.18.2 is the realistic near-term upgrade for Java users.
The vector debate goes another round
The numeric vector thread, started by Rok Mihevc in September, kept going. Jan Finis proposed an alternative: put finiteness on float elements and an optional fixed size on LIST, rather than adding a new FINITE_VECTOR type. He argued that design stays forward compatible, while a new union member would make older engines reject the column. Ryan Blue held to FINITE_VECTOR. He floated allowing both LIST and FINITE_VECTOR annotations on the same column, and argued that types carry guarantees, as UTF-8 does for strings. Finiteness lets readers skip per-element checks and catches bad data at write time.
Curt Hagenlocher pointed out that SQL Server, MySQL and SAP ASE already reject non-finite floats. Will Edwards suggested putting VECTOR on hold until there is a broader proposal for ML tensors and sparse or quantized data. He warned that writers that do not know VECTOR will rewrite files as plain lists, and repeated that Iceberg may be the better home for finiteness. Fokko Driesprong said the group is "running in circles," while noting people are willing to do the work and that Parquet should be self-contained. Rok's format PR #624 is still open. For anyone storing embeddings in Parquet today, nothing changes yet, but the decision will shape how Iceberg's own vector type is defined.
Precise definitions and safer readers
Andrew Bell asked a deceptively simple question about the difference between num_values and num_rows in DataPageHeaderV2. Adam Reeve explained that num_values counts leaf-level values including nulls, num_rows counts logical rows, and V2 pages never split a row. Antoine Pitrou went further. He noted that the null-count fields are ambiguous at nesting levels deeper than one and proposed precise definitions, in which num_values really counts levels. This could become a spec clarification, which would help every implementer who has had to reverse-engineer the answer from parquet-java.
Antoine also continued his fuzzing thread. Steve Loughran asked which code areas fuzzing hits most, and Antoine named three bug classes: unvalidated metadata such as negative lengths or invalid enums, memory allocations sized from metadata instead of actual file size, and unchecked integer overflow in arithmetic on metadata values. It is a useful checklist for anyone writing or auditing a Parquet reader. His Community Over Code Glasgow talk is linked in the thread. Julien Le Dem posted the October 21 sync logistics.
Apache Ossie
First release prep and a new committer
The incubating semantic interchange project took two steps toward maturity. Josh Klahr announced Yufei Gu as a new committer, with congratulations from Russell Spitzer, Dmitri Bourlatchkov, Chris Eubank, Ayush Saxena and others. Yufei's Polaris background is useful here, since Polaris is already validating semantic models against Ossie's schema. After the converters moved to their own repository, JB Onofré started the release-prep PRs for the first core release: version properties, source release, distribution and Maven publication. No vote has been called yet.
The spec takes shape
Chris Eubank put the Wide Table Query Interface under lazy consensus through November 4. A Layer 3 query names dimensions, metrics and a filter, and the engine lowers it to Layer 2 using AGG(). Justin Talbot proposed reserving AGG() for evaluating measures and letting dataset-scoped metrics reference source columns directly. Together these define how a BI tool or AI agent asks for "revenue by region last quarter" without writing SQL against physical tables.
John DaCosta asked what Dataset.source should guarantee. Chris said the group agrees on a source discriminator, table, query or file, with connection details kept in vendor extensions. He framed the core goal as sharing semantics across tools while querying data where it already lives. A real ERP export then started a debate over which time field slices a metric, after a 14 versus 15 mismatch traced back to that choice. Julian Hyde argued the time hint belongs on the dataset, not the metric, except for semi-additive measures like balances.
The expression portability work continued too. Richard SG Kim migrated 176 expressions to the portable form and found it 22 to 27 percent shorter. Chris told contributors that the ClickHouse and Trino dialect PRs only need a rebase, since adding a dialect is no longer a core spec change. Johannes Zemlin shared an initial ANTLR4 grammar for Ossie expressions, and Ayush Saxena fixed NOT-operator precedence. Ossie has no mail-archive.com mirror, so these links go to lists.apache.org.
Cross-Project Themes
Metadata you can trust
Look at the week's spec decisions side by side and they share one goal. Iceberg's tight_stats flag says the counts are exact. Table constraints say whether a key was actually validated for this snapshot. Stats-only fields give expression stats a declared identity. Parquet's FINITE_VECTOR debate is about whether a type should guarantee its contents. Antoine's null-count definitions would make a page header mean one thing in every reader. In each case, the project is turning metadata that is "probably right" into metadata that is guaranteed.
That shift matters most for automated consumers. A human analyst can sanity-check a suspicious number. A query planner skipping files, or an AI agent answering a question from metadata, cannot. The more of the lakehouse that carries guaranteed facts, the more work engines and agents can skip safely, and the cheaper and faster the whole stack gets. Ossie's spec work fits the same pattern one layer up, where semantic definitions become portable contracts rather than tool-specific settings.
Storage access is being rethought from both ends
Iceberg's "none" delegation value and Polaris's skip-vending options cover clients that already have storage access. Remote signing in Polaris and batch signing in Iceberg cover the opposite case, where clients should never hold storage credentials at all. The File type is forcing the question, because a column of object references multiplies the number of storage requests. Expect the REST spec and Polaris to keep moving together here.
Maintainers under pressure
The most important theme was not a spec. Russell Spitzer's dashboard shows fewer returning contributors in Iceberg Java. The Iceberg Rust group wants a roadmap partly to manage AI-generated PRs. Nic Crane found a third of Arrow PRs skip AI disclosure. Polaris is trialing an AI reviewer while contributors worry about its blind spots. Arrow's CI thread shows the shared ASF runner pool is a real bottleneck, and the Sovereign Tech Fund grant explicitly pays for tooling to handle agentic PRs.
Contributions are getting cheaper to produce, while review time and CI minutes are not. These projects are building the open infrastructure that AI agents will read from, and they are also among the first communities to feel what agents do to open source maintenance. Expect disclosure rules, PR caps and funded maintainer time to spread across ASF data projects quickly. If you contribute to any of them, disclose AI assistance and keep PRs small. It is the easiest way to keep your work out of the triage pile.
Looking Ahead
Watch for the stats-only fields vote result around October 30 and a Polaris 1.9.0 vote by October 31, including whether Directories ship as beta. A parquet-java 1.18.2 vote, the iceberg-rust 0.11 release, binding votes on the Iceberg Terraform Provider 0.2.0 RC1 and Arrow Java 20.0.0 are all possible in the next two weeks. Ossie's Wide Table lazy consensus closes November 4, and its first core release vote could follow. The Iceberg Bay Area meetup runs November 2 in Sunnyvale, and the monthly Iceberg and Spark sync starts November 11.
Resources & Further Learning
Get Started with Dremio
- Try Dremio Free: Build your lakehouse on Iceberg with a free trial
- Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow: Learn how Dremio brings the open lakehouse stack together
Free Downloads
- Apache Iceberg: The Definitive Guide: O'Reilly book, free download
- Apache Polaris: The Definitive Guide: O'Reilly book, free download
Books by Alex Merced
Top comments (0)