This Week at a Glance
- The Iceberg community opened a formal scoping discussion for the v4 table spec, with Daniel Weeks laying out three workstream categories and contributors already adding collation and column updates to the list.
- Steven Wu proposed making Iceberg v4 manifests Parquet-only, and early responses from Anoop Johnson, Russell Spitzer, and Manu Zhang all point toward a single-format future.
- PyIceberg 0.12.0rc1 drew a binding -1 from Kevin Liu after a user reported a correctness regression, so a new release candidate is coming.
- Apache Polaris shipped 1.7.0 with Kafka event publishing and GCS principal attribution, then disclosed CVE-2026-64640, a low-severity flaw in the register endpoint.
- Apache Parquet passed its versioning vote with 5 binding +1s, formalizing major versions as the vehicle for forward-incompatible changes, and released parquet-java 1.18.0.
- Apache Arrow released 25.0.1 and Arrow Rust 59.2.0, welcomed Jeffrey Vo to the PMC, and received a funded win_arm64 support offer from Microsoft and Linaro.
- Apache DataFusion Comet hit 1.0.0 after two years of incubation, and Andy Grove opened a discussion about promoting it to a top-level ASF project.
- Apache Ossie hit a fork in the road, with Justin Talbot proposing a smaller SQL-with-measures standard as an alternative to the foundational semantics document under review.
The first full week of August brought release energy across the entire stack. Four projects moved artifacts through votes while the deeper conversations turned to the shape of what comes next: Iceberg scoped v4, Parquet locked in a versioning strategy for incompatible changes, and Ossie debated what a semantic layer standard even is. The through line this week is a community deciding how to change formats without breaking the people who depend on them.
Apache Iceberg
The most consequential thread of the week came from Daniel Weeks, who opened a discussion on v4 spec scope and priorities following the community sync. Weeks grouped the active workstreams into three buckets. Content metadata updates cover the Adaptive Metadata Tree with single file commits, column statistics, relative paths, and column append. Table features cover check constraints, default value expressions, and generated columns. Data types cover the proposed file type and vector type. His point is that individual efforts are well known to sync regulars, but the community has never stated what a cohesive v4 looks like. Andrei Tserakhau responded with two additions: collation, which depends on v4-only machinery like per-collation bounds as generated-expression column stats, and a unified treatment of column append and column updates as two operations over the same column-file representation. Tserakhau also noted that parallel work on the Delta side converged on the same dense, row-aligned representation, and suggested keeping the representations compatible across formats while both are still being defined.
Closely tied to the v4 conversation, Steven Wu asked whether v4 manifests should be Parquet-only. The column update sync leaned toward dropping the Avro option because Avro cannot support projection reads on manifest files, including column stats, and forcing every integration to choose between two formats adds decision burden with no upside. Anoop Johnson pointed out that Iceberg does not track the root manifest format today, so supporting Avro root manifests requires new tracking work that buys nothing. Russell Spitzer expressed a slight bias toward Parquet-only as a step toward converging on a single file format, and Manu Zhang agreed while recalling a separate discussion about deprecating ORC. Upgraded tables keep their v3 Avro leaf manifests, so the restriction applies only to newly written v4 metadata.
That upgrade path got its own thread when Shawn Chang raised V3 to V4 migration expectations. The current design makes upgrades an O(1) operation: a v4 root manifest references existing pre-v4 manifests, and new writes produce v4 metadata. Chang worries that this shifts migration responsibility onto users who rarely run optional maintenance, a situation he compared to the equality delete problem. Anoop Johnson defended the design, noting that expensive metadata rewrites add friction and that prior version upgrades worked the same way, with tables converging over time as old data ages out. Kurtis pushed the concern forward a few versions, imagining tables in the v6 era where query performance becomes unpredictable because any given scan hits a mix of v3, v4, and v5 files at multi-petabyte scale.
Performance work delivered a concrete win this week. Varun Lakhyani's benchmarks for integrating EagerInputFile into the manifest reader show a 25 to 55 percent reduction in Parquet manifest read time on S3, with two independent result sets confirming the range. Russell Spitzer called it exciting enough to consider as a default. The design discussion then settled where to put the integration. Daniel Weeks laid out three candidate points: the FileReader API, the FileIO layer, or the InputStream at point of use. Spitzer argued for keeping it contained to the Parquet reader code, since the fix addresses a parquet-java behavior and there is no reason to trigger the same path for a Puffin file. By Tuesday the group agreed on the FileReader API, and Lakhyani committed to the Parquet work with ORC exploration in parallel.
The Read Restrictions spec neared its vote. Prashant Singh surfaced the last open question: what happens when a catalog returns column projections that overlap on nested types, for example a mask on a struct and a null-replacement on one of its subfields. Option A forbids the overlap and requires readers to fail closed. Option B defines precedence rules. Singh surveyed industry practice and found no semantics to borrow, since BigQuery forbids policy tags on structs and Redshift treats the pair as an admin-resolved conflict. Russell Spitzer closed the argument by citing precedent from the default values discussion, where the community spent weeks on nearly identical questions before disallowing the ambiguous configuration outright. His +1 went to Option A: catalogs must not emit overlapping nested projections, and readers must fail closed when they receive them.
Release trains moved on both the Java and Python sides. Neelesh Salian updated the 1.12.0 thread with a plan to cut the branch on or after August 26, keeping the 3-month cadence the community set after the 8-month gap between 1.10 and 1.11. Alexandre Dutra asked for the REST path segment encoding fix, Felix Perez Diener of Stripe asked about Flink 2.3 support, and Cheng Pan raised switching the default table version from 2 to 3, which Salian deferred past 1.12 given remaining gaps in Variant and Geo types. On the Python side, Alex Stephen proposed PyIceberg 0.12.0rc1 with view support, geometry and geography types, Python 3.14 support, and a new File Format API. Verification votes accumulated until Kevin Liu cast a binding -1 after validating a user-reported correctness regression, so expect rc2 shortly.
Encryption work produced the week's most instructive vote. Gábor Kaszab called a spec vote to deprecate the key-metadata field in table statistics and add a key-id field pointing into the table's encryption-keys list, since storing raw key material inside unencrypted table metadata defeats the purpose. The vote gathered +1s from Gidon Gershinsky, Russell Spitzer, Steven Wu, and others before Ryan Blue registered a -0 with detailed objections to how the PR couples key management changes to the v4 spec version. Blue argued that v3 statistics files carrying per-file keys should stay valid in v4 tables, with key-id added as the better option rather than a forced migration, and he flagged a mismatch between the keys table design, which expects one or two reused keys, and current practice of one key-metadata per stats file.
Community infrastructure grew on two fronts. Scott Haines proposed virtual community meetups and showcases modeled on the DataFusion series, and Elizabeth Garrett Christensen, who organizes similar events for Postgres, arrived with notes and a proposed format: 10 minutes of announcements, 20 to 30 minutes of technical content, and 15 minutes of open discussion on a monthly cadence with strict no-marketing guidance. Kevin Liu committed to making it happen. Meanwhile Neelesh Salian, Sung Yun, and Andrei Tserakhau advanced the shared conformance fixtures proposal, a language-neutral repository of test fixtures modeled on parquet-testing so every implementation checks its spec reading against a shared set. Tserakhau made the case that write verification belongs in scope early, since bugs like equality_ids typed as long instead of int live in what an implementation produces, and he linked a live cross-implementation matrix covering Go, Rust, and Java on v1 through v3 reads and writes. A bounded differential fuzz run already surfaced real bugs, including a reader that rendered a fixed type as fixed(4) where the Java reference produced fixed[4].
Two more spec conversations are worth tracking. Prashant Sharma asked about derived column support after building generated columns for the Presto Iceberg connector with table properties, and the thread pulled in Szehon Ho and Daniel Weeks around definitions, determinism, and alignment with the UDF and View specs. Alexander Löser reported alignment from the collation sync: the ICU version stays an engine decision, bounds use original strings rather than collation keys, a code-point metric enables cross-version pruning, and equality deletes either get deprecated in v4 or excluded from collated columns.
The index workstream advanced through its dedicated sync. Péter Váry summarized the Iceberg Index Support session: the group agreed to retain the history of index snapshots but not of other index properties, and discussed defining index ordering through a list of transform functions, with Daniel Weeks and Yingyi sketching a JSON structure that applies named functions like day and truncate to field references. Flavio Junqueira asked the sharpest question in the thread: since engines already prune partitions and files using partition information and per-column min-max metadata, what does capturing partitioning, sorting, and clustering as index transform functions add? He also pressed for clarity on how engines consume such an index and where the mapping from input to files happens, in Iceberg or in the engine. Those are exactly the questions a proposal needs to answer before it hardens into spec text, and the recording is on YouTube for anyone catching up.
The Rust implementation got a meaningful performance contribution from outside the usual committer circle. Stephan Berger of Hansetag filed a fix for equality delete application, which currently scales with the product of data rows and applicable delete keys. His rebuilt approach uses a RowFilter with ArrowPredicateFn, mirroring the Java strategy, and lands as a 781-line diff. When Berger worried the PR exceeded the contributing guide's 300-to-500 line preference, Shawn Chang gave the practical answer: file it as-is to showcase the solution, split later if reviewers ask. Berger also linked a companion position delete PR. Threads like this show iceberg-rust maturing from a port into a project with its own performance identity.
Smaller spec threads filled in the edges. Xiening Dai asked whether null_value_count applies only to optional fields in v4, pulling in Anoop Johnson and Eduard Tudenhöfner on the semantics of stats for required columns. 杨尚卿 proposed Puffin file reference metadata tables so operators can inspect which Puffin files a table references without walking metadata by hand. Shangqing Yang raised Parquet Page Index pruning in Iceberg's custom reader, an optimization that reads column index structures to skip pages inside row groups. Matt Topol opened a vote for the Apache Iceberg Terraform Provider v0.1.0 RC2, bringing infrastructure-as-code management to catalog resources. Kevin Liu floated using PR titles and descriptions for squash commits to improve commit history quality, and Neelesh Salian scheduled a tracking document and sync for the Variant type, the semi-structured data type that keeps coming up as a blocker for making v3 the default table version.
Step back and the Iceberg picture this week is a project running three races at once. The v4 spec race defines what the format becomes. The release race keeps 1.12.0 on a 3-month cadence so features reach users predictably. And the implementation race, spanning Java, Python, Rust, Go, and now Terraform, is where the conformance fixtures work earns its keep, because every new surface multiplies the ways implementations drift apart. The fact that a correctness regression stopped a PyIceberg release this week is the system working: verification culture caught the problem before users did.
Apache Polaris
Jean-Baptiste Onofré announced Apache Polaris 1.7.0 on August 2, and the release notes read like a security and governance wishlist. The release adds a Kafka PolarisEventListener for publishing events, GCS principal attribution for vended credentials so the Polaris principal appears in GCS Data Access audit logs, a DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED flag that gives generated table locations unique unpredictable suffixes, and an ALLOW_CLIENT_SPECIFIED_TABLE_LOCATION flag that lets operators block caller-specified locations entirely.
Days later, Alexandre Dutra published CVE-2026-64640, a low-severity vulnerability affecting Polaris through 1.6.0. The register endpoint read a caller-selected Iceberg metadata file using the catalog's storage credentials before validating that the file sat within allowed storage locations. An authenticated principal with registration privileges was able to disclose limited information from objects the catalog's credentials happened to reach. Andrea Cosentino found the issue, and the demonstrated impact is limited to confidentiality. Read alongside the 1.7.0 location controls, the disclosure shows a project systematically tightening the trust boundary between catalog and storage.
The deepest architectural thread continued around consistent multi-object changes in Polaris persistence. Robert Stupp flagged PR #5222, a retry loop for concurrent notification updates, as another example of consistency semantics being decided at individual call sites because the manager-level contract does not express them. His position: the physical backend performs one atomic attempt and returns a precise outcome, while reload, revalidation, and retry need one shared owner above it. Dmitri Bourlatchkov advanced a concrete proposal, a per-request Data Context that maps to a JDBC connection plus transaction on relational backends and to tracked reference hashes on NoSQL, with all persistence changes committed once at the end of the request. Jean-Baptiste Onofré had earlier cautioned against making the transactional metastore manager the portable target, since it holds a durable transaction open across slow external work like credential vending and does not map to NoSQL.
Stupp also questioned the future of the notification API now that Iceberg's register-table operation supports an overwrite option. The endpoint arrived with the initial code import as an inbound catalog-synchronization API, and Snowflake is its one documented consumer. Dennis Huo agreed in principle with reconciling into upstream Iceberg functionality, then laid out the real design tension: register-table with overwrite serves both a repair use case, where subsequent updates are fine, and a mirroring use case, where accepting updates creates split-brain table forking. Polaris currently keeps those separated by catalog type, with EXTERNAL catalogs serving only notifications.
Storage flexibility moved forward as Srinivas Rishindra published an updated design for multiple storage configurations per catalog. Bourlatchkov called it an excellent summary with a clean path and suggested phase 1 is ready to implement pending reviews, with the practical note that non-default storage configs work best at the namespace level. Dennis Huo recapped community sync feedback on the Open Sharing APIs, covering the need to document that historical snapshots remain visible to consumers, the requirement to avoid hard-coding an internal principal behind every ExternalConsumer, and longer-term on-behalf-of semantics for fine-grained consumer attribution. Bourlatchkov proposed splitting share management under its own URI prefix such as /api/shares/v1/.
Operational threads rounded out the week. Yong Zheng proposed pagination in the CLI, citing shared-tenant deployments where listing principals returns 40,000 entries in one response, and Yufei Gu and Ayush Saxena both +1'd handling pagination internally without changing CLI output behavior. Bourlatchkov merged the JDBC location overlap query fix in PR #5003 with a follow-up issue to remove ADD_TRAILING_SLASH_TO_LOCATION and halve the SQL conditions later. The Iceberg table encryption discussion continued as Hiroaki Kawai posted draft patches pinning the expected encryption key-id and verifying encrypted metadata revisions, addressing the two catalog security requirements Robert Stupp insisted belong in the combined design. And the tag spec proposal from EJ Wang gathered detailed REST API feedback from Bourlatchkov, who wants tags exposed to external authorizers like OPA and Ranger from day one.
Two quieter threads showed the breadth of the contributor funnel. GitHub user melin asked about managing principals, privileges, policies, and roles through Spark SQL, the kind of request that signals users want Polaris governance to feel native inside the engines they already use rather than requiring separate tooling. Eundo Lee proposed making the Relational JDBC schema name configurable, a small change that matters for shops with database naming policies. And Yong Jin Lee reported that the polaris-tools console cannot set connection-type-specific fields on EXTERNAL catalogs, the sort of tooling gap that surfaces once federation features see real use.
The OpenLineage integration also clarified its sequencing. In the follow-up thread, Adnan Hemani explained that timestamps in lineage events serve as freshness indicators rather than a queryable historical log, resolving Dmitri Bourlatchkov's data retention concern. He then mapped the two open PRs: #4667 adds the APIs required for OpenLineage compatibility and blocks the forwarding mode, while #4705 introduces scaffolding for a future local storage mode. Jean-Baptiste Onofré had asked the community to return to the original problem statement, a gateway to OpenLineage backends like Marquez, and the thread now reads like a project converging on exactly that scope with local storage progressing in parallel.
What ties the Polaris week together is a maturing security posture. The 1.7.0 location flags, the CVE disclosure, the encryption metadata integrity patches, and the consistency contract debate all attack the same class of problem: a catalog is a trust broker between engines and storage, and every gap between what it validates and what it executes is attack surface. The project is closing those gaps methodically, and the volume of Bourlatchkov's review activity this week, spanning persistence, sharing, tags, pagination, and lineage, shows how much coordination that takes.
Apache Arrow
Release machinery dominated the Arrow list. Raúl Cumplido shepherded Apache Arrow 25.0.1 through its vote, a 9-issue patch release that passed with 4 binding +1s from L. C. Hsieh, Gang Wu, Bryce Mecum, and Cumplido himself. Andrew Lamb ran the Arrow Rust 59.2.0 vote in parallel, which passed with 7 +1s and is now on crates.io. Dewey Dunnington completed the trifecta with nanoarrow 0.9.0, 38 resolved issues from 5 contributors, passing with 6 binding +1s and a post-release checklist spanning CRAN, PyPI, conda-forge, vcpkg, Conan, and homebrew.
The people news matters just as much. The PMC welcomed Jeffrey Vo as a member, with congratulations pouring in from Kevin Liu, Ian Cook, Matt Topol, Xuanwo, and others. Vo has been a steady force in the Rust implementation, and his elevation lands the same week the DataFusion community, where he also reviews, saw its own PMC addition.
The most interesting structural thread came from outside the project. Gleb Khmyznikov, a Microsoft engineer working on Python ecosystem enablement for Windows on Arm, brought a win_arm64 support plan to the list after review discussion on PR #48539. His framing is refreshingly honest about the burden question. The preconditions are reducing wheel count through abi3 so win_arm64 does not worsen the PyPI project size problem, unifying the Windows build path, and writing a support policy that names who is on the hook when Arm-only CI breaks. The commitments include a funded dedicated engineer from Linaro through the CoreCollective Windows on Arm working group, Snapdragon X-class hardware shipped to maintainers who want it, engineering time on the abi3 work itself, and a named escalation contact. The proposed starting policy makes win_arm64 wheels explicitly not a release blocker. This is the template for how platform vendors should approach open source projects: bring funding, hardware, and staffing rather than a feature request.
Two more Python-adjacent threads deserve attention. Nathan Goldbaum proposed requiring NumPy 2.0 or newer in the next Arrow release, unblocking support for NumPy's variable-width StringDType, which currently fails conversion with an ArrowNotImplementedError. The prior attempt stalled precisely because StringDType support requires targeting the NumPy 2.0 C API. And Nic Crane opened a discussion on limiting concurrent open PRs for non-committers after an uptick in AI-generated contributions where authors stop responding to feedback, leaving stale PRs that block others from picking up the work. Her quick analysis shows non-committers hold a median of 1 concurrent open PR, so a limit around 3 protects productive contributors while cutting the long tail. An ASF infrastructure PR to enable the corresponding GitHub setting is already open.
Ian Cook also posted the reminder for the Arrow community meeting on August 12 at 16:00 UTC, where the win_arm64 proposal and the NumPy 2.0 floor are natural agenda items. For practitioners, the NumPy question is the one to watch: StringDType is NumPy's answer to years of awkward object-dtype string handling, and Arrow support closes the loop so pandas and Polars users move string data across the boundary without copies or surprises. The cost is dropping NumPy 1.x support, and Goldbaum explicitly asked for real-world use cases that justify keeping the internal complexity of dual support. Silence on that thread becomes consent for the floor raise.
The AI contribution policy thread deserves a wider read than its subject line suggests. Crane's framing avoids the moral panic angle entirely and treats it as a queue management problem: a stale open PR signals that work is claimed, which blocks other contributors from picking it up, and unresponsive authors turn that signal into noise. The GitHub setting under discussion caps concurrent open PRs for accounts without write access, and her data-driven suggestion of 3 leaves the median contributor untouched while cutting the tail. Expect other Apache projects to copy whatever Arrow lands on, since every large repo faces the same flood.
Apache Parquet
Parquet made governance history this week. Julien Le Dem's second vote on using versions to release forward-incompatible changes passed with 5 binding +1s, 9 non-binding +1s, and no -1s, with late +1s from Fokko Driesprong and Ryan Blue arriving after the result. The decision formalizes major version numbers as the vehicle for bundling forward-incompatible features like new encodings, giving the ecosystem a clear signal about what a reader must support. Andrew Lamb captured the sentiment: this is a major step forward for communicating compatibility across the ecosystem. Implementation details move next to the Parquet Versioning doc.
The vote matters because the encoding pipeline behind it is full. Arnav Balyan announced that the FSST string compression proposal is moving from design to implementation after months of incorporating feedback. Devan Benz has an Arrow Rust implementation underway, Balyan has an Arrow C++ proof of concept, and the group is recruiting owners for Parquet Java and Arrow Go implementations to satisfy cross-language interoperability requirements before the formal vote. In the related OnPair string encoding thread, Prateek Gaur ran both encodings on one code base across 30 string columns and reached a genuinely useful conclusion: the dominant variable is how much of the column the writer samples before picking symbols, not code width or search algorithm. That finding pushes toward a single encoding with fewer spec knobs, where the writer trades compression against encode throughput without a format change. Andrew Lamb agreed and predicted heavy research investment in symbol table construction over the next two years.
ALP, the adaptive lossless floating-point encoding, got its conformance artifact. Andrew Lamb created a 211KB example file for parquet-testing, written with the C++ implementation and verified against the Rust one. The file's design is clever: the first two columns hold the same values PLAIN-encoded with zstd, so any reader verifies ALP columns by comparison without CSV ambiguity around NaN bit patterns. Gaur confirmed the file covers low precision, high precision, and outlier cases from the original datasets.
The proposal queue kept growing. Thomas Kissinger pushed back on the IEEE-based decimal floating-point proposal with a requirements-first argument: the type must cover 38 digits losslessly because that is the common boundary across SQL Server, Snowflake, Spark, Arrow Decimal128, Iceberg, Trino, and DuckDB, while IEEE decimal128 stops at 34 digits and decimal160 has no implementation ecosystem. His proposed 18-byte layout with a signed 128-bit significand covers all 38 digits. Julien Le Dem responded enthusiastically to Spotify's Random Access Parquet write-up, where Will Edwards described extracting metadata into a fast key-value store so AI agent point queries skip footer loading entirely. Le Dem suggested several tricks deserve first-class support: aligning pages on key boundaries when sorting, making pages splittable via zstd frames in the page header, and letting column pages be non-contiguous. Divjot Arora shipped a busy week of his own, with a PR to inline parquet.thrift into parquet-java removing the upstream parquet-format dependency, closure on forward compatibility for new sort orders with parquet-java set to emit IEEE_754_TOTAL_ORDER by default, and split spec PRs for extended precision nanosecond timestamps defining how readers handle unsupported logical and physical type combinations.
Zoom out on Parquet and the week reads as a coordinated push to make the format safe to extend. The versioning vote supplies the delivery mechanism. The ALP example file and the FSST interoperability requirements supply the verification gate. And the sort order thread supplies the compatibility playbook: before parquet-java started emitting IEEE_754_TOTAL_ORDER by default, Ed Seidl tested several implementations to confirm that older readers parse the unrecognized Thrift union value and simply ignore the stats rather than failing, exactly what the spec prescribes. Jan Finis asked whether that behavior even needs stating, and the answer from Arora is instructive: the spec already says readers should ignore stats for unknown sort orders, but the community verified real implementations honor it before flipping the default. That is what forward compatibility discipline looks like in practice, and it is the muscle the ecosystem needs before ALP, FSST, and a possible OnPair-informed encoding arrive through the new versioning process.
The random access conversation deserves practitioner attention beyond the novelty. Edwards' Spotify write-up describes serving AI agent point lookups from the data lake by storing extracted footer metadata in a key-value store, which changes the read path from load footer, search, and fetch into a direct byte-range read. Haocheng Liu chimed in that he is tackling similar random access improvements for AI use cases at his firm and pointed to Weston Pace's Lance blog series on file readers without row groups. The interest from two independent shops plus a Parquet co-creator suggests the point-query workload is becoming a first-class design input for a format built around large scans, and the concrete follow-ups Le Dem listed, key-aligned pages, splittable zstd frames, and non-contiguous column pages, give the community a menu to work through.
Fokko Driesprong closed out the Apache Parquet 1.18.0 release vote with 3 binding and 4 non-binding votes, testing against Iceberg himself and finding no regressions beyond expected NaN stat collection changes. Julien Le Dem reminded everyone the next Parquet sync lands Wednesday August 12.
Apache DataFusion
Comet crossed the milestone it has been building toward for two years. Andy Grove proposed the Apache DataFusion Comet 1.0.0 release, and the vote passed with eight +1 votes, six binding, from a verification crowd including L. C. Hsieh, Andrew Lamb, Matt Butrovich, and Oleks V. Comet accelerates Apache Spark by executing query plans through DataFusion's native Rust engine, and a 1.0.0 label tells production Spark shops the compatibility surface is stable.
Grove followed the release with a bigger question, opening a discussion on promoting Comet to a top-level ASF project. After two years incubating within DataFusion, Comet has its own contributor base, release cadence, and user community centered on Spark rather than on DataFusion itself. The discussion lives in a GitHub issue for now, and the outcome shapes how the ASF organizes the growing family of DataFusion subprojects.
That family kept shipping regardless. The Ballista 54.1.0 release vote passed with seven +1 votes, three binding, keeping the distributed DataFusion scheduler current, with verifications from Phillip LeBlanc of Spice AI and Renato Marroquín Mogrovejo among others. And the community welcomed Qi Zhu to the PMC, the most congratulated thread of the week at eleven messages. Zhu's reply focused on helping more new contributors get involved, which is exactly what you want from a new PMC member in a project growing this fast.
For readers newer to the subproject family: Comet is a Spark accelerator that swaps Spark's JVM execution for DataFusion's vectorized Rust engine under the existing Spark APIs, so teams keep their Spark code and get native-speed scans, joins, and aggregations. Ballista is the distributed scheduler that runs DataFusion plans across a cluster, filling the role Spark's driver and executors play but built Rust-native from the start. A 1.0.0 Comet plus a fresh Ballista release in the same week means the DataFusion ecosystem now offers both an embed-in-Spark path and a replace-Spark path, and the top-level project discussion is partly about giving the Spark-facing community its own governance home.
The Qi Zhu announcement also completes a pattern worth naming: DataFusion added a PMC member the same week Arrow elevated Jeffrey Vo, and both projects share reviewers, release verifiers, and infrastructure. L. C. Hsieh, Andrew Lamb, and Martin Grigorov show up in the vote threads of both communities this week. The Rust data stack behaves like one large project with several release trains, and the people pipeline reflects it.
Apache Ossie
The semantic layer project reached its most important disagreement yet, and it is a healthy one. Justin Talbot put concerns about PRs #246 and #237 on the record before any vote on the foundational semantics document and its compliance suite. His core argument is about adoption: no BI vendor has tried implementing the proposed semantics or querying through the proposed query model, and some specified behaviors around join direction, fan-out prevention, and many-to-many resolution conflict with choices tools like Tableau and Power BI already made and their users depend on. At 1,308 lines specifying the core of how a semantic layer behaves, Talbot wants broader vendor review with evidence the semantics are feasible before standardization.
Will Pugh responded with the standards-body counterargument: a standard needs a specified correct answer, join directions included, so every implementation gets the same result, and constraints can relax later when real cases demand it. He noted the sub-committee already chose to shrink the foundational scope, asked for specific feedback on the PR rather than a pause, and argued that working code surfaces semantic problems faster than review does. Talbot then filed a concrete alternative: standardize first on extending SQL with measure columns, which prevent measure duplication after joins without forcing join types or paths, letting BI tools layer their own behaviors on top. His diagnosis is that the current proposal bundles normative behaviors with opinionated ones, making adoption all-or-nothing.
The adoption question got sharper framing in the How do we expect OSI to be used discussion. Mario De Felipe argued the standard becomes relevant the first time one producer, his example being SAP, declares its semantics once and stops renegotiating them per consumer, with compliance defined behaviorally at construct level: represent it, reject it with a typed error, or declare it unsupported, but never accept-and-drop. He pointed to PR #311, which caught the repo's own dbt converter silently flattening composite keys, as proof the accept-and-drop failure mode is real. Elsewhere, Mikhail Nitsenko of Cube requested a maintainer review for the bidirectional Cube converter in PR #289, which follows the pattern of the merged WisdomAI and NVIDIA GSF converters, Ankit Tandon posted Ontology WG sync notes, and new contributor Kuladeep Sandra introduced himself offering documentation and enterprise use case help.
Cross-Project Themes
Format versioning is the connective tissue this week. Parquet formalized major versions for forward-incompatible changes, Iceberg opened v4 scoping and debated the operational reality of upgrades at petabyte scale, and Ossie argued about how much behavior a version 1 standard should pin down. All three debates are the same question at different layers: how does a format evolve when the installed base cannot move in lockstep? Parquet's answer is version bundles. Iceberg's answer is O(1) upgrades with gradual convergence. Ossie has not decided yet, and Talbot's SQL-with-measures proposal is a bet that smaller normative surfaces adopt faster.
Verification infrastructure is the second thread running everywhere. Iceberg's conformance fixtures proposal, Parquet's ALP example file with self-verifying PLAIN columns, the cross-implementation matrix Tserakhau demoed, and Ossie's construct-level compliance framing all reflect the same reality: these ecosystems now have enough independent implementations that shared test artifacts, not reference implementations, define correctness. The differential fuzzing result in the Iceberg thread, where a bounded run surfaced type-string divergence between readers, shows the payoff arrives immediately.
The third theme is the community managing AI's arrival on both sides of the contribution ledger. Arrow is designing PR limits in response to unresponsive AI-generated contributions, while Spotify's Random Access Parquet work exists precisely because AI agents issue point queries against lakehouse data. The formats are being reshaped for agent read patterns at the same time the projects are defending their review processes from agent write patterns.
What This Means for Practitioners
If you run Iceberg in production, three items from this week translate into action. Test PyIceberg 0.12.0 against your workloads when rc2 lands rather than waiting for the final, because the View support and File Format API changes touch read paths broadly. Pencil in the 1.12.0 timeline, with a branch cut on or after August 26 and a release in the weeks after, and get any must-have PRs onto the milestone now, since Salian is actively curating it. And if you rely on Avro tooling to inspect manifests, start planning for a Parquet-only v4 metadata world, because the community consensus formed fast and no one argued the other side.
If you run Polaris, upgrade to 1.7.0 for the CVE fix and evaluate the two new location flags. DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED prevents path prefix collisions between tables, which closes a class of overlap attacks, and ALLOW_CLIENT_SPECIFIED_TABLE_LOCATION set to false gives operators full control over where table data lives. Both default to safe-for-compatibility settings, so the protection is opt-in and you have to reach for it.
If you build against Parquet or Arrow, the versioning vote changes your planning horizon. New encodings like ALP and FSST will arrive bundled in a major version rather than trickling in as optional features, which means one compatibility conversation per version instead of one per feature. Track the Parquet Versioning doc as the details firm up, and if your shop writes files one engine reads and another consumes, the parquet-testing example files are the cheapest insurance available: point both implementations at them in CI and drift shows up as a test failure instead of a production incident.
Looking Ahead
Watch for PyIceberg 0.12.0rc2 with the correctness fix, the Iceberg 1.12.0 branch cut on or after August 26, and whether the Read Restrictions spec reaches its vote with Option A locked in. Polaris reviewers return from summer breaks to Rishindra's storage configuration phase 1 and Bourlatchkov's Data Context proposal. Parquet's Wednesday sync should set next steps on the versioning spec, and the FSST implementation recruitment for Java and Go tells us how fast the encoding lands. In Ossie, the response to Talbot's SQL-with-measures document decides whether the project pursues one standard or two competing philosophies, and JB Onofré returns from vacation August 20 to a stack of converter reviews.
Two broader currents also deserve a place on your radar. First, the encryption threads in Iceberg and Polaris are converging on the same design language: keys referenced by id from a managed list, metadata integrity verified through trusted storage, and raw key material banished from unencrypted files. Teams planning encrypted lakehouse deployments should read the Kaszab vote thread and the Kawai patches together, because catalog and format decisions here interlock. Second, the agent workload signal keeps strengthening. Random access Parquet, page index pruning in Iceberg's reader, and the manifest read acceleration work all serve the same emerging query shape: many small targeted reads issued by machines rather than few large scans issued by scheduled jobs. Format communities that internalize that shift early will define how the lakehouse serves AI systems for the next decade.
If you want to go deeper on Apache Iceberg, lakehouse architecture, data engineering, and AI, check out my full catalog of books at books.alexmerced.com.
Top comments (0)