Apache Iceberg 1.12.0 shipped this week, and the release set off a chain of follow-up work across the open lakehouse stack. Polaris started its upgrade to the new Iceberg client within a day. Arrow opened its 26.0.0 vote, DataFusion pushed four releases through in one stretch, and Parquet moved closer to a compromise on a numeric vector type. Apache Ossie (incubating) spent the week arguing about how AI agents should query a semantic model, which turned into one of the most useful design debates on any list this fall.
Underneath the releases sit three threads that run through every project. Contributors keep asking how AI tools fit into the way Apache projects review, test, and credit code. Vector data keeps pulling the file format, the table format, and the semantic layer toward new types and indexes. And several communities are drawing sharper lines between a specification and the code that implements it.
The Week by the Numbers
The six dev lists carried 403 messages across 102 threads between September 29 and October 7. Iceberg and Polaris tied for the busiest list with 118 messages each. Ossie followed with 88, DataFusion posted 45, Arrow had 19, and Parquet had 15. Message counts undersell Parquet and Arrow this week, since both lists handled format decisions that will shape files for years.
Release activity stood out. Iceberg Java 1.12.0 and Iceberg Go 0.7.0 both passed their votes. DataFusion Comet 1.1.0, DataFusion Ballista 55.0.0, and DataFusion Python 54.1.0 all passed, and DataFusion 55.2.0 collected enough binding votes by the end of the window. Arrow 26.0.0 RC0 drew five binding approvals before the week closed.
Apache Iceberg
Iceberg 1.12.0 ships
Neelesh Salian announced Apache Iceberg 1.12.0 on October 1 after the RC2 vote passed with four binding and ten non-binding approvals. Russell Spitzer, Kevin Liu, Szehon Ho, and Amogh Jahagirdar cast the binding votes. Russell thanked Neelesh and called the cycle a slog, and several contributors echoed the thanks. Hongyue Zhang added a hope that the project keeps a steady release cadence from here.
Neelesh followed the announcement with a release summary that reads like a map of where the format is going. Variant columns gained vectorized Parquet reads for unshredded data in Spark 4.0 and 4.1. Shredding now applies only to single-type fields, and Flink and Kafka Connect can write shredded Variant data. Geometry and geography types read and write in Parquet and Avro, with end-to-end support in Spark 4.1.
Flink gained a maintenance task that converts equality deletes into deletion vectors, plus SQL support for Iceberg views. Spark 4.1 picked up a Hilbert-curve clustering strategy for rewrite_data_files. The REST spec now describes read restrictions such as column masking and row filters. The release also lays the first V4 foundations, including a V4 manifest reader and writer, relative paths, and content statistics. Flink 2.2 and 2.3 joined the support matrix, while Spark 3.4 and Flink 2.0 left it. The release carries 777 commits from 142 contributors since 1.11.0.
The RC2 vote thread showed a new pattern in how people verify releases. Kurtis Wright opened his vote with a disclaimer that an agent skill for validating open source release candidates generated his checks. His report covered signatures, tarball contents, and the LICENSE and NOTICE files inside each of the 12 runtime jars. Kevin Liu ran the release against the PyIceberg CI and also pointed an agent at the repository to scan for correctness issues. The agent flagged a few problems, and Kevin confirmed they predate 1.12, so none blocked the release.
That detail matters for anyone running Iceberg in production. Human release managers still decide what ships, but the verification depth per voter keeps climbing. A single non-binding voter now checks third-party packages inside every bundled jar, a task few volunteers attempted by hand.
Spec changes and release notes
Péter Váry noticed something odd while reviewing the 1.12.0 notes. The Java release notes listed spec changes, such as the SQL UDF specification and geometry clarifications for V3. Péter argued that this misleads readers, since spec changes take effect when a vote passes or a spec version is finalized, not when a Java binary ships. Manu Zhang, Szehon Ho, Kevin Liu, and Hongyue Zhang agreed. Péter opened PR 18349 to separate the two, and Kevin asked for a matching changelog for the REST catalog spec, which has seen the most change lately.
Kurtis Wright pushed further and floated a separate iceberg-format repository to hold the specs and drive the website, similar to how Parquet keeps parquet-format apart from its Java code. Danny Jones supported the idea and suggested keeping commit history, as the iceberg-rust DataFusion move did recently. Daniel Weeks pushed back hard. He said the project has iterated on docs and site layouts many times, and that moving the specs adds little value while keeping them next to the reference implementation works fine. For now the practical outcome is a dedicated spec changelog, not a new repository.
REST catalog: User-Agent, delegation, and expressions
Rahul Mahadev reopened the vote on a standard User-Agent format for REST clients after taking review feedback from Alex Dutra. The proposal uses the standard HTTP header with product and version tokens, ordered from most specific to least. An example looks like Spark/4.0.0 iceberg-spark/1.9.0 iceberg-java/1.9.0 (scala 2.13.16). When a client sends the header, it must include an Iceberg library token such as iceberg-java, pyiceberg, iceberg-rust, or iceberg-go.
The vote spells out what the header is not. Servers must not reject requests for a missing or malformed header, and they must not use it for authentication. It is not a capability negotiation mechanism. Servers can refuse operations from client versions known to be unsafe. The thread collected binding approvals from Daniel Weeks, Russell Spitzer, Sung Yun, Amogh Jahagirdar, Eduard Tudenhöfner, Gang Wu, Zheng Hu, and Ryan Blue. Zheng Hu also noted CVE scanning failures on PR 17727 that need attention before merge. For catalog operators, this gives a consistent way to see which engines and library versions hit the catalog, which helps with support and with retiring unsafe clients.
A second REST thread asked what servers should do when they cannot satisfy the X-Iceberg-Access-Delegation header. Alex Dutra proposed a new value named local, so a client can say it works with its own credentials. Daniel Weeks preferred none, which tells the catalog the client does not need credentials from it, so the catalog can skip generating them. The same debate ran on the Polaris list, covered below.
Revanth Ch raised a migration question about the new expression JSON forms. The expressions spec now prefers a form with explicit left and right terms, while the Java, Go, and C++ clients still read and write only the older form. Revanth proposed conformance fixtures in the iceberg-verification repository to track the migration. Ryan Blue answered with a simple recommendation: servers should drop residuals entirely, since they cost a lot to compute and clients rarely apply them.
V4 design work
The V4 spec work produced three focused threads. Anoop Johnson opened a discussion on stats tightness. Tight bounds let an engine answer some queries from metadata alone, but once a deletion vector attaches to a file, today's rules treat every bound as loose. Anoop wanted a way to keep tightness when a DV is present. Russell Spitzer walked through the cases where per-column tightness helps and concluded that most are narrow or hypothetical. Daniel Weeks, Eduard Tudenhöfner, Hongyue Zhang, and Szehon Ho all backed a single file-wide flag.
Eduard Tudenhöfner raised a rename of avg_value_size_in_bytes to total_bytes in content stats, tracked in PR 18308. A total aggregates cleanly across files, and the average still falls out of total bytes divided by value count. The timing hurt, since avgValueSizes() landed in central Java classes and the REST spec in 1.12.0. Ryan Blue suggested storing totalBytes and computing avgValueSize from it in the Java API. Steven Wu agreed and noted that the REST spec does not yet support V4 tables, so a JSON break there is acceptable.
Gábor Kaszab pointed the list at PR 17533 on reusable encryption keys for statistics files. If the approach holds, the same pattern applies to encrypting the V4 root metadata file. Sandeep Gottimukkala proposed a one-sentence spec clarification on promotable types in PR 18404. Readers must accept any data file column whose type promotes to the field's type, whether or not the field evolved. Sandeep said some systems check for an exact type match and break on valid tables.
Alexander Loeser and Andrei Tserakhau kept the collation discussion moving. Both prefer storing bounds for multiple collation versions per data file, which helps during ICU upgrades. The open question is layout. Andrei plans to prototype the current content_stats layout against Russell's idea of derived COLLATE(field, collation, version) fields and compare on-disk size with one, two, and three live versions.
Indexes, vectors, and a new file format
Shawn Chang reopened a settled question about the index spec's non-overlapping region files. He listed the costs: write amplification on hot key ranges and more maintenance work. Péter Váry explained the design goal. The index targets fast, predictable single-key lookups, and non-overlapping regions mean one region-file read once the tracking file sits in cache. Region files stay small, around 4 MB, so writing a consolidated region costs about the same as writing an overlay. Péter later posted the index sync summary, confirming non-overlapping regions, total row ordering through sort keys (renamed from cluster keys), and a ready-for-review spec in PR 16961.
Steven Wu announced that the Thursday biweekly sync will cover both File and Vector types starting October 8. The file type work nears consensus, so most of the time will go to the Vector type design from Yan Yan. That matters because Parquet is debating its own vector type this week, and the two designs need to line up.
Martin Prammer started a mailing-list discussion on adding Vortex as an Iceberg file format. The draft now opens with an overview of Vortex and its community policies, after questions at the last community sync. Martin asked the list to confirm two things before going further: whether the general rules for adding any file format match expectations, and whether concerns remain about how Vortex meets them. Iceberg has supported Parquet, ORC, and Avro for years, so any new format faces a high bar on governance and long-term stability.
Clients and catalogs
Matt Topol's Iceberg Go v0.7.0 vote passed with four binding approvals from Russell Spitzer, Sung Yun, Fokko Driesprong, and Matt. Xuanwo reported 3,344 passing tests on Go 1.25.9 and one harmless assertion failure on Go 1.27.1. Russell joked that he complained to Matt about a Claude.md file in the release.
In Rust, Danny Jones posted a plan to drop the dependency TSV files from iceberg-rust. The files serve no license obligation and go stale between releases. Danny prefers Cargo for consumers and cargo-deny policies in CI for maintainers. Shawn Chang, Kevin Liu, and Kurtis Wright agreed, and Xuanwo, who added the files, supports removal.
Andrei Tserakhau revived a debate on catalog-provided metadata tables. Java and Spark treat metadata tables as deterministic projections of table metadata, which led to closing a labels metadata-table PR. Andrei argued that PyIceberg, iceberg-rust, and iceberg-go expose inspect as a library read API, so returning catalog labels as Arrow lets users filter many tables at once for PII, ownership, or cost tags. Manu Zhang objected that the API cannot distinguish a table with no labels from a catalog that does not support labels.
Javier Sanchez Beltran from Expedia asked why GlueCatalog writes so little into the Glue table on commit, compared with the Hive catalog. Shawn Chang said the small footprint is intentional, since Iceberg metadata stays the source of truth and Lake Formation handles authorization. Daniel Weeks named the deeper problem a split brain. Both HMS and Glue mirror some Iceberg metadata but not all of it, and tools that read the mirror instead of the Iceberg metadata get stale or lossy answers. Javier replied that his team will change its tooling to read the Iceberg metadata directly. That is the right lesson for any platform team: treat the catalog entry as a pointer, and read the table's own metadata for truth.
The Kafka Connect runtime drew attention from two directions. Confluent's connector hub sent a vulnerability notice warning that the Iceberg connector faces removal unless the project addresses flagged issues. Robin Moffatt bumped his discussion on PR 18019, which fixes the runtime LICENSE and NOTICE files and drops the Hive distribution. Jad Naous from Grepr also submitted a fix for a race condition in Flink's DynamicIcebergSink through PR 18356.
Planning 1.13
Anoop Johnson opened the 1.13 discussion and volunteered as release manager. Anurag Mantripragada had already claimed 1.13, so Anoop took 1.14 instead. Anurag proposed early January, given the holiday season, with the V4 spec PR, early V4 work, and Spark 4.2 support on his list. Xin Huang noted that the Spark 4.2 setup PR has merged while publishing stays skipped. Xin's strawman supports existing Iceberg features on Spark 4.2 and treats new Spark features such as the Transactional API as best effort. Daniel Weeks asked the group not to rule out a smaller release before year end, and Kurtis Wright added JDK 25 support to the wish list.
Community events
Three meetups landed on the calendar. Talat Uyarer opened the CFP for the Bay Area meetup on November 2 at Google in Sunnyvale, with submissions due October 19. Alex Stephen reminded the list that the Seattle meetup on October 21 still needs talks. Scott Haines announced the October virtual meetup on October 23, where Liam Earley from Zus Health will cover manifest-level work in iceberg-rust to cut scan latency on TB-scale merge-on-read tables.
Apache Polaris
Riding the Iceberg 1.12 upgrade
Alex Dutra moved fast after the Iceberg release. He marked the Polaris upgrade to Iceberg 1.12 ready for review in PR 5360 and addressed the main worry up front. Iceberg 1.12 fixes URL path segment encoding, so the Java client now sends a space as %20 instead of +. Alex explained that Polaris never used the old decoder on paths, since JAX-RS decodes path parameters per RFC 3986. The upgrade fixes the wrong URLs Polaris used to advertise and needs no migration of persisted namespace encoding. Yufei Gu urged the community to review the PR.
The same release unblocks older work. Alex asked whether Polaris should resume remote signing, which stalled while the spec changed. Iceberg 1.12 adds a remote-signing-config field to LoadTableResult, which covers nearly every remaining requirement. One gap remains: refreshing the auth data inside that structure was left out of scope, and Alex plans a separate spec proposal with the Iceberg community.
Storage: Cloudflare R2 and FileIO cleanup
The busiest Polaris thread settled how to add Cloudflare R2 with scoped credential vending. Yufei Gu worried that a free-form credentialVendingMechanism string lets a client work against one server and fail against another. Dmitri Bourlatchkov argued that an enum blocks downstream projects from adding custom mechanisms. Jean-Baptiste Onofré pointed out that only catalog administrators touch this Management API field, since Iceberg REST clients just consume vended credentials.
The community sync produced a compromise. The OpenAPI spec keeps the field as a string, documents the accepted values, and servers validate at runtime. Austen Tomek confirmed the plan for his PRs, starting with STS and adding R2 in a second PR. Yufei proposed naming the R2 mechanism after how it signs, and Austen picked R2_LOCAL_SIGNING, since token formats vary by provider. Dmitri and JB agreed.
Two threads tackled how Polaris builds its own server-side FileIO. Vignesh A. proposed dropping catalog table-default.* properties from server FileIO construction before a table-metadata cache lands. An earlier fix already stopped using caller-supplied metadata properties, tied to CVE-2026-97395. Dmitri opened a parallel thread with the same conclusion: server FileIO should not depend on default table properties.
The cache work also raised a deeper question about metadata immutability across storage endpoint changes. Yufei Gu asked whether changing an S3-compatible endpoint can make the same metadata URI return different content. Dmitri reframed it as a cache correctness issue. Polaris does not control what external stores contain, so it should not assume two endpoints serve identical bytes. JB split the difference: Iceberg treats metadata files as immutable within one storage context, but an s3:// URI is not a content hash. Robert Stupp summarized the two assumptions the cache depends on, and neither follows from the URI alone. For anyone running Polaris across several S3-compatible stores, this thread is worth reading before enabling metadata caching.
Ashok Borra raised a performance regression from validating historical metadata references on every commit. Since PR 5115, each commit checks the location of every snapshot's manifest list and every statistics file. Ashok proposed limiting the check, and Alex Dutra agreed.
Persistence: DynamoDB and multi-object commits
The DynamoDB persistence backend reached agreement. JB and Robert Stupp first argued that the NoSQL persistence layer fits DynamoDB better than the relational-style AtomicOperationMetaStoreManager. Yufei Gu and Yuewei Zhou countered that the atomic path adds little complexity, since Polaris correctness already works per entity with version checks. Dennis Huo pushed for an iterative approach that does not block progress. The sync recap from Dmitri set the terms: multiple DynamoDB implementations are acceptable, docs must explain the differences, and new backends start as beta. Yuewei asked for reviews on PR 5622, the first in a series of small changes.
That debate fed a broader one on consistent multi-object changes. EJ Wang described a race where two requests both check for overlapping table locations, see none, and create conflicting tables. EJ proposed a business-aware manager layer that runs those checks in the same transaction as the write, with database adapters that know nothing about tables or grants. The proof of concept includes adapters for FoundationDB, Spanner, and JDBC, and its concurrency tests ran on FoundationDB and PostgreSQL. JB favored multi-object conditional commits over request-scoped transactions, since they map to both JDBC and NoSQL.
Metrics, events, and observability
Polaris is building an observability surface for catalogs. Dmitri proposed merging PR 4115 for REST endpoints that expose table metrics and events, and JB confirmed all feedback was addressed. The grant changes moved to PR 5659. Yufei argued that a new TABLE_READ_METRICS privilege should stay separate from TABLE_FULL_METADATA, since metrics reports include query filters from other users. JB agreed and called metrics closer to an activity log than to table metadata.
Vignesh A. opened a thread on a shared event JSON envelope before merging a webhook listener. The group settled on CloudEvents context attributes at the top level, the Polaris payload inside data, and Kafka parity in version one. Alex Dutra supported isolating the events SPI in its own module. The goal is one payload shape across Kafka, webhooks, and CloudWatch, so downstream consumers parse events the same way regardless of transport.
New catalog features: branches, sharing, and tags
Ankur Kotwal opened a discussion on catalog-level branches for atomic multi-table promotion, tracked in issue 3469. Iceberg already branches single tables. Teams running backfills, migrations, or write-audit-publish across many tables face three problems: N calls to branch N tables, non-atomic promotion, and no cheap way to list which tables carry a branch. Ankur proposes a CATALOG_BRANCH entity that groups per-table Iceberg branches and promotes them in one transaction through the existing commitTransaction workspace. Engines need no changes, and version one supports fast-forward promotion only. This is the feature many teams adopt Nessie for today, so it deserves close attention.
Dennis Huo marked the Open Sharing API spec PR ready for review in PR 5446. Sharing builds on existing Polaris authorization, with external identity providers as a first-class case. JB asked the group to focus on the API PR first. EJ Wang also pushed updates to the Tag Spec proposal, including separate endpoints to assign and unassign tags. David Chaava proposed a validator SPI for vendor-specific REST federation, with a BigLake implementation that keeps the admin service provider-neutral.
Process: changelogs, AI disclosure, and 1.9.0
Alex Dutra proposed fixing CHANGELOG.md merge conflicts with per-PR fragment files. Ayush Saxena brought numbers: 75 of 339 commits on main in 60 days touched the changelog. JB asked whether Polaris needs a changelog at all, and suggested a label plus hand-written notes assembled at release time. Robert Stupp suggested conventional commit messages instead. Dmitri noted that fragment files can be corrected before a release, which commit messages cannot. The thread had not settled by the end of the week.
Alex also raised AI-generated code disclosure. Polaris's AGENTS.md tells agents not to include AI tool names in commit messages, while Iceberg encourages disclosure and current ASF guidance recommends it. Travis Bowen supported aligning with the ASF. Dmitri agreed that disclosure helps but objected to Co-authored-by, since GitHub then lists the tool like a human contributor. He prefers a header such as Generated-by, which matches ASF language that separates generated material from authored work.
JB proposed Polaris 1.9.0 for the end of October to keep a monthly cadence, with the Polaris Directory feature as a beta. Ayush asked to update the DOAP file after each release, since ASF community pages read from it. Dmitri wants R2 support in as beta and flagged a LICENSE and NOTICE issue that an AI review tool found in his environment. He wants it fixed so the team can rely on AI tools for release validation.
Apache Arrow
Arrow 26.0.0 heads for release
Raúl Cumplido called the vote on Arrow 26.0.0 RC0, a release with 337 resolved GitHub issues. Ruoxi Sun, Sutou Kouhei, Gang Wu, Raúl, and Adam Reeve cast binding approvals, and Xuanwo verified a byte-for-byte reproducible source tarball. Raúl flagged two non-blocking verification problems. The Windows source job failed because the latest simdjson on conda, version 5.0.1, does not compile with MSVC. The wheel verification failed on conda because Python 3.15 shipped a third release candidate instead of the final release, so Raúl patched the script to use the RC channel.
Jeffrey Vo announced a planned arrow-rs 59.3.1 patch release and invited backport requests on the release ticket.
Format work: range types and JSON schemas
Florian Hölzlwimmer opened a vote on two canonical extension types for ranges. arrow.fixed_closedness_range stores a struct of lower and upper bounds, with one closedness setting for the whole column, which maps one to one onto pandas IntervalArray. arrow.variable_closedness_range adds per-value flags for inclusive bounds, which PostgreSQL's numrange needs. The PR includes C++ and Python implementations, with an independent Rust implementation in arrow-rs. The vote stays open through October 9.
David Lee contributed to the JSON representation of Arrow schemas discussion. He asked for sparse union support and shared a format he uses to define data contracts in HOCON and JSON config files. David Li signed off on the updates.
Python packaging and contributor health
Alenka Frim responded to Raúl Cumplido's proposal to improve PyArrow modularity. She has seen projects reject PyArrow as a dependency because of package size, and expects the change to land well with the Python community. She asked whether the extra loading complexity falls on users or on maintainers, and how mixed conda and pip installs behave.
Pearu Peterson asked how to get a four-month-old PR reviewed. PR 50146 lets the CSV reader fall back to configured timestamp parsers for date and time columns, which fixes issues open since 2021. Ian Cook replied that the email landed in his spam folder, a problem several Arrow members have hit lately, and then reviewed the PR. Antoine Pitrou asked for board report input by October 8, before he leaves for Community over Code in Glasgow.
Apache Parquet
A compromise takes shape on VECTOR
The numeric vector type thread carried the most consequential debate on the Parquet list. Sebastian Baunsgaard argued against requiring finite elements in parquet-format PR 624. Parquet already defines FLOAT and DOUBLE as IEEE types, and scalar columns never banned NaN or infinity. Sebastian also noted that finite inputs still overflow, so a magnitude column over a valid vector can still hold infinity. He proposed an optional finite flag on the type.
Will Edwards suggested a more general path: standard length constraints on variable-length types, so VECTOR(N) becomes a list with matching minimum and maximum lengths. The same mechanism also expresses SQL CHAR(N) and VARCHAR(N). Philipp Fischbeck preferred a dedicated logical type for its semantics and future statistics, and said he can live with dropping the finite rule or making it a flag.
Ryan Blue then summarized the shared ground and named the split. Everyone wants a logical type now and a better physical layout later. Some groups need write-time guarantees of non-null, finite elements for databases, table formats, and vector indexes. Others want NaN, infinity, tensors, and smaller floats. Ryan proposed two types: a FINITE_VECTOR and a separate fixed-length list or vector that allows special values. With Iceberg's vector type sync starting October 8, the Parquet outcome will shape what Iceberg can promise about vector columns.
Encodings, decimals, and wire formats
The FastLanes evaluation produced an apology and a reset. Antoine Pitrou apologized to Prateek Gaur for harsh words in an earlier message and asked for a higher-level document that compares the current DELTA_BINARY_PACKED, PFOR, FastLanes, an improved delta decoder, and an interleaved variant on shared axes. Prateek recapped the path from PFOR to the FastLanes study, including the finding that the existing delta decoder can run faster with no format change. He had already trimmed the evaluation doc for readability. Julien Le Dem thanked him for the work.
Costas Zarifis posted a formal proposal for an extensible decimal floating-point type. It backs the full IEEE 754 decimal model, one logical type that keeps the supplied representation, and IEEE totalOrder for ordering and statistics. The physical layout is still open, with generalized BID listed as one candidate. Thomas Kissinger said he will review it in the doc.
Neelesh Salian answered Micah Kornfield's review of WIRE, a logical type for serialized wire-format messages such as protobuf. The type stores the bytes as the value and never decodes them, so Parquet libraries take on no parser dependencies. Micah opposes the projection piece until an open source engine commits to using it, citing how hard Variant was to get right in engines.
Andrew Lamb weighed in on a proposal to treat distinct_count as inexact. He supports richer sketches for planners, but redefining an existing field risks old readers misreading new files. He suggested new standardized metadata instead. Julien Le Dem announced the October 7 Parquet sync, open to anyone.
Apache DataFusion
A four-release week
DataFusion ran a release marathon, and the most instructive moment came from a failed vote. Andy Grove started deeper verification of Comet 1.1.0 RC1 and found regressions, some serious. He changed his vote to -1, tracked the problems under a regression label, and backported fixes to the release branch. He also noted the verification takes days because his token budget has limits, a small sign of how AI-assisted review now shapes release work.
Comet 1.1.0 RC2 passed with nine approvals, six binding, from Andrew Lamb, L. C. Hsieh, Marko Milenković, Oleks V., Matt Butrovich, and Andy. Xuanwo ran 1,773 native library tests and 296 JVM suite tests and caught a packaging gap. The staged Spark 4.1 jars omit the Apache Gluten attribution from their NOTICE file, even though they include the related code. Andy opened a PR to fix it going forward.
Ballista 55.0.0 RC1 hit a different snag. Andrew Lamb and Bhargava Vadlamani saw tests hang or fail on macOS. Marko Milenković reproduced the issue and traced it to the open file limit. Raising ulimit -n to 2048 or higher fixed verification, and the vote passed with six approvals, four binding.
DataFusion 55.2.0 RC1 collected binding approvals from Andy Grove, L. C. Hsieh, Marko Milenković, Andrew Lamb, and Tim Saucer. Erik Bogado tested it on AMD Ryzen, Intel Xeon, and ARM Neoverse-N1 hardware. Tim Saucer's Python bindings 54.1.0 passed with five approvals, four binding. Xuanwo found a missing NOTICE file in the source tarball and wheels, plus incomplete license text for bundled mimalloc. Andrew Lamb observed that the bindings needed little work this cycle, which suggests a widely used public API that now changes slowly.
Decoupling Ballista from the Python release train
Andy Grove proposed releasing Ballista's Python client separately from its Rust crates. Ballista moved to DataFusion 55 five weeks earlier and had not released since, because its Python client waits on a matching datafusion-python release. Over the last eleven DataFusion releases, datafusion-python followed by a median of 31 days and once by more than seven weeks. Meanwhile, more than 300 commits sat unreleased, including fixes for wrong results from distributed NOT IN, executors killed for running out of memory, and jobs hanging when all executors drop. The 55.0.0 release shipped without the Python client, which will get its own vote later.
Security and AI policies
Andrew Lamb noticed DataFusion had no published security policy and proposed policies for the core crate and for sqlparser-rs. He is also revising the AI contribution policy with Jeffrey, Neil, Yongting, and others in PR 24510. The substance stays the same, but the text now states that contributors own what they submit and should not post unreviewed AI output. Andy Grove proposed AI guidance for Comet contributors in PR 6734. Automated reviews can help, but a human must perform every PR approval.
Apache Ossie (incubating)
Should a semantic model ship an MCP server?
The longest Ossie thread started as a GitHub discussion about building an MCP server for Ossie. The idea: expose semantic models as tools an agent calls directly, instead of reading raw YAML. Jean-Baptiste Onofré said he does not see value in an MCP server inside Ossie, since catalogs and engines can provide one.
Markus Cozowicz reframed the pitch as an MCP specification, not an implementation. In his view, Ossie's biggest value is helping agents understand semantics and build correct queries, and an execute_query tool contract sits at the center of that. Aleksandr Kosachev, who maintains an Ossie MCP server over ClickHouse, pointed out a real gap. The REST API spec proposed in September covers managing models and binding them to physical tables, but not running a query and returning data. Markus later opened PR 529 with a draft.
Chris Eubank pressed on the semantics. He asked whether execute_query should steer clients toward portable Ossie query interfaces or also standardize vendor dialects compiled outside Ossie, since the draft ties loosely to Ossie. Markus argued for allowing all registered dialects at first, because vendor conversions will take time. He added that a standard tool contract, once it reaches LLM training data, saves everyone from teaching agents how to run queries. Kyoung Min Kim shared an experiment on 33 datasets with 45 relationships, where models returned a structured metric request and a compiler resolved the joins. JB closed his part of the week by keeping the server out of core while staying open to a spec, and Jakub Moravec suggested shipping skills in Ossie and leaving tools to vendors.
This is the most important open question in the semantic layer space right now. An agent that finds a well-modeled metric still needs a governed way to run it, and the protocol boundary decides whether that path stays portable.
Query layers and Measures in SQL
Justin Talbot's specification layers overview in PR 452 frames Ossie as a model definition layer plus distinct query interfaces. Julian Hyde shared a lesson from Looker. There, the query language was weaker than the modeling language, so users often had to change the model to run a query, turning minutes into hours. That frustration led him to propose Measures in SQL. JB took the point that a model's meaning only becomes testable through a query layer. Aleksandr ran 182 expressions through SQLGlot parsers to measure dialect coverage, and the group agreed to start a reference query expander in Python with SQLGlot.
The Relational Query Interface spec in PR 354 grounds that layer in Measures in SQL. Julian objected to the TO MEASURE syntax: it is not idiomatic SQL, its precedence is unclear, and it blocks anonymous inline measures. Justin prefers AS MEASURE, as in the original paper, and JB changed his position after spotting inconsistent parentheses in the draft. Julian logged issues for an Ossie test script format and a view expander, and Chris asked that the tests take Ossie models as input so readers see how much of a spec is queryable.
Metrics, filters, and relationships
The dataset-scoped metrics proposal from Josh Klahr in PR 343 converged. Chris Eubank proposed narrowing scope to get a clean vote. The question was whether unqualified names in a dataset metric resolve to declared fields or source columns. Justin Talbot backed fields first, with explicit precedence rules. Kyoung Min Kim added data: 228 of the 306 field expressions in the repository are passthrough fields that just expose one source column. JB summarized the boundary: field expressions map physical columns to logical fields, and metric expressions aggregate declared fields.
Chris's vote to add FILTER (WHERE ...) to the expression language gathered support. Kyoung Min re-ran the lowering rules on DuckDB 1.5 and confirmed the difference that motivated the change. A CASE workaround with ELSE 0 returns an AVG of 55.0 where FILTER returns 73.33, a silent wrong answer in any metric built the old way.
The relationship cardinality thread reached a plan. JB proposed one vocabulary across the ontology spec, core spec, and PR 354: PascalCase multiplicity values, ManyToOne as the default, and OneToMany excluded from core. Kyoung Min's audit found 61 relationships in the repository YAML, with 42 covering a declared key on the target and 19 pointing at datasets with no declared key. A related thread on dataset.column agreed that model-level expressions should reference logical fields, not warehouse columns, following the SQL view analogy Julian and Justin described.
Repository structure and extensions
JB created the ossie-converters repository and opened PRs to move the converters out of the spec repo. Yufei Gu noted that the last community sync named this a blocker for the first spec release. Kurt Stirewalt explained why semantic models were embedded inside ontology mappings: the original goal of single self-contained documents. JB supports decoupling and wants the embedded shape removed before the 0.3.0-incubating release.
Mike Carlo's proposal to make custom extension data a structured object instead of an opaque JSON string drew support from Yufei, Markus, and Chris. Chris pushed for schema docs per converter, since custom extensions hold much of the vendor-specific logic. Kyoung Min and Andrew Nepogoda agreed on semantic checks for ontology documents in validate.py. A bidirectional OpenMetadata converter also appeared.
Ossie keeps attracting new contributors. Aleksandr Kosachev introduced himself as a backend architect who kept seeing teams wire AI assistants to databases and get confident, wrong answers. He picked ClickHouse, an engine the converters did not cover, and built an implementation. His measurements showed up in four threads this week.
Cross-Project Themes
AI enters the release and review process
Every project touched AI-assisted contribution this week, and each took a slightly different stance. Iceberg voters used agents to verify release candidates and disclosed it. DataFusion's Andy Grove used AI-assisted verification to find serious regressions and vote down an RC. Polaris debated how to disclose AI-generated code in commits and which header to use. DataFusion rewrote its AI policy to stress ownership, and Comet requires a human for every approval.
The pattern is consistent. Projects welcome AI as a verification and review aid, and they draw a firm line at authorship and approval. Disclosure is moving toward the ASF's recommendation. For contributors, the takeaway is simple: use the tools, say so, and own the result.
Vectors pull on every layer
Vector data showed up at three layers at once. Parquet debated whether a VECTOR type guarantees finite elements. Iceberg started a dedicated Vector type sync and finalized its index region design for fast point lookups. Ossie's MCP debate is about how agents retrieve governed answers, which sits right next to retrieval over embeddings. The file format decides what a vector column can hold. The table format decides what it can promise and index. The semantic layer decides how agents ask for it.
Specs, implementations, and sources of truth
Several threads asked where the truth lives. Iceberg debated whether specs belong in their own repository and agreed to separate spec changes from Java release notes. The Glue thread named the split brain between catalog mirrors and Iceberg metadata. Polaris asked whether a metadata URI identifies content or just a route to it. Ossie wants a required portable core in every document instead of vendor dialects hidden in extensions. Arrow kept refining a JSON form of its schema. Each case pushes toward one authoritative definition, with everything else derived from it.
Release hygiene is a shared job
LICENSE and NOTICE problems surfaced in Iceberg's Kafka Connect runtime, DataFusion Comet's jars, DataFusion Python's wheels, and a Polaris pre-release check. Each fix came from a careful voter, often with tool help. Open source licensing compliance rarely makes headlines, but it decides whether downstream vendors can ship these projects at all.
Looking Ahead
Watch the Iceberg Thursday sync on October 8, where the Vector type takes center stage, and the Parquet community's response to Ryan Blue's two-type compromise. The Arrow range-type vote closes October 9, and the Arrow 26.0.0 result should follow. Polaris aims to cut 1.9.0 at the end of October, so the R2, DynamoDB, and Iceberg 1.12 PRs face review pressure now. In Ossie, track PR 529 on execute_query and the converter repository move, which gate the first spec release. The Iceberg 1.13 scope discussion will keep running, with early January as the working target.
Keep Learning
If this digest helps you follow the open lakehouse, my books go deeper on every layer covered here, from Apache Iceberg and Apache Polaris to Arrow, Parquet, semantic layers, and agentic analytics. Browse the full catalog at books.alexmerced.com and pick the one that fits the problem in front of you.
Top comments (0)