Half the lakehouse community spent this week in Glasgow at Community Over Code, and the mailing lists show it. Syncs got cancelled, board reports went out late, and several threads open with a nod to a hallway conversation from Lakehouse Day. The work did not slow down. Apache Iceberg ran three overlapping v4 spec votes at once, Apache Parquet put a concrete versioning proposal on the table, Apache DataFusion Comet asked to become its own top-level project, and Apache Ossie (incubating) argued about where an MCP server belongs.
The thread that ties the week together is precision. Each community is replacing a loose rule with an exact one. Iceberg is pinning down snapshot timestamps and statistics guarantees. Parquet is fixing the meaning of a statistic that never had a firm definition. Polaris is redrawing the line between catalog configuration and server storage access. Ossie is deciding which parts of a semantic model must be portable before anyone can query one. These are the unglamorous rules that let engines, catalogs, and AI agents trust the same data without asking a human first.
This issue covers Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, Apache DataFusion, and Apache Ossie (incubating). All thread links point to the official archives at lists.apache.org.
Apache Iceberg
Three v4 votes in one week
The Iceberg v4 table spec moved further this week than in any week this fall. Three separate spec votes ran in parallel, and none of them drew a single -1.
The first to close was Steven Wu's proposal to require monotonically increasing snapshot timestamps in v4. The rule is simple. Every new snapshot on a branch in a v4 table must carry a timestamp-ms strictly greater than its parent's. Root snapshots stay unconstrained, and v1 through v3 tables keep their current behavior. Steven explained that out-of-order timestamps break time travel today, because a query for "the table as of 3:00 PM" picks the wrong snapshot when clocks drift between writers. The rule also opens the door to a row timestamp metadata column inherited from the snapshot timestamp, which Steven flagged as a separate follow-up change. The vote passed with 9 binding +1s and 11 non-binding +1s. Maninder Parmar added a note worth tracking: he wants the community to consider nanosecond precision (timestamp-ns) later, so high-throughput tables do not hit collisions at the millisecond level. Xuanwo restated the scope in his +1, confirming that pre-upgrade snapshots stay exempt.
The second vote is the big one. Amogh Jahagirdar opened a vote to merge the baseline v4 AMT spec PR into the main repo. AMT is the adaptive metadata tree, the new metadata structure at the center of v4 that changes how Iceberg organizes manifests and commits. Amogh was careful about what the vote means. It is a vote to land a baseline in the repository so other v4 work stops fighting rebases, and it is not a vote to adopt the design as final. Encryption, column file update rules, and dropping the partition tuple all remain open for later iteration. The thread filled with +1s from Steven Wu, Daniel Weeks, Russell Spitzer, Eduard Tudenhöfner, Fokko Driesprong, Gang Wu, Szehon Ho, and Péter Váry, plus a long list of non-binding supporters. Gábor Kaszab volunteered to cover the encryption piece in an immediate follow-up. Contributor 전대홍 said he looks forward to building AMT in iceberg-rust, which tells you the Rust implementation intends to track v4 closely rather than trail it.
The third vote opened on October 13. Talat Uyarer, writing for the FILE type working group, called a vote to add the File type to the Iceberg v4 spec. This is a new data type for a range of bytes, stored either inline or as a reference to an external file. Its value semantics follow the FILE logical type in the Parquet spec. A file field carries six fixed, implicit sub-fields: uri, offset, size, content_type, checksum, and inline. They never appear in the schema and cannot be added, removed, reordered, or promoted. Their field IDs are offsets one through six from the file field's own ID, so adding one file column moves last-column-id forward by seven.
The restrictions matter as much as the feature. Equality, ordering, and hashing are undefined for file values. A file column cannot be an identifier field or a source for partition or sort transforms. It must default to null. A struct with the same sub-fields is not interchangeable with a file and cannot be promoted into one. And a uri makes no promise that the referenced object still exists. Relative URIs resolve against the table location, which keeps tables portable when you move them. Each sub-field gets its own column statistics, and writers should produce stats for uri, content_type, and inline. The format mappings are clean: the FILE logical type in Parquet, a record in Avro, and a struct tagged iceberg.struct-type=FILE in ORC. By the end of the window the thread had binding +1s from Russell Spitzer, Steven Wu, Daniel Weeks, Gang Wu, and Sung Yun, with no objections. The 72-hour window closes this week.
Why does a file type matter? Because the lakehouse is absorbing unstructured data. Images, PDFs, audio clips, and model artifacts already sit next to tables in object storage. Today a table points at them with a plain string column and a convention. A typed file column gives engines the offset, size, content type, and checksum they need to read, verify, and govern those bytes with the same metadata machinery Iceberg uses for everything else. For teams building AI pipelines on Iceberg, this is the bridge between "a table of embeddings" and "a table of the documents those embeddings came from."
Statistics and vectors
Two design threads moved toward consensus without a vote. Anoop Johnson's Iceberg v4 stats tightness discussion settled on a single file-level flag. Hongyue Zhang laid out why bounds stop being tight. Deletion vectors added after a file is written remove rows, so the original min and max bounds no longer describe the live data. Variable-length columns such as strings and binary also carry truncated bounds by design. Steven Wu and Szehon Ho backed the file-level approach, citing Russell Spitzer's analysis that most use cases repair every bound at once and do not need per-column tracking. Anoop summarized the upgrade path. Files inside existing v3 manifests carry no flag, so readers treat their stats as wide. New v4 files carry true when no deletion vector is attached and false when one is. This matters for query planning. An engine that knows bounds are tight can answer some aggregate queries from metadata alone and prune more files with confidence.
On the vector side, Yan Yan extended the Thursday biweekly sync to cover both the File and Vector types and posted a draft spec PR built on the Parquet community's emerging vector work. The meeting notes set a narrow first scope. Iceberg will start with a one-dimensional numeric vector of fixed length with non-null, finite coordinates, where +0 and -0 both stay valid. The initial element types are int, long, float, and double. Narrower types such as int8 are deferred as possible future Iceberg primitives, though the Parquet representation can still carry them. Distance functions and sparse vectors are out of scope for now and will return with the index design. Formats without native vector support will use a list representation with writer-side enforcement, and the group will explore an Avro vector logical type with the Avro community. ORC support remains open.
REST catalog contract
Two REST catalog threads closed out with concrete changes. Rahul Mahadev posted the result of the User-Agent vote: the recommendation for a standard User-Agent format for REST clients passed with 19 +1 votes, 10 of them binding, and no objections. The reopened vote had collected late +1s from Gang Wu, Zheng Hu, Ryan Blue, Fokko Driesprong, John Zhuge, and Huaxin Gao. Zheng Hu flagged CVE scanning failures on the PR, so watch for that before merge. A shared User-Agent format sounds minor, but it gives catalog operators a reliable way to see which engines and library versions hit their servers, which helps with deprecation planning and incident response.
The second REST thread resolved a gap in credential vending. Youngrae Kim had asked what a server should do when the X-Iceberg-Access-Delegation header cannot be satisfied. Daniel Weeks suggested adding a none value, meaning the client does not need delegated access from the catalog. Yufei Gu and Dmitri Bourlatchkov agreed. Dan and Dmitri then argued against adding a sentence that explicitly allows a server to fail the request, since a server can always reject a request anyway. Youngrae opened spec PR #18480, which adds none to the enum with one defining sentence and leaves the rest of the header description alone. Clients that bring their own storage credentials now have a clean way to say so.
Smaller threads with real impact
Sandeep Gottimukkala proposed a spec clarification that readers must accept promotable types in any data file. Type promotion lets a field evolve, for example from DECIMAL(8, 0) to DECIMAL(25, 0), without rewriting old files. The clarification states that a reader must handle the narrower physical type wherever it appears. Russell Spitzer supported it but asked to drop language about whether the field was evolved, since that detail is not part of the invariant.
Revanth Ch confirmed the migration plan for the new expression JSON forms. With Ryan Blue's guidance that servers always omit residual filters, the new expression syntax only flows from client to server in plan requests, so no client capability signal is needed. Revanth will open a PR so the reference implementation stops sending a residual on every scan task, then continue building conformance fixtures.
Hongyue Zhang and a contributor writing as dzeri96 discussed making appendUpdate in BaseTransaction public. The use case is a custom index stored in a Puffin statistics file. Today that team injects custom pending updates through reflection to control conflict detection. The request is a small API change, but it points at a bigger need: index builders want first-class hooks into Iceberg's commit protocol. The Iceberg index sync that normally covers this was cancelled for October 12 because so many people were traveling. Huaxin Gao pointed everyone to the index spec PR in the meantime, and the next sync is October 26.
EJ Song posted a status update on row-granular concurrency for v3 deletion vectors. The first step merges concurrently added deletion vectors for pure delete commits. Huaxin Gao and Uroš Bojanić have reviewed it, CI is green, and it needs one more committer to sign off on the retry semantics in MergingSnapshotProducer. When this lands, two jobs that delete different rows from the same file will stop conflicting with each other.
On the integration front, Robert Kruszewski opened a PR for upgrading arrow-java from 15.0.2 to 19.0.0, which positions Iceberg for arrow-java 20 and its expected Decimal32 and Decimal64 support. Vladislav Sidorovich asked for another review on moving Delta Lake migration to Delta Kernel after addressing all comments. K Krishna raised two PyIceberg Azure fixes that had waited two weeks. One fixes a bug where FsspecFileIO wrote an inferred storage account back into shared properties, so tables spread across two storage accounts got the wrong filesystem. Fokko Driesprong picked them up.
Alex Stephen announced a new batch of public datasets behind a public Iceberg REST catalog: 72 tables with over 640 billion rows. If you want to test an engine against real Iceberg tables at scale without loading data yourself, this is a free starting point.
Community events filled out the calendar. Scott Haines announced the October virtual meetup for October 23 at 9:00 AM Pacific, featuring Liam Earley on improving scan latency with manifest work in iceberg-rust. Kevin Liu boosted the Seattle Iceberg meetup on October 21.
Apache Polaris
Where server storage access comes from
The most consequential Polaris discussion this week was about boundaries. Dmitri Bourlatchkov opened a thread on removing default table properties from server-side FileIO configuration, and Vignesh posted a parallel thread on dropping catalog table-default.* properties from server FileIO construction. Both grew out of review on PR #5328, the table metadata cache work.
Here is the problem. Polaris builds server-side FileIO objects from a mix of StorageAccessConfig and catalog-level default table properties. Vignesh noted that an earlier fix already stopped using caller-supplied metadata properties for server FileIO, tied to CVE-2026-97395. Commit, refresh, and register paths still pass catalog table-default.* properties into the FileIO factory. Everyone agreed the target state is a clean one: server-side FileIO should come from StorageAccessConfig alone.
The debate was about custom settings. Yufei Gu pointed out that deployments need to customize FileIO in ways nobody can predict. Dmitri first proposed letting the StorageAccessConfigProvider pull relevant catalog properties in. Jean-Baptiste Onofré preferred a different shape: a properties bag on PolarisStorageConfigurationInfo for custom FileIO settings, with MinIO and S3Mock endpoints and path-style access living on the AWS storage configuration where those fields already exist. Vignesh backed JB's shape and offered to write the cleanup PR, including moving test fixtures off table-default.s3.*. Dmitri agreed with the property bag but raised an upgrade concern. Environments that rely on table defaults today will break if Polaris stops reading them overnight, so he proposed a transition path. Expect the final PR to include a deprecation window.
This connects to Yufei's question about metadata immutability across storage endpoint changes. If two S3-compatible stores hold different files at the same s3:// path, can a storage endpoint change make an existing table's metadata URI return different content? Yufei argued table metadata should not change because a FileIO client was configured differently. Dmitri reframed it as a cache correctness question: Polaris does not control what external stores contain, so it cannot assume identical content across endpoints for caching. JB split the difference. Iceberg treats metadata files as immutable once written, so the URI acts like a content identifier in spirit. But an s3:// URI is not a content hash, so Polaris cannot enforce that. Robert Stupp summarized the two assumptions the system relies on and noted that neither follows from the URI and storage configuration staying the same. The practical outcome will shape whether the storage endpoint becomes part of the metadata cache key.
Dmitri also flagged expanding the config entries in StorageCredential responses for community awareness, since it touches credential vending. PR #5731 from Adam lets Polaris return the full client-visible storage access configuration next to credentials. JB supported it, noting that Iceberg defines StorageCredential.config as a prefix-scoped map that S3 and GCS FileIO overlay on their base properties. He asked for one rule to be made explicit before merge: if the credentials entry is meant to be the complete access config, it must hold in every case, including remote signing.
On the Polaris side of the access-delegation question, the matching Polaris thread confirmed that Polaris keeps its fail-fast behavior, returning HTTP 400 when delegation cannot be satisfied. Youngrae Kim noted the vended-credentials docs already explain both 400 responses and tell clients to drop the header and configure their own storage credentials. Once the Iceberg spec adds none, Youngrae plans to add support for it in Polaris. JB also noted that Iceberg 1.12 adds a remote signing configuration, so the community will need to revisit combined requests for vended credentials and remote signing.
Labels, branches, and atomic commits
Andrei Tserakhau opened a design discussion on IRC labels in Polaris now that the labels field has merged into the Iceberg REST spec and Polaris tag support is close. He proposed treating labels as a read-only projection over Polaris tags rather than a separate entity. Tags already carry definitions, allowed values, inheritance, and reverse lookup, and both models key on Iceberg field IDs. A second labels entity creates two sources of truth. Sung Yun agreed for the producer side and asked whether federated catalogs need to store foreign labels at all or can forward them straight to clients. Andrei answered that forwarding works for engines, but Polaris users need foreign labels represented as read-only tags to see them through Polaris discovery and reverse-lookup APIs. JB, picking up a conversation from Glasgow, backed the projection approach and asked about permissions and performance. Andrei proposed reusing tag visibility rules, so loadTable only returns labels the caller can already see through the Tag API, and caching effective tags per table so the hot path does not resolve inheritance on every call. He also suggested gating the feature behind a flag at first.
Ankur Kotwal started a discussion on catalog-level branches for atomic multi-table promotion, a feature users have requested in issue #3469. Iceberg already supports branches on a single table. What users want is a way to branch an entire catalog, write to many tables in isolation, and promote all of them to main in one atomic step. That is the lakehouse version of a database transaction across tables. After Dmitri questioned the need for a new entity type, Ankur dug into the code and found that commitTransaction is already atomic on both persistence families. He opened a test and doc PR (#5740) showing a recipe built on existing pieces, then shared a design doc for a native option where a branch becomes a second named pointer next to heads/main, backed by a per-catalog registry.
That work feeds directly into the long-running thread on consistent multi-object changes in Polaris persistence. EJ Wang's updated proof of concept centralizes business logic in a shared manager instead of reimplementing it across backends. JB praised the separation of concerns and noted it fixes a maintenance burden that dates back to the move from OpenJPA to JDBC and NoSQL. Ankur ran the multi-table promotion test from #5740 against EJ's branch on H2, plus two race tests, which is exactly the kind of cross-thread validation that makes a design trustworthy. Dmitri said he needs a couple of weeks to review the PoC properly.
Yufei also endorsed a proposal on avoiding validation of historical metadata references on every table commit. When write.metadata.path changes, Polaris does not need to revalidate every historical metadata file. Leaving old files in place preserves immutability for past snapshots, and moving them belongs in a separate migration process.
Operations and extensibility
Tobias Riemenschneider reported that NoSQL persistence retry timeouts are not configurable and run shorter than the recommended MongoDB timeouts. Running Polaris 1.8.0 on MongoDB, his team sees table commits fail after 15 seconds with zero retries when MongoDB stalls behind a lagging secondary. The retry loop checks its total budget only after an attempt ends, and nothing bounds a single attempt. Robert Stupp confirmed the diagnosis: RetryConfig was always meant to be configurable, but SmallRye Config ignores it because PersistenceParams.retryConfig() is a Java default method. Tobias filed issue #5777 and PR #5779 the same day. JB noted that JDBC already exposes max retries, max duration, and initial delay, so the NoSQL side will now match.
JB opened a broader thread on Polaris server extensions and custom entities, prompted by users he met in Glasgow. Authorizers and event listeners are easy to extend by shipping a module with a Jandex index. Custom entities are not. Users currently have to edit Polaris core and add values to the entities enum. JB opened PR #5776 to document the current state and asked the community how to make custom entities a supported extension point.
Vignesh landed the prerequisite for a shared CloudEvents envelope for Polaris events. PR #5717 adds a shared mapper next to the events SPI, with CloudEvents context at the top level, Kafka-parity fields in the data payload, and lowercase type names such as org.apache.polaris.after_create_table, following Dmitri's suggestion. The webhook listener in PR #5084 will rebase on it. Dmitri also pushed the REST endpoints for table metrics and events (PR #4115) into a final review round now that metrics grants merged separately.
Yong Zheng proposed a delete-dataset simulation for the benchmark tool. Re-running benchmarks against persistent backends gives skewed results because old metadata piles up. Yufei asked whether resetting the persistence layer directly is faster. JB preferred keeping the API-based delete as the default, since it works against any backend and keeps the benchmark a plain client of a running server. Dmitri, Yufei, and JB all signed off on the approach in polaris-tools PR #286. A smaller cleanup thread on role cleanup tasks concluded that the async cleanup flag on role deletion was a leftover from the initial code import. Ayush Saxena traced it to the first commit, and JB, Dmitri, and Vignesh agreed to remove it since roles have no storage to purge and grant records already get cleaned up synchronously.
Process and people
Alex Dutra's thread on reducing CHANGELOG.md merge conflicts drew 19 messages. Ayush Saxena brought numbers: 75 of 339 commits on main over 60 days touched CHANGELOG.md, roughly one in four and a half. Alex proposed per-PR changelog fragment files. JB questioned whether a hand-maintained changelog is worth the effort at all and suggested assembling release notes from labeled PRs. Robert Stupp suggested conventional commit messages. Yufei and Alex both raised AI-assisted summarization at release time as an option, with Alex listing the tradeoffs around missing context. Dmitri clarified that the changelog was never meant to have an entry for every PR. The group settled on a first step: Yufei's PR #5733 tightens the guidance so only release-note-worthy changes get entries. If conflicts persist, the changelog.d fragment approach is next.
JB announced that Ayush Saxena is a new Polaris committer, and congratulations poured in from across the community. Ayush's fingerprints are all over this week's threads, from changelog statistics to git archaeology on role cleanup. JB also cancelled the October 15 community meeting.
Apache Arrow
Arrow 26.0.0 ships
Raúl Cumplido posted the result of the Arrow 26.0.0 release vote. It carried with six binding +1s from Rossi Sun, Sutou Kouhei, Gang Wu, Raúl Cumplido, Adam Reeve, and David Li, plus non-binding +1s from Xuanwo, Tadeja Kadunc, and Jean-Baptiste Onofré. The verification thread reads like a tour of the platforms Arrow now has to support. Gang Wu verified on an Apple M5 Pro with Clang 21. Raúl ran source, reproducible-source, integration, binary, wheel, and APT checks, including a patch to validate a Python 3.15 release candidate. Adam Reeve tested on Fedora 44 with Python 3.14. Tadeja verified on macOS with Ruby 4.0 and found a couple of GLib and Ruby script fixes she plans to send upstream. A major Arrow release touches C++, Python, R, Ruby, GLib, and more, so the breadth of hands-on verification is the real story here.
The Rust implementation moved in parallel. Jeffrey Vo opened a vote on Arrow Rust 59.3.1 RC1, and L. C. Hsieh, Raúl Cumplido, and Andrew Lamb all cast binding +1s after verifying on M4 and M3 Macs and Debian 14. Arrow Rust is the foundation under DataFusion, iceberg-rust, and a growing set of Rust query engines, so its patch releases ripple outward fast.
ADBC grows up
Ian Cook brought back a new ADBC logo proposal after more design work. Broad support led to PR #4870 in arrow-adbc, which adds the logo, a new Visual Identity page, and an updated trademark notice. Rusty Conover gave detailed feedback: the project URL on the hex badge is hard to read at small sizes, and the text spacing did not sit in harmony with the chevron mark. Ian posted a revised version that addresses both points, and Dewey Dunnington approved. Barring objections, the logo passes by lazy consensus after 72 hours.
A logo matters because ADBC is becoming a brand people build on. That showed up in a second thread: Awanish Gupta from IBM asked to contribute a native C++ ADBC driver for IBM Db2. David Li explained that the adbc-drivers GitHub organization sits outside the Apache Arrow project, offered to set up access directly, and pointed to the driver authoring docs. Asked about scope, Awanish confirmed the driver targets Db2 for Linux, UNIX, and Windows only, not Db2 for z/OS. Every new ADBC driver widens the set of databases that can hand Arrow data to a lakehouse engine without row-by-row conversion.
Format work and community
The format track kept moving. Florian split the spec text for fixed and variable closedness range canonical extension types into its own PR (#53239) so the vote covers only the format change, and he asked to keep the vote open since no votes had arrived yet. David Li signed off on the latest revision of the JSON representation of Arrow schemas, a small but useful step for tools that want to exchange schemas without the Flatbuffers dependency.
Ian Cook reviewed Pearu Peterson's long-waiting CSV date and time parsing PR and noted that Pearu's emails, like many others recently, had landed in spam folders. Andrei Tserakhau reported the same thing on the Iceberg list this week. If you posted to an Apache list recently and heard nothing, check whether your message reached people's inboxes before assuming silence means disinterest.
Antoine Pitrou filed the Arrow board report, Ian Cook ran the October 7 community meeting, and Andrew Lamb announced on behalf of the PMC that Richard Baah is a new Arrow committer.
Apache Parquet
What does "distinct" even mean?
The sharpest Parquet debate this week was over a single word. Adrian Garcia Badaracco opened format PR #628 to classify distinct_count metadata as inexact after a sync call reached consensus. Micah Kornfield asked for the reasoning, since Andrew Lamb had pushed back. Russell Spitzer explained it. Exactness of the number of distinct values is not standardized for many types. Strings with collations treat "foo" and "Foo" as equal or not depending on the system. Floating point has its own traps: NaN against NaN, and -0.0 against 0.0. Most use cases only need approximate statistics anyway, so the group chose to correct the language rather than define exact semantics for every type. Daniel Weeks added that older readers and writers already disagree about logical and physical equivalence, so assuming current values are exact was never safe. Andrew still sees changing "count of distinct values" to "estimated count" as a real change in meaning, not a clarification. Dan agreed the proposed wording is not ideal and offered alternatives on the PR.
Russell also left a reminder that deserves repeating: when you cite a decision from a sync call, post notes to the list. Decisions that live only in a video call are invisible to the rest of the community.
The vector type fight
The numeric vector type thread turned into the most substantive design argument on any list this week. A focused call had produced a proposal for a FINITE_VECTOR logical type on fixed-length lists. Ryan Blue laid out what a logical type provides. It describes data for users and readers. It signals that repetition and definition levels reflect a fixed number of elements, which implementations can use to decode faster. It guarantees no null elements. And the finite variant guarantees no NaN or infinity values, so every vector has a finite length and works for distance calculations. Steven Wu agreed that FINITE_VECTOR brings value and argued against forcing every use case into a generic fixed-length list. Alkis Evlogimenos suggested naming it VECTOR to match other engines and type systems, leaving room for a non-finite variant later. Gang Wu supported separating FINITE_VECTOR (or VECTOR) from FIXED_LENGTH_LIST and moving the vector type first.
Then Jan Finis made the case against. He argued that the finite restriction adds little value on its own and that the fixed-length part should be a general feature rather than a one-off. Existing types already handle uncompressed vectors well, he said. The real win is quantization, where storage size and processing speed improve by orders of magnitude. Will Edwards made a similar point from the engine side. He wants every engine to offer vector functions on lists of floats without a new logical type, and he sees more value in a type that describes how to unpack quantized codes, scales, and codebooks, which is where machine learning is heading.
This is a genuine disagreement about what a file format owes the AI wave. One camp wants a simple, well-defined type now so engines and table formats can align, and Iceberg's vector work is already building on it. The other wants Parquet to skip the label and go straight to the compression problem. Both sides are right about something. A narrow VECTOR type gives Iceberg a stable target this year. Quantized storage is where the cost savings live. Watch for a proposal that ships the simple type first with an explicit path to quantized encodings.
A concrete plan for versioning
Alkis Evlogimenos restarted the stalled versioning discussion with a concrete proposal. His two goals are to ship features when they are ready rather than waiting for the next bundle, and to guarantee that a reader either returns correctly interpreted results or fails, never a false read. He sorts every format change into two kinds. "Add-everywhere" changes are valid at every compatibility level, including in today's files. They are safe when any reader that does not understand them either fails where it needs them or is unaffected. A new encoding is the classic example: a reader that does not know ALP fails on the ALP column and reads every other column. "Compatibility level" changes are for things existing readers misread instead of failing on, such as metadata redefinitions, removals, or a new footer. A reader that does not support the level fails before reading anything.
The most useful part of the proposal is the field survey. Alkis checked eight readers against their current source: Arrow C++, parquet-java, arrow-rs, DuckDB, Velox, arrow-go, cuDF, and fastparquet. Unknown encodings are mostly safe, though arrow-rs rejects the whole footer and parquet-java does the same when the encoding shows up in encoding stats. Unknown logical types are safe because every reader falls back to the physical type. Unknown page types are not safe: Arrow C++, DuckDB, and Velox skip them, so a new page type carrying rows gets dropped silently. Unknown compression codecs are not safe either: Arrow C++ treats them as uncompressed and decodes compressed bytes. Andrew Lamb praised the proposal and noted a third category, forward-compatible additions that older readers simply ignore, such as new statistics. This survey alone is a valuable document for anyone writing a Parquet reader.
Encodings and new types
Prateek Gaur reported that the C++ implementation of ALP encoding for floating-point data has merged, with follow-up items coming in a later PR. ALP compresses floating-point columns far better than the general-purpose codecs, which matters for sensor data, financial data, and embeddings. Asked who is working on new encodings, Andrew Lamb listed two active efforts: Prateek on PFOR (patched frame of reference) and Aarnav on FSST and OnPair, symbol-table-based string encodings.
Micah Kornfield pushed back on part of the WIRE logical type proposal for serialized wire-format messages. He does not want to move ahead with the projection component until an open source engine commits to using it, citing how hard variant was to get right in engines. He wants documenting serialized values kept separate from projection. Thomas Kissinger agreed to continue the extensible decimal floating-point type discussion in Costas Zarifis's design doc. Julien Le Dem submitted the October board report after a week of collecting status, and Russell noted that the timing collided with Community Over Code travel. Julien also announced the October 7 Parquet sync.
Apache DataFusion
Comet asks to leave home
Andy Grove posted a proposal to establish Apache Comet as a top-level project. His argument is that Comet differs from other DataFusion subprojects. The others extend DataFusion. Comet is an accelerator for Apache Spark that uses DataFusion as its execution engine, and its community has grown largely distinct from DataFusion's. The numbers back him up: 24 releases since July 2024, including 1.0.0 in August and 1.1.0 on October 6, and 146 people with merged PRs, 81 of them in the last 12 months.
The proposal is conservative by design. The name "Apache Comet" already cleared the podling name search. The initial PMC is drawn from existing DataFusion PMC members (Andy Grove, Andrew Lamb, Liang-Chi Hsieh, Matt Butrovich, and Oleks V.) with nobody promoted as part of the move, and Andy offered to chair. Initial committers are the existing DataFusion committers who contribute to Comet. The repository moves from apache/datafusion-comet to apache/comet, with docs at comet.apache.org. Joining Comet changes nobody's role in DataFusion. Confirmations came quickly from Bhargava Vadlamani, Manu Zhang, L. C. Hsieh, Matt Butrovich, Parth Chandra, Martin Grigorov, and Chao Sun, with Andrew Lamb adding his support. Andy plans a formal vote after about a week. He corrected the timeline in a follow-up: the next DataFusion board meeting is December 16, not November 18.
Comet also shipped this week. Andy announced that Comet 1.1.0 passed its vote with nine +1s, six binding, and published the jars to Maven. He also proposed an update to the Comet contributor guide on AI in PR #6734. Automated reviews can help, the guidance says, but every PR approval must come from a human, with or without AI assistance. That is a clear, enforceable line, and other Apache projects are likely to borrow it.
Releases across the family
The DataFusion family shipped on several fronts. Oleks V. announced that DataFusion 55.2.0 passed with five binding +1s and is published to crates.io. Andy reported that Ballista 55.0.0 passed with six +1s, four binding. Tim Saucer opened the DataFusion Python bindings 55.0.0 vote, which collected binding +1s from Marko Milenković, Andrew Lamb, L. C. Hsieh, and Andy Grove. Xuanwo's careful verification caught a missing NOTICE file in the source archive, a fix requested for the next release. Tim also closed out the Python bindings 54.1.0 vote with seven +1s, five binding.
Andy also proposed removing the Spark compatibility feature from Ballista. The experimental, off-by-default feature covered only a handful of expressions. After six years on Spark-compatible engines, Andy wrote, he knows Spark compatibility is a project that never finishes because Spark keeps changing, and other projects already fill that niche. Read alongside the Comet proposal, the direction is clear. Comet owns Spark acceleration. Ballista focuses on being a distributed DataFusion engine. Each project gets a sharper identity.
Apache Ossie (incubating)
Where does an MCP server belong?
Apache Ossie is the open semantic interchange specification, the standard for moving metric, dimension, and relationship definitions between BI tools, semantic layers, catalogs, and AI agents. This week its busiest thread asked a question every semantic layer team is facing: should Ossie ship an MCP server, and where should it live?
The thread started from GitHub discussion #510 and PR #529, which proposes an execute_query contract. Chris Eubank asked the key question: how tightly is execute_query tied to Ossie semantics? Is it meant to steer clients toward portable Ossie queries, or to standardize running any vendor's query language? As drafted, the model identity was optional and the result schema loose, so it looked closer to the second. Markus Cozowicz argued for pragmatism. Vendors will take time to build conversions, and each vendor has features the standard lacks, so allowing all registered Ossie dialects is a reasonable start. He also made an interesting point: if a query execution contract lands in the Ossie standard and then in LLM training data, nobody has to teach each agent how to run queries against a semantic model.
Jean-Baptiste Onofré drew the line clearly. Ossie is an interchange spec for semantic models. A running MCP server brings execution, connections, auth, and a fast-moving protocol surface into the project. He prefers that MCP servers live in engines and catalogs, with Ossie defining the contract and possibly a REST API, but not the server implementation. Chris agreed that the dividing line is whether a query resolves against an Ossie model, and said there is interest on his side in building query capabilities over the Layer 2 and Layer 3 query interfaces once they land in the core spec. Jakub Moravec asked whether defining skills in Ossie and leaving tools to vendors is enough. Ashfaqbs responded that skills are advisory text while tools are executable contracts, and that validation belongs in Ossie while conversion and query belong to vendors. Jakub agreed with that split.
The most valuable contributions came from people running code. Kyoung Min Kim ran an experiment on a real database of 33 datasets and 45 relationships, generated an Ossie model, and asked three models to return Layer 3 query specs that a compiler turned into SQL. Aleksandr Kosachev offered his ClickHouse executor as a test bed. It now accepts the aggregation query shape from PR #246 as a JSON object and returns #246 error codes such as E_NAME_NOT_FOUND, E_NO_PATH, and E_AMBIGUOUS_PATH inside the #529 error envelope. His first report found the executor raised 10 of the 28 #246 codes as specified. A later update showed it now follows paths across several many-to-one relationships, rejects a second path as ambiguous, and merges measures across datasets, with answers matching expected rows line by line. JB took the lesson that a small shared query shape plus error codes works, and that the next real test is a second engine running the same cases. Where two engines disagree, PR #246 is underspecified, and those are the spots to fix. Markus confirmed PR #529 is now narrowed to the declarative case.
Markus also captured a practical agent problem in the same thread. Filter values are where agents fail. A user asks about "Coca & Cola" when the data says "Coca-Cola Company." He laid out three strategies: put the vocabulary in the ontology when context allows, expose a dedicated tool for looking up column values, or return suggested values reactively when a query comes back empty. Each has tradeoffs in tool count and accuracy.
Pinning down the query layer
Several threads worked on the query layer that the MCP debate depends on. Chris Eubank opened a thread on model expressions versus query semantics, pointing out that the expression language spec scopes itself to defining a model and says the SQL interface comes in a separate document. Today, expressions define a model. They do not create a query interface. Justin Talbot pushed a revision of the Relational Query Interface spec (PR #354). The biggest change renames the measure evaluation function from MEASURE() to AGG() as the single normative spelling, with other spellings allowed as non-normative synonyms. Chris agreed with the changes and suggested deferring a SQL-native syntax for declaring measures to a future extension. Justin also opened a vote on the specification layers overview (PR #452), an informational document describing Ossie's four layers: Model Definition, Relational Query Interface, Wide Table Query Interface, and Ontology.
Ananth Tirumanur reworked his proposal to add an ICEBERG expression dialect to the core spec (PR #491). Reviewers worried that a new vendor dialect without a reference guide invites parser divergence. Ananth's fix: the dialect defines no new syntax. It is the predicate string accepted by PyIceberg's expression parser, which serves as the reference reader. Chris praised anchoring a dialect to an open source parser as a precedent worth encouraging. JB asked where the dialect applies, since the schema only uses dialect expressions for fields and metrics, and a predicate-only dialect fits boolean fields and nothing else. Kyoung Min Kim ran the PR's predicate table through PyIceberg 0.6.1 through current main. Thirty of 35 forms behave the same in every version, and every unsupported form is rejected everywhere. The five that differ include BETWEEN and double-quoted identifiers, which parse from 0.10.0. That data settles what minimum version the dialect needs to pin. For lakehouse teams, an ICEBERG dialect means a semantic model can carry a filter that an Iceberg-native engine pushes straight down to a table scan.
The expression language itself grew. The vote to add the FILTER (WHERE ...) aggregate modifier collected support from JB, Klahr Joshua, and Ayush Saxena after discussion with Julian Hyde and Chris. Chris noted that the || concatenation operator now renders correctly in the spec. Aleksandr opened a discussion on NULLS FIRST and NULLS LAST in window ORDER BY (PR #577). Chris volunteered to rewrite spec wording for dataset-scoped metrics, and Kyoung Min will narrow the validator so qualified references become errors. JB asked that the rule be stated in the spec text and backed by a conformance test.
Measuring whether Ossie works
Chris Eubank raised a question every interchange standard eventually has to answer: how do we measure semantic interoperability? He ran the simplest possible model, one sales dataset and one SUM(sales.amount) metric, through Ossie to vendor and back to Ossie across 14 converters. Eight of 14 preserved the ANSI_SQL expression. Only one of 14 preserved OSSIE_SQL_2026. The rest emitted a vendor dialect, dropped the metric, or lacked a vendor-to-Ossie path. Chris was clear that this measures preservation of the portable dialect, not semantic equivalence. He proposed a shared interoperability test suite in the repository with reference models, measurement utilities, and published results for every Ossie-hosted converter. This is the right instinct. A standard that only measures file format support can look healthy while portable meaning leaks out at every hop.
The converter list keeps growing anyway. Tan Huynh from Holistics proposed a Holistics converter that runs both ways without network access, along with a HOLISTICS_AQL dialect entry. Mahmoud Hassanen proposed a native ClickHouse dialect converter, since ANSI SQL transpilation produces poor queries when ClickHouse needs functions like uniqExact and toStartOfMonth. JB explained that converters translate semantic model definitions and are not SQL transpilers, and that new converters will go to the apache/ossie-converters repository he is preparing. Mahmoud opened PR #558 to register CLICKHOUSE in the dialect enum. An OpenMetadata converter discussion continued as well.
GitHub discussions rounded out the semantic modeling agenda: metric additivity (additive, semi-additive, non-additive), where Justin Talbot noted BI tools rely on additivity to roll up results client-side and skip database round trips, certified metrics and certifying authority, a shared semantics reference vocabulary, metric trees, and a proposal on how the Open Knowledge Format relates to Ossie, which Markus Weimer welcomed as a canonical way to feed agents context from both. JB posted a revised Ossie REST API spec as PR #573, and Dmitri Bourlatchkov of Polaris endorsed it, noting Polaris as a possible implementation. Nolan Clark asked that the model management API avoid assuming a one-to-one mapping between inputs and converted outputs. Ankit Tandon posted notes from the October 13 Ontology working group.
Openness and community
Julian Hyde asked a blunt question: "Are we open?" Much of the project's work seemed to run vendor to vendor, he wrote, and it was not obvious how to attend a working group meeting. Kurt Stirewalt and Matthew Mullins pointed to the community page, Slack, and a public calendar of open working group meetings, while conceding the information is hard to find. Ayush Saxena asked for specifics. Julian clarified: Slack and the working groups are open in the sense that you can ask to join, but their content is invisible to search engines and unauthenticated readers. Aldre Jota then reported that the Slack invite links on the site were revoked or broken. Klahr Joshua invited anyone to improve the community page through the ossie-website repository, and Julia Blyumen suggested clarifying organizational versus individual membership. This is a healthy thread. Apache's rule is that decisions happen on the list, and a young project dominated by vendor working groups needs to keep proving it follows that rule.
JB proposed the first Apache Ossie workshop and meetup: a full-day hands-on workshop followed by an evening of talks. Julian supported it with the standard Apache caveats, that the workshop is for knowledge transfer rather than decisions and that recordings should be posted. Support came from Dragos Crintea, Kurt Stirewalt, Yufei Gu, Ankit Tandon, and Jochen Christ, who said a full-day onsite session is worth a flight from Europe. JB is working on logistics. Ayush Saxena announced that Chris Eubank is a new Ossie committer, a fitting recognition given how many of this week's threads he started or shaped.
Cross-Project Themes
The lakehouse is learning new data types for AI, in lockstep. Iceberg's File type follows Parquet's FILE logical type. Iceberg's vector design builds on Parquet's emerging vector representation, and the Iceberg vector sync explicitly defers to the Parquet decision. Parquet's ALP encoding targets the floating-point data that embeddings and sensor workloads produce. Arrow-java 20 is expected to bring Decimal32 and Decimal64, which Iceberg is already preparing for. These are not separate efforts. The table format, the file format, and the in-memory format are converging on one shared vocabulary for unstructured bytes and numeric vectors. The Parquet vector debate is the critical path. Whatever Parquet decides about VECTOR versus quantized encodings will set the shape of AI data in Iceberg tables for years.
Catalogs are getting exact about access. The same people show up across Iceberg and Polaris on the same problem. Youngrae Kim, Yufei Gu, Dmitri Bourlatchkov, and JB carried the access-delegation question from the Iceberg spec into Polaris behavior in one week. Polaris is separating catalog configuration from server storage access, scoping what StorageCredential responses contain, and deciding whether storage endpoints belong in cache keys. Iceberg added a standard User-Agent format. Polaris is mapping IRC labels onto its tag model. As AI agents start calling catalogs directly, these precise rules about who gets which credentials and which classifications become the security model. Loose defaults stop being acceptable.
Semantics and agents are the new frontier, and the lakehouse is the ground floor. Ossie's MCP debate, its ICEBERG dialect, and its query interface work all point the same way. Agents need a governed semantic layer to ask questions, and that layer needs to push work down to open tables. An ICEBERG dialect anchored to PyIceberg means a metric definition can carry a predicate an Iceberg engine executes directly. Polaris showing up as a possible implementer of the Ossie REST API means the catalog and the semantic layer can share one interface. This is the open lakehouse stack extending upward: Parquet for files, Iceberg for tables, Polaris for the catalog, Arrow for movement, and Ossie for meaning.
Projects are formalizing how humans and AI share the work. Comet now requires a human to approve every PR. Polaris debated AI-assisted changelog summaries and chose to tighten human guidance first. Several contributors on the Ossie and Polaris lists described writing code first and discussing second, since code is cheap now. The common thread is accountability. Apache projects welcome AI help, but they want a named human responsible for every merge and every release note.
Looking Ahead
The Iceberg File type vote closes this week, and the AMT baseline vote has no objections on record, so expect both results soon. The Comet top-level project discussion moves to a formal vote around October 16, with the board resolution targeted at the December 16 meeting. On Parquet, watch whether the vector debate converges on a simple VECTOR type or detours into quantized encodings, and whether Alkis's versioning proposal picks up a sync slot. Polaris will see the FileIO cleanup PR and the NoSQL retry fix land. Ossie's workshop logistics, the converters repository move, and a second engine running Aleksandr's ClickHouse test cases are all worth tracking. On the calendar: the Seattle Iceberg meetup on October 21, the virtual Iceberg meetup on October 23, and the Iceberg index sync returning October 26.
Go Deeper With the Open Lakehouse
If this newsletter helps you keep up with Iceberg, Polaris, Arrow, Parquet, DataFusion, and Ossie, my books go further. I have written more than 50 titles on the open lakehouse, Apache Iceberg, Apache Polaris, agentic AI, and data engineering, from the O'Reilly definitive guides to hands-on Python and architecture books.
Browse the full catalog at books.alexmerced.com and find the right book for where you are in your lakehouse journey.
Top comments (0)