This was a week of votes. Apache Iceberg passed or nearly passed three separate v4 spec changes, Apache Parquet opened and then reset a vote on a new vector type, and Apache Arrow, Iceberg Rust, DataFusion Comet and the Iceberg Terraform provider all shipped releases. The quieter story sits underneath those votes. Across six dev lists, maintainers kept running into the same two problems: specs that leave room for two correct readings, and a flood of AI-assisted pull requests that outpaces the people who review them.
The six lists produced 463 messages in the eight days ending November 4. Iceberg led with 117 messages across 37 threads. Apache Ossie, still incubating, came second with 125 messages, a sign of how much design work a young semantic standard needs before its first release. Polaris followed with 95, Parquet with 52, Arrow with 47 and DataFusion with 27.
Apache Iceberg
The v4 table format took three large steps forward in a single week. Ryan Blue closed the vote on stats-only fields for v4 with 15 +1 votes and no objections, as recorded in the result thread. The change adds a stats-fields list that assigns table field IDs to things that are not real columns, such as partition fields or an expression like to_lower_case. Engines can then store statistics for those derived values in the content_stats struct. In plain terms, Iceberg can now record min and max values for the expressions people actually filter on, not only for the raw columns they store.
The design discussion behind that vote, in the stats-only fields thread, carried a useful detail. Andrei Tserakhau raised collations, and Ryan proposed a new type with standardized metadata for those rather than an expression. His reasoning was simple: a collation always produces a string, so binding an expression adds work with no payoff. He also left the handling of unbindable expressions to implementations, arguing that a strict "must ignore" rule hides real errors.
Daniel Weeks then opened a vote on default value expressions for v4. The proposal expands write-default so a column default can be an expression from Iceberg's expressions spec, not only a literal. The thread collected binding +1 votes from Steven Wu, Russell Spitzer, Ryan Blue, Gang Wu, Eduard Tudenhöfner, Fokko Driesprong, Amogh Jahagirdar and Yufei Gu within two days. Péter Váry and Gianluca Graziadei left small review comments on the spec PR. For teams that evolve schemas often, this matters: a new column can get a computed default without a backfill job.
Steven Wu's vote to add a _last_updated_time row lineage column had the most texture. The proposal records the wall-clock timestamp of the snapshot that last changed a row, next to the existing _last_updated_sequence_number. It adds an optional snapshot_timestamp field in v4 manifests, and writers leave it null so readers inherit the snapshot's timestamp-ms. Upgrading to v4 needs no manifest rewrites. Daniel Weeks voted +1 but asked to change the type from timestamptz to timestamp, since the inherited value carries no time zone. Ryan Blue then wrote that the spec text still needs more editing before it is ready. Expect a revised PR before this one closes.
Two more v4 threads show where the format is heading. Kevin Liu asked directly whether v4 should stop writing ORC and Avro data files. Existing files stay readable. His case rests on two points. Iceberg's own format test kit already lists features those formats lack, such as variant shredding, and the draft vector type does not support ORC. He also noted that recent ORC and Avro format changes Iceberg needed came from Iceberg contributors themselves, with Gang Wu adding geo types to ORC and Fokko adding nanosecond timestamps and fixed-size UUIDs to Avro. The thread poses four questions, including what bar a format must clear to become writable again.
Pachu Shrinivas revived the deprecation of equality deletes in v4 with a practical worry. Equality deletes do not conflict with concurrent compaction, because compaction can preserve the data sequence number from planning time. Deletion vectors (DVs) are scoped to one data file. So when compaction rewrites that file, either the streaming writer or the compaction job fails. Streaming shops will want an answer to this before v4 lands.
That same theme showed up from an unexpected direction. Michalis Papadimitriou from the TornadoVM team shared a proof of concept for GPU-accelerated merge-on-read scans and table maintenance. TornadoVM compiles plain Java methods into GPU kernels. The library sits on Iceberg 1.12 public APIs, so Iceberg still plans and commits while the GPU handles per-row work. The team reports 3.9x to 4.8x faster merge-on-read scans at 5 billion rows on two A10 GPUs versus 128 CPU threads. It also reports equality-delete to DV conversion on TPC-DS SF100 in 0.77 seconds versus 81.9 seconds for Spark compaction. These are the team's own numbers. Their pitch ties directly to the v4 decision: if new equality deletes are forbidden, converting existing ones becomes required work. They asked whether the code belongs out of tree, like Comet, or in a contrib module.
Steven Wu also consolidated the Monday sync into a single "V4 Metadata" meeting. It now covers the adaptive metadata tree, column files, content stats, relative paths, constraints and row timestamps. The Tuesday column-files sync goes away. The Thursday file and vector types sync stays for a few more sessions, which Philipp Fischbeck and Russell Spitzer used this week in the file and vector types thread to agree that a vector should be a primitive type in Iceberg. Russell's reasoning: consumers treat a vector as one whole value, not a list of cells to edit.
Alex Stephen added an AI angle on top of that work with a proposal for AI functions for FILE and VECTOR types. The idea is a vendor-neutral generative function and an embedding function in the Spark and Flink connectors, so users can work with multimodal Iceberg data without picking one AI vendor's SQL dialect. Alex Cruise posted a separate proposal for pluggable Puffin blob types with a compaction-aware lifecycle. His goal is "progressive indexing": adding new index types to large, old files without rewriting the data. He framed it as complementary to the v4 secondary index proposal, and he published a gentler backgrounder alongside the formal text.
Index work also moved in a dedicated sync. Huaxin Gao summarized how databases, catalogs and REST APIs model indexes. Every catalog API she checked scopes an index to its table. SQL databases split on the question. Daniel Weeks gave the proposal a +1, and Péter Váry said he prefers nesting indexes under the table path in the REST spec.
REST catalog and auth
The REST catalog spec got a useful clarification this week, and the trigger came from Polaris. Anand Kumar Sankaran asked what the metrics reporting endpoint should return when the server cannot store a report. Eduard Tudenhöfner replied that metrics reporting is best-effort, the Java RESTMetricsReporter never surfaces errors, and clients are not expected to retry. Yufei Gu wanted that written into the spec, asking where the line sits if a failed write can return 204. Daniel Weeks drew the line clearly. A server should never return a success code for a failed request, but whether a client retries is a client choice. The spec stays as is. The practical guidance: a metrics endpoint behaves like any REST endpoint and can return a 4xx or 5xx when it fails.
Alex Dutra brought news from Community over Code in Glasgow. The group agreed to downscope the OAuth2 AuthManager v2 proposal. The reboot supports one grant (client_credentials), two client authentication methods, simplified token renewal and limited legacy v1 support. It now fits in a single PR. Christian Thiel, who backed the approach in Glasgow, put it on the catalog sync agenda.
Youngrae Kim closed the vote to add none to the X-Iceberg-Access-Delegation header values with seven +1 votes. A client can now tell the server it wants no credential vending at all. That small change unblocks a Polaris discussion covered below. Yong Zheng also proposed replacing example with examples in the REST OpenAPI document to match OpenAPI 3.1.
Spec gaps found by readers
Sylwester Lachiewicz found a real interoperability bug in the spec's text on name mapping for files that carry only some field IDs. He checked four implementations and found four behaviors. Java resolves by ID only if any field has an ID. PyIceberg uses IDs only if every field, including list elements and map keys, has one. Iceberg Rust lets the first top-level column decide. Iceberg Go resolves per field. This is not theoretical. Delta tables with column mapping, written without IcebergCompatV2, store IDs on top-level columns but not on map keys and values. Read the same table through PyIceberg and Java and you get different results. He wants the community to pick one reading and write it down.
Maintenance code drew attention too. Sebastian Daberdaku asked for eyes on a fix that makes remove_orphan_files use the catalog's Hadoop configuration. When a catalog reaches storage with different credentials than the cluster, the listing step fails with AccessDenied or sees no files. Daniel Weeks suggested reading the configuration from the FileIO instead, and Russell Spitzer backed avoiding new API surface. Sebastian showed why that falls short: S3FileIO, GCSFileIO and ADLSFileIO do not implement HadoopConfigurable. Vova Kolmakov raised a sharper risk in the file_list_view location thread. A raw startsWith check on s3://bucket/table also matches s3://bucket/table-backup, so files in a sibling directory can be treated as orphans and deleted. The fix treats the location as a directory. Munendra S N asked for review of a GCS fix that separates real delete failures from missing objects, since the current code silently swallows permission errors.
Aaron Niskode-Dossett resurfaced a stale PR that pushes aggregates down for GROUP BY on identity partition columns in Spark. Today Iceberg answers COUNT, MIN and MAX from metadata only when there is no GROUP BY. Grouping by an identity partition column is the exact case where the metadata already holds the answer.
Releases and community
Iceberg Rust 0.11.0 shipped after Danny Jones ran an RC3 vote that passed with three binding and seven non-binding votes, per the result. The release adds client-side encryption on the write path and incremental v3 support, including row lineage, nanosecond timestamps, delete vectors and Variant in metadata. The delay came from licensing. Danny closed the loop on the pyiceberg-core attribution issue: the compiled Python wheels now carry every bundled dependency's NOTICE and LICENSE text. Kevin Liu tested the RC against PyIceberg's integration suite and confirmed every crate passes a cargo publish dry run.
Matt Topol announced the Iceberg Terraform Provider v0.2.0 after a vote with four binding +1s. It adds an iceberg_tables data source, SigV4 request signing and better OAuth2 and schema evolution support, and it ships to both the Terraform and OpenTofu registries. On the Python side, Orest Tokovenko proposed dropping Python 3.10 from PyIceberg. Python 3.10 reached end of life on October 1, and PyArrow 26 no longer ships 3.10 wheels. Fokko Driesprong agreed, and the PR is up. Neelesh Salian scheduled an Iceberg Go sync for November 17 to cover v4 adoption and the remaining breaking changes before a 1.0 tag.
The most human thread of the week came from a dashboard. Russell Spitzer shared Iceberg community health stats, and several people, including Kevin Liu and Ryan Blue, noted his first email never reached them, part of an ongoing ASF mail delivery problem. Jared Yu offered a theory for a weak "first to second contribution" number. He sees new contributors arriving with AI tools, expecting fast progress, then stalling because a small group of maintainers cannot review the volume. His phrase for it was a "dopamine crash." That observation connects to almost every other list this week.
Two meetups are on the calendar: a virtual Iceberg meetup on November 13 about building an open lakehouse with Spark, Iceberg and the Polaris REST catalog, and a Tokyo meetup on December 7.
Apache Polaris
Polaris spent the week finishing 1.9.0 and arguing, productively, about where policy belongs. Jean-Baptiste Onofré posted the November board report draft, and it gives a clear snapshot of the project eight months after graduation. Polaris has 33 committers and 19 PMC members. Ayush Saxena became a committer on October 12. From August through October, 646 pull requests were opened and 529 merged. Excluding dependency bots, those numbers drop to 391 and 294. The open PR backlog grew from 102 to 137, and 76 people opened PRs this quarter, up from 57. The report says it plainly: the bottleneck moved from writing changes to reviewing them.
The report also lists what shipped in 1.7.0 and 1.8.0. Highlights include event listeners that publish Polaris events to Kafka and OpenTelemetry, GCS principal attribution so the Polaris principal shows up in Google Cloud audit logs, and dedicated privileges for semantic models. Two CVEs were published this quarter, one low and one important, and both were fixed in a release before disclosure. No release shipped in October. 1.9.0 is being finalized now and includes a first beta of Polaris Directories.
Who decides about credentials
The longest Polaris thread asked who should control credential vending. Dmitri Bourlatchkov pushed typed storage config options to skip credential vending in Azure and GCS, implemented by Vignesh in PR 5884. S3 already had this option. His use case is workload identity: engines already hold ambient credentials, so vended credentials add overhead and a second, unneeded way to reach storage. Yufei Gu countered that clients can simply not send the X-Iceberg-Access-Delegation header, as Spark and Trino do by default. Dmitri replied that a server cannot assume every client behaves. Yufei then argued that vending is a property of the catalog, namespace or table, not the storage config, since one storage config will soon be shared across many entities. Dmitri disagreed and asked for more voices. This is a real design fork, and it lines up with the new Iceberg none delegation value.
That shared storage config is its own thread. Srinivas Rishindra posted a spec-only PR for multiple storage configurations per catalog. Storage configurations become named, immutable entities under a catalog. Admins can map a namespace or table name to a configuration before the object exists. The PR replaces two earlier attempts.
The /v1/config endpoint got closer scrutiny too. The soft-roll authorization thread for GET /v1/config started as a simple merge. Yufei and JB pointed out the deeper issue: Polaris returns the entire catalog properties map as client defaults, mixing client settings like rest-page-size with server settings like default-base-location. Gating all of it on one privilege means a client with narrow grants loses settings it needs. Dmitri proposed a compromise. Vignesh dropped the second operation, kept GET_CATALOG_CONFIG behind a flag that defaults to false, and documented that the default will not flip until a follow-up defines a client-facing config. Dmitri wants it in 1.9.0 and flagged it in the release thread.
Yufei raised a sharp edge in the ALLOW_UNSTRUCTURED_TABLE_LOCATION thread. If an admin turns the flag off, a renamed table whose data still sits under the old namespace starts failing with 403 errors later. Alex Dutra and Russell Spitzer both argued against blocking the change, because it blocks a move from a less secure setup to a more secure one. Russell's framing was clean: an admin who wants a hole closed wants it closed now, and failing closed is fine. The group settled on documentation in 1.9.0 and a read-only admin report after it.
Metrics, discovery and federation
The metrics endpoint debate that reached the Iceberg list started here. Anand's PR to drop reports instead of returning 500 on a database failure logs the failure and increments a polaris.metrics.reports.dropped counter. Dmitri argued retries belong on the server, since metric reports cannot conflict with other writes. Anand agreed and offered to build server-side buffering as a follow-up. Yufei asked to settle the semantics with Iceberg first, which Anand did. Given Daniel Weeks's answer on the Iceberg list, returning 204 for a failed write now looks like the wrong call.
Dennis Huo proposed catalog discovery for data-plane clients. Today an engine has to learn the catalog name out of band, even though Iceberg's /config endpoint is multi-catalog. Polaris's existing listCatalogs is a management API that needs a root privilege. The new listAccessibleCatalogs endpoint returns only catalogs the caller can call getConfig on. JB and Dmitri approved the PR. This also fills a gap for the in-progress Polaris Shares API.
Federation and auth produced several smaller threads. Laurent Goujon proposed chain-aware authorization for views using Iceberg's optional referenced-by parameter, which Polaris accepts and then drops today. Passing the chain of views lets an authorizer support definer views. Arnab Karmakar asked for a custom STS endpoint for SigV4 federated catalog connections, matching what storage vending already supports. Austen Tomek posted a draft for Cloudflare R2 with scoped credential vending, and Tommy Bollhofer offered to test resumed remote signing work on FlashBlade storage with Trino, Spark, PyIceberg and DuckDB clients.
Qiegang Long found a correctness bug worth reading. In the conditional entity writes thread, he showed that a multi-table commit can assert a table is unchanged without including it in the atomic write. Two concurrent commits can both succeed, each changing one table while asserting the other. That is classic write skew. He opened the SPI change as a draft and brought it to the list first, as CONTRIBUTING asks for extension-point changes.
Ossie inside Polaris
Arnab also drafted semantic model JSON Schema validation, validating stored semantic models against a bundled Apache Ossie schema and capping document size at 1 MiB. JB asked him to wait for the Ossie REST API proposal so Polaris does not lock in a validator interface that the standard then changes. Arnab agreed and linked his draft on the Ossie PR. That small exchange is the clearest sign yet that Polaris plans to be a home for Ossie semantic models.
How the project works
Yufei opened a timely thread on PR quality and sharing the review load. He asked authors who use AI to act as the first reviewer of what the AI writes. Dmitri agreed and suggested sharpening AGENTS.md. JB pointed out that CONTRIBUTING.md already has a section on AI-assisted contributions and asked reviewers to link it rather than invent new rules. Youngrae Kim shared two practices from the contributor side: write the failing reproduction test first, and review PRs in areas you just touched. Yufei separately proposed preferring Java records for simple value types, noting that AI agents now write the boilerplate that once justified the Immutables library. The group agreed for new code only.
Alex Dutra also revived the CHANGELOG merge conflict thread and implemented changelog.d fragments using Dmitri's naming scheme. Robert Stupp wrote a reply worth quoting in spirit. When contributors report recurring friction, saying others hit it less often does not resolve it. A useful reply names a drawback, proposes an alternative, or agrees to try.
Apache Arrow
Arrow shipped a patch release, got new funding, and started a conversation about AI contributions that mirrors Polaris and Iceberg. Raúl Cumplido announced Apache Arrow 26.0.1 after an RC1 vote that drew detailed verification reports across macOS, Debian, Fedora and Ubuntu. The release resolves 19 GitHub issues. The headline fix is security. Javier Guerrero asked whether the Brotli CVE fix lands in 26.0.1, since Arrow statically linked Brotli 1.0.9, which is affected by CVE-2026-111497. JB confirmed during verification that the bundled Brotli is now 1.2.1. If you ship PyArrow wheels inside a product, upgrade.
The funding news came from Alenka Frim. Germany's Sovereign Tech Agency contracted research and development work for PyArrow. Alenka, Nic Crane, Rok Mihevc and Tadeja Kadunc handled the application, with Open Source Collective as fiscal host. The contract funds an average of 1.9 full-time engineers through the rest of 2026 and all of 2027. It targets work that stalled because no corporate sponsor wanted to pay for it, starting with PyArrow type annotations. Antoine Pitrou, Weston Pace, Neal Richardson, Matt Topol and others replied with congratulations. Public money for core data infrastructure is still rare, and this is a good model.
Contributions, human and otherwise
Andy Grove gave the Java community a heads-up that DataFusion Comet plans to send PRs upstream for workarounds it carried against arrow-java. He was direct about the method: a human in the loop, AI writing the code, and real issues behind each PR. David Li thanked him and asked for patience, since contributor capacity on arrow-java is low. JB started reviewing. Raúl and Nic Crane, meanwhile, discussed the CPython AI usage guidelines. Python uses one label that asks a PR author to explain the change in their own words, and a second label that closes the PR when that expectation is not met. Nic liked the "give them a chance, then close" approach and the automation that labels allow.
JB also posted a heads-up for Arrow Java 20.0.0 release prep, and Fred Park asked for review of a hostnameOverride property for the Flight SQL JDBC driver. The property lets users keep TLS certificate checks on when they connect through SSH tunnels, private endpoints or raw IP addresses. Today the only workaround turns certificate verification off.
Format and tooling
Mandukhai Alimaa bumped the canonical extension type for FILE and said a vote comes soon if nobody objects. This is Arrow's half of a cross-project effort. Parquet already merged a FILE logical type, Iceberg is designing FILE and VECTOR types, and Arrow needs a canonical in-memory form so engines can pass these values around without custom glue.
Wes McKinney shared an update on Arrow's conbench data. His team has run a Go port, now called BenchDB, in production for a couple of months, with historical data offloaded to Parquet files managed by an embedded DuckDB. He offered help porting Arrow's benchmark runner and said his company is willing to sponsor cloud costs. Arrow Rust 60.1.0 passed its vote and is on crates.io, and ADBC 25 closed its RC2 vote.
Jeffrey Vo raised the CI issue every Apache Rust and Java project now faces. In the pull_request_target migration thread, he warned that GitHub will block the trigger by default on December 2, with brownouts on November 9 and November 16. A scan found arrow, arrow-java, arrow-site and arrow-erlang still use it. Raúl opened a tracking issue the same day.
Antoine proposed an informal Arrow gathering at Compute! Paris on the morning of Thursday, November 26. JB and Raúl plan to attend. Ian Cook posted the November 4 community meeting at 17:00 UTC.
Apache Parquet
Parquet had the most dramatic vote of the week. Rok Mihevc called a vote to add the FINITE_VECTOR logical type as specified in parquet-format PR 624. The type annotates the same structure used for LIST, so readers that do not know it still see a list. Every non-null vector has exactly num_elements elements, and every element is finite: no NaN, no null, no infinity. Support came fast from Julien Le Dem, Ryan Blue, Daniel Weeks, Gang Wu, Russell Spitzer and others.
Then Antoine Pitrou voted -1 (binding). His objection was not about vectors. It was about the sentence "Writers must enforce these rules." Parquet does not require writers to validate STRING data as UTF-8, and Parquet C++ does not do so. If FINITE_VECTOR demands writer-side validation, then the natural expectation is that STRING and JSON do too. That is a much larger change in Parquet's philosophy. Antoine said writers can validate, but it should stay an implementation choice. Rok asked him to propose wording on the PR. He did. Rok updated the PR with feedback from Antoine and Andrew Lamb and moved the vote to the new commit, asking earlier voters to look again. The vote stays open for at least 72 hours.
That vote grew out of the focused-call follow-up on numeric vector physical representation. Will Edwards worried that the annotation will "orphan" huge volumes of existing embedding data stored as plain lists of floats. Daniel Weeks told the group to stop arguing about performance, since speed is not the goal of the type, and confirmed that both Iceberg and Delta want it with these semantics. Ryan Blue expects the Iceberg spec to require writers to produce the annotation. Jan Finis drafted an alternative in PR 633 with two composable building blocks: an optional elementCount on lists and an optional isFinite property on float types. Rok replied that FINITE_VECTOR is narrow by design and does not block a general fixed-size list later. Julien made the meta point that settled the tone. Consensus on new types stalls easily, and narrowing scope is how projects keep moving.
Getting the spec right
Two threads this week dealt with how Parquet changes its own rules. Julien posted a taxonomy of format changes that defines backward compatibility, forward compatibility, correct reads and "correct failed reads," where a reader errors out instead of returning wrong data. It also defines partial compatibility, where a reader can read the subset of columns it understands. Julien wants agreement on definitions before the implementation discussion resumes.
The FILE type then gave that taxonomy an immediate test. Stefan Grafberger found an ambiguity when a FILE value sets both inline and uri. If the object at the URI changes after the write, one reader returns the inline bytes and another returns the new object, and both follow the spec. He proposed "inline wins," with locator fields kept as provenance. Divjot Arora asked whether changing read rules for a released type counts as a compatible change. Julien answered directly. When the spec under-defines behavior, the project has to pick the correct reading, declare the others bugs, ship fixes, and keep compatibility with the clarified spec from then on.
Jiayi Wang posted the revised Parquet Modular Footer proposal. The new version stops treating row groups as physical boundaries. A column chunk is now just a contiguous set of pages for one column, and different columns can have different numbers of chunks. Stats are defined over row ranges, not row groups. The confusing EncodedArray concept is gone, replaced by Parquet pages as modules. Julien called it "fairly transformational" and asked everyone to review, since the project wants to avoid changing the footer again for a while.
A release, with lessons
Parquet Java 1.18.2 is close. Fokko Driesprong proposed the patch release to carry Jackson 2.22.4 and Thrift and Snappy backports, and Russell noted the Jackson bump lets Iceberg remove a Trivy CVE exception. Manu Zhang volunteered as release manager and used the new automated release scripts. RC0 failed when Gang Wu found two problems. The signing key was missing from the KEYS file, and the source zip contained an entire Thrift build tree with 41 compiled object files and an 80 MB executable. Manu fixed the script and cut RC1. His verification even ran Iceberg's CI against the RC, with the Jackson CVE ignores removed. Russell then spotted an incorrect SCM tag in the POM and pushed a fix to the release automation. New release tooling always finds its bugs on the first real run, and this team found them in public and fixed them fast.
Apache DataFusion
DataFusion's list was short but pointed. Andy Grove ran the vote for DataFusion Comet 1.2.0, the Spark accelerator that runs Spark plans on DataFusion. Manu Zhang voted -1 (non-binding) after finding that the JAR file is missing lz4 and zstd licenses, and he opened a fix. Martin Grigorov argued it should not block the release. Manu cited ASF release policy, which requires LICENSE and NOTICE files to cover everything in a package, including binary JARs. Andy approved the fix for the next release and moved forward. The vote passed with nine +1 votes, five binding. The licensing question echoes the Iceberg Rust delay a week earlier: compiled artifacts carry obligations that source releases do not.
Andrew Lamb opened two threads that set the tone for the next release cycle. He proposed slowing feature velocity to work down the bug backlog and asked for feedback on a GitHub issue. A few days later he announced that the 56.0.0 release is starting, with Xudong cutting branch-56 and an RC expected in days. He also called the vote for sqlparser-rs 0.64.0, which passed verification with 1,917 tests and no failures in one report.
CI cost shaped two more threads. Andy announced that Comet fork PRs now run CI on the contributor's fork, not on shared ASF runners, matching what DataFusion and arrow-rs already do. Andrew said the change made a large difference in the runner time the ASF gets charged for. Jeffrey Vo then posted a call to migrate away from pull_request_target before GitHub's December 2 block, and Kumar Ujjawal found a PR already in progress. Andy also opened a discussion on the future of the datafusion-spark crate after the weekly sync.
Apache Ossie
Apache Ossie, the incubating open semantic interchange standard, was the busiest list after Iceberg. The project is preparing its first Apache release, and the week's threads read like a spec hardening under real data.
Yufei Gu opened a thread on stability expectations for the first release, which will be either 0.2.0-incubating or 0.3.0-incubating. His reading: 0.x is beta, minor releases can break things, patch releases cannot, and 1.0 brings backward compatibility within 1.x. He argued for shipping now, since converters need a version to build against instead of a moving main branch. JB agreed and said 1.0 can follow quickly once something is stable and verified. Kyoung Min Kim added a practical warning. The Python models hard-code the version string 0.2.0.dev0 in a Literal type, so a patch release breaks five converters unless that changes too. JB then posted the release resources prep with three PRs that stage release scripts, bump the version to 0.3.0-incubating and align converter dependencies. Ossie will use the ASF's ATR release tooling.
What a table name means
The deepest technical thread asked what Dataset.source should guarantee. It ran 17 messages and pulled in real data from three producers. JB started from a gap one step before connections. Catalogs such as Apache Polaris allow dots in table names, yet the Ossie schema treats source as a plain string with no delimiter or quoting rule. John DaCosta proposed a list form, identifier: [prod, sales, eu.customers], with dotted strings kept as shorthand. Damian Waldron ran the proposal against 164 datasets from 31 real ThoughtSpot models. All of them stayed valid shorthand, so the change costs nothing on real data. Chad CH Kim ran 53 sources from his generator and got the same result.
Richard SG Kim then reframed the problem. The repo's own example uses DATABASE.SCHEMA.RUNWAYS, where the first two parts are placeholders. A tool reading that file looks for a database literally called DATABASE. He proposed a qualification field that separates two cases. A relative reference means the consumer must supply the missing parts. An absolute reference with too few parts is a broken document. Damian changed his mind on the list, and JB asked for three conformance behaviors written into the spec: import a relative reference the consumer can bind, reject one it cannot bind while naming the missing parts, and treat a short absolute reference as invalid. JB's key line: a converter should fail loudly, not write an empty value and exit 0.
Query semantics
Chris Eubank proposed the Layer 3 wide-table query interface. A Layer 3 query names dimensions, metrics and an optional filter. The engine picks the joins and lowers the query to Layer 2. Measure evaluation reuses Layer 2's AGG() function, so the spec has one source of truth for measures. Aleksandr Kosachev confirmed the worked examples on ClickHouse and gave it a +1. A GitHub discussion on how semantic engines should interpret relationship definitions explained the split. At Layer 2, relationships are descriptive metadata. At Layer 3, they drive join selection. Julian Hyde added that Layer 3 is limited on purpose: it lets you pick dimensions, metrics, filters and sort order, but offers no joins or unions.
The AGG() work itself moved in the dataset-scoped metrics thread. Justin Talbot split his PR in two after JB's review. The first reserves AGG() in the expression language. The second lets dataset metrics reference source columns, and it now requires the source. qualifier, so adding a field named quantity later cannot silently change what SUM(quantity) means. Julian Hyde pushed on terminology: the project needs to pick "metric" or "measure" and use it consistently. The October 28 working group notes track both workstreams.
David Schwarzenbacher proposed fixing two OSSIE_SQL_2026 rules by lazy consensus on November 9. ROUND breaks ties half away from zero, and / means true division, so 5 / 2 is 2.5. Engines differ on both today, which means the same portable metric returns different numbers on different engines. Johannes Zemlin opened the ANTLR4 reference grammar PR with valid and invalid conformance fixtures, keeping the written spec normative.
Converters, validators and the ecosystem
Validation got a careful survey. Kyoung Min Kim mapped what the 23 converters check today and where a vendor hook belongs. JB proposed two limits: a vendor checker only reads its own payload and dialects, and vendor findings never change core results. Andrew Nepogoda asked for stable, documented error codes and severities instead of free-form strings. Andrew also proposed publishing the apache-ossie Python package to PyPI, after main-branch changes broke converters he did not own. JB asked him to wait for the release prerequisites, citing legal checks on published artifacts. His related thread on shared models and model-based validation moves ontology types into the shared package.
New contributors kept arriving. Yurii Melnychuk asked how to contribute Coginiti's Rust converter, and Yufei confirmed apache/ossie-converters is the home, with thin wrappers fine. Chen Ning from Dameng Database introduced himself with a consumer-side view: a text-to-SQL system is only as good as the semantic model it is grounded on. The GASP SaaS metrics standard announced it publishes its 306 metrics as Ossie files, with 164 staged datasets. Mike Carlo asked for a vote on making custom_extensions data an object instead of a string, and Purvaja Narayanaswamy proposed an Action concept for the ontology spec to describe business operations next to terms and metrics. Two GitHub discussions explored graph-based ontology integration with an RDF mapping and whether Ossie is an interchange format or a source of truth. JB's answer to the second: both, in layers. Mario De Felipe gave the SAP view, where the ERP system owns structure and a small human-owned layer sits on top.
Cross-Project Themes
The biggest theme this week was ambiguity in specs, and how each project chose to resolve it. Iceberg found four readers giving four answers for partial field IDs. Parquet found two readers returning different bytes for the same FILE value. Ossie found that its own example file contains placeholder table names a tool will treat as real. Polaris and Iceberg disagreed for a day about whether a failed metrics write can return success. In every case, the fix followed the same pattern Julien Le Dem described on the Parquet list. Pick one correct reading, write it into the spec, and declare the other behaviors bugs. The Ossie thread went one step further by asking for conformance behaviors, so a tool that cannot do the right thing fails loudly instead of guessing. That principle belongs in every open format.
The second theme was review capacity in the age of AI-written code. Polaris's board report put numbers on it: 137 open PRs and a backlog that grew 34% in a quarter. Iceberg's health dashboard surfaced the same pattern as a drop in second contributions. Arrow discussed borrowing CPython's labels for AI-generated PRs, and Andy Grove opened his Comet contributions with a note that a human stayed in the loop. DataFusion is pausing features to fix bugs. None of these projects banned AI tools. They are converging on one rule: the person who opens the PR owns it end to end and must explain it in their own words.
The third theme was vectors and files moving across the whole stack at once. Parquet is voting on FINITE_VECTOR and clarifying FILE. Arrow is about to vote on a canonical FILE extension type. Iceberg treats vectors as primitives, proposed vendor-neutral AI functions over FILE and VECTOR columns, and is weighing whether v4 should write only Parquet. Read together, these threads describe an open lakehouse that stores embeddings and unstructured references as first-class typed data, with the same type meaning the same thing from file to memory to table. That only works because the same people sit on several of these lists. Daniel Weeks, Ryan Blue and Russell Spitzer voted on Parquet's vector type while shaping Iceberg's.
Release hygiene was the fourth thread. Iceberg Rust waited weeks to fix NOTICE files in compiled wheels. Comet shipped with a known JAR licensing gap and a fix queued. Parquet's new release automation packed a Thrift build tree into a source zip on its first run. Arrow 26.0.1 existed mostly to pick up a Brotli CVE fix. And Arrow, DataFusion and Comet are all racing GitHub's December 2 block on pull_request_target. Binary artifacts and CI pipelines now take as much care as the code itself.
What to Watch Next Week
The FINITE_VECTOR vote resets its 72-hour clock on the updated commit, so watch whether Antoine Pitrou's wording change holds the earlier +1 votes. Iceberg's default value expressions vote is ready to close, and the _last_updated_time vote needs Daniel Weeks's type change and Ryan Blue's editing pass. Polaris 1.9.0 is close, and the credential vending debate needs more voices before PR 5884 merges on November 6. Ossie's lazy consensus on ROUND and division closes November 9, and its first release candidate depends on the version string fix. DataFusion 56.0.0 should produce an RC, and GitHub's first pull_request_target brownout lands November 9, so expect red CI on any repo that has not migrated.
Keep Learning
If you want to go further on Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet and the open lakehouse, I have written a lot more on these topics. You can find all of my books, including Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide, along with my self-published guides on lakehouses and agentic AI, at books.alexmerced.com.
Top comments (0)