The open lakehouse stack spent this week arguing about what belongs in a spec and what belongs in a release. Iceberg contributors opened a formal push to remove equality deletes from the V4 spec, Parquet closed its vote on a brand new File logical type, and Polaris wrestled with how much consistency its persistence layer owes its users. Underneath the design debates, release trains kept moving: Iceberg Rust, an Iceberg Terraform provider, Arrow JS, Arrow Rust Object Store, DataFusion, Ballista, and Comet all had votes in flight. This is a week where the community showed both sides of its personality, big structural bets for the future and steady, unglamorous shipping for the present.
By the numbers, the six dev lists carried 268 messages across roughly 80 distinct threads in the past week. Polaris led with 84 messages, Iceberg followed at 71, Parquet posted 51, DataFusion 26, Arrow 20, and the young Ossie project added 16. Those raw counts undersell the range: the week included two format-level votes, five release candidates, one new committer, a persistence redesign proposal, and at least four threads that will shape spec decisions months from now. Grab a coffee. There is a lot to cover.
Apache Iceberg
The most consequential thread of the week came from huaxin gao, who restarted the conversation on deprecating equality deletes, this time framed around the V4 spec. Russell Spitzer first proposed the idea back in October 2024, and the blocker then was real: Flink streaming upserts depended on equality deletes and had no practical replacement. The new thread argues that the ground has shifted. Equality deletes are cheap to write but expensive to read, since every reader must join delete files against candidate rows across a sequence number range. Positional deletes and V3 deletion vectors read far faster, one compact bitmap per data file applied with an O(1) position check. The thread goes past performance too. Equality deletes block CDC and row lineage because the true state of a table requires a full scan while they exist. They also force full rebuilds of materialized views and secondary indexes instead of incremental maintenance. Steven Wu, Manu Zhang, Maximilian Michels, and Xin Huang all weighed in, and the discussion drew eight messages in its first days.
For readers newer to Iceberg internals, the stakes are worth spelling out. Iceberg supports two ways to delete rows without rewriting data files. An equality delete says "remove every row where id equals 42," which a streaming writer can emit in microseconds without reading anything. A positional delete says "remove row 1,507 of file X," which requires the writer to know exactly where the row lives. The first pushes all the cost onto readers, who must evaluate the predicate against huge swaths of data on every query. The second keeps reads fast at the price of more expensive writes. V3 deletion vectors compress the positional approach into one bitmap per data file. The V4 question is whether the format still needs the first option at all once conversion tooling makes the second cheap enough for streaming workloads.
The timing is not an accident. Maximilian Michels reported that the Flink equality delete to deletion vector conversion work is now complete, merged across six commits. The feature ships as a new table maintenance task called ConvertEqualityDeletes, integrated with IcebergSink. Writers keep producing data files and equality deletes as before. The converter reads them, resolves the deletes into deletion vectors using a primary key index backed by RocksDB in Flink state, and commits data files plus DVs to the target branch. Teams can stage writes on a separate branch so readers never see equality deletes at all, or run in-place conversion on a single branch. This is the workable alternative that was missing in 2024, and it lands right as the V4 deprecation push begins. Read the two threads together and you see a community clearing a path before it closes a door.
Release work stayed busy. Danny Jones and Shawn Chang called a vote on Iceberg Rust 0.10.0 RC4 after RC3 gathered votes from Kevin Liu, Amogh Jahagirdar, and others earlier in the week. Sung Yun, Renjie Liu, L. C. Hsieh, and Maximilian Michels verified RC4, which includes a check that pyiceberg-core builds and tests cleanly against the release. The Rust implementation now sits underneath the Python ecosystem, so each Rust release carries weight well beyond Rust users. That dependency chain is worth pausing on. PyIceberg increasingly delegates its performance-critical paths to pyiceberg-core, which is compiled from this Rust codebase. A bug in iceberg-rust becomes a bug in every Python notebook and Airflow DAG that touches Iceberg through PyIceberg. That is why the release checklist now explicitly verifies the Python bindings, and why voters from the Python side of the community show up on Rust release threads. The 0.10.0 line also continues the project's steady march toward feature parity with the Java reference implementation, which lowers the barrier for teams that want Iceberg without a JVM anywhere in the stack.
Infrastructure as code arrived as a first-class citizen this week. Matt Topol proposed RC0 of the Apache Iceberg Terraform Provider v0.1.0, the project's first release of the provider, with convenience binaries prepared for the Terraform and OpenTofu registries. Neelesh Salian, Alex Stephen, Sung Yun, Talat Uyarer, and Rich Bowen all participated in verification, and an RC1 vote followed as issues surfaced. Once this lands, teams can declare Iceberg resources in the same Terraform plans that manage the rest of their infrastructure. Think about what that unlocks in practice. A platform team can define namespaces, tables, and their properties in version-controlled HCL, review changes through pull requests, and roll environments forward and back with the same tooling they use for VPCs and Kubernetes clusters. Catalog drift, the gap between what the catalog says and what the last runbook did, becomes a solved problem instead of a recurring incident. It took years for databases to get credible Terraform support. Iceberg is getting there in its first decade.
AI showed up on the dev list in a very concrete form. Gang Wu asked the community about enabling ASF-managed GitHub Copilot code review on Iceberg repositories, starting with iceberg-cpp as a trial. The proposal uses the .asf.yaml setting that ASF Infra now supports, and it follows Apache Arrow, which enabled and tuned the same feature. Gang picked iceberg-cpp because reviewer bandwidth there is thin, and an automated first pass can catch simple issues before a human review. The thread drew eleven messages from Steve Loughran, Junwang Zhao, Scott Haines, and others, making it the most active Iceberg discussion of the week. The questions were practical: what does it cost in CI resources, how noisy is the feedback, and who tunes it. There is a bigger question under the surface. Open source review is the mechanism by which projects transfer knowledge, enforce standards, and grow maintainers. An automated first pass that catches typos, missing null checks, and doc gaps frees human reviewers for design feedback, which is a clear win. An automated pass that contributors treat as the review, or that buries PRs in low-value comments, erodes the very culture it was meant to help. Starting with one low-traffic repository and evaluating before expanding is the right way to find out which outcome Iceberg gets. The fact that Arrow already ran this experiment and tuned it gives Iceberg a head start on configuration.
Two type system proposals advanced. Yan Yan opened a discussion on first-class vector type support for Iceberg. Embeddings are everywhere in AI workloads, and today they live in Iceberg as list, which cannot express the invariant that every value shares one dimension. The proposal prefers a dense numeric vector type with compact schema encoding, something like float[768], with fixed dimension, non-null elements, and nullability controlled at the field level. It points at parallel work in the Parquet community on fixed-size lists, which matters because the table format and the file format need to agree for the type to pay off. Meanwhile the collation support discussion between Andrei Tserakhau, Alexander Löser, and Russell Spitzer dug into a genuinely hard question: how much cross-engine interoperability should the format guarantee for string ordering? Alexander laid out the trap in detail. ICU does not keep orderings stable across versions, so two engines on different ICU versions can sort the same strings differently, return different aggregation results, and filter different rows. A colleague of his once rolled back an ICU upgrade in production because users complained about changed sort orders. Pinning an ICU version at the table level buys consistency and costs upgrade freedom. The thread has not resolved the tension, and it is worth watching because collation touches execution, pruning, and equality semantics all at once.
The file format layer got its own existential question. Martin Prammer proposed adding Vortex as an Iceberg file format, and smartly split the draft into two parts: what criteria any candidate file format should meet, and how Vortex meets them. That framing turns a single-format request into a durable policy, which is exactly what a spec-driven project needs as more formats knock on the door.
Quality and correctness threads kept coming. Priyadarshini Mitra proposed a ValidateTableIntegrity action that walks the full metadata graph, metadata.json entries, manifest lists, manifests, data files, delete files including V3 deletion vectors, and statistics files, verifying every referenced file exists on storage. It supports a self-audit on one table and a source-versus-destination check for DR and migration scenarios, tracked across three sequential PRs. Neelesh Salian, Sung Yun, and Andrei Tserakhau merged their earlier threads into one proposal for shared conformance fixtures, a standalone language-neutral repository modeled on parquet-testing, so every Iceberg implementation checks its reading of the spec against a shared answer key instead of only against itself. Working proofs of concept already exist for pyiceberg, iceberg-rust, and iceberg-go. And Russell Spitzer moved to clarify in the spec that live manifest entries must be unique by file path, tightening language first added four years ago so writers know duplicate references are simply not allowed.
Russell also raised a question about breaking behavior in AvroSchemaUtil, where adding LocalTimestamp support changes what convert returns for local-timestamp-micros, from Long to TimestampType.withoutZone(). His position: the old behavior is a bug, and Iceberg should not preserve incorrect legacy behavior behind a flag for outside consumers. Ryan Blue engaged on the thread, and the precedent cited is the earlier NanoTimestamp change that did the same thing.
On the operational side, Oleksii Omhovytskyi asked about release timing for the encrypted deletion vector fix. On natively encrypted format-v3 tables, a merge-on-read UPDATE or DELETE wrote a deletion vector Puffin file without key metadata, and the next read failed. The fix is merged with backports staged on the 1.11.x and 1.10.x branches, and Oleksii verified the 1.11.x branch against his exact repro. He offered to test any release candidate, a nice example of a user pushing a patch release forward with evidence instead of just a request. Amogh Jahagirdar responded on timing.
Several single-message threads planted seeds worth tracking. Anurag Mantripragada proposed using Iceberg sort order metadata to improve read and compaction behavior in Spark. Iceberg tables already record their sort orders in metadata, but engines rarely exploit that knowledge at plan time, so there is free performance sitting on the table. A CDC practitioner opened a thread on row-delta commit patterns and multi-table transactions in iceberg-rust, sharing lessons from a production change-data-capture pipeline and asking what the Rust library should support natively. Oğuzhan Ünlü requested review on a PR adding typed exceptions for OAuth2 token endpoint errors in the API and core modules, small plumbing that makes REST catalog auth failures debuggable instead of mysterious. Andrei Tserakhau also pitched a series of Iceberg technical blog posts with a first draft attached, and Matt Butrovich responded. Community-written deep dives are one of the best on-ramps a project can have, so this effort deserves support. On the ecosystem edge, Piergiorgio Lucidi introduced the OpenCrawling connector, which bridges Iceberg tables into enterprise AI and RAG pipelines, one more signal that retrieval workloads now treat the lakehouse as a first-class source.
Rounding out the week: Adam Szita moved the KMS credential vending proposal into a draft REST OpenAPI spec PR, mirroring storage credential vending so REST catalogs can return short-lived scoped key-management credentials for encrypted tables. Ryan Blue weighed in on Spark routing for Iceberg Materialized Views, favoring a basic implementation in Iceberg itself that replaces a view with a table read, usable without engine APIs, with engines layering smarter freshness decisions on top over time. Alexandre Dutra opened a discussion on migrating Iceberg to Jackson 3, a mechanical but pervasive change whose hardest part is that Jackson types leak into the public API of the parser classes and REST layer, so Jackson 2 and 3 will need to coexist for a while. A contributor from Alibaba proposed adding an Alibaba Cloud auth type for the REST catalog. And Talat Uyarer announced an Apache Iceberg meetup in Austin on July 23.
Apache Polaris
Polaris had the busiest list of the six projects this week at 84 messages, and the center of gravity was persistence consistency. Dmitri Bourlatchkov opened a discussion on consistent multi-object changes in Polaris persistence, prompted by PRs from Ayush and Prithvi that surfaced real consistency gaps in the JDBC backend. Dmitri's framing is deliberately broad: rather than patching individual windows, the community should design one approach that covers concurrent validated commits, independent but consistent RBAC changes, atomic multi-entity updates, authorization-filtered listings, credential vending rooted in exact catalog state, and server-side retries for transient failures. Prithvi S made the problem concrete in a companion thread on an atomic multi-entity plus grant commit SPI. Today, grant and revoke, createCatalog, and dropEntity each compose multiple atomic SPI calls, so a server failure mid-sequence leaves partial state, a grant without version bumps or a catalog without its admin role. His draft adds a writeEntitiesAndGrantRecords method to BasePersistence that does entity writes, deletes, and grant changes in one all-or-nothing operation, and he asked the community whether to start narrow with grants only or migrate all three flows at once. Robert Stupp and Dmitri both engaged. These two threads together read like the start of a persistence redesign, and how Polaris answers will shape every backend it supports.
Identity and authorization saw the single most active thread of the week. Prithvi S, Dmitri, Alexandre Dutra, Yufei Gu, and Jean-Baptiste Onofré traded ten messages on forwarding user-defined principal properties in PolarisPrincipal. The resolution matters for anyone running external authorizers: the authentication layer will forward user information to PolarisPrincipal as optional attributes, which decouples OPA and Ranger authorizers from both Quarkus classes and PrincipalEntity. Authorizers then work with any identity provider and survive Quarkus upgrades untouched. Alexandre Dutra also followed through on standardizing vended credential property names, opening a PR that generates credential documentation straight from the StorageAccessProperty enum. Along the way he found and removed a spurious vended property, expiration-time, that matched no known credential.
The datasource architecture debate sharpened. In the Polaris-managed JDBC datasource thread, Yufei Gu clarified the two motivations, runtime loading of JDBC drivers for ASF binaries and runtime datasource creation as a building block for per-realm datasources. His proposed contract keeps Quarkus and Agroal as the default, with Polaris-managed Hikari as an escape hatch. Alexandre Dutra pushed back hard on the hybrid: alternating pools based on configuration means bugs and performance characteristics vary across deployments purely by pool choice, so if the goal is a runtime-driven architecture, commit fully and switch to Hikari unconditionally. Romain Manni-Bucau, JB, and Dmitri also weighed in. Nobody has yet closed the gap between "escape hatch" and "all or nothing."
Process and culture got real attention too. Dmitri opened a thread titled Respecting developer and reviewer cognitive work after a review dispute about introducing build warnings via intentional deprecations. JB called avoiding new warnings an implicit good practice that is acceptable when explained. Yufei argued the community should either adopt an explicit documented rule or stop letting individual reviewers enforce it case by case, since documented rules give contributors predictable expectations and prevent double standards. Adnan Hemani and Robert Stupp joined as well. Every open source project has this conversation eventually, and Polaris is having it in the open.
A vote is now live on error semantics. Following an earlier discussion, vignesh a called a vote to return HTTP 503 Service Unavailable when a table or view rename fails with TARGET_ENTITY_CONCURRENTLY_MODIFIED. The reasoning: 409 already means "target identifier exists" in the Iceberg REST spec, and 429 wrongly implies rate limiting, so 503 is the retryable option without semantic conflicts. The vote closes at 14:00 UTC on Sunday, July 19, with Robert Stupp, Alexandre Dutra, Nándor Kollár, and Dmitri participating. Nándor had teed up the choice in an earlier discussion thread on status codes for rename conflicts. Status code selection sounds like trivia until you remember that every Iceberg REST client on the planet encodes retry behavior against these codes. Pick a code that clients interpret as permanent failure and transient contention turns into user-visible errors. Pick one with the wrong retry semantics and clients hammer a struggling server. Getting this right once, by vote, spares every client library a heuristic.
The semantic layer story kept building, and it connects straight to Apache Ossie below. In the Semantic Model REST API payload thread, Robert Stupp supported Polaris hosting Ossie semantic-model documents as a beta foundation, then drew a sharp line: the merged API is namespace-and-name CRUD over opaque documents, and the project should not describe it as enabling AI tools, BI tools, or semantic discovery until clients can actually find models by table, metric, domain, or capability. His larger point is architectural. The client consumption model should drive the persistent data model, because once semantic models become durable Polaris entities, identity, versioning, indexing, and freshness semantics get very hard to change. This debate matters well beyond Polaris. The industry is converging on the idea that AI agents need a semantic layer to query data correctly, and catalogs are the natural place to host one. Whoever defines how agents discover the right semantic model, by table, by metric, by domain, by trust level, defines a big piece of how agentic analytics works. Robert's insistence on honest labeling, calling document CRUD what it is until discovery exists, protects users from building on promises the API does not yet keep. EJ Wang also shared a Polaris Tag Spec design proposal for community review, a native tag model covering tag definitions as catalog-scoped entities, assignments down to the column level, allowed values, inherited reads, and by-tag lookup, with a review slot planned for the July 23 community sync. And EJ posted updated framing for the table metrics and events REST work: the persistence refactor splits into its own PR, the metrics SPI stays in core with a no-op default, and the REST query API plus JDBC implementation land as optional extensions.
A cluster of smaller operational threads rounded out the Polaris week. Eundo Lee, Alexandre Dutra, and Yufei Gu discussed making the Relational JDBC schema name configurable, which matters for teams that run multiple services against one database and need Polaris to live in its own schema. Yong Zheng and yun zou, with input from Robert Stupp and Dmitri, weighed moving the Spark plugin regression tests from a Docker-based harness to JUnit, trading environment fidelity for speed and debuggability in CI. EJ Wang and Dmitri discussed removing PolarisMetricsManager from PolarisMetaStoreManager, part of the same untangling that the metrics SPI refactor demands. Dmitri also proposed dropping the schema-version option from the bootstrap command and making catalog ID nullable in the JDBC events tables, while Yufei opened a thread on supporting staged creates inside multi-table commitTransaction calls. None of these is glamorous. All of them are the difference between software that demos well and software that operators trust.
Polaris is also getting its own Terraform provider. Alex Stephen explained that the Iceberg provider community chose to focus exclusively on Iceberg resources, so the Polaris resources need a new home. Sung Yun called lazy consensus on creating the terraform-provider-polaris repository, a name the HashiCorp registry requires. Housekeeping continued elsewhere: Dmitri proposed deprecating TreeMapMetaStore for removal, laid out SPI development principles including where SPI classes should live to minimize dependency leaks, and Robert Stupp clarified what the first OpenLineage scaffolding merges do and do not settle, insisting on explicit usability and operational criteria for the local lineage store before schema work merges.
Apache Arrow
Arrow's headline discussion was about making schemas travel well. Matt Topol, David Li, and Dewey Dunnington continued the design conversation on a JSON representation of Arrow schemas. David argued for a representation that is unambiguous, consistent, and friendly to both humans and machines, aimed at REST APIs and ADBC, where compactness matters less than clarity. Dewey noted that abbreviations like uint32 match how implementations actually enumerate types, so they simplify parsers while shrinking payloads, and he worked through how extension types like GeoArrow should carry their metadata so an API consumer can read an extension parameter without re-parsing escaped JSON. This sounds like a small thing. It is not. A standard JSON schema form gives every catalog, REST service, and agent framework a common way to describe Arrow data without touching the binary IPC format. Today, every project that needs to express an Arrow schema over HTTP invents its own encoding, and every one of those encodings handles extension types, nested fields, and metadata a little differently. One blessed representation means an ADBC server, an Iceberg REST catalog, and a Flight service can all describe the same table the same way, and a client can validate schemas before any data moves. The care the trio is taking with edge cases now, especially extension metadata, is what will keep the format from needing a breaking revision later.
Release votes moved on two fronts. Sutou Kouhei proposed Apache Arrow JS 21.2.0 RC1, with David Li, Kent Wu, Bryce Mecum, and Hyukjin Kwon verifying. Andrew Lamb called the vote on Arrow Rust Object Store 0.14.1 RC1, and Raúl Cumplido, Krisztián Szűcs, and L. C. Hsieh returned binding +1s after running the verification script. The object_store crate sits under a huge slice of the Rust data ecosystem, including iceberg-rust and DataFusion, so patch releases here ripple outward fast. Ian Cook also hosted the Arrow community meeting on July 15.
Apache Parquet
Parquet delivered the week's biggest format decision. Burak Yavuz announced the result of the vote on the new File logical type: it passed with 18 +1 votes, 4 of them binding, from a list that includes Daniel Weeks, Russell Spitzer, Andrew Lamb, Gang Wu, Fokko Driesprong, Steve Loughran, Gunnar Morling, and more. The vote thread itself drew 19 messages and capped months of design docs, biweekly syncs, and a discussion thread, with reference implementations already open in parquet-java and arrow-rs. A File logical type lets Parquet columns carry file-like binary payloads with defined semantics, and the breadth of the voter list, spanning Iceberg, Arrow, and Parquet maintainers, says a lot about who plans to use it. The use cases are easy to picture. Multimodal AI datasets carry images, audio clips, and documents alongside tabular features today, usually as raw binary columns with semantics living in tribal knowledge or sidecar metadata. A File logical type gives readers a standard way to know that a column holds file content, opening the door to smarter tooling, previews, and type-aware processing across engines. Burak is holding the format PR open a few more days for remaining reviewer feedback from Rok, Antoine, and Gang, and a follow-up design conversation with Talat and Gaurav on version identifiers continues on the doc.
The long-running versioning debate produced structure this week. Micah Kornfield pulled format versioning changes into a standalone RFC PR, separate from the larger PARX proposal, and after feedback from Antoine Pitrou he dropped the SemVer branding entirely in favor of plain language about major version bumps on forward-incompatible features. He also tightened the recommendation for when writers should flip to a new default version, 6 to 18 months after a format release. Ryan Blue defended continuing the current vote in the Q&A thread on how format versions work, using an extended apples-versus-oranges analogy to argue the community already spent a month choosing between numbered releases and time-based presets, and the vote simply affirms that choice. Julien Le Dem scheduled an ad hoc sync with a poll to compare the proposals side by side and get to a conclusion faster. Watching Parquet design its own release governance in public is a treat for anyone who cares about how standards evolve.
Fokko Driesprong moved Apache Parquet 1.18.0 toward release, saying he plans to start the process next week after multiple requests for accumulated features, fixes, and patched CVEs. Aaron Niskode-Dossett flagged two performance PRs as candidates, a FileStatus cache in the footer path and a faster RunLengthBitPackingHybridDecoder.
Encodings advanced on two tracks. Prateek Gaur posted a status update on ALP encoding for floating point data ahead of a formal vote he intends to start within days. The spec PR is joined by a C++ implementation in Arrow that has been through several review rounds and a Java implementation by Vinoo, and the cross-language story is strong: the Arrow C++ decoder reads Java-written data bit-exactly across roughly 1.56 million values and 18 fixtures, covering V1 and V2 pages and several real datasets, with zero mismatches. Remaining work covers extreme values needing 63 to 64 bits after frame-of-reference. Meanwhile Andrew McCormick kept doing the empirical legwork on the FIXED_SIZE_LIST logical type discussion, answering Antoine Pitrou's nullability question with fresh benchmarks. His numbers show a hint-aware reader on a plain LIST landing within noise of the non-compatible vector option, around 1,430 nanoseconds per row versus 2,600 for a full Dremel decode, and the optional outer array costs nothing when no nulls are present. That is the kind of measurement that turns a format argument into a format decision, and it pairs directly with the Iceberg vector type proposal above.
The encoding pipeline has more behind ALP. Prateek also floated a PFOR encoding discussion for patched frame-of-reference integer compression, an approach with a long history in column stores that has never had a Parquet spec home. Alkis Evlogimenos proposed making path_in_schema optional in the column metadata, trimming redundant bytes from footers that large tables repeat thousands of times. Aaron Niskode-Dossett suggested passing a known file length into HadoopInputFile.fromPath, which saves a round trip to object storage on every file open, the kind of micro-fix that adds up to real money at petabyte scale. And Julien convened the regular Parquet sync on Wednesday, July 15, where several of the threads above got live discussion time.
Two compatibility threads deserve attention from anyone running mixed reader fleets. Kevin Liu followed up on how older parquet-java readers handle VARIANT columns, turning a sync discussion into a detailed format issue and a parquet-java fix PR. The plan includes backporting the fix as patch releases so older readers can open files with new logical types without a forced upgrade, plus testing other implementations, since Go and fastparquet reportedly crash on unknown logical types. Micah Kornfield also continued the extended precision nanosecond timestamp proposal, arguing for FLBA<9> over FLBA<8> because BigQuery and Trino are already moving toward picoseconds, and one width handles nanoseconds through picoseconds with the same code.
Apache DataFusion
DataFusion ran a clean release week across three subprojects. Matt Butrovich proposed DataFusion 54.1.0 RC1 with the changelog and verification steps in hand. Andy Grove called the vote on Ballista 54.0.0 RC2 after RC1 failed, and the vote drew verification from Andrew Lamb, Marko Milenković, Martin Grigorov, and L. C. Hsieh before passing. Matt also shepherded Comet 0.17.1 RC1 through its vote to a passing result, keeping the Spark-accelerator branch of the family current alongside the core engine and the distributed scheduler.
The community also grew. Andrew Lamb announced Adam Gutglick as a new DataFusion committer, and the congratulations thread filled quickly with notes from Kumar Ujjawal, Jeffrey Vo, Matt Butrovich, and Martin Grigorov. Committer announcements are easy to skim past, but they are the truest health metric an open source project has.
The shape of the release week says something about how the DataFusion family now operates. The core engine, the Ballista distributed scheduler, and the Comet Spark accelerator each cut releases on their own cadence while tracking the same 54.x line, so downstream users get a coherent version story across very different deployment models. A failed RC1 followed by a clean RC2 within days is also a sign of healthy release muscle: problems get caught by verification, not by users. With DataFusion increasingly serving as the query engine inside other lakehouse tools, that discipline pays dividends far outside the project's own repositories.
Apache Ossie
Ossie, the young semantic interchange project, spent the week doing the unglamorous work that decides whether a project scales. Yong Zheng, fresh to the codebase, flagged inconsistent module management across the converters: two converters use uv, one uses modern Python packaging, one uses a legacy requirements.txt, and two use Maven on different JDKs. He volunteered to standardize them so the repository stops feeling like six projects owned by six companies, and JB, Emil Sadek, Aniket Kulkarni, and Khushboo Bhatia joined the thread. Yong followed with a proposal to standardize converter naming on an apache-ossie-xxxxx pattern and to introduce a shared base converter abstraction, since only three of five converters follow the same file structure today.
On the spec side, Will Pugh shared the foundational semantics document from the expression language group, asking for general feedback and proposing to build a reference implementation in parallel, on the theory that evaluating semantics is easier with running code. Level-of-detail calculations, filter exclusion, and fine-grained join specifications are deliberately deferred to a later pass. And a GitHub discussion surfaced on the list asking how Ossie relates to FIBO and the financial services semantic stack, reading FIBO as a reference ontology layer and Ossie as the interchange layer that maps to it. The question lands at the right moment, given the Polaris semantic model hosting debate above. The ecosystem is deciding, in real time, which layer owns which promise.
Why does converter housekeeping deserve newsletter space? Because Ossie is a specification project, and a spec lives or dies on its converters. If exporting a dbt project, a Snowflake semantic view, or a GoodData workspace into Ossie feels inconsistent, adoption stalls no matter how elegant the core model is. A shared base converter class and a uniform packaging story lower the cost of writing converter number seven, and converter number seven is how a new tool joins the ecosystem. Yong volunteering to do this work in his first weeks on the project is exactly the kind of contribution that turns an incubating spec into infrastructure. New contributor Dragos Crintea also introduced himself to the developer community this week, and Will Pugh posted general repository guidelines to keep the growing contributor base aligned.
Cross-Project Themes
Three threads of connective tissue stood out this week. First, infrastructure as code went from wish to reality across the stack: Iceberg voted on its first Terraform provider release while Polaris reached lazy consensus on creating its own provider repository. Declarative catalog and table management is becoming table stakes, and both communities are honoring the registry naming rules rather than fighting them.
Second, the AI workload is reshaping formats from both ends. Iceberg debated a native vector type, Parquet benchmarked fixed-size lists to store those vectors well, Iceberg considered Copilot for code review, and Polaris debated how to host Ossie semantic models for AI and BI consumption. The table format, the file format, the catalog, and the semantic layer are each answering the same question at their own layer: what does an agent or an embedding pipeline need from open data infrastructure?
Third, correctness culture is compounding. Shared conformance fixtures in Iceberg modeled on parquet-testing, a VARIANT forward compatibility fix with backports in Parquet, spec language tightening on manifest uniqueness, and integrity validation actions all point the same direction. As implementations multiply across Java, Rust, Python, Go, and C++, the projects are investing in shared answer keys instead of trusting each implementation to grade itself.
A fourth pattern hides in plain sight: governance maturity. Parquet is writing down its versioning rules as an RFC instead of relying on tribal knowledge. Polaris is debating whether review norms should be documented rules or reviewer discretion. Iceberg is defining criteria for admitting new file formats before evaluating any specific one. DataFusion promoted a committer. These are the habits of projects planning to be around in a decade, and the fact that all four surfaced in one week suggests the lakehouse stack is entering its institutional phase. Institutional does not mean slow. The same week produced five release candidates and a passed format vote. It means the projects are building the decision-making machinery that lets them move fast without breaking the ecosystems that depend on them, and that machinery is the least visible, most valuable output of this community.
Looking Ahead
The Polaris 503 vote closes Sunday, July 19, and the Polaris community sync on July 23 takes up the tag spec. The same day, Iceberg contributors gather in person in Austin. Watch for the Parquet ALP encoding vote to open, for Fokko to kick off the Parquet 1.18.0 release process, and for results on Iceberg Rust 0.10.0 RC4 and the Terraform provider RC1. The equality deletes deprecation thread will keep growing, and the answers there will define a good chunk of what Iceberg V4 becomes.
Further out, keep an eye on three slow burns. The Iceberg collation discussion has to reconcile cross-engine consistency with ICU upgrade freedom, and whatever it decides will echo in every engine that sorts strings. The Polaris persistence redesign will take months, and the SPI shape it lands on determines how hard NoSQL backends are to build. And the Parquet versioning RFC, once merged, becomes the template other format projects copy when they outgrow informal release habits. None of these resolves next week. All of them reward following the threads as they develop, and the permalinks above will take you straight to the source.
If this is your first issue, a note on method: everything above links to the public Apache dev list archives, and every claim traces to a thread you can read yourself. The dev lists are where the real decisions happen, before the blog posts and the conference talks. Subscribing to even one of them changes how you understand this ecosystem.
Resources & Further Learning
Get Started with Dremio
- Try Dremio Free: Build your lakehouse on Iceberg with a free trial
- Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow: Learn how Dremio brings the open lakehouse stack together
Free Downloads
- Apache Iceberg: The Definitive Guide: O'Reilly book, free download
- Apache Polaris: The Definitive Guide: O'Reilly book, free download
Books by Alex Merced
Top comments (0)