This was a week where the open lakehouse stack spent most of its energy on contracts. Not legal contracts, but the promises that formats, catalogs, and clients make to each other. Iceberg debated whether equality deletes belong in V4 and whether incremental scan semantics belong in the spec at all. Parquet voted a new floating point encoding into the format and argued about who owns the Thrift file that defines everything. Polaris tried to ship 1.7.0, found a licensing gap in 44 staged jars, and pulled the release candidate. Arrow wrestled with what the nullable flag actually means. Across all of it, a pattern showed up again and again: the community is done with informal conventions and wants written guarantees.
Here is what happened, project by project.
Apache Iceberg
The week opened with a personnel note that matters more than personnel notes usually do. Steven Wu announced that Maximilian Michels joined the project as a committer, and the thread ran to 30 messages of congratulations from across the contributor base. Iceberg has grown into a project where the streaming and Flink side of the ecosystem carries real weight, and adding committers who live in that part of the codebase keeps the review load from concentrating on a handful of people. The size of the congratulations thread is its own signal about how many active humans are now paying attention to this list.
The most consequential technical debate of the week was about deletes. Huaxin Gao pushed forward on the proposal to deprecate equality deletes in Iceberg V4, and the thread drew 15 messages including strong support from Ryan Blue. Equality deletes are the mechanism that lets a streaming writer say "delete every row where id equals 42" without knowing which file that row lives in. They make writes cheap and reads expensive. Every scan has to carry the delete predicates forward and apply them against candidate files, and scan planning cannot prune as aggressively because the planner does not know which files a given equality delete actually touches.
Xiening Dai raised the concern that killing equality deletes shifts a burden onto writers that still have to support fast updates and deletes. Gao reframed the tradeoff. The work does not disappear, it moves. Instead of every reader paying the cost forever, the writer pays it once by maintaining an index that resolves the predicate to positions at write time. Blue backed the position directly and argued that V4 should not allow writing equality deletes at all, pointing at the Flink work as evidence that maintaining such an index is practical. He followed up with a note on migration: existing tables that already contain equality deletes keep working after an upgrade, and the win concentrates in scan planning where the current design forces conservative behavior.
This is the kind of decision that defines a format version. V2 gave the ecosystem row-level deletes. V3 gave it deletion vectors and new types. V4 looks like the version where the community trims the surface area that made earlier versions hard to implement correctly across engines. If you maintain a writer, this thread is the one to read this week.
Right behind it, Prashant Singh called a vote to align the REST catalog OpenAPI expression schemas with the Iceberg expression specification. The vote drew 17 messages and passed. The problem it solves is unglamorous and important. The REST spec described expressions one way and the table spec described them another way, which meant client authors had to guess which definition an implementation actually followed. Two documents describing one concept is a bug that produces silent incompatibility rather than loud errors, and those are the worst kind.
Alexandre Dutra also brought the vote to formalize remote signing configuration in the REST spec to a close, collecting a binding +1 from Russell Spitzer along with several non-binding votes. Remote signing is how a catalog hands a client short-lived, narrowly scoped access to object storage without shipping long-lived cloud credentials. It has been in use for a while, configured through conventions that varied by implementation. Writing it into the spec turns a working practice into a guarantee, which is exactly what enterprise security teams ask for when they audit a lakehouse deployment.
Alexander Bailey opened a related question about written guarantees: should incremental append scan semantics be part of the spec? Incremental scans let a downstream job read only what changed since a given snapshot, which is the backbone of most CDC and streaming ingestion patterns built on Iceberg. Xiening Dai pointed out that the current REST incremental scan API returns a set of files, which restricts it to the append-only case, while a full incremental scan needs to express the delta between two snapshots including deletes and updates. Ryan Blue replied that V4 already has planned changes that tighten change detection requirements across versions, so the spec will support all incremental scan types instead of relying on conventions that vary by implementation. Bailey asked the obvious follow-up question, which is where to follow that work and which community sync to attend. That question comes up often enough on this list that a better answer than "ask on dev@" is overdue.
Format convergence showed up in a thread from Nitya Kumar Sharma, who proposed a file data type for Iceberg V4 now that the Parquet FILE logical type has merged in parquet-format PR 585. The idea is to give Iceberg a first class way to reference a blob, an image, or a document that lives outside the columnar data, with the metadata about that reference stored in the table. Russell Spitzer bumped the thread out of spam and noted that a prior proposal already exists from Talat and collaborators. Daniel Weeks agreed and asked the community to consolidate around updating the original FileRef proposal instead of starting a parallel effort. Sharma agreed to wait for the original authors to return from vacation and comment on the existing proposal. This is good project hygiene. Two competing proposals for the same feature produce a worse outcome than one proposal with two sets of authors.
On the release side, Shawn Chang called a vote for Apache Iceberg Rust 0.10.1 RC1, and verification came in from Kevin Liu with a binding +1 plus non-binding checks from Xin Huang on Linux with rustc 1.94.0 and L. C. Hsieh on macOS aarch64, who reported 1839 tests passing. The patch release exists because of a real incident, which Danny Jones documented on the list. The pyiceberg-core release artifacts for 0.10.0 were built from the main branch rather than the 0.10.0 tag, which means the published package contained code that was never part of the release. Vova Kot spotted it, Kevin Liu yanked the package from PyPI, and Renjie Liu took responsibility for the manual publish that caused it.
The way the community handled this is worth calling out. Nobody got piled on. The thread moved straight from acknowledgment to the question of how to prevent a repeat, with Matt Butrovich thanking Jones for starting that conversation. Manual release steps are a standing hazard in every Apache project, and the fix is almost always automation plus a verification step that compares published artifacts against the signed tag. Expect a follow-up proposal on that front.
The Terraform provider had its own release friction. Matt Topol proposed v0.1.0 RC1 for the Apache Iceberg Terraform provider, and Kevin Liu found a schema validation issue during testing that affected table creation and had the potential to write incorrect metadata or leave table state wrong. Topol said he will fix the issues and cut a new candidate. Rich Bowen also weighed in on the thread. Catching a metadata correctness bug in a release candidate is the system working as designed, and it is a good argument for the Apache voting process against people who think it slows things down.
Performance work drew attention too. Gianluca Graziadei opened a review request for Hilbert curve clustering in rewrite_data_files, offering it as an alternative spatial clustering strategy alongside the existing Z-order implementation. Tanmay Rauth reviewed it, praised the approach for reusing existing Z-order byte encodings to keep the change small, and asked for end-to-end numbers. Graziadei came back with two things: a citation to the Moon, Jagadish, Faloutsos and Saltz analysis in IEEE TKDE showing that Hilbert curves have better clustering locality than Z-order on analytical grounds, and a full comparison report with headline numbers. For teams doing multi-dimensional filtering on large tables, the difference between Z-order and Hilbert ordering shows up directly in how many files a query has to open. This is the kind of contribution that pays for itself the first week it ships.
Several smaller threads rounded out the week. There was discussion of migrating Iceberg to Jackson 3, which matters because Jackson version conflicts are one of the most common dependency headaches in JVM data stacks. Contributors discussed integrating EagerInputFile into the manifest reader, a change aimed at cutting the number of round trips during scan planning. There was a vote on labels in the IRC read path, a proposal to add the variant type to the REST catalog spec, a request for table-level filtering in MetricsReporter, a review request for a memory leak fix on V3 Spark, and continued work on column update metadata representation. There was also a dedicated sync scheduled for Iceberg index support, which is a topic worth watching closely given how much of the V4 conversation touches scan planning cost.
One recurring theme that is not technical at all: three separate threads this week were people trying to join the Iceberg Slack workspace because the public invite link was broken. That is three people who cared enough to email a developer mailing list. The number who gave up quietly is larger. A broken front door costs a project more contributors than most people realize.
Apache Polaris
Polaris had the most eventful release week of any project on this list, and the story has a good ending even though it started with a failed vote.
Jean-Baptiste Onofré called a vote to release Apache Polaris 1.7.0 rc0. Yufei Gu voted -1 with a binding vote after inspecting all 269 staged jars in the Maven repository and finding that 44 of them lacked both META-INF/LICENSE and META-INF/NOTICE. That is a release blocking issue at the Apache Software Foundation, and it does not matter how long the problem has existed. Onofré acknowledged the catch, noted the issue predates 1.5.0 and goes back to incubation, and moved directly to a fix. He opened a pull request to correct both the jars published to Maven and the version string in the Python package, planned a backport to the 1.7.x branch, and cancelled the vote.
The lead-up to the vote was itself a good example of release management. In the thread on preparing 1.7.0, Gu asked Onofré to include a pull request that fixes a bug where dropping a valid idempotency key leads to table corruption. Onofré reviewed it, proposed a simple rule for the cut, and stuck to it: if the fix merges before Wednesday it lands in 1.7.0, if it does not, it lands in 1.8.0. Gu argued the fix was ready and that a follow-up can make the window size configurable. Table corruption bugs deserve exactly that level of urgency, and a time-boxed decision rule keeps the release from drifting while people negotiate.
The longest running technical debate in Polaris this week was about a connection pool. Yufei Gu summarized the state of the Polaris-managed JDBC datasource discussion as a choice between Hikari and Agroal, with the thread running to 21 messages. Romain Manni-Bucau asked the right question first, which is whether anyone had written down the criteria for the decision. Robert Stupp went further and argued the thread had not established consensus that Quarkus-managed data sources should be replaced at all. He walked back to the original proof of concept and separated the two things it addressed, one of which is loading optional JDBC drivers at runtime.
That distinction matters. Loading drivers dynamically is a packaging problem. Choosing a pool is a runtime behavior problem. Solving the first by rewriting the second is how projects accumulate accidental complexity. Dmitri Bourlatchkov asked Gu to explain the reasoning behind picking Hikari in the original pull request, which is the kind of question that turns a preference argument into a design discussion.
Security semantics got serious attention. In the thread on forwarding user-defined principal properties in PolarisPrincipal, Gu asked whether user-defined and system-managed attributes should live in separate namespaces to preserve provenance. Stupp agreed that provenance sits at the center of the question and pushed to settle security semantics before exposing these attributes to authorizers. His specific concern was that exposing the complete principal entity through a public attribute key invites authorizer implementations to depend on it, and once they do, the project owns that surface forever. Alexandre Dutra argued the resolver has legitimate need for the entity. Stupp narrowed his objection to placement rather than existence: the concern is that the key is declared on the public-facing type.
This is a good argument to read if you build authorization systems. The failure mode is not that someone reads a field they should not read. It is that a convenient field becomes a load-bearing part of third-party code, and then it cannot change.
Persistence correctness got its own thread. Robert Stupp picked up the discussion on consistent multi-object changes in Polaris persistence, noting Gu's clarification about where the atomicity guarantee lives. If atomicity is a property of the lower-level BasePersistence contract rather than the general metastore manager contract, callers cannot tell whether the state they read for validation and authorization is consistent. Prithvi S agreed the persistence layer needs one well-defined contract for multi-object consistency rather than a series of operation-specific patches, and clarified where their pull request sits in that picture. Onofré backed Stupp's framing. There is a related thread on atomic multi-entity and grant commits covering partial-commit windows during grant, createCatalog, and drop operations, which is the same problem seen from the authorization side.
Dependency reduction came up in the discussion about deprecating TreeMapMetaStore and friends. Gu asked for the specific security concern behind removing H2, pointing out that any library can have vulnerabilities and asking for data showing H2 has more than others. Bourlatchkov replied that the data is not the point. The question is whether a driver used only for non-production getting-started scenarios earns a place in the production dependency graph. Both positions have merit, and the resolution probably involves shipping the getting-started path as a separate artifact rather than arguing about any single library's track record.
Adam Szita moved Iceberg table encryption support from proposal to code with a pull request. Gu reviewed it, said it heads in the right direction, and asked for a split between spec changes and implementation. Stupp cross-linked a related reply about how two encryption pull requests fit together against Iceberg's two catalog security requirements. There is also a separate discussion about decrypt-only access for legacy AWS KMS keys, which is the sort of migration path detail that decides whether an encryption feature is adoptable by an existing deployment or only by greenfield ones.
Agent-facing work showed up in the thread on authentication modes in polaris-tools. Gu drew the distinction that organizes the whole problem: a local MCP server and a standalone shared MCP service are different deployment models with different security requirements. For a local single-user process, using the user's own Polaris token is enough because there is no separate service identity. Stupp agreed the deployment model is the first fork in the decision tree. Anyone wiring an AI agent to a catalog should read this thread before designing their auth story, because getting this wrong produces a service that can act as any user.
Onofré also handled community logistics. He created a public Google Calendar for Polaris and proposed a new meeting structure of 90 minutes every three weeks with timeboxed topics. Dutra agreed with the cadence but pushed back on a 10 minute timebox as too short. Onofré moved it to 20 minutes on the spot and Gu agreed. Small thread, real improvement. He also pushed the Polaris Directories proposal forward with a plan to rebase the pull request, incorporate comments, and schedule a dedicated meeting, with Bourlatchkov and Gu both supporting a live discussion.
Other Polaris threads worth a look: HTTP status codes for CommitStateUnknownException from federated catalogs, the result of the vote on returning 503 for concurrent modification during table and view rename, support for external principals, making the relational JDBC schema name configurable, adding Open Sharing APIs, the semantic model REST API payload representation, Polaris SPI principles, whether the Polaris Console belongs in the main repository, and the August 2026 board report.
One operational note from that authentication thread: a message with a July 20 date header sat in the moderation queue for three days before reaching the list. Bourlatchkov flagged it. Moderation delays make threads look abandoned when they are not, and contributors read silence as rejection.
Apache Parquet
Parquet landed a real format change this week. Prateek Gaur announced that the vote to add ALP encoding to the Parquet format passed with 11 +1 votes, 7 of them binding. ALP stands for Adaptive Lossless floating-Point. It compresses double and float columns by finding a decimal representation that maps the values to integers, then encoding those integers with existing integer techniques, and falling back to a second scheme for values that do not fit the pattern.
Floating point columns have been the weak spot in columnar compression for years. Sensor readings, prices, model scores, and embedding components all store as doubles, and general purpose compressors do poorly on them because the bit patterns look close to random. ALP takes advantage of the fact that most real-world doubles are actually decimals with limited precision that were converted to binary floating point somewhere upstream. In the vote thread, Antoine Pitrou gave a binding +1 and noted the remaining items were wording and document organization rather than substance. Micah Kornfield and Andrew Lamb also voted +1 binding, with Lamb calling the spec pleasant to read and flagging some subtlety worth follow-up. Julien Le Dem voted +1 and credited Gaur's leadership. Gaur thanked Dhirhan and Russell Spitzer for their contributions.
That vote almost did not happen on schedule, for a reason that has nothing to do with engineering. Gaur reported that at least three people found the voting email in their spam folder and asked what the protocol is. Kornfield said he had seen other votes land normally and hoped people will check. Spitzer said his own +1 was partly an attempt to pull the thread out of spam filters, and reported that two other votes went to spam for him. Le Dem said the same and noted a general uptick in Apache list mail being filtered. This is a real threat to a governance model built on email. A vote that nobody sees is a vote that fails by default.
The other big Parquet discussion was about ownership of the format definition. Divjot Arora proposed inlining parquet.thrift into parquet-java, and the thread ran to 11 messages. The current setup has parquet-java depending on a pinned version of parquet-format and pulling the Thrift file from it. That means you cannot prototype a Java implementation of a format change until a new parquet-format jar ships, which turns every experiment into a release-blocking dependency chain.
Andrew Lamb backed the proposal from experience, noting that the arrow-rs Parquet implementation keeps its own copy of parquet.thrift and has never had a problem with it. Gang Wu supported it in stronger terms, describing the current workflow as painful because continuous integration always fails and pull requests cannot merge before a new format jar is released. Pitrou weighed in as well. The tradeoff is real. A single canonical Thrift file guarantees that implementations agree. Copies drift. But copies that drift produce visible test failures, while a release-blocking dependency chain produces contributors who give up before writing the test.
Pitrou also raised a naming question inside the 1.18.0 release vote thread: whether future releases should be named Apache Parquet Java rather than Apache Parquet. That is more than cosmetics now that Rust, C++, and Go implementations are widely used and the format itself versions separately from any implementation. Calling the Java release "Parquet 1.18.0" implies it is the format, and it is not.
The 1.18.0 RC1 vote itself hit turbulence. Gábor Szádovszky reported that the build fails on the release candidate tag and the tarball with format violations, that running spotless fixes it, and that it was strange the problem got through continuous integration. Fokko Driesprong diagnosed it as a Java version issue, noting he saw the same behavior on JDK 17 and that JDK 11 worked, with the next release moving to JDK 17 and later. Peter Toth voted +1 non-binding. Build reproducibility across JDK versions keeps biting Java projects, and a formatter that behaves differently by JDK is a nasty variant of the problem because it produces a diff rather than an error.
The FIXED_SIZE_LIST discussion took an unexpected turn. Gunnar Morling reported implementing a fixed-length list fast path in Hardwood that detects effectively fixed-length lists by scanning the encoded definition and repetition level streams, then bypasses the general path. Will Edwards called the result worth reflecting on beyond a simple optimization and questioned what remains of the performance case for a new logical type. Alkis Evlogimenos said the result matches work done internally on Photon and argued the fixed size list discussion shifts away from physical performance toward something else. Morling agreed the finding is useful and said he sees more work ahead.
That exchange is a model for how format decisions should go. Someone proposed a new type on performance grounds. Someone else showed you can get most of the benefit through smarter decoding of the existing representation. The conversation then moved to whether the type earns its place on semantic grounds instead. Formats accumulate types easily and shed them never, so this level of scrutiny is correct. Related work continues on a VECTOR repetition level for fixed-size-list serialization.
Julien Le Dem bumped the FSST spec design thread to ask about next steps, specifically comparing the original 8-bit codes against variations with larger code sizes. Arnav Balyan reported general consensus on the FSST spec and credited Kornfield, Gang Wu, Lamb, Pitrou, and others for feedback. FSST is a string compression scheme that builds a symbol table of common substrings, and pairing it with ALP gives Parquet strong coverage on the two column types that general compressors handle worst.
Burak Yavuz posted updates on the File logical type after its vote passed, covering how to reason about compression for data stored inside the new type and a rename from path to uri based on feedback from Rok and Pitrou. That rename is small and correct. A path implies a filesystem. A URI does not.
Rounding out the project, Aaron Niskode-Dossett and Ismaël Mejía discussed whether Parquet should publish a test helpers artifact, with Mejía reporting a successful downstream test into Apache Spark. Steve Loughran gave a mixed assessment from experience on other projects: sharing testing tools helps consumers, and it also commits you to keeping them stable across changes like a JUnit 5 migration. There was also continued discussion on extended precision nanosecond timestamps, and the Parquet sync met on Wednesday July 29.
Apache Arrow
Arrow's headline discussion asked a question that sounds simple and is not. Antoine Pitrou opened a thread on the semantics of non-nullable fields with non-trivial types, prompted by an issue and pull request in the Arrow repository. What does the nullable flag in the IPC format actually mean when the field is a struct, a list, or a map? Does it describe the field itself, the children, or both? The thread ran to nine messages.
Raúl Cumplido linked back to an earlier mailing list thread on one of the cases, which tells you this question has been open for months. Weston Pace said he finds the nullability flag confusing in general and raised the deeper question: is an Arrow schema meant to describe the data in this specific batch, or to state a contract about all data that will ever flow through this stream? Those are different things. A batch-level description is an observation. A stream-level contract is a promise that downstream code can optimize against. David Lee, who opened the original issue, argued for an official format change and drew the distinction between a nullable array and a struct that contains nulls.
This is the same theme running through Iceberg and Polaris this week. A flag that has worked by convention for years turns out to mean different things to different implementations, and the fix is to write down which meaning is normative. Arrow also has a related open question in the thread on the variant extension spec being inconsistent with the Parquet shredding spec, which is a cross-project version of exactly the same failure.
On releases, David Li shipped ADBC 24 through a two-candidate cycle. RC1 covered 55 resolved issues and drew a binding +1 from Raúl Cumplido on Debian 14. Edgar Ramírez Mondragón noticed missing manylinux wheels in the nightly repository, and Bryce Mecum redirected that to the issue tracker as a non-blocking concern. Li then cut RC2. Matt Topol found a small non-blocking problem and filed a fix. Mecum verified on macOS 26 aarch64 and confirmed that the RC2 JNI library no longer dynamically links the driver manager. Ian Cook verified on macOS Tahoe 26.3.1 and confirmed the fix. The vote passed with four binding +1 votes from Mecum, Cook, Topol, and Cumplido, and Li worked through the release checklist and announced ADBC 24.
ADBC deserves more attention than it gets in lakehouse conversations. It gives you a database connectivity API that speaks Arrow natively, so result sets move as columnar batches instead of being converted row by row through a JDBC or ODBC layer. For any workload that pulls large result sets into Python or Rust for processing, that difference is the whole performance story.
Arrow Rust 58.4.0 also cleared its vote. Kosta Tarasov verified RC3 on Fedora 44 x86_64 with a non-binding +1, Adam Reeve added a binding +1 on the same platform, and Andrew Lamb, who proposed the release, posted the result. Arrow Go 18.7.0 was released after its own vote passed.
Demetrius Albuquerque proposed raising the minimum Swift version for apache/arrow-swift from 5.10 to 6.0. Sutou Kouhei did not object and asked for data on how many Swift users are still on 5.10. That is the right response to every minimum-version bump: not "no" and not "sure," but "who breaks?" The Arrow community meeting was scheduled for July 29 at 16:00 UTC.
Apache DataFusion
DataFusion had a quiet week on dev@, with the notable item being Matt Butrovich's announcement that the vote for Apache DataFusion 54.1.0 RC1 passed with six binding +1 votes from Butrovich, Andrew Lamb, Andy Grove, L. C. Hsieh, Marko Milenković, and Wang Xudong, plus one non-binding +1 from Martin Grigorov. No zero or negative votes.
A quiet dev list does not mean a quiet project. DataFusion does most of its design work in GitHub issues and discussions, and the mailing list carries mostly release traffic. What the 54.1.0 patch release signals is a project on a fast, predictable cadence, which is exactly what downstream projects need. Iceberg Rust depends on DataFusion for query execution, which makes DataFusion's release rhythm part of the Rust lakehouse story rather than a separate concern.
Apache Ossie (incubating)
Ossie is the newest name on this list and the one most people have not read about yet. It is an incubating project building a shared, vendor-neutral specification for semantic and ontology metadata, expressed in YAML, with converters that map it into other systems. This week it acted like a project preparing to be real.
Jean-Baptiste Onofré proposed beginning work on the first releases, targeting late August or early September. Yong Zheng responded with concerns about outstanding converter issues and about the proposed version numbering. Onofré replied that the first community meeting already discussed versioning and the group leans toward starting at 0.3.0-incubating rather than 1.0.0. Starting below 1.0 is the right call for a spec that is still absorbing feedback from multiple working groups. A 1.0 label sets expectations about stability that an incubating project should not make.
Markus Weimer asked a deceptively small question: is there a canonical file suffix for Ossie files? He had seen raw .yaml in the wild. Onofré said there is no canonical suffix today and supported introducing a .ossie suffix while keeping the YAML format. Sahil W supported keeping .yaml and liked the .ossie idea for discoverability. Weimer landed on the practical answer: use .ossie.yaml when Ossie files live alongside other files in the same folder, which keeps every YAML-aware editor and linter working while making the file's role obvious. That is a good outcome, and it took four messages.
Sahil W also proposed unified Python linting and formatting across the project's Python code and offered to own the implementation across several small pull requests. Yong Zheng gave a +1, said he had started similar work, and listed the problems he hit, including questions about ownership of formatting rules in a repository with multiple converters. Sahil pointed out that ruff supports hierarchical configuration natively, so a shared top-level config acts as a baseline that individual converters can extend. Quigley Malcolm backed standardization and endorsed starting small.
Housekeeping continued elsewhere. Yong Zheng proposed cleaning up two unused directories, Onofré confirmed they are legacy and volunteered to remove them, and Will Pugh shared a document to align the community on directory structure. Zheng also raised the Java version question, noting the Polaris converter still targets Java 11, which reached end of life a while ago. Malcolm supported the immediate alignment fix and asked for a sanity check on jumping further. Onofré supported going straight to Java 21 and explained Java 11 was set as the minimum required version at the time even though the build used JDK 17. Zheng agreed Java 21 has wider adoption than Java 25.
The governance thread of the week came from Ramya Priya of Tellius, who asked about the process for a new organization to join a working group. Quigley Malcolm welcomed her, said the questions indicate the contributing guide needs updates, and made the framing point clearly: at the Apache Software Foundation people participate as individuals, not as company representatives. Onofré echoed it and confirmed the project is vendor-neutral, then addressed what "active participant" means in practice.
Anyone who works at a vendor and contributes to open source should internalize that answer. Your employer does not join an Apache project. You do.
Other Ossie activity included notes from the Ontology working group sync on July 23 shared by Ankit Tandon, a proposal on representing reified relationships as entity types, discussion of extended metadata fields for fields and metrics, community workflows and compliance automation, a scaffold for repositories that consume the spec, a proposal to add Flink SQL support, and an independent .NET viewer with a two-way fact-based modelling converter looking for collaborators.
Cross-Project Themes
Three patterns connect the six lists this week.
Written contracts are replacing working conventions. Iceberg voted to align REST expressions with the table spec and to formalize remote signing, and it opened a debate about whether incremental scan semantics belong in the spec at all. Arrow asked what the nullable flag normatively means. Parquet renamed path to uri and pushed for clearer language in the ALP spec. Polaris argued about which layer owns the atomicity guarantee. These look like separate discussions. They are the same discussion. Each project has reached the point where multiple independent implementations exist, and the cost of an underspecified detail has flipped from "nobody notices" to "two engines disagree in production."
The mechanism behind that flip is worth naming. When one implementation dominates, the implementation is the spec, and ambiguity in the document costs nothing. When four implementations exist across four languages, every ambiguity becomes a compatibility bug that surfaces at the worst possible moment. Iceberg has Java, Rust, Python, Go, and C++ implementations. Parquet has at least as many. Arrow was multi-language from birth. The specs are catching up to a reality the code created.
Formats are converging around composite and nested data. Parquet merged a FILE logical type, and Iceberg immediately started designing a matching file data type for V4. Parquet is working through FIXED_SIZE_LIST and a VECTOR repetition level. Arrow flagged that its variant extension spec is inconsistent with the Parquet shredding spec, while Iceberg discussed adding variant to the REST catalog spec. ALP and FSST together attack the two column types that general compression handles worst, floating point and strings.
Read those together and you can see the destination. The stack is preparing for tables where a single row holds structured columns, semi-structured documents, a vector of floats, and a reference to a blob sitting in object storage, all queryable through one SQL interface with real pruning and real compression. That is the shape of data that AI applications produce and consume. The format work happening this week is what makes it possible to store that data without a separate vector database, a separate document store, and a separate blob index that nobody keeps in sync.
Catalogs are becoming security products. Polaris spent the week on principal property provenance, external principals, table encryption, decrypt-only access for legacy KMS keys, atomic grant commits, and authentication modes for its MCP server. Iceberg formalized remote signing in the REST spec. Both projects are working the same problem from opposite ends: the catalog is the only component that sees every request, so it is the only place where access control can be enforced consistently across engines.
The MCP thread in Polaris makes the stakes concrete. Once AI agents query catalogs directly, the question of whose identity an agent acts under stops being academic. Gu's distinction between a local single-user MCP process and a shared MCP service is the first fork in that design, and getting it wrong produces a service that quietly acts as any user who ever touched it. Every organization deploying agents against a lakehouse will confront this within the next year.
A fourth, smaller pattern: release engineering hygiene took a beating this week and the community responded well every time. Polaris found 44 jars missing license files and pulled the vote. Iceberg yanked a Python package built from the wrong branch. Parquet found a formatter that behaves differently across JDK versions. The Terraform provider had a schema bug caught during verification. In every case the process worked, which is a reasonable argument for the verification steps that people complain about when nothing is wrong.
And a fifth that is not technical at all: Parquet votes landed in spam folders, Iceberg contributors failed to join Slack through three separate threads, and a Polaris message sat in moderation for three days. Open source governance runs on email and chat working reliably. When they do not, participation drops in ways that never show up in a metrics dashboard.
Looking Ahead
Watch the Iceberg equality deletes thread. Ryan Blue's support gives the deprecation real momentum, and the follow-up questions about upgrade behavior for existing tables will shape how much work this creates for anyone running Flink-based ingestion. Alongside it, the V4 change detection work that Blue referenced in the incremental scan thread is the piece that determines whether incremental reads become a spec guarantee or stay a convention. The dedicated index support sync is the third leg of that story.
Polaris 1.7.0 should reach a vote quickly now that the license and notice fix is in flight, and the Polaris Directories meeting that Onofré promised to schedule is worth attending if you care about how the catalog organizes objects. The JDBC datasource decision needs written criteria before it needs a winner.
Parquet will move ALP from an approved spec toward reference implementations, and the FSST spec looks close to the same milestone. The parquet.thrift inlining discussion has enough support that a formal proposal is likely soon. Watch for the Parquet Java naming question to come back as a separate thread.
Arrow's nullability discussion has not converged, and Weston Pace's question about whether a schema describes a batch or promises a contract is the fork that needs an answer before any format change makes sense. The Swift minimum version bump waits on usage data.
Ossie is heading toward first releases at 0.3.0-incubating in late August or early September, which will be the first time anyone outside the mailing list can install and evaluate the spec. If you work on semantic layers or metric definitions, that is the moment to start paying attention.
Resources & Further Learning
Get Started with Dremio
- Try Dremio Free: Build your lakehouse on Iceberg with a free trial
- Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow: Learn how Dremio brings the open lakehouse stack together
Free Downloads
- Apache Iceberg: The Definitive Guide: O'Reilly book, free download
- Apache Polaris: The Definitive Guide: O'Reilly book, free download
Books by Alex Merced
- Architecting an Apache Iceberg Lakehouse
- Enabling Agentic Analytics with Apache Iceberg and Dremio
- The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI
- The Book on Using Apache Iceberg with Python
Browse the full catalog of 50+ books at books.alexmerced.com.
Alex Merced, Data Lakehouse and AI Evangelist
Top comments (0)