DEV Community: Alex Merced

Apache Data Lakehouse Weekly: May 21-27, 2026

Alex Merced — Wed, 27 May 2026 14:50:46 +0000

The week after a major release tends to look quiet on a project's dev list. This one did not. With Iceberg 1.11.0 and 1.10.2 both out the door the week before, and Polaris 1.5.0 shipping right at the start of this window, you might have expected the lakehouse projects to take a breath. Instead the conversation shifted from "what are we shipping" to "what do we build on top of what we just shipped," and that turned out to be a busier and more interesting set of threads. Encryption moved from Iceberg core into the catalog layer. The REST spec picked up two new client-facing extensions. Arrow took a donation across the finish line and started arguing about whether a bot should review its pull requests. Parquet finally voted on a statistics change that had been circling the list for years. Taken together, the week was about the connective tissue of the lakehouse: the catalog, the protocol, the client contract, and the unglamorous governance work that keeps four independent projects interoperable.

Apache Iceberg

The single most consequential thread of the Iceberg week was procedural rather than technical. Ryan Blue posted the RESULT for the vote to add an unregister endpoint to the REST spec, which passed with 16 +1 votes, 9 of them binding, and no dissent. That is a strong mandate for what is a deceptively important capability. Until now the REST catalog has had no standard way to drop a table's registration without also deleting its data and metadata, which matters enormously for the migration and multi-catalog scenarios that Iceberg increasingly has to support. As tables move between catalogs, or as a catalog needs to hand off ownership of a table to another system, unregister is the clean primitive that makes that safe. The breadth of binding support tells you this was not controversial in substance, only in getting the wording precise enough to standardize.

Around that vote, a cluster of REST spec discussions showed the catalog protocol entering a more mature phase where the arguments are about extensibility and forward compatibility rather than core mechanics. Prashant Singh opened a discussion on adding an X-Iceberg-Client-Capabilities header to the REST spec, which came out of the Read Restrictions community sync on May 12. The idea is to give clients a standard way to advertise what they support so that servers can adapt their responses, which is exactly the kind of negotiation mechanism a protocol needs once it has many independent implementations that ship on different schedules. In a related vein, Alexandre Dutra summarized the outcomes from the catalog sync in a thread on passing arbitrary information to request signers, refining the language around how clients hand context to the components that sign storage requests. Both threads point at the same underlying reality: the REST catalog is now the integration surface that the whole ecosystem leans on, and the community is carefully building the seams that let it evolve without breaking clients.

Release work did not stop with 1.11.0. The non-Java implementations are all moving in parallel. Matt Topol opened the vote on Apache Iceberg Go v0.6.0 RC2, which drew nine participants and active testing, while Junwang Zhao started a discussion on releasing Iceberg C++ 0.3.0 and Alex Stephen kicked off planning for PyIceberg 0.12.0 by pointing contributors at the milestone to flag anything still missing. Three client libraries in three languages all cycling toward releases in the same week is a good illustration of how the project has decoupled its implementations so they can each move at their own pace rather than waiting on the Java reference.

The post-1.11.0 design conversation also got going. Steven Wu opened a discussion on Flink version support after Iceberg 1.11.0, working through how the project should manage the matrix of supported Flink versions going forward, a recurring maintenance question that every engine integration eventually has to confront. Stepan Stepanishchev proposed adding Flink SQL procedure support to Iceberg, which would let users invoke Iceberg maintenance and management operations through Flink's CALL statement the way they already can in Spark. Noritaka Sekiyama proposed adding an OpenTelemetry-based MetricsReporter to iceberg-core that would export ScanReport and CommitReport data to any OTLP-compatible backend, a genuinely useful piece of observability plumbing that drew seven participants and nine replies. Iceberg already ships metrics reporting interfaces, but standardizing on OpenTelemetry would let operators wire table-level telemetry into the same monitoring stack they use for everything else.

Two threads dealt with the practical grind of running a large open source project. Robert Thomson, writing on behalf of ASF infrastructure, raised Iceberg's consumption of the shared GitHub-hosted Actions runners, part of a foundation-wide effort to keep CI usage within the shared pool's limits. It drew eight participants quickly, because every active committer feels the pain of CI queueing. Max Konstantinov opened a discussion on sunsetting MkDocs for the project's versioned documentation, noting that the MkDocs project itself appears effectively abandoned with no new contributions in roughly 18 months, which makes it a risky foundation for the docs that the whole community depends on. These are not glamorous threads, but they are the kind of maintenance the project has to stay on top of to keep growing.

Apache Polaris

Polaris had the cleanest headline of the week. Jean-Baptiste Onofré announced the release of Apache Polaris 1.5.0, following the vote that passed with binding +1s from Robert Stupp, François Papon, Dmitri Bourlatchkov, Yong Zheng, and Onofré himself. The release headlines Apache Ranger support as an external authorizer, alongside CLI improvements and Helm chart work. Ranger integration is a meaningful step for enterprise adoption, because it lets organizations that already run Ranger for their broader data platform extend those same authorization policies to their Iceberg catalog rather than maintaining a separate access model. That this is the first release since the project's graduation discussions, and that it shipped on a clean RC0, says good things about where Polaris is in its maturity curve.

The more forward-looking work was about authorization and delegation, which are clearly the project's center of gravity right now. Yufei Gu posted updates on the Delegation Service design document, noting that he and Onofré will co-author it and that the pull request reflects the latest direction on the pull-versus-push modes question. Sung Yun followed up on the dedicated sync on Polaris authorization, proposing to fold earlier authorization discussion into the new authorization SPI work. The throughline is that Polaris is building a pluggable authorization architecture rather than hardcoding a single model, which is the right call for a catalog that has to serve organizations with wildly different security requirements. Ranger landing in 1.5.0 is the first external authorizer; the SPI work is what makes the second and third ones tractable.

Two threads showed Polaris reaching across project boundaries. Alexandre Dutra opened an Iceberg 1.11 feature branch retrospective, evaluating the experience of maintaining a feature/iceberg-1.11 branch to stay ahead of upcoming Iceberg enhancements and deciding what to do with it now that 1.11 has shipped. Adam Szita started a discussion on Iceberg table encryption support in Polaris, picking up directly from the base encryption implementation that landed in Iceberg 1.11. This is the most important cross-project signal of the week and worth dwelling on. Iceberg shipped KMS-based key wrapping and encrypted data, delete, manifest, and manifest-list files in 1.11, but encryption is only useful end to end if the catalog knows how to manage and hand out keys. The fact that Polaris opened this thread within days of the Iceberg release shows how tightly the catalog and the table format now move together. The encryption story does not work unless both halves cooperate, and the community is treating it that way.

There was also a healthy run of operational and integration discussion. Adnan Hemani followed up on an OpenLineage proposal for lineage tracking, Bill Bejeck floated a diagnostics shell prototype to answer simple operational questions like how many tables a bootstrapped Polaris instance is managing, and Dmitri Bourlatchkov pushed on the practical question of how to land generic table delegation in the Polaris SparkCatalog. Alexandre Dutra also proposed forbidding special characters in entity names that most cloud providers reject or discourage, the kind of guardrail that prevents a class of cryptic failures down the line. And in a thread that captures where the industry's head is at, Dennis Huo proposed an agentic eval meta-skill for extensibility and maintainability, exploring how the project should think about agentic development as a first-class tool rather than something contributors do off to the side.

Apache Arrow

Arrow's week was anchored by a donation reaching its conclusion. Sutou Kouhei posted the RESULT for the vote to donate the Apache Arrow Erlang implementation, which carried with four binding +1s from Sutou, Curt Hagenlocher, Matt Topol, and David Li. The next step is the IP clearance process and a vote on the incubator general list. Benjamin Philip, who has been shepherding the contribution, had earlier worked through the grant documents for the Erlang implementation, filling out the IP clearance template, the contributor license agreement, and the software grant. The Erlang library is built on bindings to the Rust implementation, which is itself a nice illustration of how Arrow's investment in a strong Rust core is now letting new language bindings come together faster than a from-scratch implementation ever could. Every new language Arrow speaks is another place its columnar format becomes the default interchange layer, and getting there by wrapping Rust rather than reimplementing C++ is the efficient path.

The release engineering that has become Arrow's signature continued without drama. Andrew Lamb posted RESULT threads for three Rust releases in close succession: arrow-rs 56.2.1, 57.3.1, and 58.3.0, each approved with five +1 votes. Shipping three point releases across three maintenance lines in a single stretch is the kind of cadence that signals a project with its release automation thoroughly sorted out. Rok Mihevc also closed the loop on the pyarrow-stubs donation, confirming that the software grant has been formally filed and published, which concludes that process and brings type stubs for PyArrow into the project proper.

The design discussions leaned toward type system and protocol extensions. Florian Hölzlwimmer proposed an arrow.range canonical extension type for bounded ranges, filling a gap in Arrow's type vocabulary. Tornike Gurgenidze opened two threads on the Flight and ADBC side: a partitioned bulk ingest API for ADBC that would mirror the existing ExecutePartitions and ReadPartition read-side primitives on the write side, and a proposal to add dialect-related SqlInfo codes to FlightSQL so clients have a standard way to learn what SQL features a backend supports. There was also continued work on a Flight SQL field to signal whether a prepared statement is an update, with Jean-Baptiste Onofré suggesting the vote be extended to give more people time to weigh in.

The thread that will resonate beyond Arrow was Sutou Kouhei's discussion on enabling automatic GitHub Copilot review on apache/arrow pull requests. His framing was pragmatic: the project does not have enough human review bandwidth, and a Copilot pass could catch trivial problems before a human reviewer ever looks. This is the same underlying tension that Iceberg has been working through with its AI contribution guidelines, just approached from the reviewer side rather than the contributor side. It is a question every large open source project is going to have to answer, and seeing Arrow debate it openly on the dev list, weighing the value against the noise, is exactly how these norms should get set.

There was also a low-level performance thread worth a mention for the systems-minded: Dan Mattheiss opened a discussion on AVX2 SBBF probe for parquet/bloom_filter.cc, noting that arrow-go already shipped SIMD bloom filter probes and proposing the C++ side catch up. It is a reminder that the cross-language consistency Arrow promises also means keeping performance optimizations roughly in step across implementations.

Apache Parquet

Parquet's headline was the close of a long-running saga. Gang Wu opened the vote to adopt the format change for PARQUET-2249, covering IEEE 754 total order and NaN-counts, drawing eight participants. This proposal has been circulating in various forms for a long time, and the discussion thread on adding nan_count to handle NaNs in statistics had a "bumping this one last time" quality to it before the vote finally opened. The substance matters more than the procedural relief. Floating-point NaN values break the assumptions that min/max statistics rely on, which means engines either produce wrong results when pruning row groups that contain NaNs or disable statistics-based pruning entirely on float columns to be safe. Standardizing a total order and a NaN count fixes that at the format level, so every engine can prune float columns correctly. It is the kind of fix that sounds narrow and is actually load-bearing for query performance on a very common data type.

The other major structural conversation was the footer. Jiayi Wang posted the kickoff for the Parquet Footer Working Group, setting up a dedicated forum to move the footer redesign forward more efficiently after it was discussed at the Parquet sync. The footer is where Parquet stores its metadata, and how it is structured determines how quickly a reader can open a file and figure out what is in it, which is increasingly a bottleneck as files get larger and workloads get more selective. Pierre Lacave contributed to the related discussion on an alternative to the FlatBuffer footer, a lightweight byte-offset index, sharing that a similar pattern is in use in a custom file format his team is migrating toward Parquet. Standing up a working group is a signal that the community sees footer evolution as a multi-release effort that deserves focused attention rather than ad hoc threads.

Release planning got going too. Fokko Driesprong opened a discussion on an Apache Parquet 1.18.0 release, noting that a lot of work has accumulated since the last major release and that it is overdue. Ismaël Mejía proposed bumping the minimum Java version for Parquet Java to 17, pointing out that Java 17 has been the baseline LTS since September 2021 and that holding the floor at 11 is increasingly costly. Mejía was also active on the performance front, sharing encoding and decoding hot-path optimizations and asking for code reviews on work he presented at the Parquet community sync. On the safety side, Steve Loughran circulated a pull request hardening the variant readers, noting that while a malformed 1KB file triggering a multi-gigabyte allocation is not strictly a security issue, it is close enough to be worth fixing. And the community recognized that contribution work with an announcement that Ed Seidl has accepted an invitation to become a committer. There was also continued interest in the geospatial story, with Dewey Dunnington noting in a thread on geography test files with statistics that he had added geography statistics writing to SedonaDB via arrow-rs, closing a gap that had been flagged when the geospatial types blog post came out.

Cross-Project Themes

Two patterns connect these four lists this week, and both say something about where the lakehouse stack is heading.

The first is that encryption has become a cross-project program rather than a single project's feature. Iceberg shipped the base table encryption implementation in 1.11, and within the same week Polaris opened a thread on how the catalog should support it. Encryption that protects data files but leaks key management to every client is not real protection, so the catalog has to be the trusted party that wraps, unwraps, and hands out keys under policy. You cannot reason about Iceberg encryption by reading the Iceberg list alone; the design only closes when you read the Polaris thread next to it. That is the lakehouse working as a coordinated platform, where a capability is split across the format and the catalog by design and the two communities build their halves in step.

The second is that every one of these projects is now wrestling, openly and on the record, with how AI fits into its development process. Arrow debated whether to let Copilot review pull requests. Polaris explored an agentic eval meta-skill as first-class project tooling. Iceberg has its AI contribution guidelines work, and the word "agentic" is showing up in the Polaris topic cloud. These are not the same question, but they rhyme. The community is deciding, in the open, what role AI tools should play in producing and reviewing the code that underpins the open data stack, and it is doing so transparently rather than letting individual contributors quietly make those choices alone. The decisions made over the next few months will set norms that stick for years, and it matters that they are being made on public dev lists where the whole community can see the reasoning.

There is a quieter third theme worth naming: protocol and format extensibility. Iceberg's client capabilities header, Arrow's FlightSQL dialect codes, and Parquet's footer working group are all the same instinct expressed in three places. Each project has reached the point where it has many independent implementations on different release schedules, and the central task is no longer adding features but building the negotiation and versioning seams that let those implementations evolve without breaking each other. That is what maturity looks like for an interoperability standard.

Looking Ahead

The Iceberg Go v0.6.0 and PyIceberg 0.12.0 releases should close in the coming days, and the C++ 0.3.0 discussion will likely firm up into a release plan. Watch the X-Iceberg-Client-Capabilities header thread, because if it gains traction it becomes the mechanism through which a lot of future REST evolution gets negotiated. On the Polaris side, the encryption support thread is the one to follow, since it is the catalog half of the story Iceberg started in 1.11, and the Delegation Service design doc should continue to take shape. Arrow's Erlang donation moves to the incubator general list for IP clearance, and the Copilot review discussion is worth watching as a bellwether for how Apache data projects handle AI in their workflows. For Parquet, the PARQUET-2249 vote should close and move into implementation across engines, the Footer Working Group will likely publish a charter and cadence, and the 1.18.0 release planning plus the Java 17 baseline proposal will shape what the next Parquet Java looks like. The through-line for the weeks ahead is the same one that defined this week: the interesting work is increasingly in the layers that connect the projects to each other.

Resources & Further Learning

Get Started with Dremio

Try Dremio Free lets you build your lakehouse on Iceberg with a free trial.

Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow walks through how Dremio brings the open lakehouse stack together.

Free Downloads

Apache Iceberg: The Definitive Guide, the O'Reilly book, is available as a free download.

Apache Polaris: The Definitive Guide, the O'Reilly book, is available as a free download.

Books by Alex Merced

Architecting an Apache Iceberg Lakehouse

Enabling Agentic Analytics with Apache Iceberg and Dremio

The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI

The Book on Using Apache Iceberg with Python

AI Weekly: Cheaper Coding Models, Custom Chips, and a Stateless MCP

Alex Merced — Wed, 27 May 2026 14:19:26 +0000

The past week pushed three quiet shifts into the open. A coding model matched the frontier at a tenth of the cost. Custom chips started outgrowing Nvidia. And the protocol behind most AI agents got its biggest rewrite yet.

AI Coding Tools: Cursor Ships Composer 2.5 and the Price of Frontier Coding Drops

Cursor released Composer 2.5 on May 18. The headline is not the benchmark score. It is the price next to that score.

Composer 2.5 scores 79.8% on SWE-Bench Multilingual and 63.2% on CursorBench v3.1. Those numbers sit right next to Claude Opus 4.7 and GPT-5.5 on the same tests. The standard tier costs $0.50 per million input tokens and $2.50 per million output tokens. That works out to roughly one-tenth the cost per task of the frontier models it matches.

The model runs on Moonshot AI's open-source Kimi K2.5 checkpoint. Cursor spent about 85% of its compute budget on its own post-training, including reinforcement learning on 25 times more synthetic coding tasks than Composer 2 used. The base model came from a Beijing lab. The behavior that developers actually feel came from Cursor's training pipeline. That split tells you something about where value lives in 2026. The base weights are increasingly a commodity. The post-training is the product.

Composer 2.5 is built for long, tool-heavy sessions. It reads files, runs terminal commands, edits across many files, runs tests, and iterates on its own. Cursor tuned it for sustained work and better instruction following, not just raw puzzle-solving. The jump from Composer 2 was real: SWE-Bench Multilingual went from 73.7% to 79.8%, and Terminal-Bench 2.0 went from 61.7% to 69.3% in two months.

The model is not the whole story for Cursor this month. The editor is turning into something closer to a control panel for a whole team's agents. Cursor 3.3 landed on May 7 with Build in Parallel, a feature that dispatches async subagents across independent steps of a plan at the same time. Cursor 3.5 followed on May 20 with multi-repo automations and shared canvases for team artifact access. The pattern is clear. Cursor wants to be the place where you manage many agents, not just the place where you autocomplete one line.

Google answered fast. On May 19, one day after Composer 2.5 shipped, Google launched Antigravity 2.0 with Gemini 3.5 Flash at I/O 2026. Antigravity 2.0 targets the same agentic IDE seat as Cursor. It pairs multi-agent orchestration with a built-in Chromium browser, dynamic subagents, and scheduled background tasks. Two of the biggest names in the space shipped competing agentic IDEs within 24 hours of each other. That cadence is the real signal.

Here is the part worth sitting with. Composer 2.5 does not beat Opus 4.7 or GPT-5.5 outright. On CursorBench v3.1, a test built around real Cursor workflows, it edges ahead. On Terminal-Bench 2.0 it ties Opus 4.7 at 69.3% but trails GPT-5.5 at 82.7%. Opus keeps an edge on deep architectural reasoning and long single-shot generation. So the frontier still leads on the hardest work. What changed is that "good enough for most tasks" now costs a fraction of what it did, and it runs inside the editor where the work already happens.

For teams, this resets the math. A year ago the question was which single coding tool to standardize on. That question is gone. JetBrains research from April found that 90% of developers used at least one AI tool at work as of January 2026, and 74% used specialized development tools beyond plain chat. GitHub Copilot stayed the most adopted at 29% workplace usage, with Cursor and Claude Code tied at 18%. Most teams now run two or three tools in different roles. A common setup pairs Claude Code in the terminal for agentic work, Cursor or Copilot in the IDE for inline edits, and a chat window for thinking through design.

The terminal-native agents keep gaining ground at the high end. The JetBrains 2026 survey recorded Claude Code jumping from 3% adoption in April 2025 to 18% in January 2026, a sixfold rise in nine months. The starker number is senior developer preference. When JetBrains asked developers with more than ten years of experience which tool they would pick for daily work, 46% chose Claude Code and 9% chose Copilot. The anthropics/claude-code repository now counts more than 126,000 GitHub stars. Codex passed 3 million weekly active users in March, up from 2 million a month earlier. None of these tools is winning outright. Each owns a slice.

The interesting question for teams is which tool acts as the controller and which ones do the subtasks. Most teams now run a terminal-native agent like Claude Code as the controller and hand specific jobs to Codex or Cursor. That arrangement is not stable. It has shifted twice already this year and will likely shift again before December. Picking a permanent stack right now is a bet against your own future workflow.

One more change is reshaping the field, and it is about money, not models. GitHub paused new sign-ups for Copilot Pro and Pro+ in April. Copilot moves to AI Credits-based flex billing on June 1, keeping the $10 and $39 prices but swapping in credit pools. A new Copilot Max tier targets heavy individual users. Windsurf raised Pro from $15 to $20 a month and added a $200 Max plan bundling Devin. Cursor included double usage for the first week after Composer 2.5 to pull developers in for evaluation. The tools are competing on cost structure now, not only capability.

The plain advice for a working developer in 2026 holds up well. A solo developer or hobbyist gets the best entry value from Copilot at $10 a month. A full-time developer tends to pay for itself with Cursor Pro at $20 a month within the first week. A senior developer or technical founder who lives in the terminal gets the most from Claude Code on a higher tier, where the agentic depth justifies the price. Many people use more than one of these at once, and that is the sane default rather than a sign of indecision.

Microsoft Build runs June 2 and 3 in San Francisco, so its announcements land just after this issue. The smaller two-day format and the agenda point at agents as the throughline. The seven session tracks include Agents and Apps, Azure AI Foundry, and a track on working with models. Microsoft framed 2026 as the year agentic tooling moves from announced to production-ready. Expect multi-agent orchestration, new APIs for deploying autonomous agents, and updates to Microsoft's MCP integrations. We will cover the actual announcements next week.

AI Processing: Custom Chips Start Outgrowing Nvidia

A shipment forecast this week marked a turning point in AI hardware. For the first time, custom AI chips are projected to outgrow Nvidia's GPUs.

TrendForce projects 44.6% growth in ASIC shipments for 2026 against 16.1% growth for merchant GPUs. Alchip Chairman Johnny Shen confirmed the shift in comments reported by Digitimes on May 26. ASIC stands for application-specific integrated circuit, a chip designed for one job rather than general flexibility. The growth gap says buyers are moving toward purpose-built silicon faster than they are buying more general-purpose GPUs.

The reason is straightforward. Nvidia's GPUs are general-purpose processors. They are powerful and flexible, built to run nearly any AI workload, but not tuned for any single one. AI inference, the ongoing job of running trained models against live queries, has overtaken training as the dominant compute load. For inference at scale, that flexibility carries a cost you no longer need to pay. A chip built for one model architecture can run it cheaper and cooler than a GPU that can run anything.

This does not mean Nvidia is in trouble. Nvidia still holds roughly 70% to 80% of the AI accelerator market by revenue. Total Nvidia AI accelerator revenue could pass $150 billion in 2026. Losing share in percentage terms is not the same as losing money when the whole market is growing this fast. The real pressure on Nvidia is margin, not market exit. As hyperscalers diversify suppliers, they gain leverage to push pricing on next-generation parts.

The custom-chip wave has clear backers. Broadcom co-designs Google's Tensor Processing Units and chips for Meta and others. It reported $8.4 billion in AI semiconductor revenue in a recent quarter. Alchip is forecasting a return to growth in 2026 as new 3-nanometer accelerator programs hit volume in the second quarter, with about 80% of its revenue landing in the second half of the year. Every major hyperscaler now ships in-house silicon: Google with TPU, AWS with Trainium, Microsoft with Maia, and Meta with its own designs.

Nvidia is not standing still. Its Vera Rubin architecture, the successor to Blackwell, is in full production, with partner products arriving in the second half of 2026. Rubin is built on TSMC's 3-nanometer process with HBM4 memory and 336 billion transistors. Nvidia reports it cuts inference token costs by 10 times and reduces the GPUs needed to train mixture-of-experts models by 4 times compared to Blackwell. AWS, Google Cloud, Microsoft, and Oracle are among the first cloud providers set to deploy Vera Rubin instances. The architecture is tuned for mixture-of-experts models, the same design trend showing up across the field.

AMD is running its own play. The Instinct MI400 launches in the second half of 2026 with 432GB of HBM4 memory and 40 petaflops of FP4 compute. S&P Global projects the MI400 will generate $7.2 billion in its first year, and AMD's data center GPU revenue is forecast to grow 114% year over year to $15 billion. AMD also locked in a multi-generational deal with Meta covering a 6-gigawatt deployment, the first tranche using MI450-based custom GPUs.

What does this mean if you build with AI rather than sell chips? Inference is getting cheaper, and the savings will reach your bills. The same shift that lets Cursor sell near-frontier coding at a tenth of the cost is happening one layer down in silicon. Purpose-built inference chips, mixture-of-experts models that activate only the parameters they need, and architectures tuned for serving instead of training all point the same direction. Running models in production is on a steady path to costing less, which changes what is worth building.

There is a catch that keeps Nvidia ahead even as custom chips grow faster. Its CUDA software ecosystem has more than a decade of tools, libraries, and developer habits built on top of it. Moving a workload off Nvidia means rewriting or re-tuning the code that runs on it, and that switching cost is real. Custom ASICs win where the workload is fixed and the volume is huge enough to justify the engineering, which describes a hyperscaler running one model at massive scale. It does not yet describe most teams, who still benefit from the flexibility of a GPU that runs whatever they throw at it.

The other trend to watch is the move to the edge. NPUs in laptop-class chips from Intel, AMD, and Apple now deliver 40 to 50 TOPS of on-device inference. That is enough to run capable local models without a round trip to the cloud. The hybrid pattern, cloud for the hard reasoning and the device for latency-sensitive work, is becoming the default shape for AI apps rather than a niche choice.

Standards and Protocols: MCP Gets Its Biggest Rewrite, and the NSA Weighs In

Two things happened to the Model Context Protocol this week, and they pull in opposite directions. The protocol got its largest revision since launch, and the NSA published its first formal security guidance for it. Both matter if you build agents.

MCP is the open standard, created by Anthropic in late 2024, that lets AI models connect to external tools, databases, and services through one common interface. Instead of writing custom glue code for every new integration, you connect through MCP once. It has become a core building block of the agentic AI stack.

On May 21, the maintainers locked the release candidate for MCP 2026-07-28. The final spec publishes on July 28. The ten-week gap gives SDK maintainers and client builders time to validate the changes against real workloads. This is the biggest revision since the protocol launched.

The centerpiece is a stateless protocol core. The new version removes the session ID, the initialize handshake, and resumable streams. In plain terms, the protocol now runs on ordinary HTTP infrastructure without holding a connection open or tracking session state on the server. That is a major change for anyone running MCP in production. Stateless services scale across commodity servers far more easily than stateful ones, since any server can handle any request. Load balancing gets simpler, and recovery from a crashed node stops being a problem.

The release brings more than the stateless core. An Extensions framework lets capabilities ship on their own timeline instead of waiting for a full spec release. Two extensions arrive with it. MCP Apps allow server-rendered user interfaces, so a tool can return a real UI rather than plain text. The Tasks extension handles long-running work, the kind of job that takes minutes or hours instead of returning instantly. Authorization now aligns more closely with OAuth and OpenID Connect, which matters for enterprise deployments. A formal deprecation policy means the protocol can change without breaking what teams already built.

The timing of the NSA guidance is no accident. On May 20, the NSA's Artificial Intelligence Security Center published a Cybersecurity Information Sheet titled "Model Context Protocol: Security Design Considerations for AI-Driven Automation." The document runs 17 pages, carries identifier U/OO/6030316-26, and is the most careful security treatment of MCP to date.

The core warning is structural. MCP flips the usual security model. In a normal API, clients query servers for data. With MCP, servers query data and execute actions on behalf of clients. That inversion means the mental model most engineers use to reason about API security points the wrong way. Access control is optional at the protocol level, which is exactly the gap the NSA flags.

The guidance names specific risks. Serialization flaws can let bad input trigger unstable behavior. Trust boundary failures let one component over-reach into another. Unverified task propagation lets tasks pass between MCP servers without checking their origin, scope, or intent, which can leak sensitive context or fire unrelated tools. Session weaknesses can allow message replay or session hijacking. The NSA's central point is that these problems cannot be patched at isolated endpoints. They have to be addressed across the whole MCP environment.

The practical advice is concrete. Validate every tool invocation against defined schemas, expected ranges, and the intended execution context. Log all tool and model invocations with their exact parameters and the identities involved. Use a filtering outgoing proxy or enterprise data-loss prevention for external MCP connections, with resource URLs pinned tightly enough to limit leakage. Prefer a local MCP server instance when processing private data. Align tools and models with data classification zones, so public tools handle public data and sensitive tools stay segregated and explicitly controlled.

Read the two documents together and a picture forms. MCP adoption has outrun its governance. The protocol is now embedded in production workflows across finance, legal, and software, which means the NSA is describing live exposure in regulated industries, not a hypothetical. MCP stacks built in 2024 and early 2025 likely lack the authentication and privilege isolation now considered baseline. The 2026-07-28 spec hardens authorization and brings the stateless core, and the NSA guidance gives teams a checklist while they wait for it. If you ship anything touching MCP, both belong on your reading list this week.

If you run agents in production, this week gives you a short to-do list. Audit your MCP servers for the access controls the NSA flags, since the protocol will not enforce them for you. Log every tool call with its parameters and the identity behind it, because you cannot investigate what you did not record. Plan for the stateless migration ahead of the July 28 final spec, especially if your current setup leans on session IDs or resumable streams. And test at least one of the cheaper coding models against your real workload before your next billing cycle, since the cost gap is now large enough to matter at team scale.

The thread tying all three categories together is maturation. Coding models are competing on cost because capability has spread. Chips are specializing because workloads have settled into clear shapes. And the protocol layer is being rewritten for scale and locked down for security because it is running real production systems now. The experimental phase is closing. The infrastructure phase is here.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free: Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data: Dremio's agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community: Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development: Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis: A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail

Alex Merced — Sun, 24 May 2026 00:50:59 +0000

For the past decade, data engineering was synonymous with distributed clusters. If your dataset exceeded a few gigabytes, standard practice dictated spinning up an Apache Spark cluster on AWS EMR or Databricks. This distributed paradigm introduced massive operational complexity: managing JVM configurations, allocating executors, tuning shuffle partitions, and paying a substantial "serialization tax" to move data across network sockets and language runtimes.

Recently, the data engineering landscape has experienced a single-node renaissance. Rather than scaling out to distributed clusters, teams are scaling up on single machines. Modern laptops ship with 12 or more CPU cores, fast NVMe SSDs capable of multi-gigabyte-per-second read throughput, and up to 128 GB of RAM. Cloud providers offer single virtual machines with hundreds of cores and terabytes of memory for a fraction of the cost of a Kubernetes or Spark cluster.

This physical hardware evolution is only half the story. The true catalyst is a new generation of data technologies built on Apache Arrow, vectorized execution, and out-of-core memory management. Tools like DuckDB, Apache Arrow DataFusion, Polars, and LakeSail enable a single laptop or VM to process hundreds of gigabytes—and even terabytes—of data. You can now execute complex analytical pipelines locally or on a single node without the overhead of a distributed JVM runtime.

The Core Foundations: Columnar Memory and Apache Arrow

To understand how single-node data engineering can process datasets that previously required hundreds of cluster nodes, you must look at how data is structured in memory.

Traditional databases and processing runtimes designed for transactional workloads (OLTP) use row-oriented layouts. They store all fields of a single record contiguously in memory: [User_ID, Age, Name], followed by the next record. When executing analytical queries (OLAP) that only target a subset of columns (such as calculating the average age of users), a row-oriented engine must scan the entire record structure from memory. This process loads irrelevant data (like names and IDs) into the CPU's L1/L2 caches, leading to cache pollution and wasted memory bandwidth.

Columnar query engines solve this inefficiency by storing data contiguously by column: [Age, Age, Age] in one buffer, and [Name, Name, Name] in another. The CPU only reads the specific columns required by the query.

Row-Oriented Layout (OLTP):
┌──────────────────────────────┬──────────────────────────────┐
│ ID 1 │ Age 1 │ Name 1        │ ID 2 │ Age 2 │ Name 2        │
└──────────────────────────────┴──────────────────────────────┘

Columnar Layout (Arrow/OLAP):
┌──────────┬──────────┐ ┌──────────┬──────────┐ ┌──────────┬──────────┐
│ ID 1     │ ID 2     │ │ Age 1    │ Age 2    │ │ Name 1   │ Name 2   │
└──────────┴──────────┘ └──────────┴──────────┘ └──────────┴──────────┘

Apache Arrow standardizes this columnar memory layout. It defines an open-source, language-independent specification for in-memory columnar data. By establishing a shared memory format, Arrow eliminates the serialization tax that historically slowed down data pipelines.

In traditional architectures, passing data between a Python script and a Java or C++ engine required serializing the data into a byte stream (like JSON or Protobuf) and deserializing it on the other side. This serialization tax frequently consumed up to 80% of the total query execution time.

Arrow enables zero-copy Inter-Process Communication (IPC). Because Arrow represents data in memory exactly the same way across Python, Rust, and C++, different processes can memory-map (mmap) the same physical memory buffers. An engine can pass a dataset to Python for machine learning or visualization by exchanging memory pointers. No bytes are copied, and no serialization occurs.

Furthermore, Arrow's contiguous memory alignment matches the layout of modern CPU cache lines, making it straightforward to utilize Single Instruction, Multiple Data (SIMD) instruction sets (such as AVX-512 on Intel/AMD or Neon on ARM). SIMD allows the CPU to apply a single instruction (such as a filter comparison or an arithmetic addition) to a vector of data points in a single clock cycle. This hardware-level parallelism turns data processing from a memory-bound or CPU-bound bottleneck into an efficient operation running directly on the processor.

In-Process SQL Powerhouse: DuckDB Architecture & Features

DuckDB has become the standard database engine for single-node SQL analytics. Designed as an in-process analytical database, DuckDB runs directly inside the host process (such as a Python interpreter or a CLI binary) rather than as a separate server daemon. This eliminates the network socket latency and IPC overhead of client-server databases like PostgreSQL or Snowflake.

DuckDB's execution engine utilizes a vectorized query execution model. Rather than processing data one row at a time (the Volcano iterator model) or processing entire columns at once (which overflows L1/L2 caches for large tables), DuckDB processes data in small, cache-friendly vectors. These vectors typically contain 2048 elements.

Volcano Model:       [Row 1] ──► [Operator] ──► [Row 2] ──► [Operator]
Column-at-a-time:    [Entire Column (10M rows)] ──► [Operator] (Overflows Cache)
Vectorized Model:    [Vector of 2048 rows] ──► [L1/L2 CPU Cache] ──► [Operator]

By keeping these vectors small enough to fit inside the CPU's L1/L2 cache, DuckDB minimizes memory bandwidth bottlenecks. The CPU executes operations on the vectors using SIMD instructions, keeping the execution pipelines saturated with data.

To handle datasets that exceed physical RAM, DuckDB implements out-of-core execution. When memory consumption reaches a user-defined limit, DuckDB's buffer manager automatically spills intermediate query states (such as hash join tables, sorting buffers, or aggregation states) to temporary disk files. This spilling mechanism uses a block-based buffer pool that page-faults data to disk, allowing you to run queries on datasets that are multiple times larger than your system's RAM.

In the latest v1.5.3 release (May 2026), DuckDB has introduced several updates that expand its single-node utility:

Quack Remote Protocol: DuckDB now ships with a core extension implementing the Quack protocol. This protocol allows users to run DuckDB in a client-server configuration when needed, facilitating remote attachments and remote query orchestration without losing the simplicity of the engine.
Ecosystem and Format Updates: The Iceberg extension has been upgraded to support MERGE INTO operations, making it possible to execute complex delta updates on Iceberg tables directly from a local DuckDB session.
AWS Security and IRSA: Native support for IAM Roles for Service Accounts (IRSA) has been added, simplifying secure S3 access when running DuckDB inside containerized single-node pipelines.
Static Linking: The distribution now statically links jemalloc on Linux platforms, improving memory allocation speed and reducing fragmentation during heavy out-of-core spilling.

The following Python script illustrates how to configure DuckDB's memory limits, register an S3 credential using the new AWS extension features, and run a query that spills to disk:

import duckdb

# Initialize DuckDB connection
con = duckdb.connect(database='local_cache.db')

# Set memory limit to force out-of-core spilling on smaller datasets
con.execute("SET max_memory='8GB';")
con.execute("SET temp_directory='./duckdb_temp';")

# Load S3 and AWS extensions (built-in in v1.5.3)
con.execute("INSTALL aws;")
con.execute("LOAD aws;")

# Autodetect AWS credentials from environment (supports IRSA)
con.execute("CALL load_aws_credentials();")

# Query a large Parquet dataset directly on S3 with predicate pushdown
# DuckDB only downloads the columns and row groups that match the filter
query = """
    SELECT 
        user_id, 
        COUNT(event_id) as event_count,
        AVG(session_duration) as avg_duration
    FROM read_parquet('s3://my-lakehouse/bronze/events/**/*.parquet')
    WHERE event_date >= '2026-01-01'
    GROUP BY user_id
    HAVING event_count > 1000
    ORDER BY avg_duration DESC
"""

# Execute and stream results
result = con.execute(query).fetchdf()
print(result.head())

DuckDB's combination of SQL support, vectorized performance, and out-of-core stability makes it a core tool for local analytical workloads.

Extensible Rust Processing: Apache Arrow DataFusion

While DuckDB is packaged as an analytical database, Apache Arrow DataFusion is designed as an extensible query engine framework. Written in Rust and utilizing Apache Arrow as its native memory format, DataFusion is widely used to build other databases, query engines, and custom data platforms (including Bauplan, Spice.ai, and LakeSail).

DataFusion's design is modular. It decouples the query planning, optimization, and execution stages. If you are building a custom data tool, you can register custom catalogs, write user-defined logical optimization rules (like custom predicate pushdowns), or plug in custom physical execution nodes.

For thread-level parallelism, DataFusion utilizes Rust's asynchronous Tokio runtime. Rather than pinning execution to a fixed number of threads, DataFusion distributes physical plan fragments (represented as asynchronous streams of Arrow RecordBatch objects) across a Tokio worker thread pool. This allows the engine to adapt to multi-core architectures and avoid thread contention under heavy I/O loads.

In the recent v53.x and v54.x releases (early-to-mid 2026), the DataFusion community has introduced several optimizations:

Datetime Predicate Preimages: DataFusion now optimizes queries containing datetime functions (like date_trunc and date_part) by evaluating their mathematical "preimages." Instead of executing the datetime function on every row, the optimizer rewrites the filter predicate against the raw partition bounds, enabling partition pruning.
Sort Pushdown Phase 2: The engine now sorts file groups by physical statistics before executing sort operators. If a set of Parquet files contains non-overlapping sorted ranges, DataFusion skips the global sort merge step, reducing planning and CPU execution times.
Null-Aware Anti-Joins: Support has been optimized for null-aware anti-joins, which frequently occur in SQL queries containing NOT IN clauses.
Variant Type Integration: The planner has introduced initial support for the binary VARIANT format, laying the groundwork for format-agnostic semi-structured data querying.

The following Rust code snippet demonstrates how to initialize a DataFusion context, register an in-memory Arrow table, and execute a query programmatically:

use datafusion::prelude::*;
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::arrow::schema::{DataType, Field, Schema};
use datafusion::arrow::array::{Int32Array, StringArray};
use std::sync::Arc;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    // Create a local execution context
    let ctx = SessionContext::new();

    // Define a simple schema
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int32, false),
        Field::new("name", DataType::Utf8, false),
    ]));

    // Create Arrow arrays
    let id_array = Int32Array::from(vec![1, 2, 3, 4, 5]);
    let name_array = StringArray::from(vec!["Alice", "Bob", "Charlie", "David", "Eve"]);

    // Build the record batch
    let batch = RecordBatch::try_new(
        schema.clone(),
        vec![Arc::new(id_array), Arc::new(name_array)],
    )?;

    // Register the record batch as an in-memory table
    ctx.register_batch("users", batch)?;

    // Execute SQL query
    let df = ctx.sql("SELECT name FROM users WHERE id > 2").await?;

    // Print the physical execution plan
    df.show().await?;

    Ok(())
}

This library-first model makes DataFusion the preferred choice for teams building specialized, high-performance data systems.

Vectorized DataFrames: Polars Eager & Lazy Pipelines

For developers working in Python, Rust, or JavaScript, DataFrames are the preferred API for data manipulation. While Pandas has been the standard in Python for a decade, it is single-threaded, has a high memory footprint (often requiring 5–10x the dataset size in RAM), and does not support query optimization.

Polars is a Rust-native, Arrow-backed DataFrame library designed to replace Pandas. It is optimized for multi-core execution, utilizing a custom work-stealing CPU scheduler that distributes execution chunks across available cores.

Polars offers two execution modes:

Eager API: Executes operations immediately, step-by-step, mimicking Pandas' behavior. This mode is useful for interactive debugging in Jupyter Notebooks.
Lazy API: Builds a logical Directed Acyclic Graph (DAG) representing the pipeline. When you call .collect(), Polars passes the DAG through a query optimizer. The optimizer applies several rules:
- Projection Pushdown: Only reads the columns explicitly referenced in the query.
- Predicate Pushdown: Moves filter operations as close to the storage layer as possible (pushing them down into the Parquet reader).
- Common Subexpression Elimination: Identifies duplicate calculations and executes them once.

Eager: Load File (All Columns) ──► Filter Rows ──► Select Columns
Lazy:  Query Planner ──► Push Filter & Select Into File Reader ──► Load File (Filtered & Pruned)

In 2026, the Polars team officially stabilized its streaming execution engine. This engine allows out-of-core DataFrame execution on datasets that exceed physical memory limits. The streaming engine now supports:

Streaming Merge and AsOf Joins: Useful for temporal alignments (such as joining financial tick data or IoT sensor metrics).
Streaming Aggregations: Complex statistical calculations (including skew, kurtosis, and entropy) can now run in streaming mode.
Direct Cloud Sinks: Polars can stream data directly back to storage formats like Delta Lake (sink_delta) and Apache Iceberg (sink_iceberg) without materializing the intermediate tables.

To enable the streaming engine, developers configure Polars to use the streaming execution path:

import polars as pl

# Enable streaming engine affinity globally
pl.Config.set_engine_affinity("streaming")

# Define a Lazy pipeline querying a folder of compressed CSVs
lazy_query = (
    pl.scan_csv("./data/raw_metrics/*.csv")
    .filter(pl.col("metric_type") == "cpu_utilization")
    .with_columns(
        (pl.col("metric_value") * 100).alias("percentage")
    )
    .group_by(["host_id", "timestamp"])
    .agg([
        pl.col("percentage").mean().alias("mean_cpu"),
        pl.col("percentage").skew().alias("skew_cpu") # Uses new streaming aggregations
    ])
    .sort("mean_cpu", descending=True)
)

# Execute the query out-of-core using the streaming engine
# This will process files in batches, avoiding Out-Of-Memory (OOM) crashes
result_df = lazy_query.collect(streaming=True)
print(result_df.head())

Polars' combination of an expressive DataFrame API, lazy query optimization, and stabilized streaming makes it a powerful engine for Python and Rust developers.

Comparative Analysis: Evaluating Single-Node Engines

Choosing the right tool requires evaluating their architectural differences and primary API surfaces:

Key Tradeoffs to Consider

API Choice: If your team writes standard SQL, DuckDB is the logical starting point. If you write procedural code, Polars' expression language is more expressive and easier to parallelize than SQL.
Extensibility vs. Out-of-the-Box Utility: DuckDB and Polars are complete user-facing applications. DataFusion is an engine framework. You use DataFusion if you are building a custom database or need to modify how the physical query execution layer functions.
Memory Footprint: DataFusion and Polars generally maintain a lower memory footprint than DuckDB for in-memory operations due to Rust's memory management model and direct mapping to Arrow structures. However, DuckDB's buffer manager is more mature for highly complex queries that require massive disk spilling.

Zero-JVM Spark: High-Performance Pipelines with LakeSail

For teams with existing data codebases, the primary blocker to adopting single-node tools is the legacy API footprint. Many organizations have thousands of lines of Apache Spark code written in PySpark. Rewriting these pipelines to DuckDB SQL or Polars DataFrames is expensive and introduces validation risks.

Historically, running PySpark locally required spinning up a local Spark cluster. This cluster runs on the Java Virtual Machine (JVM), which introduces significant configuration complexity and memory overhead. A default local Spark session can easily consume 4 GB of RAM just to start, even when processing a 10 MB CSV file.

Furthermore, PySpark operates via a Py4J gateway bridge. When your PySpark code calls a Python User-Defined Function (UDF), the data must be serialized, sent from the JVM to a Python worker process, processed, serialized again, and sent back to the JVM. This cross-process serialization tax makes Python UDF execution in Spark slow.

Traditional PySpark UDF Path:
[JVM Executor] ──(Serialize via Py4J)──► [Python Worker] ──► [Run UDF] ──(Serialize)──► [JVM Executor]

LakeSail (specifically the open-source Sail engine) solves this constraint. Sail is a Rust-native, JVM-free compute engine designed as a drop-in replacement for Apache Spark. It implements the Spark Connect protocol, allowing existing PySpark and Spark SQL applications to run unmodified by connecting to a Sail server over gRPC.

LakeSail PySpark Connect Path:
[PySpark Session] ──(Spark Connect gRPC Logical Plan)──► [LakeSail Rust Server] ──► [DataFusion Physical Execution]

Under the hood, Sail replaces Spark's JVM-based Catalyst optimizer and Tungsten execution engine with Apache DataFusion and Apache Arrow. This architecture provides several advantages:

Zero JVM Overhead: Sail starts in milliseconds and has a negligible idle memory footprint. You can run Spark code on small single-core VMs or local laptops.
Zero-Copy Python UDF Execution: Sail embeds a Python interpreter directly into its Rust binary using PyO3. When executing a Python UDF, Sail passes pointers to the Arrow memory buffers directly to the Python interpreter. The UDF executes in-process without serialization, eliminating the cross-process Py4J bottleneck.
Native Open Formats: Sail includes native Rust-based support for Delta Lake, Apache Iceberg, and Parquet, integrating directly with AWS Glue, Unity Catalog, and Polaris REST catalogs.

To run your PySpark pipelines against a local Sail session, you install the packages and point the session builder to the local Sail gRPC port:

# Install pysail and PySpark client supporting Spark Connect
pip install pysail pyspark

Start the Sail server from your terminal:

# Start local Sail gRPC server on port 50051
sail spark server --port 50051

In your Python code, connect the SparkSession to the local Sail server using the standard remote connection string:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, udf
from pyspark.sql.types import IntegerType

# Connect to the local Sail Rust-native server over Spark Connect protocol
spark = SparkSession.builder \
    .remote("sc://localhost:50051") \
    .getOrCreate()

# Load a local Parquet dataset using standard Spark DataFrame API
df = spark.read.parquet("./data/raw_orders")

# Define a standard Python UDF
@udf(returnType=IntegerType())
def calculate_tax(amount):
    # This runs in-process via Sail's PyO3 integration
    # Zero serialization tax is paid between Rust and Python
    return int(amount * 0.08)

# Execute transformations and show results
processed_df = df.filter(col("status") == "COMPLETED") \
                 .withColumn("tax", calculate_tax(col("total_amount")))

processed_df.show()

By keeping the Spark API surface while replacing the execution engine, LakeSail allows teams to modernize their legacy PySpark pipelines and run them on single nodes without the overhead of a JVM.

The Threshold of Scale: When Does Single-Node Break?

While single-node data engineering has expanded the scale of data that can be processed on a single machine, it is not a silver bullet. At a certain point, physical resource constraints make single-node architectures impractical.

The primary bottleneck is I/O. During out-of-core execution, spilling data to disk shifts the bottleneck from memory capacity to disk read/write bandwidth. Even on fast NVMe SSDs, writing and reading hundreds of gigabytes of intermediate join tables or sorting buffers introduces latency. If a query spends more time reading and writing temporary blocks to disk than it does executing CPU cycles, the system is I/O-bound.

The second bottleneck is query planning and CPU execution scaling. If your query must scan multiple terabytes of data, even a vectorized engine running on 64 cores will take minutes to complete the scan. If your business SLAs require sub-second or low-second query latencies, you need to distribute the scanning and processing work across multiple machines in parallel.

The third bottleneck is organizational concurrency. If a single VM hosts your analytical database, and hundreds of analysts or BI dashboards query it simultaneously, the CPU cores will experience thread starvation, and lock contention will slow execution times for all users.

To guide your architectural transitions, use the following operational decision framework:

The MPP Landscape: Scaling to Spark, Dremio, Bauplan, SpiceAI, and MotherDuck

When your data scale, latency requirements, or concurrency needs exceed single-node limits, you must transition to a distributed MPP (Massively Parallel Processing) architecture. The modern MPP landscape offers several pathways, depending on your workflow patterns.

MotherDuck (Dual Execution)

For teams who want to scale their DuckDB workloads to the cloud without managing infrastructure, MotherDuck provides a serverless platform built on DuckDB.

MotherDuck's core architectural pattern is Dual Execution (formerly hybrid execution). When you submit a query, MotherDuck's query planner evaluates the locations of the datasets. It splits the query plan: executing parts of the query locally on your laptop CPU using local cached data, and executing other parts on MotherDuck's cloud compute nodes (for cloud-hosted Parquet or Iceberg tables). The engine joins these streams dynamically using specialized "bridge" operators.

In early 2026, MotherDuck added a native PostgreSQL wire protocol endpoint. This allows BI tools and legacy applications to connect directly to MotherDuck using standard PostgreSQL drivers, eliminating the need to install the DuckDB runtime on the client machine. Additionally, MotherDuck features Pulse (serverless) billing with one-second increments and DuckLake integration for scaling storage to the petabyte range.

Bauplan (Serverless Python Pipelines)

For data engineers building pipeline workflows on Apache Iceberg, Bauplan provides a serverless, "zero-infrastructure" execution engine.

Instead of managing Spark or Kubernetes clusters to run scheduled data transformations, you define your pipeline steps as standard Python or SQL functions. Bauplan spins up stateless, ephemeral compute on-demand to execute the code and shuts down immediately after, utilizing a pay-per-invocation model.

Bauplan integrates Apache Iceberg with Project Nessie, providing a "Git-for-data" experience. Developers and AI agents can create isolated branches of the lakehouse, run experimental Python pipelines to verify changes, and merge the updates atomically back into production without risking data corruption or paying for idle staging compute.

Spice.ai (Federated Query Acceleration)

Spice.ai (SpiceAI) targets the data access layer for high-performance applications and AI agents. It functions as a federated query runtime that accelerates slow data queries by materializing "hot" data sets locally.

Spice.ai implements a tiered caching model. It caches query results in-memory and caches active working sets of data in high-performance local engines like DuckDB or Cayenne (a native columnar engine).

In its recent v2.0 updates, Spice.ai introduced a prefix-aware list-files cache that speeds up data lake scans, a statistics cache for file metadata, and native Change Data Capture (CDC) syncing that streams updates from databases (like PostgreSQL WAL streams) directly into the local acceleration cache. This keeps the local cached tables updated in real-time without requiring complex Kafka or Debezium setups.

Dremio (Distributed MPP Lakehouse Platform)

For enterprise-scale BI, multi-source data federation, and semantic layer management, Dremio serves as the central engine of the lakehouse.

Dremio is built from the ground up on Apache Arrow, eliminating the serialization tax entirely. When Dremio queries data, the physical execution plan processes memory structures natively in Arrow columnar format and streams results to clients (such as Python scripts or BI tools) using Arrow Flight.

Dremio achieves sub-second performance on massive cloud data lakes through three architectural layers:

Columnar Cloud Cache (C3): Automatically caches data blocks from object storage (like AWS S3 or Azure ADLS) onto local NVMe drives at execution nodes, turning remote cloud I/O into local disk read speeds.
Reflections: Dremio’s query planner automatically and transparently substitutes physically optimized, pre-computed Iceberg materializations to accelerate user queries. As of Dremio v26, Reflections store data exclusively in Iceberg format, deprecating legacy formats to streamline the storage path. Dremio's Autonomous Reflections use AI to observe query patterns over a rolling 7-day window, automatically creating, updating, and dropping Reflections to maintain optimal dashboard performance without manual administration.
Open Catalog (Powered by Apache Polaris): Dremio's built-in catalog is built on Apache Polaris, which graduated to a top-level Apache project in 2026. The Open Catalog implements the Apache Iceberg REST specification, allowing other engines (like Spark or Flink) to query the same tables securely. It provides Fine-Grained Access Control (FGAC) including column-masking and row-level filtering.

Dremio’s AI Semantic Layer allows teams to define virtual datasets (views) once and reuse them across all BI and AI applications. This layer embeds descriptions, wikis, and tags directly onto columns and datasets. The semantic layer teaches AI models the business context of your data, allowing AI agents to generate correct, governed SQL queries rather than hallucinating generic code. Dremio also embeds generative AI features to auto-generate wiki descriptions and suggest tags based on schema patterns.

Architectural Selection Framework and Conclusion

Modern data engineering is no longer about choosing between a local script and a massive cluster. It is about matching your toolchain to your data volume, latency SLAs, and organizational needs.

To guide your selection, follow this decision tree:

Is your workload running locally or on a single node?
- If you prefer writing SQL for analytical queries: Use DuckDB. It requires zero configuration and handles larger-than-memory data via out-of-core spilling.
- If you are writing procedural Python or Rust DataFrame pipelines: Use Polars. Its lazy optimizer and stabilized streaming engine provide rapid execution.
- If you have legacy PySpark or Spark SQL code but want to avoid JVM overhead: Use LakeSail. It executes Spark Connect gRPC logical plans natively in Rust.
- If you are building a custom query engine or analytical tool: Use Apache Arrow DataFusion as your modular compiler framework.
Does your workload exceed single-node capabilities (multi-TB scale, high concurrency, or cross-source BI)?
- If you want a serverless, hybrid extension of your DuckDB SQL code: Use MotherDuck.
- If you need to build serverless Python pipelines directly on Iceberg with Git-like version control: Use Bauplan.
- If you need to cache and accelerate federated data for local AI/RAG applications: Use Spice.ai.
- If you need enterprise-scale BI, semantic governance, multi-source federation, and sub-second SQL queries on Iceberg: Use Dremio.

Single-node data technologies have shifted the boundary of what is possible on a single machine. By utilizing Apache Arrow for zero-copy memory layouts, compilers like DataFusion, and vectorized execution engines, you can process workloads that previously required a complex distributed cluster.

As you design your next data platform, start by evaluating if your workload can run on a single node. Modern columnar engines let you build, test, and run pipelines with minimal infrastructure complexity. When your data scale or organizational concurrency requires a distributed architecture, transition incrementally using open standards like Apache Iceberg and Apache Arrow.

Accelerate Your Lakehouse Skills

To deepen your understanding of modern data architectures, consider the following next steps:

Read Lakehouse Reference Materials: Explore "Architecting an Apache Iceberg Lakehouse" and other technical publications that cover partition tuning, catalog design, and query optimization at books.alexmerced.com.
Build Your Own Local Pipeline: Start by downloading pysail or polars and testing them against a local Parquet dataset. Compare the query planning time and CPU memory footprint against your existing frameworks.
Evaluate Dremio Cloud: If your local query engines are hitting limits or you need to federate data across multiple sources, deploy Dremio directly on your S3 data lake. Try Dremio Cloud free for 30 days at dremio.com/get-started.

An In-Depth Overview of the Apache Iceberg 1.11.0 Release

Alex Merced — Sat, 23 May 2026 16:52:51 +0000

Apache Iceberg 1.11.0 was officially released on May 19, 2026, marking a major milestone in the evolution of open data lakehouse architectures. While minor point releases often focus on small bug fixes and dependency bumps, this release introduces fundamental structural changes. The community has completed major initiatives to improve security, extend file format capabilities, and optimize query planning overhead.

This release represents a convergence of two development focuses. First, it introduces structural changes to the core metadata specification to support advanced security features and lay the groundwork for future format revisions. Second, it stabilizes several feature sets in the Iceberg format specification, moving them from experimental status to fully stable defaults.

This post analyzes the most critical improvements in the Apache Iceberg 1.11.0 release. We will examine the specific GitHub pull requests, explain the underlying mechanics of each feature, and review what these changes mean for data engineers and platform architects.

Manifest List Encryption (PR #7770, #15813)

Security in open data lakehouses has historically focused on encrypting the actual data files stored in object storage. While file-level encryption prevents unauthorized users from reading raw Parquet or ORC data, the table metadata has remained exposed. In a default setup, anyone with read access to the storage bucket could inspect the metadata JSON, manifest lists, and manifest files.

These metadata files contain sensitive structural details. An attacker scanning an unencrypted manifest list can extract file paths, column names, partitions, partition bounds, and exact null value counts. In highly regulated industries such as healthcare or financial services, this structural exposure constitutes a major data leak.

To resolve this vulnerability, PR #7770, introduced by @ggershinsky, adds native encryption for manifest lists. This change works alongside follow-up improvements in PR #15813. Manifest lists can now be encrypted using the Galois/Counter Mode (GCM) stream cipher.

Metadata JSON (Contains encryption state references)
       │
       ▼
Manifest List (Encrypted via GCM Stream Cipher) ◄── Decrypted in-memory during planning
       │
       ▼
Manifest Files (Point to encrypted Parquet data files)

The table encryption configuration can be defined during table creation or updated via table properties:

When a query engine plans a scan against an encrypted table, it performs the following sequence:

The client queries the catalog to fetch the table metadata.
The catalog returns the metadata location along with the required decryption keys.
The query engine reads the encrypted manifest list from object storage.
Using the catalog keys, the engine decrypts the manifest list in-memory.
The engine processes the decrypted partitions and statistics to prune manifest files.

This approach ensures that the manifest list is never written to disk in plain text. It implements a model of envelope encryption: each metadata file is encrypted with a unique data encryption key (DEK), and these DEKs are encrypted using the table's master key managed by the Key Management Service (KMS). Even if an attacker gains raw access to the storage bucket, they find only encrypted bytes, protecting both the table contents and its structural metadata.

Pluggable File Format API and V4 Spec Foundations (PR #15049)

Historically, Apache Iceberg hardcoded its support for data file formats. The core library contained format-specific code paths for Parquet, ORC, and Avro. If you wanted to query or write a table, the engine executed internal code blocks tailored to those exact structures.

This hardcoded design created a major bottleneck for format innovation. If a team wanted to test a next-generation format, they had to modify the core engine codebase, extending complex switch statements and format-dependent utilities.

PR #15049, introduced by @anoopj, restructures this architecture. It introduces a pluggable File Format API that decouples Iceberg core metadata management from physical storage layouts.

┌────────────────────────────────────────────────────────┐
│                  Iceberg Core Engine                   │
└───────────────────────────┬────────────────────────────┘
                            │
                            ▼
┌────────────────────────────────────────────────────────┐
│            File Format API Interface Layer             │
└──────┬────────────┬─────────────┬─────────────┬────────┘
       │            │             │             │
       ▼            ▼             ▼             ▼
  ┌─────────┐  ┌─────────┐   ┌─────────┐   ┌─────────┐
  │ Parquet │  │   ORC   │   │ Vortex  │   │  Lance  │
  └─────────┘  └─────────┘   └─────────┘   └─────────┘

The File Format API provides a clean plugin interface. A file format is defined as a plugin that implements standard reader and writer interfaces. Iceberg core negotiates table transactions, schemas, and partition specs, while delegating the physical file access to the registered plugin.

This decoupling makes it practical to support next-generation formats:

Vortex: A general-purpose, modular format designed as a successor to Parquet. It is optimized for high-performance analytics, utilizing fixed-width columns with bitmap masks for nulls. This enables Single Instruction Multiple Data (SIMD) filtering directly on memory-mapped files without CPU decompression cycles. The community is actively using the new API to build a Vortex-backed Iceberg plugin.
Lance: A layout built for machine learning and AI workloads. It is optimized for high-dimensional vector search and random access to nested embeddings, implementing index structures such as Inverted File with Product Quantization (IVF-PQ) directly in the file format to enable fast query planning.
Nimble: A format optimized for wide tables containing thousands of feature columns. Nimble prioritizes fast decoding over high compression ratios, opting for lightweight run-length and bit-packing compression schemes. This reduces the CPU overhead of ML training loops that consume millions of rows per second.

Additionally, PR #15049 introduces the foundational Java interfaces and types for the upcoming V4 manifest specification. These changes prepare Iceberg for format-agnostic manifest storage, ensuring the metadata layer can scale to tables with millions of files without hitting Java memory overhead limits.

REST Client Protocols and Extended Headers (PR #12194)

The REST Catalog protocol has become the standard interface for managing Iceberg tables across multiple processing engines. It isolates clients from catalog catalog details and provides a unified API for schema management, snapshot commits, and credential vending.

However, as deployments scale inside large enterprises, catalogs need to process custom client context. For example, a platform team might want to track which business unit submitted a query, pass custom security tokens, or inject correlation IDs for distributed tracing. In previous versions, the standard RESTClient did not allow clients to send custom HTTP headers.

PR #12194, written by @gaborkaszab, solves this constraint by extending header support inside the RESTClient implementations.

┌────────────────────────────────┐
│      Iceberg REST Client       │
│  (Spark, Flink, Trino, etc.)   │
└───────────────┬────────────────┘
                │
                │  POST /v1/namespaces/db/tables/events
                │  Custom-Headers:
                │    - X-Trace-Id: trace-98421
                │    - X-Tenant-Id: finance-billing
                │
                ▼
┌────────────────────────────────┐
│      REST Catalog Server       │
│  (Parses headers for auditing)  │
└────────────────────────────────┘

With this update, client engines can configure and inject custom headers into every REST call. The client-server handshake follows this sequence:

The client initializes the REST catalog using the properties map.
The client specifies static custom headers using the prefix header.custom.:

   header.custom.X-Tenant-Id=finance-billing
   header.custom.X-Trace-Id=system-trace-99

During request execution, the RESTClient intercepts the HTTP call and injects these custom headers.
The REST catalog server processes the headers to apply dynamic authorization, audit logging, or request routing.

This change enables the following capabilities:

Auditing and Governance: Engines can pass tenant identifiers or user profiles in the HTTP headers, allowing the REST catalog server to log catalog operations with full user context.
Distributed Tracing: Tracing headers such as W3C Trace Context can propagate from client engines through the catalog server, providing end-to-end trace visibility for query planning operations.
Dynamic Authorization: Clients can send custom authorization tokens that the REST catalog server evaluates dynamically to enforce fine-grained access control.

The properties are configured during catalog initialization using the standard configuration map, making it simple to roll out headers across existing query platforms.

Overwrite-Aware Table Registration (PR #15525)

In multi-tenant data platforms, multiple engines frequently access and modify the same table metadata. When registering a new table or importing an existing table state into the catalog, concurrency conflicts can occur.

If two separate processes attempt to register or overwrite a table reference at the same location simultaneously, a naive catalog might register the second request, silently overwriting the first. This creates data inconsistencies where the catalog points to an outdated or incorrect metadata.json file.

PR #15525, written by @sririshindra, adds overwrite-aware table registration to the catalog API.

Writer 1: Commits events_v1 ────► [Catalog Table Pointer] ◄──── Writer 2: Commits events_v2
                                            │
                                            ├────────► If conflict: Catalog rejects Writer 2
                                            └────────► Prevents silent metadata overwrites

This implementation leverages Optimistic Concurrency Control (OCC) at the catalog level. The conflict resolution sequence proceeds as follows:

Writer A and Writer B both read the current table state pointing to snapshot v1.
Writer A writes new data files, generating metadata version metadata_v2.json.
Writer B writes new data files in parallel, generating metadata version metadata_v3.json.
Writer A calls the catalog's /v1/namespaces/db/tables/events/register endpoint, stating that the expected base location is metadata_v1.json.
The catalog verifies the base matches, registers the new pointer to metadata_v2.json, and updates the table version.
Writer B attempts to register its state, listing metadata_v1.json as its expected base.
The catalog detects that the current pointer is now metadata_v2.json.
The catalog rejects Writer B's request, returning a HTTP 409 Conflict. Writer B must re-read the updated table state, resolve any overlapping partition commits, and retry the registration.

This validation ensures that table registration is safe and prevents silent metadata overwrites in highly active environments.

Deletion Vector Pruning in Snapshot Validation (PR #15653)

One of the major highlights of the V3 format specification is the stabilization of deletion vectors. Deletion vectors improve row-level delete performance by replacing positional delete files with Roaring bitmaps. Instead of writing a new delete file for every minor update, the engine updates a binary bitmap linked directly to the data file.

These deletion bitmaps are stored in the Puffin file format. You can inspect active deletion vector locations using metadata system tables:

SELECT file_path, pos, row_position, deletion_vector
FROM TABLE(table_files('my_catalog.schema.events'));

However, as tables grow to hold millions of data files, validating these deletion vectors during query planning can introduce latency. During scan planning, the query engine must ensure that the deletion vectors linked in the metadata are valid and match the corresponding data files.

In earlier versions, this validation was executed across the entire table snapshot during plan initialization. If you had a 50 TB table and queried a single day, the planner still spent time validating deletion vectors for the entire table.

PR #15653, introduced by @anoopj, optimizes this process. It adds manifest partition pruning to deletion vector validation inside the MergingSnapshotProducer.

Query Filter: WHERE event_date = '2026-05-23'
       │
       ▼
Partition Pruning Step
       │
       ├─► Skip Partition '2026-05-22' ──► Skip Deletion Vector Validation
       │
       └─► Read Partition '2026-05-23'  ──► Run Deletion Vector Validation

With this change, the query planner matches the query filter predicates against partition bounds before executing deletion vector checks. If a partition is pruned out, the engine skips validating the deletion vectors for the files in that partition. This change reduces planning CPU cycles and improves scan startup times for partitioned tables.

For a detailed look at how hidden partitioning helps the query engine perform partition pruning and reduce metadata scan sizes, read the Apache Iceberg Hidden Partitioning Post.

Scheduled Credential Lifecycle Refresh (PR #15678, #15732, #15696)

To security-harden data lakehouses, platforms avoid using long-lived storage credentials. Instead, query engines authenticate using temporary tokens vended by the REST catalog or cloud identity providers. These credentials typically have short lifespans, often expiring after one hour.

This security model creates issues for long-running operations. If a massive query runs for 90 minutes, or a streaming Flink sink runs continuously, the temporary credentials expire mid-job. When the client attempts to write new files or fetch manifests after the expiration window, the storage client throws an authentication exception, failing the job.

The 1.11.0 release resolves this lifecycle problem. PR #15678 (by @danielcweeks) and PR #15732 (by @nastra) add scheduled refresh threads to the S3FileIO client. A parallel change in PR #15696 (by @nastra) implements the same capability for GCSFileIO.

Query Thread (Reads/Writes Data)
       │
       ├───────► Token Expiration Approaching (e.g. at 50 minutes)
       │
Background Refresh Thread
       │
       ├───────► Send Request to Catalog ──► Fetch New Credentials
       │
       └───────► Update S3FileIO/GCSFileIO Credentials In-Memory
       │
Query Thread (Continues without interruption)

The credential refresh system runs a background daemon thread that tracks token expiration times. The lifecycle is controlled by the following properties:

Before the active credential expires, the background thread automatically polls the catalog's /v1/tokens endpoint for refreshed tokens and updates the file system client in-memory. The main query and write threads continue to run without interruption, eliminating query failures caused by expired credentials.

Spark Streaming Triggers and Z-Ordering (PR #13824, #15706)

Apache Spark remains the primary engine for heavy write workloads and batch compaction in Iceberg tables. Version 1.11.0 includes several updates to improve Spark streaming and layout optimization.

Trigger.AvailableNow Support (PR #13824, #14026)

PR #13824, introduced by @alexprosak, adds support for the AvailableNow trigger in Spark Structured Streaming. This change was also backported to Spark 4.0, 3.5, and 3.4 in PR #14026.

Continuous Trigger:
[Read Batch 1] -> [Write] -> [Wait] -> [Read Batch 2] -> [Write] -> (Runs indefinitely)

AvailableNow Trigger:
[Scan All Available Data] -> [Process Batch 1] -> [Process Batch 2] -> [Write All] -> [Graceful Shutdown]

In Spark streaming, the default trigger runs continuously in the background, consuming resources even when no new files are arriving. The alternative Once trigger processes only a single batch and shuts down, which can leave data unprocessed if a large backlog has accumulated.

The AvailableNow trigger combines the benefits of both approaches. It scans the source for all available data, splits the workload into consecutive micro-batches, processes them all in a single run, and then shuts down the streaming context. This is configured in PySpark as follows:

# Configure Trigger.AvailableNow with Iceberg source and sink
query = spark.readStream \
    .format("iceberg") \
    .load("prod_catalog.db.events") \
    .writeStream \
    .format("iceberg") \
    .trigger(availableNow=True) \
    .option("checkpointLocation", "/mnt/checkpoints/events") \
    .toTable("prod_catalog.db.events_compacted")

This trigger configuration allows data platforms to run streaming ingestion pipelines as scheduled cron jobs, reducing cluster idle time.

Z-Order Column Collision Validation (PR #15706)

PR #15706, introduced by @YanivZalach, addresses a failure mode during Z-order layout optimization. Spark uses the internal column name ICEZVALUE during Z-order sorting. If a user table already contained a column named ICEZVALUE, the compaction process failed or generated incorrect sort orders.

The update adds strict schema validation that checks for column name collisions before running Z-order compactions, preventing data corruption.

Flink Post-Commit Maintenance and Branch Compaction (PR #15566, #15672, #14148)

Apache Flink is the standard engine for real-time streaming ingestion into Iceberg tables. Streaming ingestion has different write characteristics than batch ingestion, often writing many small files at high frequency. Iceberg 1.11.0 adds features to manage these files directly within Flink pipelines.

Flink Post-Commit Maintenance (PR #15566, #15667)

PR #15566, written by @mxm, adds support for arbitrary post-commit maintenance tasks inside the Flink IcebergSink builder. This is also backported to active Flink branches in PR #15667.

During streaming ingestion, Flink commits data to the Iceberg table at every checkpoint. These frequent commits generate a large number of small manifest files. With the new post-commit interface, you can attach background maintenance tasks directly to the sink:

// Configure Flink sink with post-commit compaction
IcebergSink.forRowData(dataStream, tableLoader)
    .table(icebergTable)
    .tableLoader(tableLoader)
    .writeParallelism(4)
    .distributionMode(DistributionMode.HASH)
    .postCommitMaintenance(
        PostCommitMaintenance.builder()
            .optimizeDataFiles(true)
            .rewriteManifests(true)
            .build()
    )
    .append();

After a commit succeeds, Flink runs compaction and manifest cleaning tasks in the background, keeping the table structure optimized without requiring external scheduler jobs.

Flink Stream Ingestion
       │
       ▼
[Commit Data File (Checkpoint)]
       │
       ├───────► Post-Commit Trigger
       │
       ▼
[Background Maintenance Action (RewriteDataFiles / Compaction)]

Flink Branch Compaction Support (PR #15672, #15690)

PR #15672, also written by @mxm, adds branch support to the Flink RewriteDataFiles action.

Historically, Flink's background compaction actions could only run on the table's main branch. In modern architectures, engines often ingest experimental data or staging runs into separate table branches. Flink can now run file compaction directly on these non-main branches, keeping staging and experiment branches organized before they are merged back.

Flink Metadata Columns (PR #14148)

PR #14148, introduced by @Guosmilesmile, exposes metadata columns to Flink readers.

Flink applications can now read the _row_id and _last_updated_sequence_number system columns. This is useful for CDC (Change Data Capture) reconciliation pipelines that need to track the exact ingestion sequence of rows.

JSON to Variant Mapping and Spec Cleanups (PR #13137, #14045)

The Variant type is a key part of the Iceberg V3 specification, designed to store semi-structured data using a binary representation that supports predicate pushdown. Iceberg 1.11.0 refines this integration across multiple engines.

Variant Type Validation (PR #13137, #14081)

PR #13137 (by @manirajv06) and PR #14081 (by @geruh) add schema validation and filtering rules for the Variant type in Parquet metrics.

These updates ensure that Parquet file readers can extract column-level statistics from nested variant structures. This allows the query engine to prune files based on nested variant fields, improving query performance.

Trino Variant Type Mapping

In parallel, query engine connectors are adopting these changes. Trino now maps its native JSON type to Iceberg's Variant type in V3 tables. This means you can write JSON data from Trino and query it with predicate pushdown, avoiding the performance penalties of plain string JSON.

Positional Deletes with Row Data Deprecated (PR #14045)

PR #14045, written by @pvary, deprecates positional delete files that embed row data.

In Iceberg V2, positional delete files could store the actual deleted row data alongside the file path and row offset. While this design saved a join step during reads, it duplicated data in the delete files, increasing storage costs and metadata complexity.

The community has deprecated this option in favor of Deletion Vectors, simplifying the V3 read path.

Table Upgrade Path and Connector Compatibility

All V3 features: manifest list encryption, deletion vectors, Variant types, geospatial types, and nanosecond timestamps: require upgrading your tables to format version 3.

Existing V2 Table
       │
       ├───────► Run: ALTER TABLE events SET TBLPROPERTIES ('format-version' = '3')
       │
Upgraded V3 Table
       │
       ├───────► New writes use Deletion Vectors and Variant type
       └───────► Existing data files are left untouched (no rewrite required)

The upgrade is a metadata-only operation executed using SQL:

-- Upgrade an existing table to Iceberg V3 format version
ALTER TABLE my_catalog.schema.events
SET TBLPROPERTIES ('format-version' = '3');

This operation updates the format-version pointer in the table's metadata JSON. It does not rewrite your existing data files, which remain in place and continue to be readable.

New writes to the table will adopt V3 features automatically. For example, subsequent update or delete statements will write deletion vectors instead of positional delete files.

Lifecycle Status Updates

Before planning your migration to V3, review the engine compatibility changes in Iceberg 1.11.0:

Java 11 Support Dropped: Iceberg 1.11.0 drops support for Java 11. Core libraries and engine connectors now require Java 17 or Java 21.
Spark 3.4 Support Deprecated: Support for Spark 3.4 is deprecated. Teams should migrate to Spark 3.5 or Spark 4.0+.
Flink 1.19 Support Removed: Flink 1.19 is no longer supported. The release adds support for Flink 2.1.0.

Make sure all query engines and toolchains in your lakehouse deployment support Iceberg V3 and Java 17 before upgrading production tables.

For more on managing query performance optimizations and table format versions inside Dremio, refer to the Dremio Autonomous Performance Blog.

Conclusion

Apache Iceberg 1.11.0 is a significant release for the project. It moves beyond incremental enhancements to deliver major architectural updates.

The unified File Format API restructures how Iceberg interacts with physical storage formats. This change makes it easier to integrate next-generation codecs designed for AI and high-performance workloads.

At the same time, the stabilization of V3 features provides a production-ready path for deletion vectors, Variant data, geospatial types, and nanosecond precision. These features help organizations optimize query performance and reduce operational overhead.

If you are running Iceberg V2 tables in production, evaluate your workloads to identify tables that will benefit from a V3 upgrade. In particular, tables with active update patterns or large JSON columns will see immediate performance gains.

Build Your Data Lakehouse Expertise

If you are designing, building, or managing modern data platforms, staying ahead of formatting specifications is critical. To deepen your understanding of these technologies, consider reading:

"Architecting an Apache Iceberg Lakehouse": An architectural guide to designing open lakehouse platforms, managing catalog architectures, and optimizing table layouts.
Other Data Lakehouse Publications: Practical books covering hidden partitioning, schema evolution, and query acceleration engines.

Find these books and other lakehouse learning resources at books.alexmerced.com.

To query your Iceberg V3 tables with automatic file layout optimization, background compaction, and zero infrastructure management, start a free trial of Dremio Cloud at dremio.com/get-started.

Semantic Layer Best Practices: 7 Mistakes to Avoid

Alex Merced — Fri, 22 May 2026 18:03:06 +0000

Semantic layers don't fail because the technology is wrong. They fail because of design decisions made in the first two weeks — choices that seem reasonable at the time and create compounding problems for months afterward.

Here are the seven mistakes that kill semantic layer projects, and how to avoid each one.

Mistake 1: Defining Metrics in Multiple Places

What happens: Revenue is defined in a Tableau calculated field, a Power BI DAX measure, a dbt model, and a SQL view. Four sources of truth. None of them agree.

Why it's common: Teams adopt new tools without migrating metric definitions. Each tool gets its own model. Over time, the definitions drift.

The fix: Every metric gets exactly one canonical definition in the semantic layer. All downstream tools query that definition. No exceptions. When someone needs Revenue, they query business.revenue, not their own formula.

This principle extends to AI agents. If your AI generates its own metric formulas instead of referencing the semantic layer, you've just added another source of truth — the least trustworthy one.

Mistake 2: Skipping the Bronze Layer

What happens: A data engineer creates a Silver view that joins raw source tables directly, mixing data cleanup (type casting, column renaming) with business logic (filters, calculations) in a single query. When the source schema changes — a column is renamed, a type is modified — the Silver view breaks.

Why it's common: The Bronze layer feels redundant. It's just a 1:1 mapping of the source. Why add a layer that doesn't change anything?

The fix: The Bronze layer absorbs schema changes. When a source renames col_7 to order_date_utc, you update one Bronze view. The Silver and Gold views above it don't change. This insulation is worth the tiny overhead of maintaining passthrough views.

Bronze views also standardize data formats. Timestamps normalized to UTC. Strings cast to consistent encodings. Column names made human-readable. This cleanup happens once, at the bottom of the stack, and every view above benefits.

Mistake 3: Using SQL Reserved Words as Column Names

What happens: A Bronze view exposes a column called Date. Now every downstream query must reference "Date" with double quotes. Analysts forget. AI agents don't quote it at all. Queries break intermittently. Debugging is frustrating because the error messages are cryptic.

Why it's common: Source systems often use generic names. Date, Timestamp, Order, Group, Role — all are SQL reserved words. Bronze views that don't rename them propagate the problem to every consumer.

The fix: Rename early. In the Bronze layer, map Date to TransactionDate, Timestamp to EventTimestamp, Order to CustomerOrder. Use domain-specific prefixes that are unambiguous and never conflict with SQL keywords.

This small decision saves hundreds of hours of debugging across the life of the semantic layer. It also dramatically improves AI agent accuracy, since language models generating SQL rarely add appropriate quoting for reserved words.

Mistake 4: Building Without Stakeholder Input

What happens: A data engineering team builds 50 Silver views based on the database schema. They expose every table, every column, every possible metric. Business users look at the result, don't recognize any of the terms, and go back to their spreadsheets.

Why it's common: Data engineers understand the schema. They assume the schema structure maps to business needs. It usually doesn't.

The fix: Start with a metric glossary co-created with stakeholders from Sales, Finance, Marketing, and Product. Ask them: What are your top 5 metrics? How do you calculate them? What decisions do they drive? Build the Silver layer around those answers, not around the database schema.

This step feels slow. It's the fastest path to adoption. A semantic layer that uses business language and models business concepts gets adopted. A semantic layer that mirrors the database schema gets ignored.

Mistake 5: Treating Documentation as Optional

What happens: Views are created with no Wikis, no column descriptions, no Labels. The semantic layer works for the person who built it. Everyone else — analysts, AI agents, new team members — can't figure out what the views mean.

Why it's common: Documentation takes time. Deadlines are tight. Teams plan to "add documentation later." Later never comes.

The fix: Make documentation part of the view creation process, not a follow-up task. At minimum, every view gets:

A one-sentence description of what it represents
Labels for governance (PII, Finance, Certified)
Column descriptions for any non-obvious field

Modern platforms reduce this burden with AI-generated documentation. Dremio's generative AI samples table data and auto-generates Wiki descriptions and Label suggestions. The AI provides a 70% first draft. The data team adds domain context for the other 30%.

Undocumented views are invisible to AI agents. If the Wiki is empty, the AI agent has no context to generate accurate SQL. Documentation isn't just nice to have. It's an accuracy requirement.

Mistake 6: Applying Security at the BI Tool Level Only

What happens: Row-level security is configured in Tableau so regional managers only see their region. Then an analyst opens a SQL client, queries the underlying table directly, and sees all regions. The security was enforced in the dashboard, not in the data.

Why it's common: BI tools make it easy to apply filters and security rules. Data platforms require more setup. Teams take the easy path.

The fix: Enforce access policies at the semantic layer, not the BI layer. Row-level security and column masking should be applied on the virtual datasets (views). Every query path — dashboard, notebook, API, AI agent — inherits the same rules.

Dremio implements this through Fine-Grained Access Control (FGAC): policies defined as UDFs at the view level. A regional manager queries business.revenue and automatically sees only their region, regardless of how they access the data. No security gaps between tools.

Mistake 7: Trying to Model Everything at Once

What happens: The team commits to building a complete semantic layer covering every source, every table, and every metric. The project takes six months. By the time it launches, requirements have changed, stakeholder interest has waned, and half the views are out of date.

Why it's common: Ambitious leaders want a "complete" solution. Data teams want to avoid rework. Neither wants to ship an incomplete layer.

The fix: Start with 3-5 core metrics that the organization actively debates (usually Revenue, Active Users, Churn). Build one Bronze → Silver → Gold pipeline per metric. Validate that the same question produces the same answer across two different tools.

Once those metrics are stable, expand incrementally. Add new sources, new views, new metrics — one at a time. Each addition is low-risk because the layered architecture isolates changes. A new Gold view doesn't affect existing Silver views.

The fastest semantic layers reach 80% organizational coverage not by modeling everything up front, but by proving value quickly and expanding from momentum.

What to Do Next

Pick one mistake from this list. Check whether your semantic layer (or your plan for one) is making it. Fix that one thing this week. Then come back for the next one.

Try Dremio Cloud free for 30 days

How a Self-Documenting Semantic Layer Reduces Data Team Toil

Alex Merced — Fri, 22 May 2026 17:57:43 +0000

Every data team knows documentation is important. And almost every data team has a backlog of undocumented tables, unlabeled columns, and outdated descriptions that nobody has time to fix. The problem isn't motivation. It's that manual documentation doesn't scale.

A self-documenting semantic layer changes the equation. Instead of asking humans to describe every column in every table, the platform generates descriptions automatically, suggests governance labels from data patterns, and propagates context through the view chain. Documentation becomes a byproduct of building the semantic layer, not a separate project.

The Documentation Problem Nobody Solves

Industry surveys consistently find that 70% or more of enterprise data assets are undocumented or poorly documented. The result: analysts spend 30-40% of their time searching for data and trying to understand what it means before they can start analyzing it.

This isn't just a productivity problem. Undocumented data is a governance risk. A column named status with values 0, 1, 2, and 3 could mean anything. An analyst guesses. An AI agent guesses worse. Nobody verifies. The wrong assumptions get baked into dashboards that drive business decisions.

Data teams respond with documentation sprints. They burn a week writing Wiki pages for their top 50 tables. Two months later, half the descriptions are outdated because schemas have changed. The cycle repeats.

What Self-Documenting Actually Means

A self-documenting semantic layer generates and maintains documentation with minimal human effort. Three mechanisms work together:

AI-generated descriptions: The platform samples data in a table and generates human-readable descriptions for each column and the table itself.

Automated label suggestions: The platform analyzes column names, data types, and value patterns to suggest governance labels (PII, Finance, Certified).

Metadata propagation: When a Silver view references a Bronze view, column descriptions flow downstream automatically. Documentation written once at the Bronze level appears everywhere the column is used.

Human oversight is still essential. AI provides a 70% first draft. Data engineers add the domain-specific context that only they know: business rules, edge cases, known data quality issues. The point isn't to eliminate human documentation. It's to eliminate the blank page.

AI-Generated Descriptions

Modern semantic layer platforms can sample a table's data and generate meaningful descriptions automatically.

Consider a column named cltv in a table called customers. The AI samples values (1200.50, 3400.00, 780.25), examines the column name and table context, and generates:

cltv: Customer Lifetime Value in USD. Represents the total revenue attributed to this customer from their first purchase to the current date, excluding refunded transactions.

Not every generated description will be this precise. But most are useful enough to replace the current state: an empty description that tells the analyst nothing.

More examples:

A column with values "US", "UK", "DE" → "ISO 3166 alpha-2 country code for the customer's billing address"
A DATE column named created_at in a subscriptions table → "Date the subscription was created"
A FLOAT column named mrr → "Monthly Recurring Revenue in the account's base currency"

Automated Label Suggestions

Labels categorize data for governance and discovery. Manually tagging every column in a data warehouse with hundreds of tables is impractical. AI-based label suggestion makes it manageable:

Columns containing email-like patterns (text with @ symbols) → suggested label: PII
Columns with phone number patterns → suggested label: PII
Columns named price, total, amount, revenue → suggested label: Finance
Columns in tables marked "Certified" → suggested label propagated to downstream views

Dremio's approach combines these suggestions with human approval. The AI proposes labels. A data engineer reviews and accepts or rejects. Over time, the catalog fills up with accurate, useful labels without dedicated labeling sprints.

Metadata Propagation Through Views

In a well-designed semantic layer, documentation shouldn't need to be written more than once. The Bronze-Silver-Gold view architecture creates a natural propagation path:

Bronze layer: Document the CustomerID column as "Unique identifier for the customer, sourced from the CRM system."
Silver layer: A Silver view references CustomerID. The description propagates automatically. No re-documentation needed.
Gold layer: An aggregated Gold view groups by CustomerID. The description carries through.

This propagation is especially valuable for join columns, filter columns, and commonly used dimensions that appear in dozens of views. Write the description once at the source, and it follows the column everywhere.

How This Reduces Toil

The impact on data team productivity is measurable:

Documentation Task	Manual Approach	Self-Documenting
Column descriptions	Write each by hand	AI generates draft, human refines
Governance labels	Manual tagging sprint	AI suggests from data patterns
Downstream view docs	Re-write for each view	Propagated from upstream
Schema change updates	Manually check and update	AI re-scans and flags changes
New table onboarding	Create from scratch	AI generates baseline immediately

The net effect: documentation coverage goes from 30% (what the team could manage manually) to 80-90% (AI baseline + human refinement). The team spends hours instead of weeks on documentation. And the documentation stays current because the AI can re-scan when schemas change — flagging outdated descriptions instead of waiting for someone to notice.

For AI agents, this improvement is material. A richer, more accurate semantic layer means the AI generates better SQL, hallucinates less, and requires fewer corrections. Self-documentation isn't just a productivity feature. It's an AI accuracy feature.

What to Do Next

Pick your most-used table. Open it in your data platform. How many columns have descriptions? How many have governance labels? If the answer is "not many," calculate how long it would take to document the entire table manually. Then consider a platform that does 70% of that work for you.

Try Dremio Cloud free for 30 days

Migrating to Apache Iceberg: Strategies for Every Source System

Alex Merced — Fri, 22 May 2026 17:48:08 +0000

This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern.

Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, and Parquet directories. Moving this data to Iceberg is not an all-or-nothing project. The best migrations happen incrementally, one dataset at a time, with no disruption to existing consumers.

Three Migration Strategies

1. In-Place Migration (Metadata Only)

In-place migration creates Iceberg metadata over existing Parquet or ORC files without copying or moving them. The data files stay exactly where they are; only new Iceberg metadata is created to track them.

Spark example:

CALL system.migrate('db.existing_hive_table')

This converts a Hive table to Iceberg by scanning its files and creating the Iceberg metadata tree (metadata.json, manifest list, manifest files) that references them. The Parquet files are untouched.

Pros: Fast. No data movement. The table becomes queryable as Iceberg immediately.

Cons: The existing file layout (sizes, partitioning, sort order) is inherited. If the original files are poorly organized, you inherit those problems. Requires the original files to be in Parquet or ORC format.

2. Full Rewrite (CTAS)

A full rewrite reads data from any source and writes it as a new Iceberg table with optimal partitioning and file sizes:

-- Spark
CREATE TABLE iceberg_catalog.analytics.orders
USING iceberg
PARTITIONED BY (day(order_date))
AS SELECT * FROM hive_catalog.legacy.orders

-- Dremio
CREATE TABLE analytics.orders
PARTITION BY (day(order_date))
AS SELECT * FROM legacy_source.public.orders

Pros: Best result. Optimal file sizes, correct sort order, proper partitioning. The table is perfectly organized from day one.

Cons: Requires reading and writing all data, which takes time and compute resources. The source system must be available during the migration.

3. Shadow Migration (Build and Swap)

Shadow migration builds the Iceberg table alongside the existing source, then swaps consumers from old to new when ready:

Create a new Iceberg table with the desired schema and partitioning
Backfill historical data from the legacy source
Set up incremental sync to keep the Iceberg table current
Validate data quality between old and new
Swap consumer views from legacy to Iceberg

Pros: Zero downtime. Consumers never see a disruption. You can validate the migration before committing to it.

Cons: Temporarily doubles storage costs. Requires maintaining two copies during the transition.

Choosing the Right Strategy

Source	Recommended Strategy
Hive table (Parquet files)	In-place migration, then compact
Data warehouse (Snowflake, Redshift)	Full rewrite via Dremio federation
CSV/JSON files in S3	Full rewrite with COPY INTO
PostgreSQL/MySQL	Full rewrite or shadow migration
Delta Lake tables	In-place conversion or rewrite
Production system (no downtime)	Shadow migration with view swap

The View Swap Pattern

The view swap pattern is the recommended approach for production migrations. It uses Dremio's semantic layer to create an abstraction between consumers and the underlying data:

Phase 1: Federation

Create views in Dremio that point to the legacy data source:

CREATE VIEW analytics.orders AS
SELECT order_id, customer_id, order_date, amount, status, region
FROM postgres_source.public.orders

All consumers (dashboards, reports, notebooks) query through these views. They do not know or care where the data physically lives.

Phase 2: Build Iceberg

Create and populate the Iceberg table:

-- Create the Iceberg table
CREATE TABLE iceberg_data.analytics.orders (
    order_id BIGINT, customer_id BIGINT,
    order_date DATE, amount DECIMAL(10,2),
    status VARCHAR, region VARCHAR
) PARTITION BY (day(order_date))

-- Backfill from the legacy source
INSERT INTO iceberg_data.analytics.orders
SELECT * FROM postgres_source.public.orders

Phase 3: Validate

Compare the two datasets to confirm data integrity:

SELECT
  (SELECT COUNT(*) FROM postgres_source.public.orders) AS legacy_count,
  (SELECT COUNT(*) FROM iceberg_data.analytics.orders) AS iceberg_count

Beyond row counts, validate aggregates (total amounts, distinct customer counts) and spot-check individual records. A comprehensive validation script should compare:

Total row count
Column-level checksums or hash aggregates
Distinct value counts for key columns
Boundary values (MIN/MAX) for numeric and date columns
Sample of specific records matched by primary key

Only proceed to the swap after all validation checks pass.

Phase 4: Swap

Update the view to point to the Iceberg table:

CREATE OR REPLACE VIEW analytics.orders AS
SELECT order_id, customer_id, order_date, amount, status, region
FROM iceberg_data.analytics.orders

Consumers notice nothing. The view name is the same. The query interface is the same. But now the data is served from Iceberg with all of its advantages: time travel, hidden partitioning, metadata-driven pruning, and automatic optimization.

Migrating One Table at a Time

The view swap pattern enables incremental migration. You do not need to migrate everything at once:

Week 1: Migrate the highest-value table (e.g., orders)
Week 2: Migrate the next table (e.g., customers)
Continue until all critical tables are on Iceberg

During the transition, Dremio's federation queries legacy and Iceberg tables together. A join between a PostgreSQL table and an Iceberg table works the same as a join between two Iceberg tables. The migration is invisible to consumers.

Post-Migration Checklist

After migrating each table:

Run OPTIMIZE TABLE to ensure optimal file sizes
Set up automatic optimization through Dremio Open Catalog
Add wikis and tags for the AI agent
Verify metadata table health checks
Decommission the legacy source after the retention period

Common Migration Pitfalls

Migrating without testing query performance: Always benchmark critical queries against the new Iceberg table before switching production traffic. Iceberg's partition layout and file organization affect performance, and a migration can make some queries faster but others slower if the partition strategy is wrong.

Skipping the validation phase: Data discrepancies between the old and new systems are more common than expected. Schema differences, timezone handling, null semantics, and data type precision can all cause subtle mismatches. Validate thoroughly.

Migrating everything at once: Large "big bang" migrations carry high risk. If something goes wrong, rolling back is complex and time-consuming. Migrate one table at a time, validate each one, and build confidence incrementally.

This completes the Apache Iceberg Masterclass. The series covered table formats, metadata, performance, partitioning, writes, catalogs, maintenance, tooling, and migration. For hands-on practice, start a Dremio Cloud trial and follow the workflow in Part 14.

Books to Go Deeper

Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)
Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced
Constructing Context: Semantics, Agents, and Embeddings by Alex Merced
Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced
Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

Free Resources

Hands-On with Apache Iceberg Using Dremio Cloud

Alex Merced — Fri, 22 May 2026 17:19:31 +0000

This is Part 14 of a 15-part Apache Iceberg Masterclass. Part 13 covered streaming approaches. This article is a practical walkthrough of working with Iceberg on Dremio Cloud, covering table creation, data ingestion, optimization, semantic layer construction, and AI-powered analytics.

Getting Started

Step 1: Sign Up and Connect Storage

Create a Dremio Cloud account (free trial available)
Add a cloud storage source (S3, ADLS, or GCS) through the Sources panel
Configure credentials and target bucket

Dremio creates an Open Catalog for your Iceberg tables automatically. This Polaris-based catalog handles metadata management, access control, and automatic optimization.

Step 2: Create Iceberg Tables

CREATE TABLE analytics.orders (
    order_id BIGINT,
    customer_id BIGINT,
    order_date DATE,
    amount DECIMAL(10,2),
    status VARCHAR,
    region VARCHAR
)
PARTITION BY (day(order_date))

This creates a table with hidden partitioning by day. Users query on order_date naturally; the engine handles partition pruning automatically.

Step 3: Ingest Data

From files in object storage:

COPY INTO analytics.orders
FROM '@my_s3_source/raw/orders/'
FILE_FORMAT 'parquet'

From another table or source:

INSERT INTO analytics.orders
SELECT * FROM postgres_source.public.orders
WHERE order_date >= '2024-01-01'

Dremio's federation can query data in PostgreSQL, MySQL, Oracle, MongoDB, S3 files, and other sources directly. You can migrate data into Iceberg tables with a single INSERT...SELECT statement.

The Dremio Platform

Columnar Cloud Cache

Dremio's Columnar Cloud Cache (C3) stores frequently accessed Iceberg data on local NVMe SSDs attached to the query engine nodes. When a query accesses data for the first time, Dremio caches the relevant columns locally. Subsequent queries against the same data read from local SSD instead of remote object storage, reducing latency from hundreds of milliseconds to single-digit milliseconds.

C3 operates transparently. You do not need to configure which data to cache. Dremio tracks access patterns and caches the most-queried data automatically.

Connecting BI Tools

Dremio exposes Iceberg data through ODBC, JDBC, and Arrow Flight endpoints. Any BI tool (Tableau, Power BI, Looker, Superset) can connect to Dremio and query Iceberg tables as if they were a traditional database. The semantic layer ensures consistent governance and naming across all connected tools.

Semantic Layer

Dremio's semantic layer lets you create governed SQL views that serve as the interface between raw data and consumers:

CREATE VIEW analytics.customer_orders AS
SELECT
    o.customer_id,
    c.customer_name,
    c.region,
    SUM(o.amount) AS total_spend,
    COUNT(*) AS order_count
FROM analytics.orders o
JOIN analytics.customers c ON o.customer_id = c.customer_id
GROUP BY o.customer_id, c.customer_name, c.region

Add wikis and tags to views and tables through the Dremio UI. These descriptions help other users find and understand data, and they power the AI agent's ability to generate accurate SQL from natural language.

Reflections (Query Acceleration)

Dremio Reflections are precomputed materializations that automatically accelerate queries without requiring changes to your SQL. When you create a reflection on a view or table, Dremio precomputes the results and stores them as optimized Iceberg tables on fast storage:

-- Create an aggregation reflection for fast dashboard queries
ALTER TABLE analytics.customer_orders
  CREATE AGGREGATE REFLECTION customer_orders_agg
  USING DIMENSIONS (region, order_date)
  MEASURES (total_spend SUM, order_count SUM)

When a query matches the reflection's definition, Dremio serves it from the precomputed data instead of scanning the full table. Queries that take 30 seconds against raw data can complete in under 1 second with reflections. The query optimizer chooses the reflection transparently, so users and applications do not need to know reflections exist.

Data Governance

Dremio provides column-level access control and row-level filtering directly in the semantic layer:

-- Create a view that masks PII for non-privileged users
CREATE VIEW analytics.orders_masked AS
SELECT
    order_id,
    CASE WHEN is_member('finance_team') THEN customer_name
         ELSE '***MASKED***' END AS customer_name,
    order_date,
    amount
FROM analytics.orders

Governance policies defined in the semantic layer apply consistently regardless of which tool (BI dashboard, Python notebook, AI agent) queries the data. This approach is more maintainable than duplicating access policies in every consuming application.

Query Federation

One of Dremio's unique capabilities is querying Iceberg tables alongside data in other systems:

-- Join Iceberg table with a PostgreSQL table
SELECT i.order_id, i.amount, p.payment_status
FROM analytics.orders i
JOIN postgres_source.public.payments p
ON i.order_id = p.order_id

This eliminates the need to move all data into Iceberg before you can query it. You can start with federation and migrate incrementally. Federation is especially useful during migration: query legacy systems and Iceberg tables side by side, then swap the underlying source when you are ready.

Essential SQL Operations

Table Optimization

-- Compact small files
OPTIMIZE TABLE analytics.orders REWRITE DATA USING BIN_PACK

-- Compact with sorting for better file skipping
OPTIMIZE TABLE analytics.orders REWRITE DATA USING SORT (order_date, customer_id)

-- Expire old snapshots
ALTER TABLE analytics.orders EXPIRE SNAPSHOTS OLDER_THAN = '2024-04-01 00:00:00'

For tables managed by Open Catalog, Dremio runs automatic table optimization in the background, handling compaction, expiry, and orphan cleanup without user intervention.

Time Travel

-- Query the table as of a specific timestamp
SELECT * FROM analytics.orders
AT TIMESTAMP '2024-03-01 00:00:00'

-- Compare current data to a previous snapshot
SELECT
    current_data.region,
    current_data.total - old_data.total AS growth
FROM (SELECT region, SUM(amount) AS total FROM analytics.orders GROUP BY region) current_data
JOIN (
    SELECT region, SUM(amount) AS total
    FROM analytics.orders AT TIMESTAMP '2024-01-01'
    GROUP BY region
) old_data ON current_data.region = old_data.region

Metadata Inspection

-- Check table health
SELECT AVG(file_size_in_bytes)/1048576 AS avg_mb, COUNT(*) AS files
FROM TABLE(table_files('analytics.orders'))

-- Review recent snapshots
SELECT committed_at, operation, summary
FROM TABLE(table_snapshot('analytics.orders'))
ORDER BY committed_at DESC LIMIT 5

AI-Powered Analytics

Dremio's built-in AI agent converts natural language questions into SQL queries using the semantic layer's wikis and tags as context:

"Show me the top 10 customers by total spend this quarter"
"What was the month-over-month revenue growth by region?"
"Which products had the highest return rate last month?"

The AI agent generates standard SQL, meaning the results are transparent and auditable. Users can see exactly what SQL was generated, verify it, and refine it. This is different from black-box AI analytics tools that hide the underlying logic.

MCP Server for External AI Agents

The MCP Server extends Dremio's data access to external AI agents and tools through the Model Context Protocol. LLMs running in Claude, ChatGPT, or custom agent frameworks can query your Iceberg lakehouse through MCP, inheriting all the governance, semantic context, and optimization that Dremio provides.

This positions Dremio as the data layer for agentic AI workflows: the AI agent asks questions in natural language, MCP translates them into governed SQL, and Dremio returns the results from optimized Iceberg tables.

Part 15 covers strategies for migrating existing data into Iceberg.

Books to Go Deeper

Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)
Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced
Constructing Context: Semantics, Agents, and Embeddings by Alex Merced
Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced
Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

Free Resources

Approaches to Streaming Data into Apache Iceberg Tables

Alex Merced — Fri, 22 May 2026 16:53:14 +0000

This is Part 13 of a 15-part Apache Iceberg Masterclass. Part 12 covered Python and MPP engines. This article covers the three primary approaches to streaming data into Iceberg tables and the operational trade-offs each creates.

Iceberg was designed for batch analytics, but most production data arrives continuously. Streaming ingestion bridges this gap by committing data to Iceberg tables at regular intervals. The challenge is that frequent commits create the small file problem, and managing that trade-off between data freshness and table health is the central concern of streaming to Iceberg.

Three Streaming Architectures

Spark Structured Streaming

Spark Structured Streaming processes data in micro-batches and commits to Iceberg at configurable intervals:

df = spark.readStream.format("kafka") \
    .option("subscribe", "events") \
    .load()

df.writeStream.format("iceberg") \
    .outputMode("append") \
    .option("checkpointLocation", "s3://checkpoint/events") \
    .trigger(processingTime="60 seconds") \
    .toTable("analytics.events")

Each trigger creates a new Iceberg commit with the accumulated data. A 60-second trigger produces 1,440 commits per day, each adding a small number of files.

Latency: Seconds to minutes (configurable via trigger interval).
Small file impact: Moderate. Longer trigger intervals produce fewer, larger files.
Best for: Teams already using Spark for batch processing who want to add near-real-time ingestion.

Apache Flink Iceberg Sink

Flink processes events continuously and commits to Iceberg at checkpoint intervals:

-- Flink SQL
INSERT INTO iceberg_catalog.analytics.events
SELECT event_id, event_time, payload
FROM kafka_source

Flink's checkpointing mechanism determines commit frequency. A 30-second checkpoint interval produces commits every 30 seconds with whatever data has accumulated.

Exactly-once semantics: Flink's checkpoint mechanism provides exactly-once delivery guarantees to Iceberg. If a Flink job crashes, it recovers from its last checkpoint and replays any data that was not yet committed to Iceberg. This means no duplicate records and no data loss, which is critical for financial and transactional data pipelines.

Partitioned writes: Flink can route events to partitions dynamically based on partition transforms. Combined with Iceberg's hidden partitioning, this means streaming data lands in the correct partition directory automatically without any special logic in the streaming application.

Upserts and CDC: Flink supports changelog streams (insert, update, delete operations) and can write them to Iceberg as equality deletes and data files. This enables CDC (change data capture) patterns where a database's transaction log is streamed directly into an Iceberg table, maintaining a near-real-time copy.

Latency: Seconds (tied to checkpoint interval).
Small file impact: High. Frequent checkpoints produce many small files.
Best for: Teams needing the lowest-latency streaming with exactly-once semantics and CDC support.

Kafka Connect Iceberg Sink

The Iceberg Sink Connector reads directly from Kafka topics and writes to Iceberg tables:

{
  "name": "iceberg-sink",
  "config": {
    "connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
    "topics": "events",
    "iceberg.catalog.type": "rest",
    "iceberg.catalog.uri": "https://catalog.example.com",
    "iceberg.tables": "analytics.events"
  }
}

Latency: Minutes (Kafka Connect batches records before committing).
Small file impact: Lower than Spark/Flink because commits are less frequent.
Best for: Organizations with existing Kafka infrastructure that want a managed connector approach.

Apache Iceberg Sink Connector: The community-maintained Iceberg Sink Connector for Kafka Connect supports schema evolution from Kafka's Schema Registry, automatic table creation, and partition routing. It reads records from Kafka topics, buffers them in memory, and commits to Iceberg in configurable batch intervals.

Operational simplicity: Kafka Connect is a managed framework. You deploy the connector configuration, and Kafka Connect handles scaling, offset management, and fault recovery. There is no custom application code to write or maintain. For organizations that already run Kafka Connect for other sinks (databases, search indexes), adding an Iceberg sink is straightforward.

The Streaming + Compaction Cycle

Every streaming approach shares the same fundamental problem: frequent commits produce small files. The solution is to pair streaming ingestion with aggressive compaction.

A typical production pattern:

Stream data in via Flink or Spark with 60-second commit intervals
Run compaction every hour to merge small files from the last hour into optimally-sized files
Expire snapshots daily to clean up the accumulated snapshot metadata

Dremio's automatic table optimization handles this compaction automatically for tables managed by Open Catalog. AWS S3 Tables also provides built-in compaction for streaming workloads.

The Latency vs. Maintenance Trade-off

The key insight: you do not always need sub-second latency. Most dashboards refresh every 5-15 minutes. If your consumers can tolerate 5-minute data freshness, using a 5-minute trigger interval produces 90% fewer small files and dramatically reduces compaction overhead.

Production Streaming Architecture

A production streaming-to-Iceberg pipeline typically includes four components:

Message queue (Kafka, Kinesis, Pulsar): Buffers events from source systems
Stream processor (Flink, Spark Streaming): Transforms and writes to Iceberg
Compaction service (Dremio auto-optimization, Spark scheduled jobs): Merges small files on a recurring schedule
Monitoring (metadata tables): Tracks file counts, sizes, and commit frequency

The most common mistake in streaming Iceberg architectures is deploying the stream processor without the compaction service. Without compaction, query performance degrades within days. Always deploy both together.

Choosing the Right Approach

Monitoring Streaming Health

After deploying a streaming pipeline, monitor these metrics daily using metadata tables:

Commit frequency: How many snapshots are being created per hour?
Average file size: Is the small file problem growing?
Compaction lag: Are compaction jobs keeping up with the write rate?
End-to-end latency: How long between an event occurring and it being queryable in Iceberg?

A well-tuned streaming pipeline commits every 1-5 minutes, produces files of 32-128 MB per commit, and has compaction running every 30-60 minutes to consolidate the small files into 256 MB targets.

Part 14 provides a hands-on walkthrough of Iceberg on Dremio Cloud.

Books to Go Deeper

Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)
Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced
Constructing Context: Semantics, Agents, and Embeddings by Alex Merced
Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced
Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

Free Resources

Using Apache Iceberg with Python and MPP Query Engines

Alex Merced — Fri, 22 May 2026 16:35:48 +0000

This is Part 12 of a 15-part Apache Iceberg Masterclass. Part 11 covered metadata tables. This article covers the two main ways to access Iceberg data: directly from Python libraries and through MPP (massively parallel processing) query engines.

The Python Ecosystem for Iceberg

PyIceberg: Native Python Access

PyIceberg is the official Python library for Apache Iceberg. It reads Iceberg metadata directly and can scan data files without an external query engine.

from pyiceberg.catalog import load_catalog

# Connect to a REST catalog
catalog = load_catalog("my_catalog", **{
    "type": "rest",
    "uri": "https://catalog.example.com",
})

# Load and scan a table
table = catalog.load_table("analytics.orders")
scan = table.scan(row_filter="amount > 100")
df = scan.to_pandas()

PyIceberg leverages Iceberg's metadata-driven pruning: the row_filter is pushed down to manifest evaluation, so only relevant data files are read. For reading subsets of large tables into Python for analysis or ML training, this is remarkably efficient.

PyIceberg also supports writes (appending data from Arrow tables), schema evolution, and table management operations. It connects to any catalog that implements the REST protocol, including Dremio Open Catalog.

DuckDB: SQL-Based Python Analysis

DuckDB can read Iceberg tables through its Iceberg extension:

import duckdb

conn = duckdb.connect()
conn.execute("INSTALL iceberg; LOAD iceberg;")

df = conn.execute("""
    SELECT customer_id, SUM(amount) as total
    FROM iceberg_scan('s3://warehouse/orders')
    GROUP BY customer_id
""").fetchdf()

DuckDB processes the query locally using its columnar execution engine, which is significantly faster than pandas for analytical queries. It supports Iceberg's partition pruning and column statistics for file skipping. DuckDB runs entirely in-process, so there is no separate server to manage. This makes it a strong choice for local analysis, CI/CD data validation, and notebooks where starting a Spark cluster would be overkill.

DuckDB also supports reading Iceberg metadata tables, which means you can use it for table health diagnostics without standing up a full query engine.

Polars: High-Performance DataFrames

Polars can read Iceberg tables through its scan_iceberg method, providing lazy evaluation and parallel processing:

import polars as pl

df = pl.scan_iceberg("s3://warehouse/orders").filter(
    pl.col("amount") > 100
).collect()

Polars uses a lazy evaluation model: the scan_iceberg call does not read data immediately. Instead, it builds an execution plan. When collect() is called, Polars optimizes the plan (predicate pushdown, column pruning, parallel reads) and executes it. For large Iceberg tables, Polars can scan data several times faster than pandas because it uses all available CPU cores and processes data in Apache Arrow columnar format.

Writing from Python

PyIceberg supports writes through Apache Arrow tables:

import pyarrow as pa

# Create an Arrow table with new data
new_data = pa.table({
    "order_id": [1001, 1002, 1003],
    "amount": [150.00, 275.50, 89.99],
    "order_date": ["2024-03-15", "2024-03-15", "2024-03-16"],
})

# Append to the Iceberg table
table.append(new_data)

This creates a new Iceberg commit with the data files, manifests, and metadata. PyIceberg handles the entire write lifecycle, including partition assignment based on the table's partition spec.

For bulk writes from Python, using PyIceberg with Arrow is often simpler than setting up Spark. However, PyIceberg runs on a single machine, so it is not suitable for writing terabyte-scale datasets. For that, use an MPP engine.

MPP Query Engines

For production workloads at scale, Python libraries running on a single machine are not sufficient. MPP engines distribute query execution across multiple nodes, handling petabyte-scale tables with sub-minute response times.

Dremio

Dremio provides full Iceberg support with several unique capabilities: query federation across Iceberg and non-Iceberg sources, automatic table optimization through Open Catalog, a semantic layer for governed access, and AI-powered analytics through its built-in agent and MCP server.

For Python users, Dremio exposes data through Apache Arrow Flight, which is a high-performance data transfer protocol. Arrow Flight sends data in columnar Arrow format directly to the client, avoiding the serialization overhead of JDBC/ODBC. This makes it 10-100x faster than traditional database connectors for large result sets:

from dremio_simple_query import DremioConnection

conn = DremioConnection("https://your-dremio.cloud", token="...")
df = conn.query("SELECT * FROM analytics.orders WHERE amount > 100")

The result is a pandas DataFrame populated via Arrow Flight. Because the data stays in Arrow format end-to-end (Iceberg Parquet to Dremio to Arrow Flight to pandas), there are no format conversion bottlenecks.

Dremio also provides a Columnar Cloud Cache that stores frequently accessed data on local NVMe drives, making subsequent queries against the same Iceberg data dramatically faster without requiring reflections or materialized views.

Spark

Apache Spark is the most mature Iceberg engine for both reads and writes. It handles batch ETL, streaming ingestion (Part 13), and all maintenance operations. Most Iceberg production pipelines use Spark for data ingestion because of its extensive connector ecosystem (Kafka, JDBC, file formats) and its ability to process large volumes across a distributed cluster.

Spark supports all Iceberg operations: CREATE, INSERT, MERGE, DELETE, UPDATE, schema evolution, partition evolution, and every maintenance procedure (compaction, snapshot expiry, orphan cleanup).

Trino

Trino (formerly PrestoSQL) is optimized for interactive, ad-hoc queries with low latency. It reads and writes Iceberg tables and supports the REST catalog protocol. Trino is popular for exploration and dashboarding workloads where sub-second response times matter and data is being read rather than written. Its architecture keeps no persistent state, making it easy to scale up and down based on query demand.

Other Engines

Several other engines provide Iceberg support: AWS Athena (serverless, AWS-native), Snowflake (read-only for external Iceberg tables), StarRocks (sub-second analytics), and Doris (real-time analytics). The Iceberg community maintains a compatibility matrix showing which engines support which operations.

Choosing the Right Approach

The key takeaway: Python libraries (PyIceberg, DuckDB, Polars) are best for local analysis and development. MPP engines (Dremio, Spark, Trino) are necessary for production-scale analytics. Many teams use both: PyIceberg for data science experimentation, and Dremio for production dashboards and governed access.

Part 13 covers how to stream data into Iceberg tables.

Books to Go Deeper

Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)
Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced
Constructing Context: Semantics, Agents, and Embeddings by Alex Merced
Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced
Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

Free Resources

Apache Iceberg Metadata Tables: Querying the Internals

Alex Merced — Fri, 22 May 2026 15:45:10 +0000

This is Part 11 of a 15-part Apache Iceberg Masterclass. Part 10 covered maintenance operations. This article covers the metadata tables that let you inspect Iceberg table internals using standard SQL.

Iceberg exposes its internal metadata as queryable virtual tables. You can use them to check table health, debug performance issues, audit changes, and build monitoring dashboards. No special tools required, just SQL.

The Seven Metadata Tables

Snapshots

The $snapshots table lists every snapshot in the table's history. Each row represents a committed transaction.

-- Dremio syntax
SELECT * FROM TABLE(table_snapshot('analytics.orders'))

-- Spark syntax
SELECT * FROM analytics.orders.snapshots

Key columns: snapshot_id, committed_at, operation (append, overwrite, delete), summary (files added/removed counts).

History

The $history table shows the timeline of which snapshot was current at each point in time.

SELECT * FROM TABLE(table_history('analytics.orders'))

Files

The $files table lists every data file in the current snapshot with detailed statistics.

SELECT file_path, file_size_in_bytes, record_count, partition
FROM TABLE(table_files('analytics.orders'))

This is the primary diagnostic table for checking file sizes and identifying the small file problem.

Manifests

The $manifests table lists the manifest files for the current snapshot.

SELECT path, length, added_data_files_count, existing_data_files_count
FROM TABLE(table_manifests('analytics.orders'))

Partitions

The $partitions table provides statistics per partition: row counts, file counts, and size.

SELECT partition, record_count, file_count
FROM TABLE(table_partitions('analytics.orders'))

Practical Use Cases

Monitoring: Average File Size

SELECT
  AVG(file_size_in_bytes) / 1048576 AS avg_file_mb,
  MIN(file_size_in_bytes) / 1048576 AS min_file_mb,
  COUNT(*) AS total_files
FROM TABLE(table_files('analytics.orders'))

If avg_file_mb drops below 64, schedule compaction.

Debugging: Files Per Partition

SELECT partition, COUNT(*) AS files, SUM(record_count) AS rows
FROM TABLE(table_files('analytics.orders'))
GROUP BY partition
ORDER BY files DESC
LIMIT 20

Partitions with hundreds of files are compaction candidates. Use this query as a daily health check and pipe the results into your monitoring system.

Debugging: Sort Order Effectiveness

Column statistics in the files table reveal whether your sort order is effective:

SELECT
  file_path,
  lower_bounds['customer_id'] AS min_customer_id,
  upper_bounds['customer_id'] AS max_customer_id
FROM TABLE(table_files('analytics.orders'))

If the min/max ranges overlap heavily across files, the sort order has decayed and compaction with sorting (Part 10) will restore effectiveness.

Monitoring: Commit Velocity

Track how frequently the table is being written to:

SELECT
  DATE_TRUNC('hour', committed_at) AS hour,
  COUNT(*) AS commits,
  SUM(CAST(summary['added-data-files'] AS INT)) AS files_added
FROM TABLE(table_snapshot('analytics.orders'))
WHERE committed_at > CURRENT_TIMESTAMP - INTERVAL '24' HOUR
GROUP BY DATE_TRUNC('hour', committed_at)
ORDER BY hour

High commit velocity (hundreds of commits per hour) indicates a streaming workload that needs aggressive compaction.

Auditing: Recent Changes

SELECT committed_at, operation, summary
FROM TABLE(table_snapshot('analytics.orders'))
ORDER BY committed_at DESC
LIMIT 10

This shows the last 10 operations: how many files were added or removed per commit.

Time Travel

Metadata tables enable time travel queries. Use the snapshot list to find the snapshot ID for a specific point in time, then query the table at that snapshot:

-- Query the table as it existed on February 15
SELECT * FROM analytics.orders
AT SNAPSHOT '1234567890123456789'

-- Or by timestamp
SELECT * FROM analytics.orders
AT TIMESTAMP '2024-02-15 00:00:00'

Time travel is useful for debugging data issues ("what did this table look like before yesterday's pipeline ran?"), auditing ("what was the account balance at end-of-quarter?"), and reproducible analysis ("run this report against last month's data").

Incremental Reads

Metadata tables also enable incremental processing. By comparing two snapshots, you can identify which files were added between them and process only the new data:

-- Find files added in the last snapshot
SELECT file_path, record_count
FROM TABLE(table_files('analytics.orders'))
WHERE file_path NOT IN (
  SELECT file_path FROM TABLE(table_files('analytics.orders'))
  AT SNAPSHOT '1234567890'
)

This pattern is the foundation for CDC (Change Data Capture) on Iceberg tables: read only what changed since the last processing run, rather than re-scanning the entire table.

Rollback

If a bad write corrupts your table, use the snapshot list to rollback:

-- Find the last good snapshot
SELECT snapshot_id, committed_at, operation
FROM TABLE(table_snapshot('analytics.orders'))
ORDER BY committed_at DESC

-- Rollback to it (Spark)
CALL system.rollback_to_snapshot('analytics.orders', 1234567890)

Rollback does not delete data. It simply changes the current snapshot pointer to an earlier snapshot, making the table appear as it was at that point. The rolled-back data files remain in storage for potential recovery.

Dremio supports all Iceberg metadata table queries through its TABLE() function syntax and provides time travel in both SQL and its semantic layer.

Building a Health Dashboard

Combine metadata table queries into a scheduled monitoring job:

-- Table health summary
SELECT
  (SELECT COUNT(*) FROM TABLE(table_snapshot('analytics.orders'))) AS snapshots,
  (SELECT COUNT(*) FROM TABLE(table_files('analytics.orders'))) AS files,
  (SELECT AVG(file_size_in_bytes)/1048576 FROM TABLE(table_files('analytics.orders'))) AS avg_mb,
  (SELECT COUNT(*) FROM TABLE(table_manifests('analytics.orders'))) AS manifests

Set alerts when snapshots exceed 1,000, average file size drops below 64 MB, or manifest count exceeds 500.

Engine Syntax Variations

Different engines use different syntax for metadata tables:

The underlying data is identical; only the SQL syntax differs. Regardless of which engine you use, these metadata tables are the key diagnostic tool for understanding and maintaining Iceberg table health.

Automating Decisions with Metadata

You can use metadata table queries to drive automated maintenance decisions. For example, a scheduler can check whether compaction is needed before running it:

-- Only compact if average file size is below threshold
SELECT CASE
  WHEN AVG(file_size_in_bytes) / 1048576 < 64 THEN 'COMPACT_NEEDED'
  ELSE 'HEALTHY'
END AS table_status
FROM TABLE(table_files('analytics.orders'))

This avoids running compaction on tables that are already well-organized, saving compute costs and preventing unnecessary data rewrites.

For production environments, integrate these checks into your orchestration tool (Airflow, Dagster, Prefect). Schedule a daily metadata scan across all tables, collect the health metrics, and trigger maintenance jobs only for tables that need them. This approach scales to hundreds of tables without manual oversight. Dremio's autonomous optimization automates this entire workflow for tables managed by Open Catalog.

Part 12 covers using Iceberg from Python and MPP query engines.

Books to Go Deeper

Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)
Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced
Constructing Context: Semantics, Agents, and Embeddings by Alex Merced
Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced
Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

Free Resources

Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup

Alex Merced — Fri, 22 May 2026 15:33:50 +0000

This is Part 10 of a 15-part Apache Iceberg Masterclass. Part 9 covered how tables degrade. This article covers the four maintenance operations that keep Iceberg tables healthy and the three approaches to running them.

The Four Maintenance Operations

1. Compaction (File Rewriting)

Compaction reads small files, merges them into optimally-sized files (128-512 MB), and optionally re-sorts the data. It is the most impactful maintenance operation because it directly addresses the small file problem and restores sort order effectiveness.

In Spark:

CALL system.rewrite_data_files('analytics.orders')

In Dremio:

OPTIMIZE TABLE analytics.orders REWRITE DATA USING BIN_PACK

Compaction with sorting rewrites files so that column values are ordered, tightening the min/max statistics and making file skipping far more effective:

OPTIMIZE TABLE analytics.orders REWRITE DATA USING SORT (order_date, customer_id)

2. Snapshot Expiry

Snapshot expiry removes old snapshots from the metadata. After expiry, the snapshot and its exclusive data files are eligible for cleanup. You typically retain snapshots for a window (e.g., 7 days) to support time travel, then expire everything older.

-- Spark
CALL system.expire_snapshots('analytics.orders', TIMESTAMP '2024-04-22 00:00:00')

-- Dremio
ALTER TABLE analytics.orders EXPIRE SNAPSHOTS OLDER_THAN = '2024-04-22 00:00:00'

3. Orphan File Cleanup

After snapshots are expired, the data files they exclusively referenced become orphans. Orphan cleanup scans the storage directory, compares files against the current metadata, and deletes files that are not referenced by any snapshot.

-- Spark
CALL system.remove_orphan_files('analytics.orders')

This operation should run after snapshot expiry and with a safety delay (e.g., files older than 3 days) to avoid deleting files from in-progress writes.

Running orphan cleanup too aggressively can delete files from long-running write operations. A 3-day safety window ensures that any write operation has had time to complete before its files are considered orphans.

4. Manifest Rewriting

Over many commits, manifests accumulate. A single snapshot's manifest list might reference hundreds of small manifests from individual commits. Manifest rewriting consolidates them into fewer, larger manifests.

-- Spark
CALL system.rewrite_manifests('analytics.orders')

This speeds up scan planning because the engine reads fewer manifest files. Each manifest file requires a separate I/O operation to read, so reducing the count from 500 to 20 eliminates 480 I/O round trips during query planning.

Sort-Order Compaction

Standard compaction (BIN_PACK) merges small files without changing the data order. Sort-order compaction rewrites files with data sorted by specified columns, which tightens the min/max statistics and makes file skipping more effective:

-- Dremio sort-order compaction
OPTIMIZE TABLE analytics.orders REWRITE DATA USING SORT (order_date, customer_id)

-- Spark sort-order compaction
CALL system.rewrite_data_files(
  table => 'analytics.orders',
  strategy => 'sort',
  sort_order => 'order_date ASC NULLS LAST, customer_id ASC NULLS LAST'
)

Sort-order compaction is more expensive than BIN_PACK because it reads, sorts, and rewrites all data. However, the performance improvement for queries that filter on the sorted columns is substantial: file skipping can eliminate 90%+ of data files when the sort columns match common query filters.

Data Retention Policies

Decide how long to keep historical data accessible through time travel:

Longer retention means more snapshots, more metadata, and more storage consumed by old data files. Shorter retention reduces costs but limits time travel capabilities.

Three Approaches to Maintenance

Manual (Scheduled Jobs)

Run maintenance operations on a schedule using Spark, Trino, or Dremio. A typical pattern:

Run compaction daily for heavily-written tables
Expire snapshots older than 7 days
Remove orphan files older than 3 days
Rewrite manifests monthly

Pros: Full control over timing and configuration. Cons: Requires operational effort; forgotten or broken jobs lead to degradation.

Semi-Automated (Scheduled with Monitoring)

Build a monitoring layer that checks table health metrics (Part 9 diagnostics) and triggers maintenance only when thresholds are exceeded (e.g., average file size drops below 64 MB).

Fully Automated

Use a platform that handles maintenance autonomously. Dremio's automatic table optimization runs compaction, expiry, and cleanup for tables managed by Open Catalog without any user configuration. AWS S3 Tables provides built-in compaction.

Recommended Maintenance Schedule

For most teams, starting with Dremio's autonomous optimization and only adding manual jobs for tables with unusual requirements is the most practical approach.

Common Maintenance Pitfalls

Running compaction during peak query hours: Compaction reads and rewrites data files, which competes with analytical queries for I/O bandwidth. Schedule compaction during off-peak hours, or use a separate compute cluster (Spark on EMR) that does not share resources with your query engine.

Expiring snapshots too aggressively: If you expire snapshots while a long-running query is using one of them, the query can fail because the data files it needs might be cleaned up. Always keep snapshots for at least as long as your longest-running query.

Forgetting orphan cleanup: Many teams run compaction and snapshot expiry but forget orphan cleanup. Without it, compacted and expired data files accumulate indefinitely. Set up orphan cleanup as a weekly job with a 3-day safety window.

Not monitoring after migration: Tables migrated from Hive or other formats (Part 15) often inherit poor file layouts. Run an immediate compaction pass after any in-place migration.

Part 11 covers how to query the metadata tables that power diagnostics.

Books to Go Deeper

Architecting the Apache Iceberg Lakehouse by Alex Merced (Manning)
Lakehouses with Apache Iceberg: Agentic Hands-on by Alex Merced
Constructing Context: Semantics, Agents, and Embeddings by Alex Merced
Apache Iceberg & Agentic AI: Connecting Structured Data by Alex Merced
Open Source Lakehouse: Architecting Analytical Systems by Alex Merced

DEV Community: Alex Merced

Apache Data Lakehouse Weekly: May 21-27, 2026

Apache Iceberg

Apache Polaris

Apache Arrow

Apache Parquet

Cross-Project Themes

Looking Ahead

Resources & Further Learning

AI Weekly: Cheaper Coding Models, Custom Chips, and a Stateless MCP

AI Coding Tools: Cursor Ships Composer 2.5 and the Price of Frontier Coding Drops

AI Processing: Custom Chips Start Outgrowing Nvidia

Standards and Protocols: MCP Gets Its Biggest Rewrite, and the NSA Weighs In

Resources to Go Further

Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail

The Core Foundations: Columnar Memory and Apache Arrow

In-Process SQL Powerhouse: DuckDB Architecture & Features

Extensible Rust Processing: Apache Arrow DataFusion

Vectorized DataFrames: Polars Eager & Lazy Pipelines

Comparative Analysis: Evaluating Single-Node Engines

Key Tradeoffs to Consider

Zero-JVM Spark: High-Performance Pipelines with LakeSail

The Threshold of Scale: When Does Single-Node Break?

The MPP Landscape: Scaling to Spark, Dremio, Bauplan, SpiceAI, and MotherDuck

MotherDuck (Dual Execution)

Bauplan (Serverless Python Pipelines)

Spice.ai (Federated Query Acceleration)

Dremio (Distributed MPP Lakehouse Platform)

Architectural Selection Framework and Conclusion

Accelerate Your Lakehouse Skills

An In-Depth Overview of the Apache Iceberg 1.11.0 Release

Manifest List Encryption (PR #7770, #15813)

Pluggable File Format API and V4 Spec Foundations (PR #15049)

REST Client Protocols and Extended Headers (PR #12194)

Overwrite-Aware Table Registration (PR #15525)

Deletion Vector Pruning in Snapshot Validation (PR #15653)

Scheduled Credential Lifecycle Refresh (PR #15678, #15732, #15696)

Spark Streaming Triggers and Z-Ordering (PR #13824, #15706)

Trigger.AvailableNow Support (PR #13824, #14026)

Z-Order Column Collision Validation (PR #15706)

Flink Post-Commit Maintenance and Branch Compaction (PR #15566, #15672, #14148)

Flink Post-Commit Maintenance (PR #15566, #15667)

Flink Branch Compaction Support (PR #15672, #15690)

Flink Metadata Columns (PR #14148)

JSON to Variant Mapping and Spec Cleanups (PR #13137, #14045)

Variant Type Validation (PR #13137, #14081)

Trino Variant Type Mapping

Positional Deletes with Row Data Deprecated (PR #14045)

Table Upgrade Path and Connector Compatibility

Lifecycle Status Updates

Conclusion

Build Your Data Lakehouse Expertise

Semantic Layer Best Practices: 7 Mistakes to Avoid

Mistake 1: Defining Metrics in Multiple Places

Mistake 2: Skipping the Bronze Layer

Mistake 3: Using SQL Reserved Words as Column Names

Mistake 4: Building Without Stakeholder Input

Mistake 5: Treating Documentation as Optional

Mistake 6: Applying Security at the BI Tool Level Only

Mistake 7: Trying to Model Everything at Once

What to Do Next

How a Self-Documenting Semantic Layer Reduces Data Team Toil

The Documentation Problem Nobody Solves

What Self-Documenting Actually Means

AI-Generated Descriptions

Automated Label Suggestions

Metadata Propagation Through Views

How This Reduces Toil

What to Do Next

Migrating to Apache Iceberg: Strategies for Every Source System

Table of Contents

Three Migration Strategies

1. In-Place Migration (Metadata Only)

2. Full Rewrite (CTAS)

3. Shadow Migration (Build and Swap)

Choosing the Right Strategy

The View Swap Pattern

Phase 1: Federation

Phase 2: Build Iceberg

Phase 3: Validate