DEV Community: Alex Merced

AI Weekly: Fable 5 Returns, ZCode Deuts, MCP Grows Up

Alex Merced — Thu, 09 Jul 2026 00:09:54 +0000

The first week of July delivered three stories that will shape how developers work for the rest of the year. Anthropic restored its most capable model after a 19-day export control suspension. A Chinese lab shipped the first open-weight frontier coding environment. And the Model Context Protocol moved two big pieces into place for enterprise adoption. Underneath all three runs the same current: governments, vendors, and open communities are now negotiating the rules of AI in real time.

AI Coding Tools: Fable 5 Returns as Open-Weight Rivals Arrive

The restoration everyone waited for

Anthropic restored Claude Fable 5 on July 1 across Claude.ai, the Claude Platform, Claude Code, and Cowork. The model had been suspended under export controls since June 12, a 19-day outage for the strongest coding model on public benchmarks. Fable 5 leads SWE-bench Verified at 95.0% and SWE-bench Pro at 80.3%. Its sibling Mythos 5 remains limited to approved partner organizations.

The restoration came with strings attached. Anthropic's updated privacy policy took effect on July 8, requiring government-issued ID verification for access under the redeployment agreement. Usage credit billing for Fable 5 started the same day. Developers now face a new reality where frontier model access involves identity checks and government coordination.

Think about what the suspension taught the market. Teams that built their entire workflow around one frontier model lost their best tool for almost three weeks. The lesson landed hard. Build on models and tools that stay available, and keep a fallback path warm. The suspension also handed a marketing gift to every vendor selling models without US export exposure. One of them cashed in this week.

The business backdrop makes the restoration stakes clear. Fortune reported Anthropic overtaking OpenAI on revenue during the same news cycle, driven heavily by Claude Code and enterprise API growth. Google's CFO acknowledged that Anthropic codes close to 100% of its own work with AI while Google sits near 50%. A company whose flagship revenue engine is agentic coding cannot afford three-week outages of its best coding model. That pressure explains why Anthropic accepted identity verification terms to bring Fable 5 back, and it previews the compromises every lab will weigh as capability and regulation keep colliding.

ZCode makes open-weight agentic coding real

Z.ai, the international brand of Zhipu AI, launched ZCode on July 2. The company positions it as the first open-weight frontier agentic coding environment. That claim holds up better than most launch-day framing. ZCode wraps GLM-5.2, which scores 62.1% on SWE-bench Pro. That beats GPT-5.5 at 58.6% on the same benchmark. The model weights carry an MIT license with no regional restrictions.

The pricing tells its own story. Z.ai charges $1.40 per million input tokens and $4.40 per million output tokens. Claude Sonnet 5 runs $2 and $10 at introductory pricing. Claude Opus 4.8 runs $5 and $25. For teams running heavy agentic workloads, that gap compounds fast. An agent that burns 50 million tokens a week costs real money, and a 60% discount changes budget conversations.

ZCode ships a native terminal agent, a browser control agent, and a file system agent in one environment. That matches the tool suite Claude Code offers. The strategic pitch is explicit: agentic coding without US-origin model dependency. After the Fable 5 suspension, that pitch writes itself. Non-US enterprises watched a US regulator switch off the best model in the world for 19 days. Some of them will not forget it.

The open-weight wave extends beyond Z.ai. Meituan open-sourced LongCat-2.0 under MIT license in early July. The 1.6 trillion parameter model trained entirely on Chinese domestic chips. It turned out to be the anonymous "Owl Alpha" model that topped OpenRouter usage rankings for weeks before Meituan revealed its identity. Developers voted with their API calls before they knew whose model they were calling. For enterprise teams that need strong coding capability with full weight access, GLM-5.2, LongCat-2.0, and DeepSeek V4-Pro now form a credible open-weight bench.

The adoption numbers say this is bigger than a niche. CNBC confirmed Chinese AI models now account for 30% to 46% of US enterprise usage, depending on segment. Read that number twice. American companies are running Chinese open-weight models at scale, right now, mostly for cost reasons. The MIT licenses remove legal friction, self-hosting removes data residency worry, and the benchmark gap keeps shrinking. Enterprises that banned these models on principle are increasingly debating the policy in procurement meetings. Whatever your position, the era when frontier-adjacent capability belonged only to three US labs ended this quarter.

The incumbents keep shipping

Anthropic's other big move landed right at the window's edge. Claude Sonnet 5 launched June 30 and became the default model for every Free and Pro user on July 1. Anthropic calls it the most agentic Sonnet ever built, and the numbers back the claim. Sonnet 5 scores 63.2% on SWE-bench Pro, close behind Opus 4.8 at 69.2%, at a fraction of the price. Introductory pricing runs $2 per million input tokens and $10 per million output tokens through August 31. That undercuts the older Sonnet 4.6.

The positioning answers a specific pain. Enterprises recoiled from agentic AI bills in the second quarter as always-on agent workloads burned through annual budgets in weeks. TechCrunch framed the Sonnet 5 launch as a cheaper way to run agents, and that framing is the whole strategy. Anthropic wants the default answer to "which model runs our agents" to be a model most teams can afford to leave running. Free users getting near-flagship capability as the default also resets expectations for what baseline AI assistance means.

A quick word on reading this week's benchmark numbers, because three different tests got quoted above and they measure different things. SWE-bench Verified asks a model to fix real GitHub issues from popular open source projects, with human-checked solutions. SWE-bench Pro raises the difficulty with harder, longer-horizon issues, so scores drop across the board. Terminal-Bench measures how well an agent operates a command line to finish tasks, which tests the tool harness as much as the model. A model can lead one chart and trail another. Fable 5 tops both SWE-bench lists, while Codex CLI on GPT-5.5 holds the Terminal-Bench lead with Claude Code on Fable 5 a fraction behind. Match the benchmark to your workload before you match the model to your budget.

GitHub Copilot added Moonshot AI's Kimi K2.7 Code to its model roster this week. That puts a Chinese open-weight model inside Microsoft's flagship developer product. Copilot's move to usage-based billing on June 1 continues to settle in. Pro includes $15 per month in AI credits, Pro+ includes $70, and Max includes $200. Premium model picks draw from the credit pool, so the model roster now doubles as a price list.

Cursor shipped its iOS beta, with a launch promotion that ended July 5. Mobile agentic coding sounds like a gimmick until you kick off a refactor from the train, check the diff over coffee, and merge from your phone. The agent does the typing, so the phone stops being the wrong tool. Cursor Teams also added MCP support, which connects team workflows to the same tool ecosystem the rest of the industry standardized on. More on that protocol below, because it had a big week.

OpenAI kept adjusting its consumer funnel. Codex access is now bundled across Free, Go, and Plus plans. ChatGPT Go rose from $6 to $8 per month in July. The budget tier still undercuts Plus at $20 and gives light coding sessions a cheap entry point. The pattern across all three vendors is the same. Flat subscriptions are dying, and metered agentic usage is replacing them.

One quieter development deserves a paragraph, because it changes how teams invest in agent tooling. The SKILL.md convention, a folder of instructions and scripts that teaches an agent a repeatable job, now works across every major coding agent. A skill written once runs in Claude Code, Codex, Cursor, and the open source agents. That portability matters more than any single model release. Prompts locked inside one vendor's product are sunk cost. Skills that travel between agents are assets. Teams building agent workflows should write skills, not vendor-specific configuration, wherever the two options compete.

Anthropic tightened Claude Code's safety posture. A July update made manual permission mode the default and refreshed the AskUserQuestion flow. The agent now pauses for human sign-off more often out of the box. Power users will flip the setting back. New users get protected from an agent deleting the wrong directory on day one. Read that change next to the week's scariest story, and the timing makes sense.

JADEPUFFER changes the security conversation

Security researchers documented the first end-to-end AI-driven ransomware operation this week. The attack, tracked as JADEPUFFER, used an AI agent to exploit a remote code execution flaw in Langflow, then automated a database ransomware attack from initial access through extortion. Multiple security outlets confirmed the core finding. No human operator drove the individual steps.

Every developer running coding agents should sit with that for a minute. The same capabilities that let an agent fix your failing tests let a hostile agent chain an exploit. Agent autonomy is a dial, not a switch, and this week the industry saw what the far end of the dial looks like in criminal hands.

The defensive playbook starts with boring basics, so here is a concrete checklist. Patch internet-facing AI tooling fast, because Langflow-style frameworks now sit in attacker scan lists. Scope agent permissions to the minimum each task needs, and treat write access to production systems as a formal grant, not a default. Log every tool call your agents make, since agent activity without an audit trail is invisible until it is expensive. Separate the credentials agents use from the credentials humans use, so revocation is surgical. And run your own agents against your own systems before someone else's agent does, because AI-assisted penetration testing is now table stakes on both sides. None of this is exotic. All of it is suddenly urgent.

There is a sharp irony in the timing that security researchers noticed. The same model class the US government restricted over cybersecurity risk found a 29-year-old bug that every human security team missed, with a 90.6% confirmation rate on independent sampling. AI agents are simultaneously the newest attack tool and the best defensive researcher available. The asymmetry favors whoever adopts faster, and defenders usually adopt slower. Close that gap inside your own organization.

The JADEPUFFER story also explains the week's regulatory posture. Two frontier model launches in June involved direct government action. GPT-5.6 launched June 26 with government-approved limited access. Fable 5 spent 19 days under export suspension. Autonomous attack tooling is exactly the risk regulators cite. Expect the coordination between labs and governments to deepen, not fade.

AI Processing: Everyone Wants Their Own Chip

DeepSeek joins the custom silicon club

Reuters broke the biggest hardware story of the week on July 7. DeepSeek is developing its own AI chip, according to three sources familiar with the effort. The chip targets inference, the stage where a trained model answers user queries, rather than training. DeepSeek has depended on Nvidia and Huawei silicon to train and serve its models. An in-house inference chip cuts that dependence on both fronts at once.

The strategic logic runs deeper for DeepSeek than for its Western peers. US export controls bar Chinese companies from buying Nvidia's most advanced chips. Beijing keeps pressing its technology champions to build domestic alternatives. Founder Liang Wenfeng called chip export controls a challenge for the company back in 2024. Building your own inference silicon answers both the commercial problem and the political one.

The move also reshuffles China's domestic chip market. Reuters notes a DeepSeek chip adds pressure on Huawei, whose Ascend line currently anchors Chinese AI inference. Huawei targets 600,000 units of the Ascend 910C this year, and DeepSeek's own testing pegged that chip near 60% of Nvidia H100 inference performance. Good enough for serving at scale, and now facing competition from its highest-profile customer. When a country's AI champion decides to compete with its chip champion, the domestic ecosystem has reached a maturity that export controls were supposed to prevent. That is the uncomfortable read for policymakers in Washington, and analysts on both sides of the Pacific spent the week making it.

Put the week's chip news in one list and a pattern jumps out. OpenAI unveiled Jalapeño, its first custom inference chip built with Broadcom, last month. Anthropic opened preliminary talks with Samsung about manufacturing a custom accelerator, reported by The Information on July 2. The discussions involve Samsung's 2nm process and advanced packaging, and Anthropic has already hired specialized silicon engineers. CNBC separately confirmed Anthropic is in early talks with Microsoft to run Claude inference on Maia 200 chips through Azure. The Maia 200 launched in January on TSMC's 3nm process and claims over 30% better performance per dollar for inference.

Every major lab now pursues the same play. Train wherever the compute lives, then control your own inference economics. The distinction matters enough to spell out. Training is a rare, giant expense, a months-long run that produces the model. Inference is the meter that runs every time any user sends a prompt, forever. Training costs make headlines. Inference costs make or break the business, because they scale with success. A lab serving a billion requests a day feels every percentage point of chip performance in its margins. Inference is the recurring cost that scales with every user, so a 30% performance-per-dollar gain compounds into billions at frontier scale. Nvidia still dominates training. The inference market is fracturing in real time, and this week added two more fractures.

The infrastructure spending behind all this kept pace. Anthropic signed a $19 billion AI data center lease with TeraWulf, one of the largest compute commitments on record. Google reported data centers driving a record 37% jump in its electricity use. Global startups raised $510 billion in the first half of 2026, with OpenAI and Anthropic absorbing a huge share of it. The custom chip programs, the mega-leases, and the power draw are one story told three ways. The labs are converting capital into physical capacity as fast as the supply chain allows, and the supply chain is straining. Which brings us to memory.

Speed becomes the new battleground

OpenAI confirmed plans to deploy GPT-5.6 Sol on Cerebras wafer-scale hardware in July for select customers. The target is 750 tokens per second. Standard GPU serving of frontier models runs near 50 tokens per second. That is a 15x jump, and it comes from architecture rather than incremental tuning. Cerebras chips process entire transformer layers on a single wafer. That design removes the memory transfers between GPUs that throttle generation speed today.

The Cerebras hardware itself explains the numbers. The WSE-3 is a single chip covering an entire silicon wafer, with 4 trillion transistors and 900,000 AI cores on TSMC's 5nm process. It carries 44GB of on-chip SRAM delivering 21 petabytes per second of memory bandwidth. Conventional GPUs hit a memory wall shuttling data between chips. The wafer keeps the whole layer local. Cerebras signed a framework deal with OpenAI worth over $20 billion for up to 750 megawatts of inference capacity by 2029, and AWS already pairs CS-3 hardware with Trainium inside Bedrock. Wafer-scale inference has moved from research curiosity to purchase order.

Speed at that level changes what agents can do. An agent loop that takes 40 seconds per step feels like batch processing. The same loop at 3 seconds per step feels interactive. A coding agent that iterates 15 times on a fix finishes in under a minute instead of ten. Developers who design agentic workflows should watch the wafer-scale deployments closely. Latency budgets that seem fixed today will look quaint by winter.

LongCat-2.0 carries a hardware headline inside its model story too. Meituan trained the full 1.6 trillion parameter model entirely on Chinese domestic chips. A year ago, skeptics doubted domestic silicon handled frontier-scale training runs at all. That debate is now closed. The chips that trained LongCat-2.0 exist, they work, and their output sits under an MIT license on Hugging Face.

The memory shortage hits everyone's budget

The week's least glamorous hardware story will touch the most wallets. A global memory chip shortage is driving broad price increases across consumer and data center hardware. AMD notified board partners including Sapphire, ASUS, and XFX that Radeon GPU and memory bundle prices rise 10% starting in July. Intel raised suggested prices on Arrow Lake Refresh desktop CPUs by 10% to 17%, only three months after launch.

The data center numbers are wilder. The Information reports Nvidia AI server prices climbing for months, with some component costs swinging 40% in a single week. One GPU cloud executive described rack prices rising 2% to 3% weekly. Grace Blackwell 300 racks now cost 10% to 15% above baseline, with a fully loaded 72-system rack near $5 million. Cloud providers have started passing those costs to AI developers through higher rental prices.

For working developers, the shortage translates into three practical effects. Cloud GPU pricing keeps drifting up through the year. Token prices from providers that rent their compute face upward pressure. And the value case for smaller, cheaper models gets stronger every month the shortage runs. Teams that trimmed agent token budgets during the Q2 cost squeeze made the right call twice over.

The shortage also rewrites the model selection math in a specific way. When compute was cheap, teams defaulted to the biggest model and reasoned about quality alone. With rack costs rising weekly, the question becomes cost per completed task, and that metric favors different tools. A cheaper model that finishes a task in three attempts beats an expensive model that finishes in one, whenever three times the cheap price stays under the premium price. Sonnet 5 at $2 and $10, GLM-5.2 at $1.40 and $4.40, and the free-tier agents all exist because vendors read the same spreadsheet. Route routine work to cheap models, reserve frontier models for the tasks that defeat everything else, and measure the routing. Every serious agent platform added model routing features this year for exactly this reason.

The memory squeeze traces back to AI demand itself, which gives the whole story a circular quality. Data center buildouts absorb high-bandwidth memory supply, GDDR6 prices climb, consumer GPUs and CPUs get repriced, and the cost flows back to the same developers whose AI workloads started the cycle. Nobody sees relief before new fabrication capacity lands, and fabs take years. Plan budgets on the assumption that compute stays expensive through 2027.

One acquisition that did not happen

The Qualcomm and Tenstorrent story closed this week. The Information reported in mid-June that Qualcomm was in talks to acquire Tenstorrent for $8 billion to $10 billion. Tenstorrent CEO Jim Keller publicly denied the talks on June 30, stating the company has no acquisition discussions underway and plans to focus on its own business.

The underlying story survives the denial. Tenstorrent builds AI chips on RISC-V, the open, royalty-free chip architecture that lets any company build processors without ARM or x86 licensing fees. Its Galaxy Blackhole platform packs 32 accelerators with 768 RISC-V cores each into a 6U data center unit. Qualcomm already bought RISC-V designer Ventana Micro Systems in December 2025 and paid $3.92 billion for Modular, the AI deployment platform. Open chip architectures are pulling serious capital, deal or no deal. The same open-versus-controlled tension that runs through model licensing now runs through silicon.

Rounding out the funding news, Raja Koduri's Oxmiq Labs raised $35 million to build a unified chip architecture combining graphics, CPU, and tensor processing. Koduri, the former Intel chief architect, wants to license chip IP in a model that echoes Arm. Nvidia meanwhile rolled out revenue-sharing and credit-support structures that let AI cloud providers access GPUs without paying everything upfront. The chip giant is now financing its own demand.

Standards & Protocols: MCP's Enterprise Month

New readers deserve thirty seconds of context before the news. The Model Context Protocol is the open standard, created by Anthropic in late 2024, that connects AI agents to tools and data. Before MCP, every AI product built custom integrations to every system it touched, an N-times-M explosion of glue code. MCP collapses that to one protocol. A tool exposes an MCP server once, and every MCP-capable agent can use it. The protocol won the standards race in about a year, and this week showed what winning looks like: enterprise features, a major spec overhaul, security scrutiny, and platform vendors building certification programs on top.

Enterprise-Managed Authorization goes stable

The Model Context Protocol team promoted its Enterprise-Managed Authorization extension to stable status, InfoQ reported July 6. The extension gives organizations a central way to control access to MCP servers through their existing identity provider. Anthropic, Microsoft, and Okta have adopted it, along with a growing set of MCP servers.

Here is the problem it solves, in plain terms. MCP servers are the connectors that let AI agents reach tools and data, things like your ticketing system, your data warehouse, or your document store. The original authorization model was user-scoped. Every person approved every server through interactive consent prompts. That works fine for one developer with three connectors. It collapses at a company where 5,000 employees need 40 approved connectors and the security team needs to revoke one instantly when a vendor gets breached.

Enterprise-Managed Authorization replaces the prompt parade with a zero-touch flow. Users sign in once through the company identity provider. Approved servers just work after that, and administrators control the approved list centrally. The MCP team says repeated authorization prompts ranked among the loudest enterprise complaints. Stable status means IT departments can now build on the extension without fear of breaking changes. Watch for identity vendors beyond Okta to ship support fast, because this extension turns MCP governance into a product category.

For platform and security teams, the stable release changes the adoption conversation this quarter. Shadow MCP usage already exists in most engineering organizations, with developers wiring personal connectors to work systems one consent prompt at a time. Enterprise-Managed Authorization gives security teams a path to bring that activity under identity provider control without banning it. The practical first step is an inventory. Find the MCP servers your teams already use, sort them into approved and prohibited lists, and pilot the centralized flow with one high-value connector. Governance that arrives as convenience gets adopted. Governance that arrives as prohibition gets routed around.

Three weeks until the biggest MCP release ever

The final MCP 2026-07-28 specification lands on July 28. The release candidate locked on May 21, and SDK maintainers are deep in the ten-week validation window right now. This is the largest revision since the protocol launched in November 2024, and it earns that label on substance.

The headline change makes MCP stateless at the protocol layer. Earlier versions required a session handshake, and servers tracked session IDs across requests. That forced production deployments into sticky sessions, shared session stores, and gateways that understood MCP internals. The new spec makes each request self-contained. A remote MCP server now runs behind a plain round-robin load balancer like any other web service. Routing keys move into Mcp-Method and Mcp-Name headers, so infrastructure routes traffic without inspecting request bodies.

Several supporting changes matter for builders. List results now carry ttlMs and cacheScope fields modeled on HTTP caching, so clients know how long a tools list stays fresh. W3C Trace Context propagation is documented, which means a trace follows a tool call from host application through server and into downstream systems. A formal lifecycle policy guarantees at least twelve months between feature deprecation and removal. And new capabilities like MCP Apps, which render server-provided interfaces, and Tasks, which handle long-running work, ship as opt-in extensions rather than core requirements.

The stateless rework asks real migration effort from server authors. It pays back in operational simplicity. Scaling an MCP server starts to look like scaling any web service, and that ordinariness is the point. Protocols that demand special infrastructure stay niche. Protocols that run on boring HTTP conquer the world.

For teams planning the migration, the sequencing matters. The old 2025-11-25 version stays supported under the new twelve-month deprecation policy, so nothing breaks on July 29. Tier 1 SDKs are expected to ship 2026-07-28 support within the validation window, which means most server authors upgrade by bumping a dependency and adjusting session handling. Servers that stored per-session state need the real work: converting hidden session data into explicit handles that clients pass back as tool arguments. Audit that state now, before the SDK update forces the question. Client and host developers get new patterns like multi-round-trip requests, which enable richer server-to-client interactions than the old model allowed. Read the changelog against your own architecture rather than trusting summaries, including this one.

The security bill arrives with the features

Security researchers spent the validation window reading the new spec closely, and Akamai published a detailed analysis. The findings cut both ways. The stateless design ends session hijacking as a class of attack. Unsolicited server-initiated prompts go away, and authentication standards get stronger.

New surfaces open in exchange. The new MCP headers create desync risk where proxies and servers disagree about a request. Developers who accidentally map secrets into headers expose them to every load balancer and logging system on the path. MCP Apps bring classic web risks like stored cross-site scripting into agent interfaces. And Tasks create a denial-of-service vector, because task creation costs the client little and the server a lot. The pattern is familiar from every protocol that grew up: the spec removes vulnerability classes while shifting more security responsibility onto implementers. Teams adopting 2026-07-28 should budget security review time, not just migration time.

Microsoft goes all-in on the MCP catalog

Microsoft published its July Dataverse update on July 6, and the MCP commitment runs deep. The company now offers a catalog of more than 60 ready MCP servers spanning productivity tools, developer systems, and business applications. The same servers work across Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry, GitHub Copilot, and any MCP-compatible client.

Two pieces of the announcement deserve attention. Partner MCP certification creates a trusted path for ISVs to ship connectors into the Microsoft ecosystem, complete with security and governance review. And Bring Your Own MCP lets organizations register internal servers for their proprietary systems under enterprise controls. Register the server once, make it discoverable to the right agents, and manage it with normal admin approval flows.

Step back and the standards story of the week snaps into focus. Anthropic created MCP. Microsoft now certifies MCP servers as a partner program. Okta ships identity integration for it. The protocol crossed the line from developer convention to enterprise infrastructure, and the 60-server catalogs and certification programs are what that crossing looks like in practice. History offers a useful rhyme. Docker created the container format, but Kubernetes certification programs and enterprise registries turned containers into the default deployment unit. MCP is walking the same path on a faster clock, and the certification layer is where platform vendors capture value from a standard they did not invent. Anyone building commercial MCP servers should study the Microsoft certification requirements now, because they preview what enterprise buyers will demand everywhere.

The web writes rules for agents too

The standards story extends past MCP to the open web itself. Cloudflare launched granular AI bot management that lets website owners separately control three crawler categories: Search, Agent, and Training. New defaults block Agent and Training bots on ad-supported pages while allowing Search. Starting September 15, all new domains get these restrictions by default.

That September date deserves a circle on every agent builder's calendar. A meaningful slice of the web is about to become opt-in territory for agents. Products that assume free browsing access will hit walls, and the polite path forward runs through identified agents, negotiated access, and standards for agent traffic. The web spent thirty years developing norms for search crawlers. It is now compressing that process into months for agents, and infrastructure providers like Cloudflare are writing the default rules.

Governments enter the standards business

One more standards story sits outside any protocol spec. The Financial Times reported the US government in advanced talks with major AI companies over voluntary standards for releasing powerful new models. The discussions cover benchmarks, release timelines, and access rules inside and outside the United States. An announcement is possible within weeks.

June already previewed the new normal. GPT-5.6 launched with government-approved limited access. Fable 5 spent 19 days suspended under export controls, and its return came with identity verification requirements. Model releases at the frontier now involve regulators the way drug launches involve the FDA. For developers, the practical takeaway repeats the coding section's lesson. Do not assume day-one access to any frontier model, and design your stack so a model swap is a configuration change rather than a rewrite.

The Agent2Agent protocol rounds out the standards picture with quieter progress. A2A, the Linux Foundation project that lets independent agents delegate work to each other, keeps growing its supporter list past 150 organizations, spanning every major hyperscaler plus enterprise vendors like Salesforce, SAP, ServiceNow, and PayPal. The division of labor between the two protocols is settling into common wisdom. MCP connects agents to tools, the structured things with known inputs and outputs. A2A connects agents to agents, the autonomous things that negotiate and delegate. An orchestrator agent uses A2A to hand a research task to a specialist, and that specialist uses MCP to reach the databases it needs. Add the skills format that lets one instruction set work across Claude Code, Codex, Cursor, and the open agents, and the interoperability stack for agentic software is filling in layer by layer. Tools, agents, and know-how each got a portable standard, and all three matured this year.

One prediction flows from the pattern. The next standardization fight covers agent identity and payment. Cloudflare's crawler rules already ask agents to identify themselves. Enterprise authorization already binds agents to human identity providers. The missing piece is a standard way for an agent to prove who sent it and settle what it owes for access. Expect proposals before the year ends, and expect the same coalition dynamics MCP and A2A already showed. Whoever ships the boring, adoptable version first wins.

What This Week Means

Three threads tie the week together. First, model access became a supply chain concern. The Fable 5 suspension, the ID verification rules, and the government standards talks all say the same thing. Frontier capability now flows through political channels, and resilient teams treat models like any other dependency with a failover plan.

Second, open weights and open silicon advanced together. ZCode, LongCat-2.0, RISC-V investment, and lab-built inference chips all push against concentration, each from a different angle. The open lakehouse community has watched this movie before. Open standards win when the proprietary alternative starts feeling like a single point of failure, and this week supplied fresh evidence on both the model and hardware layers.

Third, the agentic stack is standardizing fast. MCP gained enterprise authorization, a stateless core, a Microsoft certification program, and a hard security review in a single week. Boring infrastructure work like this is what turns agent demos into agent deployments. The teams that learn these standards now will ship the production systems everyone else studies next year.

If you carry one action item from this issue, make it a dependency audit. List the frontier model, the coding agent, the protocol version, and the cloud GPU pool your team assumes. Then write down the fallback for each one. The Fable 5 suspension, the memory shortage, and the September crawler defaults all punished assumptions this quarter. Cheap insurance exists for every one of these risks, and it costs an afternoon of planning. The teams that did that planning in June spent the first week of July shipping. Everyone else spent it in status meetings.

Looking Ahead

Four dates anchor the next stretch. July 28 brings the final MCP 2026-07-28 specification, and the SDK releases that follow will show how smooth the stateless migration really is. The White House voluntary standards announcement is possible within weeks, and its benchmark and access provisions will shape every frontier release after it. August 31 ends the Sonnet 5 introductory pricing window, which forces a budget decision on every team that adopted it as the agent default. And September 15 flips Cloudflare's agent-blocking defaults for new domains, the first hard deadline in the agent-versus-web negotiation.

The open questions matter as much as the dates. Watch whether GPT-5.6 exits its government-gated preview and at what price, because Terra at half of GPT-5.5's cost resets the mid-market. Watch whether the ZCode launch pulls meaningful enterprise workloads onto open weights, and whether the 30% to 46% Chinese model usage figure grows or triggers a policy response. Watch which identity vendor follows Okta into MCP enterprise authorization. And watch the DeepSeek chip effort for a tape-out timeline, since a working chip turns this week's strategy story into next year's market story. This newsletter will track all of it.

Resources to Go Further

The AI world changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free: Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data: Dremio's agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community: Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development: Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis: A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

Apache Data Lakehouse Weekly: July 1 to July 8, 2026

Alex Merced — Thu, 09 Jul 2026 00:00:30 +0000

The lakehouse community spent this week deciding how change itself should work. Apache Parquet opened a formal vote to adopt versioned releases for breaking changes, borrowing a governance model that Iceberg refined over years. Apache Polaris canceled a vote on its semantic model API so it can align with Apache Ossie, the freshly incubating semantics project that opened its dev list this week. And across Iceberg, Arrow, and Parquet, contributors debated who owns statistics, which format features deserve to survive, and how far the specs should bend to serve AI and machine learning workloads. The connective tissue this week is governance. These communities are building the rules for evolving open formats without breaking the millions of tables that already depend on them. That work is invisible when it goes well, which is exactly why it deserves a close read while it happens. Releases also kept pace with the design debates: Polaris shipped 1.6.0, Arrow Rust shipped 59.1.0, and both Iceberg Rust 0.10.0 and Arrow 25.0.0 entered their final voting rounds.

Apache Iceberg

The Rust implementation dominated release activity this week, and the path was not smooth. For readers who mostly touch Iceberg through Spark or a query engine, iceberg-rust deserves a moment of framing. The Java implementation remains the reference, but the Rust library has become the foundation for a second generation of tooling: pyiceberg-core binds it into Python, DataFusion integrates it for query processing, and a wave of lightweight services use it to read and write tables without a JVM. When this library ships a bug, the blast radius crosses language ecosystems, which explains the ceremony you are about to read. Danny Jones and Shawn Chang opened the vote for Iceberg Rust 0.10.0 RC2 on July 1, drawing verification from L. C. Hsieh, Matt Butrovich, Neelesh Salian, Renjie Liu, and Xin Huang across a twelve message thread. Issues surfaced during verification, and by July 8 Jones was back with a vote on RC3. The willingness to cut a third candidate rather than wave through a flawed second one says something about how seriously the Rust community takes its release checklist, which covers everything from ASF license headers to a clean build of the pyiceberg-core bindings. The Rust library now sits underneath a growing stack of Python and query engine integrations, so the caution pays forward.

The week's most consequential design debate concerned table statistics, a topic that sounds dry until two engines start fighting over the same files. Some background helps here. Query engines rely on statistics, things like distinct value counts and data distributions, to choose good execution plans. Join two tables in the wrong order and a query that should take seconds takes minutes, so statistics quality translates directly into compute cost. Iceberg lets engines write statistics files and attach them to a snapshot in table metadata. That design works cleanly when one engine owns a table. The trouble starts in the multi-engine deployments Iceberg was built for, where Spark handles ingestion, Trino or Dremio serves interactive queries, and Impala or Flink sits somewhere in the mix, each with its own idea of which statistics it wants. Dzeri96 raised concerns about how Iceberg handles existing statistics files, and Gábor Kaszab pushed back on several proposed remedies across a nine message exchange. The core problem: when engine A writes statistics for a snapshot and engine B later computes its own, B can overwrite A's work. One proposal would create a new snapshot whenever statistics get computed. Kaszab argued this breaks the mental model, since snapshots today mark data changes, not metadata additions, and a snapshot X holding stats for snapshot Y confuses more than it clarifies. Another proposal would bind multiple statistics files to one snapshot, keyed by engine. Kaszab questioned how readers match files to engines in practice. Does Impala version X.Y know whether it can read stat files written by Spark version A.B? Would engine IDs live in the Iceberg spec or in tribal knowledge? The thread closed the week without resolution, which is fine. Multi-engine interoperability is Iceberg's whole reason for existing, and the statistics gap is a real hole in that story that deserves a careful fix rather than a fast one.

Security and access control produced a second substantial discussion. A quick primer for readers newer to the REST catalog: when a client asks the catalog for a table, the catalog can vend temporary storage credentials scoped to that table, so the client reads data files directly from object storage without holding permanent cloud keys. This delegation model is one of the REST catalog's best ideas, since it centralizes access decisions in the catalog while keeping the data path fast. William Hyun proposed extending it with file-level access delegation, with Kevin Liu joining the exchange. Today, delegated access in the REST catalog operates at table scope, and that granularity is the limitation Hyun wants to fix. If a consumer should only see a subset of partitions, administrators either over-provision access or fragment tables to match access boundaries, and both options create operational pain. Hyun's proposal uses pre-signed URLs during scan planning to make partition-scoped sharing practical without restructuring tables. Concretely, a catalog planning a scan for a restricted consumer returns signed links only for the files that consumer is entitled to see, and the storage layer enforces the boundary because unsigned paths simply fail. The idea builds on an earlier delegation thread and moves Iceberg closer to the fine-grained sharing models that commercial platforms offer on top of the format. That last point carries strategic weight. Fine-grained access control is one of the few remaining capabilities where proprietary lakehouse platforms hold a clear edge over the open spec, and standardizing it in the REST catalog narrows that gap for every open implementation at once. Nevin Zheng added a related thread on updating the Read Restrictions proposal to adopt the Expressions Spec and IDReference, which points toward a more unified security model across these proposals.

The primary key conversation resurfaced with new energy. Anyone coming from the database world finds this gap surprising, so it deserves a plain statement: Iceberg tables have no primary keys. The format grew up serving analytical scans over immutable files, where keys matter less, and it handles row-level changes through delete files rather than key-addressed updates. That works, but change data capture pipelines, which replicate every insert, update, and delete from an operational database into the lakehouse, spend enormous effort reconstructing key semantics the format never promises. Chandra Sekhar K continued the discussion on first-class primary key tables, describing production experiments with primary-key-oriented semantics for CDC and mutable-data workloads. His team treats the primary key as table metadata and then builds behavior on top of it: key-aware write semantics, storage organization, compaction strategy, and changelog generation for incremental processing. He framed this as complementary to the existing constraints proposal, which limits PRIMARY KEY and UNIQUE to informational metadata and leaves enforcement out of scope. The distinction matters because it sketches a two-layer future. The spec standardizes how keys are declared, and engines opt into richer key-aware behavior above that line. Anyone watching the upsert and CDC space, where formats like Apache Hudi and Paimon built key handling in from the start, should track this thread closely. A related note from st...@steveis.com asked about row-delta commits and multi-table transactions in iceberg-rust from a CDC producer's perspective, showing the same pressure arriving through the Rust door.

Geospatial work is quietly becoming one of Iceberg's most active frontiers. Sunmin Lee proposed declaring row-level bounding box covering columns in Iceberg metadata, which lets readers prune data files using spatial bounds. Seyed Muhammad Mahdi Hoseini followed with a separate proposal for QuadTree-inspired physical spatial partitioning, with Tanmay Rauth engaging on the bbox thread. Together these ideas sketch a spatial stack for the format: partition data by spatial cells on write, prune by bounding boxes on read. The economics mirror what min-max statistics did for numeric columns years ago. A query asking for events inside a city polygon should never open files whose contents sit on another continent, and today that pruning depends on engine-specific tricks rather than portable metadata. Iceberg v3 introduced geometry and geography types, so the type system groundwork exists, and these proposals show the community moving from type support to the performance engineering that makes spatial workloads economical. Fleet telemetry, logistics, climate data, and location-based applications all stand to benefit, and they represent exactly the data volumes where file pruning changes the bill.

The variant type effort keeps a steady drumbeat. Variant is the shredded, binary-encoded type for semi-structured data that lets JSON-shaped payloads live in tables with columnar performance, and it spans both the Iceberg and Parquet specs, which makes coordination the hard part. A field promoted to a shredded column by one writer has to remain readable by every engine, and the type touches file format, table format, and engine layers at once. That is why the community runs a dedicated sync for it. Neelesh Salian posted notes and a recording from the July 2 Variant Sync, the recurring gathering that coordinates work on semi-structured data support across implementations, with action items now tracked in a shared document. The cadence of these syncs since April shows the feature moving through the unglamorous middle stage of standardization, past the exciting design documents and into the grind of cross-implementation agreement. Anurag Mantripragada opened a related design question about column update file representation, which touches how partial column updates get written to storage.

Spec hygiene rounded out the Iceberg week. Daniel Weeks opened threads on clarifying schema JSON type string serialization and clarifying valid source types for the identity transform. Sung Yun raised a write-path gap for field-id-bound policy during schema evolution and a question about field id handling in the REST spec. These threads rarely make headlines, but they are the reason independent implementations of Iceberg in Java, Rust, Python, Go, and C++ produce the same answers on the same tables. Every ambiguity closed on the list is a bug that never ships, and the current volume of clarification work reflects the v3 and v4 feature waves pushing the spec into corners nobody had to define precisely before. Spec text that a single reference implementation can leave fuzzy becomes load-bearing the moment a second implementation reads it differently.

A few more items deserve a sentence each. Szehon Ho called a vote to add specific-name to the UDF spec, advancing the effort to make user-defined functions portable across engines. Alexander Löser continued the collation support discussion, which matters for anyone whose sort order needs to survive engine boundaries. Daniel Weeks flagged an upcoming Iceberg Terraform provider release with Alex Stephen, bringing infrastructure-as-code workflows to table management. Tanmay Rauth proposed a table_properties_log metadata table to expose property history for audit and debugging. And Renjie Liu suggested trimming CI runner time by running JDK 21 checks only on main and nightly builds, a small change that respects contributor time on every pull request.

Apache Polaris

The Polaris story of the week is a vote that did not happen, and the cancellation matters more than most votes that pass. First, the context. A semantic model defines business concepts, metrics like revenue or monthly active users, dimensions like region or customer segment, and the logic connecting them to physical tables. Today those definitions live scattered across BI tools, dbt projects, and application code, and every copy drifts from the others. The industry calls the result semantic drift, and it is why two dashboards in the same company report different numbers for the same metric. The Open Semantic Interchange effort, backed by a broad vendor coalition, produced a specification for expressing these definitions in a portable, machine-readable format. Catalogs are the natural home for such definitions, since they already serve as the shared source of truth for tables, and that is where Polaris comes in. Yufei Gu had called a vote to accept the OSI Semantic Model API Specification, the proposal that would let Polaris store and serve business metric definitions alongside its catalog duties. Jean-Baptiste Onofré asked the community to pause. His reasoning: the Open Semantic Interchange specification is transitioning into the Apache Ossie project, which entered incubation on June 22, and an initial Ossie spec release is expected soon. Implementing the OSI spec now and the Ossie spec later means doing the work twice. Robert Stupp and Adam Christian weighed in across the thirteen message thread, and Gu canceled the vote after gathering the feedback. He noted the practical middle path along the way: most of the semantic model work in Polaris can proceed in parallel, since the main dependency on Ossie is a JSON validator that arrives later, and the API can ship marked as beta with an explicit plan to converge on the Ossie specification.

Gu then did exactly that kind of parallel work, opening a focused design thread on the semantic model REST API payload representation. The question sounds technical but shapes years of API stability: should Polaris represent a semantic model as a raw string, as an opaque JSON document, or as a fully modeled structure in the REST specification? Gu argued for the first or second option. His concern with full modeling is coupling. The Ossie schema is versioned and expected to evolve, and baking its structure into the Polaris REST spec means every Ossie schema change ripples through the REST specification, generated clients, and client applications. Keeping the payload opaque separates the long-term REST contract from the fast-moving semantics spec. This is the kind of API design judgment that determines whether integrations built in 2026 still work in 2029.

Meanwhile, the release train kept rolling. Jean-Baptiste Onofré prepared and opened the vote for Apache Polaris 1.6.0 rc1, which drew verification from Dmitri Bourlatchkov, Yong Zheng, Francois Papon, Robert Stupp, and Ajantha Bhat. The vote passed on July 8, keeping Polaris on the steady release cadence it has maintained through the year. For a project that graduated to top-level status only in 2025, the rhythm of regular, verified releases is itself a signal of maturity. Release votes at Apache are more than ceremony. Each binding vote represents a committer who downloaded the artifacts, checked signatures and licenses, and built from source, which is why the roll call of verifiers in these threads reads like a project health report. Six people independently verifying a release candidate within days tells you a community has depth beyond its most visible names.

Persistence flexibility got real engineering attention. Yufei Gu shared a proof of concept for a Polaris-managed JDBC datasource, which lets Polaris create its own Hikari connection pool from configuration instead of depending on the Quarkus-managed datasource. Three details make this more interesting than typical plumbing. Different configurations can create independent datasources, which lays groundwork for future per-realm datasource routing, a meaningful capability for multi-tenant deployments. JDBC drivers can load at runtime from a jar rather than living on the build-time classpath, which solves a genuine licensing problem for drivers like MySQL that carry Apache-incompatible licenses. And the change is opt-in, since Polaris keeps the existing Quarkus path when no JDBC URL is configured. Dmitri Bourlatchkov, Robert Stupp, and Onofré worked through the design across seven messages.

Configuration cleanup and operational clarity threaded through the rest of the week. Gu proposed deprecating ALLOW_EXTERNAL_TABLE_LOCATION after code review showed it acts mostly as a metadata-location escape hatch that overlaps with ALLOW_EXTERNAL_METADATA_FILE_LOCATION while inviting confusion with ALLOW_UNSTRUCTURED_TABLE_LOCATION. His proposal keeps the old flag as a backward-compatible alias and clarifies the documentation, including a doc mismatch he found where one setting claims to require a flag the implementation never checks. Location flags in a catalog are security boundaries, so naming confusion here is not cosmetic. In the same spirit, Alexandre Dutra proposed deprecating TreeMapMetaStore and related classes for removal, trimming legacy persistence code, and Dmitri Bourlatchkov opened a discussion on standardizing vended credential property names, which affects how engines consume the temporary storage credentials Polaris hands out.

Two threads pointed at the operational future. Bourlatchkov floated a Kafka events publisher, building on the earlier discussion Gu, Bourlatchkov, and Dutra held about REST endpoints for table metrics and events. Catalogs sit at the choke point of lakehouse activity, so streaming catalog events into Kafka opens the door to audit pipelines, cache invalidation, and reactive workflows. And the Terraform wave that hit Iceberg reached Polaris too, with Alex Stephen, Gu, Nándor Kollár, and Sung Yun discussing a Polaris Terraform provider across six messages. The same contributor raising Terraform providers on two project lists in one week is worth pausing on. It signals that lakehouse components have crossed a maturity threshold in the eyes of platform engineering teams, who now expect to manage catalogs, principals, grants, and table definitions the way they manage VPCs and Kubernetes clusters: declared in code, reviewed in pull requests, applied by CI. When the infrastructure-as-code ecosystem starts building first-class providers for your project, it means production adoption arrived ahead of the tooling, and the tooling is catching up.

Community texture rounded things out. Rich Bowen interviewed project members for the ASF's PlusOne series, Prithvi S pushed forward on clearer 403 authorization messages, Ayush Saxena asked about staged creates in multi-table transactions, and Gu proposed entity-level filtering for list operations, which matters once catalogs hold thousands of entities and clients need server-side filtering rather than full listings.

Apache Arrow

Arrow entered release season on two fronts at once. Raúl Cumplido opened the vote for Apache Arrow 25.0.0 RC1 on July 6, drawing verification reports from Bryce Mecum, Sutou Kouhei, Gang Wu, Adam Reeve, and L. C. Hsieh. On the Rust side, Andrew Lamb ran the vote for Arrow Rust 59.1.0 RC1 and announced its passage on July 7 with binding approvals from Hsieh, Mecum, and Kevin Liu among others. The arrow-rs crate underpins DataFusion and a wide slice of the Rust data ecosystem, so its steady minor-release cadence quietly services a lot of downstream projects. The two release lines also illustrate how differently the same project can move in different languages. The C++ line ships large coordinated releases spanning a dozen language bindings, while the Rust line ships small and often, and both models work because each matches its ecosystem's expectations. Downstream users planning upgrades should note that Arrow major releases occasionally carry ABI-relevant changes for the C data interface consumers, so the 25.0.0 release notes deserve a read before rollout.

The format-level discussion of the week asked a question every mature project eventually faces: which features deserve to keep living? Arrow's IPC format is the wire and file protocol that moves Arrow data between processes, and it was designed primarily around RecordBatch messages, the tabular unit that nearly every Arrow user touches. Alongside those, the format carries two specialized message types for N-dimensional arrays. Antoine Pitrou proposed deprecating the Tensor and SparseTensor IPC messages. His case is thorough. The Tensor message arrived in 2017 and SparseTensor in 2019, both remain marked experimental, neither appears in the cross-implementation integration test suite, neither is reachable from Flight RPC or the C++ Dataset API, and a GitHub code search turned up no third-party usage. Meanwhile the messages carry maintenance cost, including recent security reports about missing validation on untrusted input. Arrow now has canonical extension types that carry dense tensors as ordinary RecordBatch columns, with room to add sparse variants if demand appears. Rok Mihevc and Weston Pace joined the discussion. Deprecating unused format surface is unglamorous work, but every retired feature shrinks the attack surface and the implementation burden for the ecosystem.

Kent Wu opened a discussion that fills a long-standing usability gap: a canonical JSON representation of Arrow schemas. Arrow schemas today only serialize canonically as IPC binary, which is awkward wherever humans or JSON APIs get involved. Try describing an expected schema in an API contract, or hand-writing one in a test fixture, or diffing two schemas in a code review, and the binary-only reality starts to chafe. The immediate motivation comes from ADBC, Arrow's database connectivity standard, where the 1.2 milestone includes new metadata APIs and contributors found returning schemas as IPC blobs unsatisfying. Wu drafted a proposal document and invited comments, and Dewey Dunnington engaged from the ADBC side. Wu also noted the topic has surfaced repeatedly over the years, which is usually the tell that a gap is real rather than hypothetical. A readable schema format helps API contracts, hand-authored fixtures, and configuration files, and it lowers the barrier for tools that want to speak Arrow without linking an Arrow library. Anyone who remembers how much JSON Schema did for the JSON ecosystem understands the shape of the win.

The rest of the Arrow week was community maintenance in the best sense. Robert Kruszewski asked for a spec clarification on field names, drawing responses from Felipe Oliveira Carvalho and Weston Pace. Pitrou and Mihevc discussed the status of Arrow conbench data and the conbench OSS project, the continuous benchmarking infrastructure that guards against performance regressions. Nic Crane asked for extra hands closing out old issues, and Pitrou collected contributions for the quarterly board report. Ian Cook hosted the July 1 community call.

Apache Parquet

Parquet produced both the busiest thread and the most consequential vote of the week, and the two stories reinforce each other.

The busiest thread first. The discussion on introducing a FIXED_SIZE_LIST logical type ran to twenty four messages, making it the most active conversation across all five projects this week. The motivation is the AI workload wave. Machine learning embeddings, feature vectors, and scientific arrays are fixed-length by nature. An embedding column holds, say, exactly 1,536 floats in every single row, forever. Yet Parquet today stores those values as variable-length lists using its Dremel-derived repetition and definition levels, the bookkeeping that records where each list starts and ends and which values are null. That bookkeeping is essential for genuinely variable data and pure waste for arrays whose length never changes. Readers pay to decode structure that carries zero information, and vector search and feature store workloads read a lot of these columns. The thread weighed three design options, and Andrew McCormick made a detailed case for what participants call Option C. Option A breaks backward compatibility and loses encoding quality. Option B is the design a greenfield format would choose but demands a rewrite-scale effort in Parquet's layered reader and writer stack. Option C stays fully backward compatible, keeps the existing encodings, and needs only modest reader and writer changes. Old readers see a normal list, new readers see the fixed-size annotation and take a fast path. The remaining cost is that repetition levels still hit the disk, but under run-length encoding a constant-length list compresses to almost nothing. McCormick brought receipts: on a benchmark of one hundred thousand rows of four-thousand-element float arrays, the fast reading path decompressed 2.3 times faster than the baseline, with roughly another 1.5x available when the writer records the fixed-size hint so readers skip verification. Russell Spitzer, Adam Reeve, Antoine Pitrou, Alkis Evlogimenos, and Philipp Fischbeck all engaged. Watch this one. Vector-heavy workloads are becoming a defining Parquet use case, and this type closes a real efficiency gap against newer formats designed with ML in mind.

Now the vote. Ryan Blue moved the long-running versioning debate to a decision, opening a vote to use version numbers for releasing forward-incompatible changes. The mechanics are straightforward. Forward-incompatible changes accumulate against the next major version of the Parquet spec, new breaking changes automatically target the version after the current one, and the community votes to close and adopt each major version as a unit. Blue noted the lineage with a smile: Iceberg contributors call this the Iceberg model, though Iceberg originally inherited the idea from Parquet. His summary in the discussion thread captured the state of debate honestly, acknowledging that Micah Kornfield and Antoine Pitrou hold differing views on the ideal mechanism while observing that nobody argued versions cannot work. The community sync surfaced no dissent either. Micah Kornfield, Russell Spitzer, Daniel Weeks, Gunnar Morling, and Pitrou participated as the vote thread gathered ten messages. This decision unblocks a queue of format work, starting with specifying how files carry the format version. Andrew Lamb kept a related question moving in the SemVer for parquet-format releases thread, which concerns versioning the specification artifacts themselves.

Why does versioning deserve this much ink? Because Parquet sits at the bottom of nearly every analytics stack on earth, and it has historically evolved through feature flags and reader capabilities rather than clean version boundaries. Forward incompatibility is the scary direction of change. Backward incompatibility breaks old files, which communities avoid at all costs. Forward incompatibility breaks old readers, meaning a file written with a new feature fails, or worse, silently misbehaves, in software that predates the feature. With Parquet readers embedded in everything from Spark clusters to embedded databases to decade-old ETL jobs nobody dares touch, the community needs a way to ship new capabilities without playing compatibility roulette. Recent additions like Variant, Geometry, and new encodings strain the old model past its limits. A predictable major-version mechanism means query engines can advertise Parquet 3 support as a coherent unit, vendors can test against a fixed target, and users can reason about compatibility without memorizing a feature matrix. The INT96 story this week shows the flip side of format ambiguity: Micah Kornfield closed the vote defining ordering for INT96 timestamps, pinning down semantics for a legacy type that engines interpreted loosely for years.

The ecosystem around the format had its own headline. Gunnar Morling announced Hardwood 1.0, a new Parquet reader for the JVM. Hardwood targets a gap that has annoyed JVM users for a decade: parquet-java carries heavy Hadoop dependencies, and plenty of applications want to read Parquet without dragging in a distributed filesystem stack. A microservice that reads a few Parquet files from S3 should not inherit hundreds of megabytes of Hadoop transitive dependencies and their security advisories. The C++, Rust, and Go ecosystems solved this with clean standalone readers years ago, and Hardwood brings the JVM the same option. Community response was warm and practical. Pritam Pan asked about Spark integration, and Morling sketched a plausible path where an engine keeps its own decoders but adopts a Hadoop-free metadata, IO, and filtering layer underneath. Steve Loughran connected Hardwood to his pending parquet-testing fixtures for malformed files, and Morling reported the results candidly: Hardwood rejected nearly everything, but several rejections happened incidentally rather than for the right reasons, a few fixtures exposed missing validation, and one surfaced a latent bug. That exchange is open source working as designed, with a new implementation and a shared test corpus hardening each other in public. It also echoes the security thinking in Loughran's separate thread asking how deep a realistic Variant value nests, part of an ongoing effort to harden Variant handling against hostile inputs. Kevin Liu, Loughran, and Ed Seidl also worked through expected behavior when older parquet-java readers encounter VARIANT columns, another compatibility question the new versioning policy will make easier to answer in the future.

Encodings research added a forward-looking note. Prateek Gaur opened threads on ALP encoding for floating point data and PFOR encoding. ALP, short for adaptive lossless floating point, exploits the fact that most real-world floats are decimals in disguise, encoding them as scaled integers that compress far better than raw IEEE bits, with an exact fallback for values that resist the trick. It has shown strong results in the research literature and in modern engine formats, and floats dominate ML feature data, so the pairing with the FIXED_SIZE_LIST discussion is natural. Between the fixed-size type, float-native encodings, and the footer redesign work, a picture emerges of Parquet systematically retooling for the vector era rather than ceding that ground to specialized formats. The healthy sign is that each piece arrives through the normal proposal process, with benchmarks attached, rather than as a rushed response to competitive noise. Divjot Arora proposed extended precision nanosecond timestamps, Daniel Weeks continued the complete FILE proposal, Jiayi Wang convened session four of the Parquet Footer Working Group, and Eduard Tudenhöfner and Gunnar Morling discussed adopting AssertJ for test assertions in parquet-java. Julien Le Dem gathered input for the July board report. Add it up and Parquet had one of its most substantive weeks of the year.

Apache Ossie (Incubating)

Apache Ossie opened its doors this week, and readers of this newsletter got a preview of why it matters in the Polaris section above. Ossie comes from Open Semantic Interchange, and it defines a vendor-neutral specification for expressing business metrics, dimensions, and their relationships, so a definition like monthly active users means the same thing to every tool that touches it. The design philosophy matters as much as the format. Rather than point-to-point field mappings between tools, Ossie standardizes the ontology and lets systems read semantic metadata straight from the source, which means the meaning travels with the data instead of being retranslated at every boundary. The project ships two main components, the specification itself plus bindings and converters from existing formats, so teams with definitions locked inside today's tools have a migration path. Ossie entered the Apache Incubator on June 22 with a mentor bench that signals how seriously the data community takes it: Jean-Baptiste Onofré, Zili Chen, Russell Spitzer, and Holden Karau, names readers of this newsletter will recognize from Iceberg, Polaris, and Parquet threads.

The dev list came alive on July 8. Onofré posted the welcome message, announcing that the website is live at ossie.apache.org and the GitHub repository has been populated. He followed with a proposal for a bi-weekly community meeting on Wednesdays at 6pm Pacific, and the proposed norms deserve notice: meetings host discussion only, decisions happen on the mailing list, sessions get recorded, and summaries return to the dev list. That is the Apache Way applied cleanly from day one, and it keeps the project legible to contributors in every time zone. Quigley Malcolm joined the thread, an early signal that the semantic layer community beyond the founding group is paying attention. With Polaris explicitly holding its semantic API in beta until the Ossie spec lands, the podling starts life with a waiting integration partner, which is a rare and useful forcing function for a specification project. Most standards efforts spend their first year searching for an implementer willing to bet on them. Ossie has one on day one, plus a coalition of semantic layer and BI vendors already invested in the OSI work it inherits.

For readers wondering whether to invest attention in a four-committer podling, consider the pattern this newsletter has tracked before. Polaris itself went from incubation to top-level project to production deployments in under two years because it standardized something the ecosystem had already decided it needed. Ossie targets the same kind of gap. Every data team maintains metric definitions somewhere, none of those definitions travel, and AI agents querying the lakehouse have made the cost of ambiguous metrics suddenly concrete. An agent asked for monthly active users has to resolve that phrase to logic, and today the answer depends on which tool it asks. A shared specification is the boring, correct fix, and boring, correct fixes are what Apache projects do best. This section will cover Ossie weekly as list activity warrants.

Cross-Project Themes

Three threads of connective tissue stood out this week. The first is that format governance is converging on a shared playbook. Parquet voted to adopt versioned releases for breaking changes, the model Iceberg has used to sequence v2, v3, and the emerging v4 work. Arrow moved to deprecate format surface that never found users, applying the discipline of subtraction, and its stated policy of extension types over new message kinds is the same lesson expressed structurally. Iceberg spent the week closing spec ambiguities one thread at a time. These communities share contributors, and the people carry the lessons across projects. Ryan Blue, Russell Spitzer, Daniel Weeks, Antoine Pitrou, Micah Kornfield, Kevin Liu, and Andrew Lamb all appeared in multiple project discussions this week, and the versioning vote text itself acknowledged the cross-pollination with its nod to the Iceberg model that Iceberg originally borrowed from Parquet. For practitioners, the payoff of this convergence is predictability. When every format in the stack evolves through the same version-and-vote mechanism, upgrade planning becomes one skill instead of four.

The second thread is that semantics is becoming lakehouse infrastructure. Polaris pausing its own API vote to wait for Ossie is a small procedural act with a large implication: the community wants one shared semantic specification, not a catalog-specific dialect. Add the week's AI-adjacent format work, from fixed-size vectors and float encodings in Parquet to variant hardening and geo partitioning in Iceberg, and the direction is clear. The open lakehouse stack is being fitted for a world where the primary consumers of data include agents and models, and where meaning has to travel with the data rather than living in a BI tool's private configuration.

The third thread is that fresh implementations are stress-testing old specifications, to everyone's benefit. Hardwood 1.0 ran a shared corpus of malformed Parquet files and exposed validation gaps in a week-old 1.0 release, in public, with the author reporting the findings himself. Iceberg Rust burned two release candidates rather than ship a flawed one. Arrow Rust shipped 59.1.0 on schedule while the C++ line worked through 25.0.0 verification. Every additional independent implementation of these formats converts spec ambiguity from a latent hazard into a visible bug report, and the volume of clarification threads across Iceberg, Arrow, and Parquet this week shows that feedback loop running at full speed. Multiple implementations are not a maintenance burden for open formats. They are the immune system.

Looking Ahead

Watch for the Parquet versioning vote result and the first concrete steps toward carrying a format version in files, since that decision unlocks the queue of pending format work including FIXED_SIZE_LIST. The Iceberg Rust 0.10.0 RC3 vote should conclude within days, and the Arrow 25.0.0 vote is in its verification window, so expect two release announcements shortly. On Polaris, the payload representation discussion should settle the shape of the semantic model API, and the newly released 1.6.0 starts reaching production deployments. The first Ossie community meeting is expected once the calendar details land, and the podling's first spec draft is the artifact the whole semantics conversation now waits on. The Iceberg statistics ownership debate has no obvious resolution yet, which makes it the design thread most worth reading closely next week, and the file-level access delegation proposal deserves the same attention from anyone running multi-tenant lakehouses. Quarterly board reports for Arrow and Parquet land this month, which usually surface useful project health summaries for both communities.

Resources & Further Learning

Get Started with Dremio

Try Dremio Free: Build your lakehouse on Iceberg with a free trial
Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow: Learn how Dremio brings the open lakehouse stack together

Free Downloads

Apache Iceberg: The Definitive Guide: O'Reilly book, free download
Apache Polaris: The Definitive Guide: O'Reilly book, free download

Books by Alex Merced

The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade

Alex Merced — Wed, 08 Jul 2026 22:56:06 +0000

Apache Parquet is thirteen years old, holds more of the world's analytical data than any other format, and, for most of its life, has been the least dramatic project in the data stack. It sat at the bottom, it worked, and the interesting arguments happened in the layers above it.

That era is over. As someone who reads the Parquet dev mailing list every week for my newsletter, I can report that 2026 is the busiest, most consequential stretch of Parquet development in a decade. The past year alone brought a native variant type for semi-structured data, first-class geospatial types, a new format release, an active redesign effort for the file footer, proposed types for embeddings and unstructured blobs, a new floating-point encoding, and the single longest discussion thread I have seen on that list: eighty-plus messages debating nothing less than the future of Parquet versioning itself.

Why is the quiet format suddenly loud? Two forces converged. The lakehouse era made Parquet the shared substrate under every table format, so every ambition of Iceberg and Delta eventually becomes a demand on Parquet. And the AI era arrived with workloads, embeddings, semi-structured context, wide feature tables, unstructured payloads, that the format's 2013 assumptions never anticipated. Parquet is being renovated while fully occupied, which is the hardest kind of engineering and the most interesting kind to watch.

This article is my 2026 state of the project: a proper refresher on how Parquet works, what shipped this year, what the dev list is fighting about, how the ecosystem of implementations has reorganized, and what it all means for the people building on top. As always, the goal is that the logic clicks, so the next Parquet headline you see explains itself.

Thirteen Years in Four Chapters

A compressed history sets the stakes for the present, because Parquet's current renovation only reads correctly against what came before.

Chapter one, 2013 to 2015, was the founding. Engineers at Twitter and Cloudera, with Julien Le Dem among the creators, built a columnar file format for the Hadoop ecosystem, drawing on the record-shredding ideas from Google's Dremel paper to handle nested data properly, something earlier columnar attempts had fumbled. The bet was that analytical storage should be columnar, compressed, and self-describing, and that an open format would beat every vendor's private one. The bet was not obviously right at the time. Row-oriented formats dominated, and columnar was a warehouse-vendor specialty.

Chapter two, 2016 to 2019, was the victory. Spark made Parquet its default, every SQL-on-Hadoop engine standardized on it, cloud object storage made file formats matter more than databases, and the Arrow project arrived as the in-memory complement, with the two communities intertwining from the start. By the end of the chapter, Parquet was less a choice than an assumption, and the exabytes began accumulating.

Chapter three, 2020 to 2024, was the lakehouse consolidation. Iceberg, Delta, and Hudi all chose Parquet as their substrate, which quietly changed the format's job description: it was no longer just a file people read, it was the physical layer of a transactional table abstraction, and every table-format ambition, statistics for planning, encryption for governance, column-level everything, arrived as a requirement on Parquet. Development in this chapter was steady and unglamorous: page indexes, bloom filters, modular encryption, better encodings, the compounding infrastructure work that made the layers above possible.

Chapter four is now, and its character should be clear from everything above: the AI era arrived with new shapes of data and new intensities of access, the lakehouse's next generation is negotiating with the format layer in real time, and a community that spent a decade in maintenance mode is running its most ambitious renovation, in public, with the versioning question as the constitutional debate that will govern how all the rest lands. Thirteen years in, the format's second act is genuinely more interesting than its first.

A Proper Refresher: How Parquet Actually Works

Everything in this article depends on the file anatomy, so let me build the mental model quickly and honestly, because most Parquet explanations stop one level too shallow.

A Parquet file organizes a table in three nested layers. The file is divided horizontally into row groups, each holding some slice of the rows, commonly in the hundreds of thousands. Within each row group, data is organized vertically into column chunks, one per column, so all the values of one column in that row slice sit together. Within each column chunk, values are stored in pages, the smallest unit of encoding and compression, typically around a megabyte.

This nesting is the whole trick. Row groups let engines parallelize and skip horizontally. Column chunks let queries read only the columns they touch. Pages let the format apply the right encoding per batch of values: dictionary encoding when values repeat, run-length encoding when they repeat consecutively, bit packing for small integers, delta encodings for sorted data, with general compression like Zstandard layered on top. The encodings are why Parquet files routinely land at a fraction of the size of the same data as CSV or JSON, and the layout is why queries can ignore most of a file's bytes.

Then comes the part this year's arguments revolve around: the footer. At the end of every Parquet file sits a metadata block, serialized with Apache Thrift, describing everything a reader needs: the schema, the location of every row group and column chunk, and statistics, minimum and maximum values, null counts, per column per row group. Readers open a Parquet file by reading the footer first, and the statistics power the pruning that makes analytics fast: a filter on date can skip every row group whose date range cannot match, before decompressing a single page. Auxiliary structures extend the same idea: page indexes push min and max tracking down to page granularity, and bloom filters answer "is this value definitely absent" for high-cardinality columns.

Hold two design facts from this tour, because the rest of the article pulls on them. First, the footer must be read before anything else, and it must be substantially decoded even when a query wants one column of a thousand, a property of the Thrift serialization that was harmless when tables were narrow and files were opened rarely. Second, the format's power comes from types and statistics: every capability Parquet gains arrives as a new logical type, a new encoding, or a new statistic, which is exactly the shape of everything that shipped this year.

What Shipped: Variant, Geospatial, and Format 2.13

Start with the ratified and released, because 2026 opened with Parquet formally announcing two of the biggest type-system additions in its history.

The variant type went official. In February 2026, the Parquet community announced native support for the Variant type, the binary encoding and shredding specifications for semi-structured data that I have written about at length in the Iceberg context, because Iceberg v3's variant support is built directly on these Parquet specs. The design lives at the Parquet layer on purpose: the binary encoding replaces JSON text with a compact, offset-navigable representation, and shredding extracts frequently occurring fields into real Parquet columns with real statistics, with engines able to supply an explicit shredding schema when read patterns are known or let inference decide. Because the specification is Parquet's, every engine and table format that implements it shares one physical representation, which is why Spark can write shredded variants that Dremio reads transparently. The dev list traffic since the announcement shows a spec in the hardening phase: threads on realistic variant depth limits, on how pre-variant readers should behave when they meet variant columns, and on where shared components like the JSON parser should live. This is what success looks like for a format feature: the arguments move from "should it exist" to "what happens at the edges."

Geospatial became a first-class citizen. Also in February, Parquet announced native geometry and geography logical types. For years, spatial data lived awkwardly outside mainstream analytics, in specialized formats or as opaque well-known-binary blobs that columnar engines could store but not understand, with the GeoParquet community convention bridging the gap admirably from the outside. The native types bring coordinates, spatial reference systems, and geometry semantics into the format itself, with statistics such as bounding boxes enabling spatial pruning the same way min and max enable numeric pruning. The follow-on dev list work has the same healthy hardening shape: clarifying coordinate reference system string formats, aligning with the parallel geospatial work in Iceberg v3, which adopted geometry and geography types in the same wave. Logistics, climate, mobility, and location intelligence workloads just got a columnar home that the whole ecosystem shares.

And the release train delivered. Parquet format 2.13.0 went through its release process this spring, carrying the accumulating spec work, while parquet-java shipped its 1.17 line with a 1.17.1 patch following. The community also passed a set of small, telling votes: defining ordering for the legacy INT96 timestamps, adopting IEEE 754 total ordering with NaN counts so floating point statistics finally handle NaN values coherently, and making a redundant schema-path field optional to shave footer weight. Individually minor, collectively these are the format sanding down decade-old ambiguities, the kind of work that only happens when implementers compare notes at scale.

The Eighty-Message Thread: The Future of Parquet Versioning

Now the argument that towers over the season's dev list: what does a Parquet version even mean, and how should the format evolve from here?

The problem is one Parquet earned through success. The format nominally has versions, and files carry a version marker, but the marker long ago stopped describing reality. Features landed in the specification one by one over a decade, implementations adopted them at wildly different speeds, and "Parquet 2" ended up meaning different things to different writers. The practical result is that nobody negotiates compatibility by version number. Engines make conservative feature-by-feature choices about what to write, defaulting to the lowest common denominator because they cannot know what readers will meet their files. That conservatism has a real cost: excellent features like delta encodings and modern statistics sit underused for years because writers dare not emit what some reader somewhere might choke on.

The thread, running past eighty messages with contributors from across the implementer ecosystem, is wrestling with the way out, and the option space is instructive. One direction formalizes feature flags: files declare exactly which capabilities they use, readers declare what they support, and compatibility becomes a checklist rather than a version comparison, an approach with clear precedent in how table formats above Parquet handle the same problem. Another direction argues for meaningful version milestones, a genuine "Parquet 3" that bundles the modern feature set, footer improvements, new types, better defaults, into a named target that the ecosystem can rally around and test against, with the marketing clarity that a decade of accumulated features has lacked. The companion thread on documenting which features belong to which versions shows the community doing the archaeology either path requires.

I will not predict the outcome, but I will name what is actually at stake, because it is bigger than labeling. The versioning decision determines Parquet's metabolism: how fast the format can absorb the AI-era additions discussed below without fracturing into dialects. A format read by thousands of independent implementations has one asset above all others, the guarantee that a Parquet file is a Parquet file, and the versioning thread is the community redesigning how to grow without spending that asset. It is the most important boring argument in the data stack right now.

The Footer Problem: Renovating the Front Door

The second great campaign of 2026 attacks the footer, and this one comes with a working group, regular sessions, and competing proposals.

The complaint, precisely stated: the Thrift-serialized footer must be parsed monolithically. A reader wanting the schema and the location of three columns must decode metadata for all one thousand columns, because Thrift's compact protocol does not support jumping selectively into the structure. When tables were dozens of columns and files were opened once per long scan, nobody noticed. Then came the modern workloads: feature tables thousands of columns wide, machine learning pipelines opening thousands of files per second, interactive engines where footer decode time is visible in query latency, and metadata-heavy features, page indexes, bloom filters, variant shredding statistics, all growing the footer they attach to. Measurements across the ecosystem put footer decoding at a startling share of some scan workloads, and the wide-table AI cases suffer worst.

Two remedial philosophies are on the table, and the contrast is a beautiful engineering study. The first replaces Thrift with FlatBuffers, a serialization format designed for zero-copy access: readers map the footer bytes and jump directly to the pieces they need, decoding nothing they do not touch. It is the thorough fix, and it is also a breaking change to the most compatibility-critical bytes in the analytics world, which is why it has been debated carefully for over a year. The second philosophy, advanced this spring as an alternative, keeps Thrift but adds a lightweight byte-offset index, a small directory that tells readers where each column's metadata lives inside the footer so they can decode selectively, buying much of the win with a fraction of the disruption. Alongside both runs a proposal to support non-contiguous pages, loosening layout constraints so writers can organize data and metadata more flexibly.

The footer working group has been convening openly, sessions announced on the list, notes flowing back, and the discussion has the flavor of the Iceberg v4 metadata debates, which is no coincidence: the layers are co-evolving. Iceberg's efficient-column-update ambitions lean on Parquet metadata getting cheaper, and Arrow's ecosystem supplies much of the implementation muscle. My read as of July: consensus that the problem is real and urgent is total, consensus on the remedy is not yet formed, and the versioning thread's outcome will shape which remedy is even deliverable. Watch these threads together, because they are one renovation.

The AI-Era Type System: Vectors, Floats, and Files

The third theme of 2026 is the format learning the shapes of AI data, and three proposals carry it.

FIXED_SIZE_LIST for embeddings. Vector embeddings are the defining data type of the AI era, and today Parquet stores them as generic variable-length lists, an encoding that pays offset overhead to express variability that embeddings never use, since every vector in a column has identical dimensionality. A well-supported discussion proposes a fixed-size list logical type: declare the dimension once, store the values as a dense contiguous block, and gain both compactness and the alignment that vectorized readers and GPU consumers want, mirroring the fixed-size layout Arrow has offered in memory for years. It sounds small. Multiplied by the billions of embeddings landing in lakehouses for retrieval workloads, it is one of the highest-impact storage changes on the board, and it pairs naturally with the parallel conversations about vector indexes in the layers above.

ALP for floating point. Floating point data, sensor readings, metrics, model outputs, coordinates, has always compressed poorly under Parquet's classic encodings, which were designed with integers and strings in mind. The community has been evaluating ALP, adaptive lossless floating point compression, a modern technique from the database research world that exploits how real-world floats cluster, with the dev list working through the remaining spec-level questions. Encodings are Parquet's quietest superpower, and adding a float-native one addresses what practitioners have long known as the format's weakest compression story, right as float-heavy AI and observability data becomes a dominant share of what gets written.

A File type for unstructured data. The boldest proposal of the spring introduces a new File logical type: a way to store whole unstructured payloads, documents, images, audio, model artifacts, as first-class values inside Parquet, with the format understanding that a value is a file with a media type rather than an anonymous blob. The motivation is the multimodal AI pipeline, which today shuttles metadata in tables and payloads in object-store sprawl, joining them by fragile path convention. Bringing the payloads into the columnar world, with the metadata, statistics, and governance that implies, would collapse that split. The thread has been appropriately spirited, because the proposal stretches Parquet's identity: row groups and pages were sized for analytical values, not hundred-megabyte videos, and the boundary between "table format problem" and "file format problem" gets genuinely blurry here, the same layering question the Iceberg column-update debate keeps meeting from the other side. Whether File lands, shrinks, or migrates upward, the pressure it responds to is real and not going away: AI made unstructured data everyone's analytical problem.

Add the variant hardening work to these three and the pattern is unmistakable. Parquet's type system is being extended along exactly the axes AI workloads demand: semi-structured context, dense vectors, efficient floats, and raw payloads. The 2013 format assumed data was numbers, strings, and dates. The 2026 format is learning that data is whatever a model touches.

The Ecosystem: Implementations, Old and New

A specification is only as real as its implementations, and the implementation story reorganized quietly over recent years in ways worth understanding.

The historical center, parquet-java, remains the reference for the JVM world that Spark, Flink, Hive, and Iceberg's Java core inhabit, and 2026 finds it in a deliberate modernization push: the 1.17 line shipping, a discussion to raise the minimum Java version to 17 in line with the platform-wide JDK modernization wave, sustained performance optimization work openly seeking reviewers, testing modernization, and automation of the release process itself. The community health signals surrounding it are good: a new committer welcomed this spring in Ed Seidl, a prolific contributor across the C++ and Rust ecosystems, regular open sync meetings, coordination with the Iceberg community's calendar, and, in a sign of the times I could not have invented, a thread on adding an AGENTS.md file so AI coding assistants contribute to parquet-java under proper guidance, mirroring the AI-contribution policy conversations running across the Apache data projects.

The newer centers of gravity live inside the Arrow project, a structural fact I flagged in my Arrow state-of piece: the official C++, Rust, and Go Parquet implementations are developed within Arrow's repositories, maintained by overlapping communities, and shipping on Arrow's brisk cadences. The Rust implementation in particular has become the engine room for the new wave of lakehouse tooling, and much of the footer and encoding experimentation draws its benchmarks from there. The practical meaning for the ecosystem: Parquet evolves as a joint venture between two Apache communities, with the format specification governed in Parquet and the highest-velocity implementations governed in Arrow, an arrangement that has worked because the people substantially overlap.

And the edges keep growing new implementations, which is the surest sign a format remains alive. This spring brought the announcement of Hardwood 1.0, a new independent Parquet reader for the JVM, built for modern Java and modern performance expectations, arriving on the dev list to a welcome rather than a turf war. Thirteen years in, people still choose to write new Parquet readers from scratch. Formats die when that stops happening.

A Worked Example: One File, One Query, Every Layer Visible

The anatomy section gave you the parts. Let me now assemble them into a single concrete story, one file and one query, with the 2026 developments highlighted along the way, because the renovation only makes sense once you can feel where the time goes.

The file: a day of e-commerce order events written by a compaction job into an Iceberg table. Two million rows, forty columns, one gigabyte on object storage. The writer split it into four row groups of half a million rows each. Inside each row group, the country column chunk dictionary-encoded its two hundred distinct values down to a few kilobytes of dictionary plus tightly bit-packed indexes. The order_total column, floats, compressed less impressively, which is exactly the gap the ALP encoding work targets. A properties column holds semi-structured attributes as a variant, with the writer shredding properties.channel and properties.campaign into typed subcolumns because they appeared on nearly every row. The footer records all of it: schema, the byte locations of one hundred sixty column chunks, and min, max, and null counts for each, a few hundred kilobytes of Thrift at the tail of the file.

The query: total revenue from the mobile channel in Germany, yesterday afternoon.

Step one, the engine reads the footer. Today that means decoding metadata for all forty columns to use the four it needs, the exact inefficiency the footer working group is attacking, whether by FlatBuffers or by the byte-offset index that would let the reader decode only four entries. On this narrow table the cost is a rounding error. On the thousand-column feature table next door, opened thousands of times an hour by a training pipeline, it is the dominant cost, which is why that constituency is driving the redesign.

Step two, pruning. The filter wants Germany, mobile, and an afternoon time window. The engine checks row group statistics: two of the four row groups have event_time ranges entirely in the morning, skipped without touching a byte. Within the survivors, page indexes narrow further, and the country column's statistics and bloom filter rule pages in or out. The shredded properties.channel subcolumn has its own min and max, so the mobile filter prunes like any typed column, the entire payoff of the variant shredding spec in one sentence: a filter on a JSON field just skipped physical bytes. A year ago, before the shredding spec, this predicate meant decoding every properties blob in every surviving row group.

Step three, reading. The engine fetches exactly the surviving pages of exactly four columns, event_time, country, order_total, and the channel subcolumn, a few dozen megabytes of the gigabyte file, decompresses and decodes them with vectorized kernels into Arrow batches, and aggregates. The other thirty-six columns were never touched. The result returns in the time a row-oriented format would still have spent parsing.

Now run the same story forward two years, with the 2026 proposals landed. The footer read decodes four entries instead of one hundred sixty. The order_total chunk is thirty percent smaller under a float-native encoding. The recommendation team added a 768-dimension embedding column, stored as a fixed-size list, dense and aligned, that the training pipeline reads straight into GPU memory. The support team attached call recordings through a File-typed column, governed and pruned like everything else, instead of orphaned in a bucket joined by path string. Same format, same guarantee that every reader can open it, and a file that serves workloads the 2013 designers never imagined. That continuity, renovation without rupture, is the whole game, and it is what the versioning thread exists to protect.

Parquet and the Layers Above: One Co-Evolution

I write constantly about Iceberg, and a theme of this year's series has been how often Iceberg's frontier turns out to be Parquet's frontier wearing a different name. Let me make the co-evolution explicit, because it is the strategic frame for everything above.

The variant type is the cleanest case: Iceberg v3 declared the type, Parquet defined the encoding and shredding, and the interoperability that makes variant valuable exists precisely because the physical layer standardized once for everyone, Delta included. Geospatial ran the same play in the same release wave. The footer renovation is entangled with Iceberg v4's ambitions, since efficient column updates and richer per-file statistics presuppose metadata that is cheap to read and extend. The versioning debate mirrors, and will interact with, how table formats negotiate feature support with their readers. Even the File type debate is at bottom a negotiation over which layer owns which problem, the same negotiation the Iceberg column-family discussion conducts from above.

The lesson I keep drawing for practitioners: the lakehouse stack is one organism with three specification layers, memory in Arrow, files in Parquet, tables in Iceberg, and capabilities flow through all three or arrive hobbled. When you evaluate a roadmap claim at any layer, ask what it requires of the layer below. And when you want to see eighteen months into Iceberg's future, read Parquet's dev list today, which is, not coincidentally, why my weekly newsletter covers both.

How Parquet Decides: The Machinery Behind the Threads

Since this article leans so heavily on dev list threads, working groups, and votes, a short guide to the project's decision machinery will make everything above more legible, and it doubles as a template for reading any Apache format project.

Parquet's constitution is unusual among the projects I cover, because the specification and the implementations have different centers of gravity. The format itself lives in the parquet-format repository as documentation plus Thrift definitions, governed by the Parquet PMC, and changes to it are the highest-stakes decisions in the project: a spec change binds every implementation, forever, against exabytes of existing files. The implementations then live in several places, parquet-java under the Parquet project directly, and the C++, Rust, and Go implementations inside Arrow's repositories under Arrow's cadence, with heavily overlapping contributors keeping the two communities aligned.

Ideas move through a recognizable pipeline. They surface as DISCUSS threads, often paired with a design document and a GitHub issue, and the message count is a decent proxy for how contested the design space is, which is how you should read the eighty-message versioning thread and the fourteen-message File type thread differently. Sustained topics graduate to working groups with scheduled sessions and posted notes, the footer effort being this year's example. Decisions land as VOTE threads on the list, spec changes like the NaN-count statistics and the INT96 ordering passing this way in recent months, and releases of both the format and the libraries go through release-candidate votes with public verification.

Two practical habits follow for anyone tracking the project. First, weight artifacts by their stage: a merged spec change outranks a passed vote outranks a working group draft outranks a lively thread, and vendor blog posts rank below all of them. Second, watch the implementation gap deliberately, because Parquet features become real for you only when your engines' Parquet libraries ship them, which typically trails the spec by quarters, and the community's own effort to document feature support per version exists precisely because that gap has historically been foggy. The monthly community syncs, coordinated openly enough that scheduling around the Iceberg Summit made the list, are where the two halves, spec ambition and implementation reality, reconcile.

If the Iceberg and Polaris articles in this series taught the lesson that open governance is slow on purpose, Parquet is the senior example: a thirteen-year-old format still making every consequential decision in a searchable public archive, one thread at a time, with the patience of a project that knows its files must outlive every company reading them.

What Practitioners Should Do in 2026

Grounded guidance, in the order I would give it to a platform team this quarter.

Harvest the shipped features. If semi-structured data lives in your tables as JSON strings, the variant encoding with shredding is production-real across a growing engine set, and the read-side wins are large. If spatial data lives in your stack as blobs or sidecar systems, the native geospatial types are the migration target to plan for as engine support arrives through the year. Neither requires waiting on any of the debates above.

Audit your compression and encoding posture. Most teams write Parquet with whatever defaults their engine chose in 2019. A periodic pass on compression codec, dictionary behavior, row group sizing, and statistics settings routinely recovers double-digit storage percentages and scan speedups, and the arrival of new encodings like ALP will reward teams that know their current baseline. Your files are read thousands of times more than they are written, so write-side care is the cheapest performance you can buy.

If you run wide tables or file-per-second workloads, follow the footer work actively, and quantify your own footer overhead now, because you are the constituency the working group is designing for, and because the interim mitigations, trimming unneeded statistics, moderating column counts, caching decoded metadata, are available today.

And calibrate your format expectations to reality: Parquet changes slowly on purpose, features reach you through your engines rather than from the spec directly, and the version marker on a file tells you little. Track capabilities, not versions, which is, after all, exactly the conclusion the eighty-message thread is circling.

Questions I Hear Most Often

The recurring questions from meetups, the podcast, and customer conversations.

Is anything actually going to replace Parquet? The research world keeps producing candidate successors, and the honest reading of the past few years is that their best ideas get absorbed rather than their names winning. New encodings like ALP come straight from that research current, the footer work responds to the sharpest criticisms the newer formats raised, and the AI-era types address the workload gaps challengers pointed at. Parquet's moat was never technical perfection, it is the thousands of independent implementations and the exabytes already written, and the community's observable strategy is to metabolize the challengers' insights faster than the challengers can build ecosystems. My bet remains on the incumbent that learns.

Should I store embeddings in Parquet today? Yes, with eyes open. Variable-length list storage works, the ecosystem does it at enormous scale already, and keeping vectors next to their metadata in governed tables beats a separate store for many retrieval architectures. The fixed-size list work will make it meaningfully better, and the vector-index conversation in the layers above will decide how much of similarity search belongs near the storage. For high-QPS low-latency vector serving, a specialized index still earns its place, with Parquet as the durable system of record beneath it, the same hot-and-cold pattern I described for streaming.

What about page-level and file-level encryption? Parquet modular encryption is mature, specified, and implemented: columns can be encrypted with separate keys, footers can be encrypted or signed, and integrity verification is built in. Adoption has historically lagged the capability, mostly because key management is an organizational problem before it is a format problem. The rising governance pressure around AI data is changing that calculus, and the encryption machinery is one of the format's underused assets worth a fresh look.

Row groups, file sizes, what should I actually choose in 2026? The classic guidance holds: row groups sized so a scan unit is meaningful, files in the hundreds of megabytes to low gigabytes for object storage economics, achieved through your table format's compaction rather than hand-tuning writers. The new wrinkle is metadata weight: very wide tables now pay footer costs per file, which argues for fewer, larger files and pruning unneeded per-column statistics until the footer work lands. As ever, measure on your workload, because the defaults were tuned for someone else's.

How does the versioning debate affect files I have already written? It should not, and that constraint shapes the whole debate. Backward compatibility, old files readable forever, is the community's non-negotiable, reaffirmed constantly in the thread. What is being decided is forward evolution: how new capabilities get declared, negotiated, and adopted without fragmenting readers. Your existing exabytes are the constituency every proposal must serve, which is precisely why the argument is long.

Where do I follow all this without reading eighty-message threads myself? The dev list archives at lists.apache.org are the source of truth, the monthly community syncs are open with notes posted, and the parquet.apache.org blog now carries substantive feature announcements like the variant and geospatial posts. For the digest version, this is exactly what my weekly Apache Data Lakehouse newsletter exists for: I read the Parquet, Iceberg, Polaris, and Arrow lists so you can spend your hour on your own stack.

Closing Thoughts

The state of Apache Parquet in 2026 is a paradox worth savoring: the most settled format in the data stack is having its most unsettled year, and both halves of that sentence are good news. Settled, because the moat of implementations and exabytes has never been deeper, the release train runs, and the community handles decade-old ambiguities with quiet competence. Unsettled, because the lakehouse above it and the AI era around it are demanding renovations, richer types, cheaper metadata, a saner evolution model, and the community is conducting all of them in the open, working groups and eighty-message threads and all.

Formats are where the industry's real decisions get made, slowly, byte by byte, in public. The picture of a table in memory was decided this way in Arrow. The picture of a table's metadata was decided this way in Iceberg. The picture of the data itself, the bottom of the entire stack, is being redecided right now in Parquet, and everything built above will live with the outcome for the next decade.

If you want the kind of foundation that makes these layers legible, from file internals through table formats, catalogs, and the AI workloads reshaping them all, that is what my books are for. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, with further titles on lakehouse architecture, data engineering, and agentic analytics.

Browse the full collection of my books on data and AI at books.alexmerced.com.

The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere

Alex Merced — Wed, 08 Jul 2026 22:54:35 +0000

In February 2026, Apache Arrow turned ten years old. The first commit landed on February 5th, 2016, and the anniversary passed the way Arrow itself operates: quietly, while running inside nearly every data tool you touched that day.

I have a house interest to declare here too, and in this case it is practically genetic. Arrow was co-created by people who founded and built Dremio. Jacques Nadeau, Dremio's co-founder, was Arrow's original PMC chair, and Dremio's founding engineers were among the earliest contributors, building the company's engine Arrow-native from day one, years before that was a fashionable phrase. The project's origin braided together several threads: the Apache Drill community's in-memory format work, Wes McKinney's frustrations with pandas performance, and the Parquet community, where Julien Le Dem and other Parquet founders joined the early design sessions to build the in-memory complement to their on-disk format. Ten years later, that braid runs through pandas, Spark, Snowflake, DuckDB, Polars, and the AI stack.

So this article is my 2026 state of the project: what Arrow actually is beneath the buzzword, the full surface area it now covers, what the development pulse looks like this year, how deep the penetration really goes, the honest challenges, and why the next decade of Arrow is being written by AI workloads. Arrow suffers from a strange fame problem: everyone has heard of it and few can explain it. Let me fix the second part.

What Arrow Actually Is: The Cost of Copies

Strip away everything else, and Arrow is an answer to one question: what should a table look like in memory?

That sounds too small to matter. Here is why it is enormous. Before Arrow, every system answered the question privately. Pandas had its internal layout, Spark had another, every database had its own, and every file format had its own. So whenever data crossed a boundary between systems, and analytics is nothing but data crossing boundaries, it had to be converted: serialized out of one representation, parsed into another, row by row, allocation by allocation. Studies and painful experience suggested that big data systems were spending most of their CPU cycles not computing anything, just translating data between representations of itself. The industry was paying a permanent tax to disagreement.

Arrow's founding move was to make the in-memory representation a standard rather than an implementation detail. It specifies, byte for byte, how columnar data sits in memory: values of a column contiguous in fixed-width buffers, variable-length data like strings in value buffers with offset arrays pointing into them, null tracking in validity bitmaps, nested types built by composing these primitives. The layout is language-independent and machine-friendly, designed so modern CPUs can rip through it with vectorized instructions, and defined identically whether the process is C++, Java, Python, Rust, or Go.

The payoff is the elimination of translation. When two systems both speak Arrow, handing data between them is not a conversion, it is a pointer. The C Data Interface lets two libraries in the same process share Arrow data with zero copies, literally passing memory addresses. The IPC format lets processes and machines exchange Arrow data as a stream of buffers that the receiver uses as-is, no parsing step at all. The tax to disagreement drops to zero because the disagreement is gone.

The analogy I have used for years: before shipping containers, every transfer between ship, train, and truck meant unloading and repacking cargo by hand, and ports spent more effort repacking than moving. The container standardized the box, so cargo stopped being touched at boundaries. Arrow is the shipping container for tables. Nobody gets excited about the box. Everybody's goods move faster because the box is boring and identical everywhere.

One more foundation worth setting, since the two are eternally confused: Arrow and Parquet are complements, not competitors. Parquet is how a table sits on disk, optimized for compact storage and selective reading. Arrow is how a table sits in RAM, optimized for computation and exchange. Data typically lives as Parquet, gets read into Arrow, is computed on and shared as Arrow, and lands back as Parquet. The two communities have intertwined to the point that most official Parquet implementations, in C++, Rust, and Go, are literally developed inside Arrow repositories today. The founders designed them as two halves of one interoperability answer, and a decade later the halves have effectively merged their engineering.

A Decade in Six Chapters

Before surveying the present, a compressed history helps, because Arrow's current shape only makes sense as accumulation. Six chapters, roughly chronological.

Chapter one, 2016 to 2017, was the founding bet. The format specification, the first C++ and Java implementations, and the founding argument that a shared memory layout would pay for itself. Adoption was speculative, and the early integrations, pandas interchange, Spark's Python acceleration, were proofs of concept that the copy tax was real and removable.

Chapter two, 2018 to 2019, was language expansion. Rust, Go, JavaScript, and more joined, establishing the principle that Arrow's value scales with the square of its implementations, since every new language can now exchange with every existing one. Flight arrived at the end of this stretch, extending the standard from memory to the network.

Chapter three, 2020 to 2021, was the compute era. A 1.0 format with stability guarantees, the C++ compute kernels and what became Acero, Gandiva for expression compilation, and DataFusion growing inside the Rust repository. The project tested how far up the stack a standard should climb.

Chapter four, 2022 to 2023, was connectivity. Flight SQL matured, ADBC launched with its 1.0 specification, and nanoarrow appeared at the opposite extreme, Arrow as two embeddable C files. The project's center of gravity shifted from "represent data well" to "move data everywhere," which in hindsight was the decisive strategic turn.

Chapter five, 2024 to 2025, was restructuring and resilience. Language implementations moved to independent repositories, DataFusion graduated to its own Apache Top-Level Project, format additions like string views and run-end encoding landed for modern workloads, and the community absorbed the wind-down of its largest corporate patron without missing a release. The first Arrow Summit in Paris closed the chapter with the community meeting itself in person.

Chapter six is now: the anniversary, the AI reinterpretation, ADBC's adoption knee, and a contributor intake that skews toward Rust and toward the connectivity layers. Each chapter built on the last without discarding it, which is the quiet discipline that separates standards that endure from projects that pivot.

The Surface Area: Far More Than a Format

People who last checked on Arrow in 2019 think of it as a memory spec with a Python library. The 2026 project is a family of standards and libraries covering the full lifecycle of data in motion. A tour of the surface, layer by layer.

The format layer remains the core: the columnar specification itself, now rich with a decade of carefully added types. Recent years brought string view and binary view layouts, which store short strings inline and long strings by reference for dramatically faster string-heavy workloads, list view variants, run-end encoding for compressed representation of repetitive data, and smaller decimals down to 32 and 64 bits. Alongside the layout spec sit the interchange standards: the C Data Interface for zero-copy in-process sharing, the IPC streaming and file formats for crossing process and network boundaries, and canonical extension types so the ecosystem can agree on things like UUIDs, JSON, and geospatial data without forking the format. The discipline here is the story: format changes move slowly, through votes, because a hundred systems have to agree byte for byte, and the community has kept that discipline for ten years.

The implementation layer is where the past two years reorganized the project. The main apache/arrow repository now holds the format specification and the C++ implementation with its Python, R, Ruby, and GLib bindings, while the other languages graduated to their own repositories: arrow-java, arrow-rs for Rust, arrow-go, arrow-js, arrow-dotnet, arrow-swift, and arrow-nanoarrow. That split, mundane as it sounds, unclogged release cycles and let each language community move at its own pace, and the results show in the cadence: the main line shipped 23.0.0 in January 2026, a 23.0.1 security patch for the C++ IPC reader in February, and 24.0.0 in April, while Java shipped its own 19.0.0 in March with a proposal to raise the floor to JDK 17 for the next major, aligning with the broader Java modernization wave across the data stack.

Two implementations deserve their own sentences. Arrow-rs, the Rust implementation, has become an ecosystem force in its own right, the foundation under DataFusion, InfluxDB 3, and much of the new-generation lakehouse tooling in Rust, and in 2025 it attracted more first-time contributors than the main repository itself, 132 to 125. And nanoarrow, at the tiny end, ships the format as a pair of C files that any library can embed without adopting a heavyweight dependency, and its steady release train, 0.8.0 in February 2026 with string view building, LZ4 decompression, and broader packaging, has made "just embed Arrow" viable for R packages, database drivers, and anything else that wants the standard without the toolkit.

The transport layer is where Arrow stopped being only about memory. Arrow Flight is an RPC framework for moving Arrow data over the network at scale, skipping serialization on both ends. Flight SQL layers database semantics on top, queries, prepared statements, catalogs, so a database can expose one wire protocol that any Flight SQL client can consume, and Dremio, InfluxDB 3, and a growing set of engines ship it as a first-class interface. The long-running effort to build a Flight SQL ODBC driver, with sustained contribution from BigQuery-affiliated engineers among others, aims the same protocol at the vast installed base of ODBC tooling.

The connectivity layer is the newest and, I would argue, the most strategically important: ADBC, Arrow Database Connectivity, which deserves its own section.

And the compute layer tells a graduation story. Arrow C++ ships Acero for query execution, and the Java lineage included Gandiva for expression compilation. But the headline is DataFusion, the Rust query engine that began life as an Arrow subproject and grew until it graduated into an independent Apache Top-Level Project, now the embedded engine inside a wave of commercial and open systems. Arrow's compute ambitions succeeded so thoroughly that they left home, which is the best possible outcome for a foundational project: the standard stays neutral, and the engines built on it compete above it.

ADBC: Fixing the Last Slow Mile

If I had to pick the single most consequential Arrow initiative of this decade so far, it is ADBC, because it attacks the last place where the copy tax survived untouched: the database driver.

The absurdity it fixes is easy to state. We spent fifteen years building columnar databases, columnar file formats, and columnar memory, and then connected them to applications through JDBC and ODBC, APIs designed in the early 1990s that hand data over row by row. A columnar database executing a columnar query for a columnar client would pivot results into rows at the driver boundary so the client could pivot them straight back into columns. Every analytical tool in the world was paying this toll on every query, and it was invisible only because it was universal.

ADBC is the columnar replacement: a vendor-neutral API where applications ask for data and receive Arrow, full stop. Drivers for Arrow-native systems pass data through essentially untouched. Drivers for row-oriented systems do the conversion once, inside the driver, where it can be optimized, instead of in every application. It deliberately complements rather than replaces Flight SQL: ADBC is the client-side API, Flight SQL is a wire protocol a server can speak, and an ADBC driver can use Flight SQL, a native protocol, or anything else underneath.

The 2026 status is a project hitting its adoption knee. The libraries shipped version 23 in April, with the API specification itself at 1.1 and a 1.2 milestone underway focused on richer metadata and catalog capabilities. The driver roster now covers Snowflake, BigQuery, DuckDB, PostgreSQL, SQLite, Flight SQL generally, and a newly contributed Go driver for Databricks, with a startup called Columnar, founded around core Arrow contributors including Ian Cook, launching commercial ADBC drivers for Redshift, MySQL, SQL Server, and Trino to fill the long tail. Adoption stories carry real numbers: DuckDB reported query time reductions beyond 90 percent in many applications versus the old driver path, Microsoft adopted ADBC for Power BI connectivity, and the driver manager work now spans Python, Java through JNI bindings to native drivers, and as of this spring, Node.js on NPM. A decade from now, I suspect we will look at row-oriented drivers for analytics the way we now look at hand-rolled CSV parsers: a thing everyone did, and nobody could quite explain why.

The Development Pulse: A Community That Outgrew Its Patron

The health of a ten-year-old open source project is a fair question, and 2025 gave Arrow a stress test with an unusually clean answer.

Voltron Data, the company that had employed a large concentration of Arrow maintainers and hosted project infrastructure, wound down operations in 2025. A project overly dependent on one patron dies or drifts when this happens, and the fact that Arrow's year-end community report treats the event mostly as an infrastructure-migration item, benchmarking systems and nightly builds moved to Arrow-managed accounts, driven by community members, is the tell. The contributor base had diversified past the point of single-sponsor risk years earlier.

The numbers behind that resilience: across the main language implementations, more than 300 new contributors showed up in 2025 alone, with the Rust repository leading the intake. The first Arrow Summit convened in Paris in October 2025, bringing maintainers and users together for the project's first dedicated conference, a milestone that usually marks a community graduating from mailing-list scale to movement scale. Governance has been steady, with the PMC publishing anniversary retrospectives and community highlights that read like a project comfortable in its own skin, celebrating things like stale-issue cleanup and packaging work, the deeply unglamorous maintenance that only healthy communities bother to celebrate.

The release rhythm tells the same story. Major versions land roughly quarterly on the main line, the language repositories ship on their own cadences, ADBC and nanoarrow run their own trains, and a February security advisory for the C++ IPC reader was disclosed, patched, and shipped in 23.0.1 with the boring competence you want from infrastructure. Ten years in, the pulse is strong and, more importantly, distributed.

Penetration: The Part Everyone Underestimates

Here is where I get to make the claim that sounds like advocacy and is just arithmetic: Arrow is plausibly the most widely deployed piece of data infrastructure created in the past decade, and most of its users have never typed the word.

Walk the Python data stack. Pandas ships Arrow-backed data types and leans on Arrow for I/O, and every pandas user reading a Parquet file is running Arrow code. Polars is Arrow-native to its bones. DuckDB exchanges Arrow data zero-copy with everything around it. The result is that essentially every Python data scientist runs Arrow daily, knowingly or not.

Walk the big engines. Spark uses Arrow to make pandas UDFs and Python interchange fast. Snowflake returns results in Arrow through its drivers. Dremio has been Arrow-native since inception, with Flight and Flight SQL as its high-speed interfaces, and the same story repeats across BigQuery interfaces, InfluxDB 3, and the wave of Rust-based systems built on arrow-rs and DataFusion. GPU computing standardized on the same layout through cuDF and its relatives, which is exactly the point of a memory standard: the same bytes that a CPU engine produced can land on a GPU without reshaping.

Walk the AI stack, because this is where the past two years added a whole new continent. Hugging Face's datasets library is built on Arrow, meaning a meaningful fraction of the world's model training data flows through Arrow buffers on its way into GPUs. Vector and embedding workloads lean on Arrow's fixed-size list and tensor extension representations. Observability pipelines adopted Arrow through the OpenTelemetry Arrow protocol to cut telemetry bandwidth dramatically. Geospatial computing built GeoArrow on top of the extension type machinery. And the lakehouse world, my daily habitat, runs on the Arrow-Parquet partnership end to end: Iceberg tables store Parquet that engines read into Arrow, with the newest features like the variant type specified jointly across the Parquet and Iceberg communities with Arrow-resident implementations.

The pattern across every walk: Arrow won by disappearing. It is not a product anyone chooses at a whiteboard. It is the agreement underneath the products, and agreements compound. Every system that adopts Arrow makes Arrow more valuable for the next system, which is why the adoption curve has never had a plateau, only accelerations.

A Worked Example: One Dataset's Day, With Arrow Highlighted

Claims about ubiquity land better traced than asserted, so let me follow one dataset through one ordinary day at a composite company and highlight every moment Arrow is present. Nobody in this story thinks about Arrow once. That is the point.

Morning. Overnight clickstream events landed in an Apache Iceberg table as Parquet files on object storage. A Spark job starts the day's feature engineering, and as it reads those files, the Parquet decoder, developed inside the Arrow project's repositories, decompresses disk pages into Arrow columnar batches in memory. The job's Python UDFs run over pandas batches that Spark hands across the JVM-to-Python boundary as Arrow buffers, the interchange that made this pattern fast enough to be standard practice. First boundary crossed, zero rows pivoted.

Midmorning. An analyst opens a notebook to investigate a metric dip. Her query goes to the lakehouse engine, Dremio in my telling, which plans and executes entirely over Arrow batches internally, then returns results over Arrow Flight. Her notebook's client lands those buffers zero-copy and wraps them as a Polars frame, Arrow-native, so exploration is instant. She pulls a reference table from PostgreSQL through an ADBC driver, receiving Arrow directly instead of paying the row-by-row toll of the old driver stack, and joins the two frames without either being converted, because both were the same bytes-level format all along. Three more boundaries, still zero conversions.

Afternoon. The data science team trains a ranking model on last quarter's data. Their training pipeline loads examples through a datasets library built on Arrow, streaming record batches from Parquet shards into tokenizers and collators, feeding accelerators at a rate that per-record deserialization would have throttled. Meanwhile, the platform team's observability pipeline has been shipping the day's telemetry using the OpenTelemetry Arrow protocol, cutting the bandwidth bill for metrics that describe all of the above.

Evening. The new agentic assistant fields a question from a sales director: how did the campaign perform by region? The agent authenticates through the catalog, issues SQL to the lakehouse, and receives a result set that, in the emerging pattern, travels as Arrow over Flight into the agent's runtime, where it is summarized for a human and forwarded, still columnar, to a charting service. The day's final boundary crossed the same way the first one was.

Count the crossings: Parquet to Spark, JVM to Python, engine to notebook, database to dataframe, frame to frame, storage to trainer, service to collector, lakehouse to agent. A decade ago every one of those was a serialization event, each burning CPU and each an opportunity for types to mangle in translation. Today every one is the same buffers changing hands. Multiply this day by every company running a modern data stack and you have the honest measure of Arrow's penetration: not a market share number, but a tax that an entire industry quietly stopped paying.

Arrow and the AI Era: The Next Decade's Pull

Every technology gets reinterpreted by the era it survives into, and Arrow's reinterpretation is happening now. Julien Le Dem's talk at this spring's Iceberg Summit carried the thesis in its title, column storage for the AI era, and the argument deserves unpacking because it will drive the project's next decade.

AI workloads are, beneath the glamour, the most data-movement-intensive workloads ever deployed. Training pipelines shovel tokens and tensors from storage to accelerators at rates where any per-record overhead is fatal. Feature engineering joins petabytes across systems. Retrieval pipelines move embeddings between stores, indexes, and models continuously. Every one of those movements is a boundary crossing, and boundary crossings are exactly the thing Arrow exists to make free. The stack noticed: the training-data path standardized on Arrow years ago through the datasets ecosystem, and the format's newer types, tensors, views, run-end encoding, read like a list of AI workload accommodations.

The newer and more interesting frontier is agents. Agentic analytics, the pattern where AI agents query, transform, and act on data autonomously, is mostly plumbed today with JSON over HTTP, a format that is to data movement what smoke signals are to fiber optics. Core Arrow contributors have been making the public case that agent-to-data and agent-to-agent channels should carry Arrow, with throughput advantages over JSON-based transport that are not incremental but categorical, especially once results stop being three rows of chat context and start being real analytical payloads. When an agent asks a lakehouse a question, the answer should travel as Arrow over something like Flight, land zero-copy in the agent's runtime, and flow onward without ever being stringified. The pieces all exist. The standardization conversations, including how Arrow-native transport fits alongside agent protocols like MCP, are the ones to watch through 2027.

I will say the Dremio-flavored version plainly, since I have already declared my colors: we built an engine on Arrow a decade ago because moving data without copies was the right architecture for BI. The agent era is that same argument with the volume turned up, because agents issue more queries, chain more systems, and tolerate less latency than humans ever did. The infrastructure bet Arrow's founders made in 2016 turns out to have been a bet on 2026's workloads.

Honest Challenges

A state-of address that skips the hard parts is a press release, so here are the real ones.

Format evolution is a permanent tension. Every new layout, views, run-end encoding, new decimals, makes the format better and fragments it temporarily, because a hundred implementations adopt at a hundred speeds, and data written with new types meets readers that lack them. The community manages this with format versioning, capability negotiation in the protocols, and deliberate slowness, but the tension never resolves, it is only governed. The practical advice for builders: track the canonical extension types and check reader support before adopting the newest layouts on shared boundaries.

Implementation parity is unevenly funded. The C++ and Rust lines are thriving, Go is healthy, and the JavaScript, .NET, and Swift implementations move more slowly, with Java in a modernization push, the JDK 17 floor discussion, repository independence, ongoing Flight SQL investment, that still trails the energy of arrow-rs. Since Arrow's value is precisely its everywhere-ness, the gaps between implementations are the project's most strategic surface, and where new contributors can matter most.

The compute question stays deliberately unresolved. With DataFusion graduated and Acero maintained, Arrow-the-project hosts less of the execution story than it once seemed destined to, and some observers read that as retreat. I read it as focus: the standard stays neutral and universal, engines built on it compete freely, and the project's scarce attention goes to formats, transport, and connectivity, the layers where a neutral steward is irreplaceable. But it is a choice with trade-offs, and reasonable people in the community still argue it.

And invisibility cuts both ways. A standard nobody sees is a standard nobody funds marketing for, and Arrow perpetually punches below its adoption weight in mindshare, conference keynotes, and, frankly, corporate sponsorship of maintenance. The post-Voltron community proved it can carry the load. The load is still real, and the anniversary retrospectives' emphasis on thanking maintainers for janitorial work is both charming and a quiet fundraising pitch the ecosystem should hear.

Getting Hands-On: Where to Start by Role

Since "Arrow is everywhere" can leave people unsure where to actually touch it, here is my starting map by role, each entry a first project you can finish in an afternoon.

If you are a data analyst or scientist, make the invisible visible. Take a workflow you already run, a Parquet read into pandas or a pull from a warehouse, and switch the pandas path to Arrow-backed dtypes or swap the connection to an ADBC driver, then measure. The before-and-after on a wide result set is the fastest way to internalize what the copy tax was costing you, and the code change is usually a handful of lines. Then hand the same data between pandas, Polars, and DuckDB in one notebook and notice that the handoffs cost nothing, because nothing is being converted.

If you are a data engineer, adopt the interchange consciously. Next time two services in your pipeline exchange data through JSON or CSV over HTTP, prototype the same hop as Arrow IPC or Flight and benchmark both directions. Then look at your driver layer: every JDBC or ODBC connection feeding an analytical workload is a candidate for an ADBC swap, and the drivers for the major warehouses are mature enough to trial in a day.

If you are a platform or infrastructure builder, the leverage points are Flight SQL on your serving side and nanoarrow at your edges. Exposing Flight SQL puts your service one protocol away from every ADBC, JDBC, and eventually ODBC client at once, and embedding nanoarrow gives you the format in constrained environments, drivers, plugins, embedded readers, without a heavyweight dependency. And if you write Rust, arrow-rs plus DataFusion is the fastest path from zero to a working columnar engine that exists today, which is exactly why so many new systems start there.

If you want to contribute, the community highlights each year effectively publish the wish list: implementation parity in the smaller languages, the Flight SQL ODBC driver, packaging and docs, and triage in the busiest repositories. More than 300 people made their first Arrow contribution in 2025. The water is warm, and the maintainers are famously welcoming to newcomers who show up with a reproduction or a benchmark in hand.

The Standards Lesson Arrow Keeps Teaching

One more reflection before the questions, because Arrow's decade doubles as the best case study we have in how open data standards actually win, and the lesson generalizes to every layer of the stack I write about.

Arrow never had a killer feature. At any given moment, some proprietary format was faster for some workload, some vendor's interchange was more convenient inside its own walls, and a benchmark could always be constructed where Arrow looked merely fine. What Arrow had instead was a property no proprietary alternative could offer: neutrality with staying power. It was governed at a foundation, implementable by anyone, and guaranteed not to tilt toward any vendor's interest, which made it the only format that competitors could all adopt without arming each other. Snowflake and Databricks and Dremio and Google do not agree on much. They can all agree on Arrow, precisely because agreeing on it concedes nothing.

That property compounds in a way features do not. A feature advantage erodes as competitors copy it. A neutrality advantage grows with every adopter, because each new system that speaks Arrow raises the cost of speaking anything else. Economists call it a network effect. I prefer the plainer framing: standards win by making disagreement expensive, and Arrow spent a decade patiently raising the price of disagreement about memory layout until nobody could afford it.

The same play has now run twice more in my corner of the industry. Parquet raised the price of disagreement about analytical storage. Iceberg raised it for table metadata, and its REST protocol is raising it for catalogs, with Polaris as the neutral implementation. In every case, the winning move was the same: put the specification where no one controls it, let competitors co-invest safely, and wait for compounding to do what marketing cannot. When people ask me why I bet my career on open lakehouse architecture, this is the answer in miniature. The open standard is not the idealistic choice that costs performance. Over any horizon longer than a budget cycle, it is simply the winning strategy, and Arrow is the decade-long proof.

It also explains the correct posture toward the challenges I listed above. Format evolution tension, uneven implementation funding, and the invisibility problem are not signs of weakness. They are the operating costs of neutrality, the price a standard pays for belonging to everyone. Ten years of receipts say the community pays those costs willingly, and the ecosystem is enormously richer for it.

Questions I Hear Most Often

The recurring questions, answered directly.

Is Arrow a database? A file format? What do I install? Neither. Arrow is a specification for in-memory columnar data plus a family of libraries implementing it, and mostly you do not install it, it arrives inside tools you already use. You reach for Arrow directly when you build data infrastructure: a driver, an engine, a pipeline between systems, a service returning analytical results. For everyone else, Arrow is why your pandas-to-Spark handoff or your DuckDB-to-Polars handoff stopped costing anything.

Arrow versus Parquet, one more time? Disk versus memory. Parquet optimizes for storage: heavy compression, encodings that trade CPU for size, layouts for selective reads from slow media. Arrow optimizes for computation: layouts a CPU can vectorize over, no decode step, zero-copy sharing. You want both, they are designed as a pair, and their implementations increasingly live in the same repositories maintained by the same people.

Does Arrow make my queries faster? Indirectly and honestly: Arrow removes overhead between systems rather than accelerating the computation inside one. If your workload never crosses a boundary, Arrow gives you little. The moment data moves, between libraries, processes, machines, drivers, or accelerators, Arrow converts a copy-and-convert cost into approximately zero. Since real pipelines cross boundaries constantly, the savings are ubiquitous, which is different from magical.

Should my team adopt Flight SQL or ADBC? Both, at different layers, and the confusion is common enough to spell out. ADBC is the API your applications code against, one interface for many databases, always yielding Arrow. Flight SQL is a protocol a server exposes on the wire. A database that speaks Flight SQL serves ADBC clients beautifully, and ADBC drivers exist for systems that speak nothing of the sort. If you build clients, adopt ADBC. If you build servers, expose Flight SQL. If you do neither, watch your vendors do it for you, which they are.

Is Arrow still healthy after the Voltron Data wind-down? The 2025 evidence says clearly yes: 300-plus new contributors across the implementations in a single year, infrastructure migrated to community-managed accounts without visible disruption, releases on cadence across every repository, a security response handled cleanly, and the first Arrow Summit convened. Concentration risk was the fair worry five years ago. The record since is a case study in a community outgrowing its founding patrons, which, I will note with a smile, is also the Arrow origin story itself.

Where does Arrow fit in my lakehouse architecture? Everywhere data moves and nowhere you have to draw it. Parquet under your Iceberg tables on disk, Arrow inside every engine that reads them, Flight or Flight SQL when results cross the network, ADBC when applications and notebooks connect, and increasingly Arrow again when agents consume the answers. The lakehouse pitch has always been open formats at every layer so no layer locks you in. Arrow is that pitch applied to the layer nobody used to think about: the memory between systems.

Closing Thoughts

Ten years ago, a group that included Dremio's founders, the Parquet creators, and the author of pandas agreed on something unusual: the industry's biggest performance problem was not inside any system, it was between them, and the fix was not a product but an agreement. Everything since, the format, the languages, Flight, ADBC, the graduated engines, the AI-era reinterpretation, is that agreement compounding.

The state of Arrow in 2026 is that the agreement won so completely it became ambient. Three hundred new contributors a year tend a standard that billions of daily operations flow through unnoticed, the copy tax that once consumed most of analytics' CPU cycles has been engineered down toward zero across an entire industry, and the workloads of the next decade, agents and models moving data at rates humans never demanded, are pulling the project forward rather than past it. Infrastructure this successful stops being news. It becomes assumption, and assumptions are the most durable things in computing.

If you want to build the kind of understanding that lets you see through the stack this way, from memory formats through table formats through the catalogs and engines above them, that is exactly what I write books for. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, and my other titles cover lakehouse architecture, data engineering, and the agentic analytics wave now reshaping all of it.

Browse the full collection of my books on data and AI at books.alexmerced.com.

The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse

Alex Merced — Wed, 08 Jul 2026 22:53:22 +0000

I have a personal stake in this one, so let me declare it up front. Apache Polaris was co-created by Snowflake and Dremio, I work at Dremio, and I co-authored Apache Polaris: The Definitive Guide for O'Reilly. I have watched this project from the first commit, through donation to the Apache Software Foundation in August 2024, through eighteen months of incubation, and past its graduation to a Top-Level Project in February 2026. I am not a neutral observer. What I can promise instead is accuracy, receipts from the dev list, and honesty about what is finished versus what is still forming.

The short version of where Polaris stands in July 2026: the project graduated, the release train runs monthly, federation and governance have gone from roadmap slides to shipped extensions, and the community's current arguments are about semantic layers, lineage, and serving AI agents, which tells you the foundation underneath is no longer in question. The catalog conversation that dominated 2025, which implementation should you trust, has largely resolved into a different and better question: how much of your lakehouse should the catalog govern?

This article walks through all of it. What Polaris is and why the catalog layer became the architecture decision of this era. What has shipped release by release. What the dev list is debating right now, in July, and why those debates matter. Where adoption actually stands across engines and vendors. And what I would do if I were choosing a catalog this quarter. As always, my goal is that the logic clicks, so that every future Polaris announcement makes sense to you on arrival.

Why the Catalog Became the Decision That Matters

A quick foundation for anyone arriving fresh, because the stakes only make sense with the architecture in view.

An Apache Iceberg table is files plus metadata, and something has to hold the authoritative pointer that says which metadata file is current. That something is the catalog. Every commit routes through it. Every engine finds tables through it. It is a small service with an outsized position: whoever controls the catalog controls access, and whoever controls access controls governance.

For years this layer was an afterthought, a Hive Metastore inherited from another era or a cloud service adopted by default. Two things changed that. First, the Iceberg REST Catalog specification turned the catalog into a protocol rather than a library, meaning any engine speaking HTTP could work with any compliant catalog, which made the catalog choice consequential and portable at the same time. Second, the format war ended. With Iceberg established as the shared substrate across Snowflake, Databricks, AWS, Google, Microsoft, and the open source engines, the table format stopped differentiating anyone. Competition moved up a layer, to the catalog, and 2025 became the year of what many called the catalog wars.

Polaris exists as the open answer to that fight. The argument for it mirrors the argument for Iceberg itself a few years earlier: commercial catalogs serve their creators' ecosystems well, but the ecosystem needs a catalog primitive that no single vendor controls, governed at a neutral foundation, extensible by anyone. Snowflake and Dremio co-created it, donated it to the ASF in August 2024, and an incubation community that grew to include contributors from Google, Microsoft, Confluent, AWS, and dozens of other organizations carried it to graduation. Polaris implements the Iceberg REST specification and layers on the things a production catalog needs that the spec does not define: multi-catalog management, role-based access control, credential vending, federation, and a policy store.

That is the setup. Now the state of play.

Graduation, and Why It Mattered More Than a Badge

In February 2026, Apache Polaris graduated from the Apache Incubator to become a Top-Level Project. It is worth pausing on what that actually certifies, because the ceremony obscures the substance.

Incubation at the ASF is not a waiting room. It is an audit. A project must demonstrate that its community is diverse enough to survive any single vendor walking away, that its governance follows the Apache way of open decision-making on public lists, that its releases meet legal and procedural standards, and that committership grows on merit. Projects fail incubation regularly. Polaris passed in eighteen months, which for a project born from two competing vendors is fast, and the diversity requirement is the one I would highlight. The committer and PMC rolls now span cloud providers, engine vendors, governance vendors, and independents, and the new-committer announcements have kept arriving through the spring, with names like Christopher Lambert and Nandor Kollar welcomed in recent months.

Graduation also changed the project's posture in ways you can observe. The first post-graduation board report went out in March under the coordination of Jean-Baptiste Onofré, the release cadence tightened, and the dev list picked up the kind of process threads that signal a project settling in for the long haul: merge-button policies, code organization for supporting multiple Spark lines, dependency modernization threads on Jackson 3 and Quarkus 4 readiness, and a debate over the future of the regression test infrastructure. None of that is glamorous. All of it is what a ten-year project looks like in year two.

For adopters, the practical meaning is risk reduction. A TLP with a diverse PMC cannot be strategically strangled or quietly abandoned by any one company, including the two that created it. That property, more than any feature, is what enterprises were waiting to see before betting governance infrastructure on the project.

The Release Train: 1.0 Through 1.6 in One Year

The clearest way to see Polaris maturing is to walk the releases, because the arc from mid-2025 to mid-2026 tells a coherent story.

Version 1.0 arrived in the summer of 2025 as the first production-ready release: a single downloadable binary, a published Helm chart for Kubernetes, support for external identity providers like Okta and Google alongside the built-in identity system, and the first version of the policy store, a persistent home for policies like compaction and snapshot expiration with REST endpoints for managing their lifecycles. It also planted three seeds explicitly labeled experimental: generic tables for non-Iceberg formats, an event listener framework, and catalog federation. Hold those three in mind, because the next year of releases is largely the story of those seeds growing up.

Version 1.3.0 shipped in January 2026 and matured two of them. Generic tables went generally available, letting Polaris reliably catalog Delta Lake and Hudi tables alongside Iceberg in the same namespaces, a meaningful step for every organization mid-transition between formats. Observability arrived through native Iceberg metrics reporting: engines can now push query-level execution metrics, rows scanned, bytes read, commit activity, back to the catalog through the Iceberg REST API, turning Polaris from a passive metadata store into a source of operational signal about how tables actually get used. And the release introduced integration with Open Policy Agent, the first step toward externalized, auditable authorization beyond static role grants.

Version 1.4.0, in April, was the first release after graduation, and it read like a hardening release for regulated environments. Credential vending, the mechanism by which Polaris hands engines short-lived, scoped storage tokens instead of letting credentials sprawl across client machines, gained AWS STS session tags so storage access can be correlated in CloudTrail audits, plus storage-scoped credentials and S3 KMS encryption support. CockroachDB joined the persistence backend options. Metrics gained database persistence. And federation, the third seed from 1.0, reached Hive Metastore, AWS Glue, and external Iceberg REST catalogs.

Version 1.5.0, in May, pushed federation further with Google BigQuery Metastore support contributed through the community, meaning a single Polaris instance can now project tables living in GCP's metastore as standard Iceberg REST endpoints next to everything else it manages. The credential vending payload was restructured into a unified format with consistent expiration semantics, the kind of unglamorous refinement that matters enormously when distributed query executors need consistent session keys mid-job. And version 1.6.0 landed on schedule at the end of June, keeping the monthly-to-six-weeks cadence the project has now sustained for a year.

Step back from the individual items and the arc is unmistakable. The 2025 question was "does it work." The 2026 releases answer operator questions: can I audit it, can I encrypt it, can I run it on my database, can I point it at the catalogs I already have. That progression, from capability to operability, is what production adoption actually requires, and the release notes show the project's center of gravity has moved there.

Federation: The Catalog of Catalogs Became Real

Of everything that shipped this year, federation deserves the deepest look, because it changed what kind of thing Polaris is.

The original pitch for any catalog is centralization: put your tables in me. The problem with that pitch is that no real enterprise starts from zero. Tables already live in Glue because the AWS account predates the lakehouse strategy. Tables live in a Hive Metastore because the Hadoop era happened. Tables live in BigQuery Metastore because one division runs on GCP. A catalog that demands migration before it delivers governance has priced itself out of most organizations.

Federation inverts the pitch. Polaris registers external catalogs, Hive, Hadoop, Glue, BigQuery Metastore, other Iceberg REST catalogs, as federated sources and projects their contents through its own endpoints, governed by its own access model. The repository now carries dedicated extension modules for Hive, Hadoop, and BigQuery federation, and the dev list through the spring has been working the practical edges: how credentials pass through to federated sources, including an active question on STS token passthrough for federated catalogs, and how multiple data sources can be configured with runtime activation, one of the busier threads of the past two months.

The strategic meaning is that Polaris stopped competing with your existing catalogs and started offering to govern them. You adopt it as a layer, not a destination. Migration becomes optional and gradual rather than a prerequisite, with the separate catalog migrator tooling available when consolidation does make sense. In my conversations with platform teams, this reframing has done more for Polaris adoption than any single feature, because it converts a rip-and-replace proposal into an additive one. It is also the architecture that multi-cloud reality demands: nobody's tables live in one place, so governance has to be the thing that spans places.

Dremio's own product architecture reflects the same philosophy, for what it is worth: the Dremio Open Catalog is Polaris at the core with federated sources around it, presenting one governed namespace across Iceberg tables, databases, warehouses, and external catalogs. Snowflake's Open Catalog offers Polaris as a managed service from the other co-creator's side. Both companies betting their catalog products on the same open core is exactly the outcome the donation was designed to produce.

Governance: From RBAC to Policy Engines to Portable Policy

The second major thread of the year is governance depth, and it layers up nicely.

The base layer, shipped and stable, is Polaris's native model: principals, principal roles, and catalog roles, with grants at catalog, namespace, and table level, enforced identically no matter which engine comes through the REST API. Add credential vending and you get the property that makes security teams exhale: engines never hold long-lived storage credentials at all, they receive short-lived scoped tokens per operation, now with the audit correlation and encryption support from 1.4.

The middle layer, maturing fast, is externalized authorization. The OPA integration that arrived in 1.3 lets authorization decisions route to an external policy engine, so access rules can be expressed as policy code, versioned, tested, and aligned with the policy systems an organization already runs elsewhere. A Ranger extension sits alongside it in the repository for shops standardized on that ecosystem, and the community has been running dedicated syncs on Polaris authorization to work the design forward, with fine-grained access control, row and column level, as the destination the roadmap has pointed at all along.

The top layer, and the one to watch skeptically and hopefully at once, is policy portability across systems. In April 2026, Snowflake publicly committed to governance portability through Polaris, the idea that access policies authored in one system could be enforced by another, with Polaris as the exchange point. I flagged then, and maintain now, that the announcement was directional: as of this writing, the policy exchange mechanics are still more proposal than shipped engineering, and governance federation remains in preview territory rather than general availability. But the direction is the right one, and it is the same direction the open format movement has always pointed: the policies about your data should be as portable as the data itself. If that vision lands, it lands through a neutral catalog, which is precisely the position Polaris was built to occupy.

My honest scorecard on governance: the base layer is production-proven, the policy engine layer is real and deployable with engineering effort, and the portability layer is a promise with credible momentum. Plan accordingly.

The July Dev List: Semantic Layers, Lineage, and the Next Perimeter

Now the freshest material, because the current dev list traffic shows where the community's ambition points next, and the past two months have been remarkably busy.

The discussion that would have surprised me a year ago is semantic layer support in Apache Polaris, one of the highest-traffic threads of the season, running alongside a formal vote that passed to accept an open semantic model API specification into the project's orbit. The reasoning: a catalog already knows every table, every schema, and every permission. Metrics definitions, business entities, and model semantics are metadata of the same kind, and the rise of AI agents querying lakehouses directly has made a machine-readable semantic layer urgent rather than nice-to-have. An agent that knows only tables and columns guesses at business meaning. An agent that can ask the catalog what "active customer" means stops guessing. The community deciding that this belongs near the catalog, governed like everything else, is a genuinely important scope expansion, and it echoes the Table Sources direction discussed earlier in the year of extending Polaris toward a universal registry for tables, views, functions, metrics, and models.

Lineage is moving too, through a sustained OpenLineage proposal thread exploring how Polaris should emit and integrate lineage information, which pairs naturally with the proposal for REST endpoints exposing table metrics and events and a companion thread on forwarding Iceberg scan and commit metrics through the event system. Assemble those pieces and you can see the shape being built: the catalog as the observability plane of the lakehouse, knowing not just what exists but what happened, who touched it, and where it flows. There is even a thread on mechanisms to purge the events and metrics tables, which tells you people are running this at enough volume to worry about retention.

The protocol-level work continues alongside. An idempotency-key design discussion for the Iceberg REST API has been converging on a preferred model, addressing the retry-safety problems that any high-commit-rate deployment eventually meets. A thread on status codes for rename conflicts sounds trivial and is exactly the kind of precision that makes a spec implementable twice. Iceberg table encryption support has its own active design discussion. And a long thread on non-IRC endpoints appearing in IRC config responses, the single busiest subject of the past two months, is the community carefully negotiating how Polaris-specific capabilities should surface without polluting the standard Iceberg REST surface, which is precisely the discipline that keeps an implementation from drifting into a dialect.

Two more threads round out the picture of a project growing into its user base. A discussion on whether the Polaris web console should live in the main repository signals that the human interface, long the gap between Polaris and the commercial catalogs, is becoming a first-class concern. And a scale question from the field, asking about the feasibility of one realm per tenant at ten thousand tenants, tells you who is showing up: platform builders embedding Polaris inside multi-tenant products. When your issue tracker fills with questions about the tenth thousand tenant rather than the first table, the adoption story is telling itself.

Adoption: Engines, Vendors, and the Competitive Field

So who actually uses this thing? The picture in July 2026 has three rings.

The inner ring is engine support, which is now broad enough to stop listing apologetically. Polaris's REST implementation is exercised by Apache Spark, Apache Flink, Trino, Apache Doris, StarRocks, Dremio, and the growing family of Iceberg clients in Python, Rust, and Go, since anything that speaks the Iceberg REST protocol speaks Polaris. This was always the design bet: implement the standard faithfully and inherit the ecosystem, rather than courting integrations one by one.

The middle ring is managed offerings. Snowflake Open Catalog delivers Polaris as a managed service with Snowflake's Horizon governance integrated around it, and Snowflake's own engine reads and writes Iceberg through it. Dremio ships Polaris at the core of its catalog experience, wrapped with query federation and semantic layer capabilities, provisioned by default in Dremio Cloud. The pattern to appreciate: both co-creators sell managed Polaris and compete on what surrounds it, which keeps both invested in the core while keeping the core neutral. Meanwhile self-managed deployment has gotten honestly easier, with the single binary, Helm chart, and a widening set of persistence backends, though running any highly available JVM service with a database remains real operational work, and teams should budget for it.

The outer ring is the competitive field, and it deserves a sober paragraph rather than a victory lap. Unity Catalog remains the center of gravity for Databricks-first shops and has genuine strengths in AI asset governance and business semantics that Polaris is only now moving toward, exactly the gap the semantic layer work addresses. Apache Gravitino, a TLP since mid-2025, pursues the federated metadata lake vision with an AI model catalog and agent-facing features, overlapping Polaris most directly on the catalog-of-catalogs story. Lakekeeper offers a lean Rust-native REST catalog for teams that want minimal footprint. Nessie continues to serve the git-for-data niche, and the cloud-native catalogs, Glue, BigLake Metastore, OneLake, keep improving their REST faces. The stabilizing truth across all of it is that the REST protocol is the meeting point: because engines configure against the protocol, catalog choices have become reversible in a way they never were in the Hive era, and federation makes coexistence a strategy rather than a compromise. Polaris's differentiated claim in that field is the one it was founded on: the most standards-faithful, vendor-neutral implementation, governed where no roadmap can be unilaterally captured.

What I Would Do: Guidance for Teams in 2026

Let me translate the state of things into decisions, the way I would advise a platform team this quarter.

If you are standing up a new Iceberg lakehouse, Polaris is the default I would start from, either self-managed if you have the operational muscle or through a managed offering if you do not. You get the standard protocol, credential vending, RBAC, and a federation path for everything you cannot migrate, and the TLP governance de-risks the decade-long horizon. Configure engines against the REST protocol and you preserve the option to change your mind, which is worth more than any feature comparison.

If you are living with Glue, Hive, or a warehouse-native catalog today, you no longer face a migration decision. Register what you have as federated sources, put Polaris in front as the governance and discovery plane, and migrate tables opportunistically or never. Run the numbers on the operational side honestly: a federated Polaris is one more service on your critical path, so it should earn its place through consolidation of access control, not architectural fashion.

If you are betting on AI agents against your lakehouse, and increasingly everyone is, watch the semantic layer and events work closely and design toward it. Agents need three things from a catalog: discovery of what exists, enforcement of what they may touch, and semantics for what things mean. Polaris ships the first two today and is building the third in the open. Structuring your metadata, descriptions, and access model now, with that trajectory in mind, is cheap insurance.

And wherever you land, hold every catalog, Polaris included, to the same tests I apply throughout this article: does it implement the standard without dialect drift, can another vendor's engine read and write through it without permission or plugins, and can you leave it without rewriting your platform. Catalogs are the control point of the lakehouse. The entire point of the open movement is that control points should be things you choose, continuously, rather than things that choose you.

Credential Vending, Explained Properly

Since credential vending shows up in nearly every Polaris conversation and most explanations of it stay at the bullet-point level, let me give it the accessible treatment, because it is the single feature that most changes the security posture of a lakehouse.

Start with the problem. In a lakehouse, data lives in object storage and compute lives everywhere else: Spark clusters, Flink jobs, BI tools, notebooks, and now AI agents. Traditionally, every one of those compute environments needed storage credentials, an IAM key or service account with rights to the buckets. Multiply that across teams, tools, and environments and you get credential sprawl: long-lived secrets sitting in cluster configs, notebook environments, and CI pipelines, each one a breach waiting for a leak, each one scoped more broadly than any single job needs because narrow scoping across that many holders is unmanageable. Security teams hate this picture for good reason, and it has quietly blocked more lakehouse projects than any performance concern ever did.

Credential vending inverts the model. Engines hold no storage credentials at all. They hold one thing: an identity with Polaris. When an engine wants to read or write a table, it asks Polaris, Polaris checks the RBAC grants for that principal against that table, and only then does Polaris turn to the cloud provider and mint a short-lived, narrowly scoped token, rights to exactly the storage paths that table occupies, expiring in minutes or hours. The engine receives the token, does its work, and the token dies on schedule. Nothing long-lived ever leaves the vault.

The analogy I use on stage: the old model is giving every hotel guest a master key because managing hundreds of room keys is annoying. Vending is the front desk checking your reservation and cutting a key card that opens your room only, and expires at checkout. Nobody audits master keys. Everybody can audit key cards.

The 2026 releases show what happens after the concept works and the auditors arrive. Session tags flowing into CloudTrail mean every vended credential can be correlated to the principal and operation that requested it, so storage access logs finally answer "who" rather than "which shared key." Storage-scoped credentials and the unified payload format tighten exactly what each token can touch and keep distributed executors consistent mid-query. KMS support extends the model to encrypted buckets. And the dev list thread on a GCP counterpart to the AWS session tags work shows the same treatment spreading across clouds. This is what I mean when I say Polaris crossed from capability to operability this year: vending existed in 2024, and in 2026 it satisfies an audit.

One more implication worth spelling out, because it connects to where the ecosystem is going. Credential vending is the mechanism that makes multi-engine governance real rather than rhetorical. Access rules enforced only inside one engine evaporate the moment a different engine touches the same files. Rules enforced at the catalog through vending bind every engine equally, including engines that did not exist when the rules were written, including the AI agents now arriving. The catalog can be the policy point precisely because it is the credential point.

A Worked Example: One Company, Three Stages of Adoption

Abstract capability lists never convince anyone, so let me run a composite scenario drawn from the adoption patterns I actually see, one fictional company moving through three honest stages.

The company: a retailer with a data estate that grew by accretion. Core analytics tables live in Iceberg on S3, cataloged in AWS Glue because that was the default. A legacy Hive Metastore governs an older Hadoop-era warehouse that finance still queries. One acquired business unit runs on GCP with tables in BigQuery Metastore. Spark handles ETL, Dremio serves BI, a growing crew of data scientists uses Python and DuckDB, and this year the CTO wants agents answering questions against governed data.

Stage one: govern without moving anything. The platform team deploys Polaris, either self-managed on Kubernetes via the Helm chart with PostgreSQL behind it, or through a managed offering. They do not migrate a single table. Glue, the Hive Metastore, and BigQuery Metastore register as federated catalogs, and Polaris projects all three through one set of Iceberg REST endpoints. Engines repoint their catalog configuration at Polaris, a config change, not a code change, precisely because everything speaks the REST protocol. The immediate win is a single discovery plane: for the first time, one namespace answers "what tables exist," spanning three clouds' worth of history. The second win lands with security: principals and roles get defined once, and credential vending replaces the zoo of IAM keys living in Spark configs. Nothing about the data moved. The governance moved.

Stage two: consolidate where it pays. With federation carrying the legacy, the team makes migration a business-case decision instead of a prerequisite. New tables get created directly in Polaris-managed internal catalogs. The highest-value Glue tables migrate with the catalog migrator tooling when their pipelines get touched anyway. The Hive Metastore is left to age in place behind federation, queried but frozen, on a path to eventual retirement that no longer blocks anything. Meanwhile the operational muscles build: metrics reporting flows from engines back into Polaris, so the team can finally see which tables are hot, which are dead, and which governance rules actually get exercised. Policies for retention and maintenance land in the policy store. OPA arrives when the security team wants access rules expressed as reviewable policy code alongside the rest of their infrastructure policies.

Stage three: the agent era. The CTO's agents arrive, and the earlier stages turn out to have been the preparation. Each agent gets a principal with narrow role grants, exactly like a human analyst, and vending ensures an agent can never touch storage beyond its grants, no matter how creatively it composes queries. Discovery endpoints give agents the map of what exists. And as the semantic layer work lands in Polaris, the definitions the BI team curated, what counts as an active customer, how margin is computed, become machine-readable context the agents consume from the same governed catalog, rather than folklore embedded in prompts. The audit story that took stage one and two to build now covers human and machine access identically, which is the only version of agent governance that survives contact with a compliance review.

Three stages, no big-bang migration, each stage justified by its own returns. That is the adoption shape federation made possible, and it is why I keep saying the 2026 Polaris story is less about features than about the removal of reasons to wait.

How to Follow and Join the Project

A short practical section, because "get involved" advice usually stays too vague to act on, and Polaris is at the stage where new participants shape real outcomes.

For following along, the dev mailing list is the source of record, archived and searchable at lists.apache.org, and its traffic is manageable, a few hundred messages in a busy month across perhaps thirty active subjects. Skim subjects weekly and open anything tagged DISCUSS, PROPOSAL, or VOTE that touches your concerns. The GitHub discussions carry the roadmap and the longer design documents, the community syncs on topics like authorization are announced on the list with notes posted back, and the project publishes its board reports publicly now that it reports directly to the ASF board. An hour a month keeps you genuinely current, and my weekly Apache newsletter exists for people who would rather have that hour done for them.

For contributing, the honest advice is to start where the project's growth is creating gaps. Federation extensions want more sources, and the BigQuery module arriving from a community contributor is the template: a well-scoped extension with tests, developed in the open. The console, documentation, Helm chart refinements, and the testing infrastructure threads all welcome hands that are not ready to touch the authorization core. Client-side work in the Iceberg language ecosystems, Python, Rust, Go, exercises Polaris constantly and surfaces protocol issues worth reporting. And if your organization runs Polaris at any interesting scale, writing up what you hit, the way the ten-thousand-tenant question arrived on the list, is a contribution the project visibly values, because production truth is the scarcest input any young project has.

The committer announcements this year all followed the same path: sustained, visible, useful participation. There is no other door, which is rather the point of the Apache way.

Questions I Hear Most Often

The recurring questions from meetups, the podcast, and customer conversations, answered directly.

Is Polaris just a Snowflake project wearing an Apache jacket? The graduation process exists to answer exactly this, and the observable evidence says no. The PMC and committer base span competing vendors and independents, contribution flow comes from well beyond the two founders, with the BigQuery federation work arriving from the community being a nice concrete example, and decisions happen on a public list where anyone can watch vetoes and votes. Snowflake and Dremio remain the largest investors of engineering time, which is normal for young ASF projects, and the trajectory of committer announcements points the right direction. Judge it by the archives, which are public, rather than by anyone's assurances, including mine.

Polaris or Unity Catalog? The honest answer is that this is mostly a question about your center of gravity. Deep Databricks estates get real value from Unity's integration and its maturity on AI asset governance. Multi-engine, multi-vendor estates get real value from Polaris's neutrality and standards fidelity. The two increasingly meet at the Iceberg REST boundary, and federation features on both sides make coexistence practical. What I push teams to reject is the framing that this choice is permanent: configure against the protocol, keep your governance rules exportable, and the decision stays revisable.

Can Polaris handle non-Iceberg tables well enough to be my only catalog? Generic tables are generally available and genuinely useful for cataloging Delta and Hudi assets in one namespace, with discovery and access control unified. Be precise about what that does and does not mean: Polaris governs the entry point, but format-specific capabilities still depend on the engines reading those tables, and write-path parity across formats remains an evolving story, with directions like Delta write support and file-based tables living on the roadmap rather than in releases. For a mixed estate mid-migration, it is already the right tool. For a permanently multi-format strategy, watch the generic table and directories threads closely.

What is the operational footprint really like? A Quarkus JVM service, a relational persistence backend with PostgreSQL the common choice and CockroachDB now supported, and the usual high-availability engineering around both. The Helm chart makes Kubernetes deployment straightforward, and the diagnostics and testing threads on the dev list show operability getting steady attention. It is not heavy by data infrastructure standards, and it is not free. The managed offerings exist precisely for teams that want the protocol without the pager.

Does Polaris support the newest Iceberg features like v3 tables? Polaris tracks the Iceberg REST specification, and its job is to broker metadata rather than interpret every table feature, so support for format-version capabilities mostly rides on the Iceberg libraries and the engines. The catalog-relevant edges, like the idempotency work, metrics reporting, and the encryption discussion, are active on the dev list. In practice, teams are running v3 tables through Polaris with current engines today, and the catalog has not been the bottleneck in that story.

What should I watch for the rest of 2026? Four things. The semantic layer implementation following the accepted specification, because it defines whether Polaris closes the AI-governance gap. The authorization work maturing toward fine-grained policies, because that is the enterprise checklist item most often asked about. The events and lineage endpoints hardening, because the observability plane is the next competitive front. And the release notes of whatever ships after 1.6 on the now-reliable cadence, because in this project, the release notes have become the roadmap made visible.

Closing Thoughts

Two years ago, Polaris was an announcement and an argument: that the catalog layer, like the table format before it, was too important to belong to anyone. One year ago it was an incubating project with a 1.0 and a lot of experimental flags. In July 2026 it is a Top-Level Project on a monthly release train, with federation and credential vending in production, governance layering up from RBAC through policy engines toward portability, and a community whose current debates, semantics, lineage, agents, tenancy at the ten-thousand scale, are the debates of a project whose fundamentals are settled.

The catalog wars framing is fading, and what replaces it is quieter and better: a protocol everyone implements, a neutral reference implementation anyone can run, and competition moving to the layers above. That is how Iceberg won, and it is the path Polaris is walking a few years behind, co-created by two competitors precisely so that it could belong to neither.

If you want to go deeper than an article can take you, that is what the book is for. I co-authored Apache Polaris: The Definitive Guide and Apache Iceberg: The Definitive Guide for O'Reilly, and I have written further titles on lakehouse architecture, data engineering, and agentic analytics, all built to turn these systems from vocabulary into working knowledge.

Browse the full collection of my books on data and AI at books.alexmerced.com.

The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough

Alex Merced — Wed, 08 Jul 2026 22:50:51 +0000

The most common architecture question I get in 2026 is no longer "should we use Iceberg." That one is settled. The question now is "how fresh can our Iceberg tables be," followed immediately by "and what do we do when that is not fresh enough."

Those two questions deserve a serious answer, because the space has gotten crowded. Open source engines, Kafka Connect sinks, brokers that write Iceberg natively, managed vendor pipelines, and streaming databases all now claim the same job: moving events from a stream into an Iceberg table. They differ wildly in latency, cost, operational burden, and what they quietly break. Picking among them without understanding the underlying physics is how teams end up with a pipeline that technically works and a table that queries terribly.

So this article is my July 2026 map of the territory. We will start with the physics, because every option is a different negotiation with the same constraints. Then we will walk the options one by one, open source and vendor, with honest latency expectations and honest trade-offs. And we will finish with the part most articles skip: the architectural patterns for workloads where even a well-tuned Iceberg pipeline is not fast enough, and how to serve sub-second freshness without giving up the lakehouse.

As always, my goal is that the logic clicks. You should leave able to reason about any new streaming-to-Iceberg product from first principles, because new ones arrive monthly and the physics never changes.

The Physics: Why Iceberg Has a Freshness Floor

Everything in this article follows from three facts about how Iceberg works. Get these into your head and every product claim becomes easy to evaluate.

Fact one: data is invisible until committed. An Iceberg table is defined by its metadata. A writer can push Parquet files to object storage all day, but no query sees a single row until a commit publishes a new snapshot that references those files. The commit is the moment of visibility. So end-to-end freshness is never just "how fast can I write data." It is "how often do I commit," plus the time the data spent buffering before the commit, plus the time a query engine takes to notice the new snapshot. Any latency number that ignores the commit cadence is marketing.

Fact two: commits are not free. Every commit writes new metadata, a fresh metadata file, manifest list, and manifests under the current v3 format, and performs an atomic swap through the catalog. That work takes real time against object storage, typically measured in seconds, and concurrent committers to the same table contend and retry. This creates a practical floor. Commit every second and metadata work dominates, the catalog becomes a chokepoint, and snapshot history balloons. The workable range for commit intervals today runs from a few seconds at the aggressive end to minutes at the comfortable end. This is exactly the write amplification problem that the Iceberg v4 proposals around single-file commits and adaptive metadata trees are designed to shrink, and it is worth knowing that help is coming at the format level. But you are building on v3 today, and on v3 the floor is real.

Fact three: frequent commits create small files, and small files poison reads. Commit every ten seconds and you produce 8,640 commits a day, each adding files sized by whatever trickled in during those ten seconds. Thousands of tiny Parquet files mean thousands of object store requests per query, bloated metadata, and planning that slows week by week. Streams with updates and deletes add delete files or deletion vectors on top. The only cure is maintenance: compaction to merge small files and snapshot expiration to trim history. Every streaming pipeline into Iceberg is therefore really two pipelines, the ingestion path and the maintenance path, and the most common failure I see in the field is deploying the first without the second. Freshness you cannot query is not freshness.

Hold those three facts and the whole vendor bake-off becomes legible. Every option below is a different answer to the same three questions: who buffers the data, who decides when to commit, and who runs the maintenance.

The Open Source Workhorses

Start with the three options that dominate real deployments, all open source, all mature, each occupying a distinct point on the latency-versus-effort curve.

Apache Flink: the low-latency standard

Flink is the reference answer for the lowest-latency open path into Iceberg, and it earned that position. A Flink job consumes from Kafka or another source, processes events continuously, and writes to Iceberg with commits tied to Flink's checkpoint cycle. Checkpoints align with Iceberg snapshot commits, which is what gives Flink exactly-once delivery into the table: either a checkpoint completes and its data becomes a committed snapshot, or neither happens.

Latency lands wherever you set the checkpoint interval, and in practice that means seconds. Teams commonly run 10 to 60 second checkpoints against Iceberg, with aggressive setups pushing toward the low end of that range. Between commit floor physics and checkpoint overhead, think of well-run Flink-to-Iceberg freshness as roughly ten seconds to a minute, with the aggressive end paying more maintenance tax.

Flink's second superpower is change data capture. Flink CDC connects directly to database transaction logs, and Flink writes changelog streams, inserts, updates, and deletes, into Iceberg using equality deletes for the rows it cannot cheaply locate. This makes Flink the standard tool for maintaining a near-real-time Iceberg mirror of an operational database. The cost of that convenience is the equality delete backlog, which slows reads until maintenance resolves it. The good news in 2026 is that this exact pain is getting format-level and engine-level attention: v3 deletion vectors made positional deletes cheap to read, and work is active in the community on having Flink convert equality deletes into deletion vectors closer to write time, shrinking the window where readers pay the matching tax.

The trade-offs are operational. Flink is a distributed stateful system that you must size, checkpoint, upgrade, and debug, and Flink expertise is its own hiring line. Small file pressure is high at short checkpoint intervals, so the maintenance pipeline is mandatory. Choose Flink when you need seconds-level freshness, exactly-once guarantees, CDC semantics, or in-stream transformation, and you have or can rent the operational muscle. Managed Flink offerings from Confluent, AWS, Ververica, and Decodable exist precisely for teams that want the engine without the pager duty.

Spark Structured Streaming: the pragmatic middle

Spark Structured Streaming writes to Iceberg on a micro-batch model. You set a trigger interval, Spark accumulates data, and each trigger produces one Iceberg commit. A 60 second trigger yields 1,440 commits a day, each with reasonably sized files. Freshness lands at seconds to minutes depending on the trigger, with a minute being the comfortable default.

The case for Spark is continuity. Most data teams already run Spark for batch, already know its APIs, and already operate its clusters. Adding a streaming ingestion job is an increment, not a new platform. The trigger interval gives you a single clean dial between freshness and file health: lengthen it and files fatten, shorten it and freshness improves. Spark also brings the full transformation library to the stream, and the same job pattern serves backfills.

The case against is the latency ceiling and the cost profile. Micro-batching means Spark will not chase Flink to the aggressive end of the freshness range, and always-on streaming clusters are frequently overprovisioned, with a large share of allocated compute idling between triggers. Choose Spark when a minute of freshness is fine and your team is already a Spark team. That describes a lot of teams, which is why this path is more common in practice than the discourse suggests.

The Iceberg Kafka Connect sink: the no-code path

The community-maintained Iceberg sink connector for Kafka Connect reads topics, buffers records, and commits to Iceberg on a configurable interval, with a control-topic mechanism coordinating commits across connector tasks so the table gets clean, consistent snapshots. It supports automatic table creation, schema evolution driven by Schema Registry, and routing records to different tables or partitions.

There is no application code. You deploy a JSON configuration into a Kafka Connect cluster, and Connect handles scaling, offsets, and fault recovery. For an organization already running a Connect estate for other sinks, adding Iceberg is an afternoon. Freshness lands in the minutes range at typical configurations, since the connector's economics favor fewer, larger commits, and that gentler commit cadence is also why its small-file pressure runs lower than aggressive Flink or Spark setups.

The trade-offs: latency is minutes, not seconds, transformation capability is limited to lightweight single-message transforms, and CDC upsert flows are less natural here than in Flink, though supported patterns exist. And if you do not already run Kafka Connect, standing up a Connect cluster just for this erases much of the simplicity argument. Choose the sink when you have Kafka, you have Connect, minutes are acceptable, and you want the pipeline nobody has to babysit.

The Broker-Native Wave: When Kafka Itself Writes Iceberg

The most interesting structural development of the past two years is the collapse of the pipeline itself. A generation of Kafka-compatible platforms now materializes topics as Iceberg tables from inside the broker layer, no Flink job, no Connect cluster, no separate ingestion service. The pitch is "stream once, query forever," and the implementations differ in ways that matter.

Redpanda's Iceberg Topics persist topic data directly into Iceberg format from the broker, with automated housekeeping like snapshot expiration and custom partitioning. Time-to-value is excellent, since enabling a table is a topic-level switch. StreamNative's Ursa engine takes a lakehouse-native approach, storing data through a write-ahead log for ingestion and Parquet for analytics, speaking the Kafka protocol on a leaderless architecture. AutoMQ, a stateless S3-native Kafka, offers table topics with a similar promise, and pairs it with query-time federation ideas aimed at hiding the gap between not-yet-committed stream data and committed table data. Bufstream targets Protobuf-heavy shops with schema governance built in and Iceberg as the landing format.

Now the honest physics. Moving the Iceberg writer into the broker does not repeal the commit floor. These systems still buffer, still write Parquet, still commit snapshots, and the freshness of the queryable table still lands in the seconds-to-minutes range depending on configuration, typically closer to minutes at sane settings. What broker-native designs actually buy you is the removal of an entire operational tier, and that is genuinely valuable. What they can cost you shows up in the fine print. Some zero-copy designs that make Iceberg the broker's primary storage push producer latency up several fold versus classic Kafka, since produces now ride object storage economics. Some produce tables that are effectively read-only from the outside, or that lack automatic compaction and snapshot hygiene, which quietly hands the maintenance pipeline back to you. And Kafka protocol compatibility varies at the edges, transactions and compacted topics being the classic gaps, which matters if the rest of your stack assumes full Kafka semantics.

My guidance: broker-native Iceberg is the right default for append-only event streams landing in a lakehouse when you are already choosing one of these platforms for other reasons. Interrogate three things before committing: producer latency impact, who runs compaction and expiration, and whether the resulting tables are first-class citizens that any engine can maintain and evolve, or a materialized view you can only look at.

The Managed Layer: Vendor Pipelines and Cloud Services

Above the open source and broker options sits a thick layer of managed offerings, and in 2026 this is where most net-new pipelines I encounter actually get built. A tour of the ones that come up most.

Confluent's Tableflow is the highest-profile entry: a checkbox-level feature in Confluent Cloud that materializes Kafka topics as Iceberg tables, handling schema mapping, type conversion, file sizing, compaction, and catalog publication, with Delta Lake as an alternate output and integration into Confluent's governance stack. Paired with Confluent's managed Flink for in-stream transformation, it forms a complete stream-to-lakehouse path where you never touch a cluster. The trade is the classic managed trade: meaningful cost at scale and deep attachment to one vendor's ecosystem. Freshness is in the minutes class, governed by its materialization and compaction cadence. WarpStream, under the Confluent umbrella, offers its own TableFlow with a bring-your-own-cloud model that can source from any Kafka-compatible cluster, trading some polish for openness and cost control.

On AWS, the native path got legitimately good. Kinesis Data Firehose delivers streams directly into Iceberg tables with buffering measured in tens of seconds to minutes, and S3 Tables provide Iceberg storage with built-in automatic compaction and snapshot management, which removes the maintenance pipeline that self-managed teams forget. Glue and EMR cover the Spark and Flink routes with v3 support. The result is a fully AWS-native stream-to-Iceberg story with freshness in the one-to-several-minutes class and very little to operate, at the price of AWS coupling.

Snowflake's Snowpipe Streaming writes row-level streams into Snowflake-managed Iceberg tables with seconds-to-minute visibility, and those tables remain readable by external engines through the Iceberg REST protocol, which makes it a real option when Snowflake is already your center of gravity. Databricks reaches the same destination from the Delta side of the house with UniForm and managed Iceberg support in Unity Catalog.

And a healthy ecosystem of specialist pipeline vendors, Estuary, Streamkap, Decodable, Upsolver among them, sells CDC-to-Iceberg as a product: connect a database, get a continuously maintained Iceberg table, with freshness typically in the seconds-to-low-minutes band and the equality-delete and compaction machinery handled for you. For teams whose entire streaming need is "mirror these operational databases into the lakehouse," these are often the shortest path to done.

The pattern across the managed layer: you are buying the maintenance pipeline and the on-call rotation, not a better latency floor. The physics is the same physics. Evaluate managed offerings on the completeness of their table hygiene, the portability of the tables they produce, and cost at your volume, because the freshness numbers cluster tightly across vendors.

Honest Latency Expectations, All Options on One Page

Let me compress the tour into the summary I wish every evaluation started with. These are realistic end-to-end freshness ranges, event occurrence to queryable in Iceberg, for well-configured deployments in 2026.

Tuned Flink with aggressive checkpoints lands around 10 to 30 seconds, the fastest sustainable open path, paid for with heavy small-file maintenance. Standard Flink and aggressive Spark Structured Streaming land in the 30 seconds to 2 minutes band. Comfortable Spark, the Kafka Connect sink, broker-native table features, Tableflow-class managed materialization, and Firehose land in the 1 to 15 minute band depending on buffer and commit settings. Specialist CDC vendors mostly quote and deliver the seconds-to-low-minutes band for database mirroring. Anything promising meaningfully sub-ten-second queryable freshness in an Iceberg table today deserves your sharpest questions, because it is either redefining "queryable," serving reads from somewhere that is not yet the table, or accepting a commit cadence whose maintenance bill arrives later.

One more component of end-to-end latency that evaluations routinely forget: the read side. A committed snapshot still has to be noticed. Query engines cache table metadata, dashboards poll on intervals, and a BI tool refreshing every minute adds a minute of perceived staleness no ingestion tuning can remove. When a stakeholder says the data is slow, profile the whole path. I have watched teams shave their commit interval from 60 seconds to 15, quadrupling their file count, to fix what turned out to be a five-minute dashboard cache.

When Seconds Are Not Enough: Architectures for the Lowest Latency

Now the question the second half of this article exists for. Some workloads genuinely need sub-second or low-single-digit-second freshness: fraud scoring, operational monitoring, user-facing analytics, inventory decisions. The commit floor means you should not try to serve those directly from an Iceberg table on object storage today. The wrong conclusion is that Iceberg does not belong in those architectures. The right conclusion is that Iceberg plays a specific position, durable, open, queryable history, and something else plays the hot position. Three patterns dominate, and choosing among them is the real design decision.

Pattern one: the hot and cold split, unified at query time. Recent data lives in a low-latency serving layer, historical data lives in Iceberg, and the query layer spans both. The classic build is Kafka feeding a real-time OLAP engine, Apache Pinot, StarRocks, ClickHouse, or Apache Druid, for the hot window, while the same stream lands in Iceberg for history, increasingly with the OLAP engine itself able to read the Iceberg tables directly so one query surface covers both tiers. StarTree has been public about running exactly this Kafka to Iceberg to Pinot shape, keeping Iceberg as the single source of truth while Pinot answers production queries at high concurrency. The streaming world's version of the same idea pairs a sub-second stream storage layer like Apache Fluss with table-format cold storage and automatic tiering between them. This pattern gives you genuine sub-second freshness and warehouse-scale history with one logical dataset. Its cost is running the serving layer and keeping the seam honest: the boundary between hot and cold must be defined, monitored, and invisible to queries.

Pattern two: the streaming database in front. Systems like RisingWave and Materialize consume streams, maintain incrementally updated materialized views with sub-second internal freshness, and sink results continuously into Iceberg tables for durability and downstream analytics. Applications needing instant answers query the streaming database's views. Everything else, ad hoc analysis, ML training, BI, reads Iceberg. This shines when the low-latency need is for derived results, aggregates, joins, feature values, rather than raw events, because you get the derivation and the serving in one system and the lakehouse gets clean, already-shaped tables. The trade is another stateful system in the critical path and view semantics to reason carefully about.

Pattern three: stream-table federation. Keep exactly one copy of the stream, and make the query layer union the not-yet-committed tail from the broker with the committed body from Iceberg. Broker-native platforms are pushing here, AutoMQ's query-time federation being an explicit example, and it is philosophically the cleanest answer: no second serving store, no duplicated data, the seam handled by the reader. In 2026 I classify it as promising and young. It requires the query engine and the streaming platform to cooperate closely, engine support is narrow, and the operational story under failure is less proven than the older patterns. Watch it, pilot it where the stack aligns, and be honest about its maturity.

Across all three patterns, the strategic constant is Iceberg's role: the open, engine-neutral system of record that every hot layer drains into. Hot layers are increasingly replaceable components. The table format underneath is the twenty-year decision.

A Worked Example: One Stream, Three Builds

Abstractions settle best in a story, so let me run one concrete scenario through three different freshness requirements and watch the architecture change shape each time.

The scenario: you run data platform for a food delivery company. Order events flow through Kafka at a few thousand events per second, with a steady stream of updates as orders progress from placed to assigned to delivered to occasionally refunded. Three internal customers want this data. Finance wants daily and hourly reporting. Operations wants dashboards that track order flow by city with a freshness target of about a minute. And the dispatch team wants a live view of active orders per courier zone that must reflect reality within about a second, because humans make routing decisions from it.

Build one: finance only. If the minute-level and second-level customers did not exist, this is barely a streaming problem. The Iceberg Kafka Connect sink, committing every five minutes into an orders table, covers hourly reporting with enormous headroom. Files land at healthy sizes because five minutes of buffering at this volume produces real Parquet files, not confetti. A nightly compaction and weekly snapshot expiration keep the table tidy. Total new operational surface: one connector configuration and two scheduled maintenance jobs. On AWS, Firehose into S3 Tables gets the same result with the maintenance handled for you. The lesson of build one is that most streaming requirements are secretly this build, and teams that recognize it save themselves a distributed systems project.

Build two: finance plus operations. The minute-level dashboard changes the ingestion tier but not the philosophy. The Connect sink's comfortable cadence now sits too close to the requirement, so the pipeline moves to Flink or Spark Structured Streaming committing every 20 to 30 seconds, which lands end-to-end freshness comfortably under the minute target after you account for dashboard refresh. Because orders update as they progress, this is a CDC-shaped stream, and Flink's changelog handling makes it the natural pick, writing updates through the merge-on-read path. Now the maintenance pipeline earns its keep: compaction must run frequently enough to fold the delete backlog and merge the small files that 30 second commits produce, and you monitor commit rate, file sizes, and compaction lag as first-class health metrics. Same lakehouse, same table, roughly triple the operational attention. The dashboard reads the Iceberg table directly, and everyone is happy, including finance, who quietly benefits from fresher hourly numbers.

Build three: all three customers. The dispatch view breaks the pattern, and the correct response is to stop pushing the table harder. Chasing one-second freshness with one-second commits would produce 86,400 snapshots a day, a small-file blizzard, and a catalog under siege, and it would still miss the target once query-side latency is counted. Instead the architecture splits the serving, not the truth. The same Kafka stream now also feeds a hot layer: a real-time OLAP engine like Pinot or StarRocks holding the last few hours of order events with sub-second ingestion, serving the dispatch view at high concurrency, or a streaming database like RisingWave maintaining the active-orders-per-zone aggregate as an incrementally updated materialized view. The Iceberg pipeline from build two continues unchanged as the durable spine, and where the hot engine can read Iceberg directly, historical questions from the dispatch team run against the same tables everyone else uses. Iceberg remains the single source of record. The hot layer is a serving detail, chosen for the moment and replaceable without migrating history.

Three builds, one stream, one table format. The freshness requirement, stated as an honest number, was the only variable, and it alone determined whether the right answer was a connector, an engine, or an architecture.

The Half of the System Nobody Demos: Maintenance

I have said it throughout, and it deserves its own section because it is the difference between streaming pipelines that survive and those that quietly rot.

A streaming Iceberg deployment is ingestion plus maintenance, always. The maintenance side has three standing jobs. Compaction merges the small files that frequent commits necessarily produce, and for CDC streams it also resolves delete backlogs, folding equality deletes into deletion vectors and rewriting heavily deleted files. Snapshot expiration trims the history that thousands of daily commits generate, without which metadata grows unboundedly and storage fills with unreachable files. And monitoring watches the health metrics that predict query pain before users feel it: commits per hour, average file size, compaction lag, and end-to-end freshness measured honestly from event time to queryability.

The v3 format quietly improved this picture. Deletion vectors keep read performance stable between maintenance runs in ways v2 position deletes never did, which relaxes how aggressively compaction must chase ingestion. Managed table services increasingly bundle maintenance, S3 Tables' automatic compaction being the cleanest example, and engines and platforms, Dremio among them, ship automated optimization so the maintenance pipeline is configuration rather than custom Spark jobs. And the v4 work on single-file commits aims at the root cause, making small frequent commits dramatically cheaper at the metadata layer. The trajectory across format versions is unmistakable: streaming is being promoted from tolerated workload to first-class citizen. But trajectory is not present tense. On the format you run today, schedule the maintenance before you celebrate the ingestion.

A Decision Framework You Can Actually Use

Strip everything above down to the sequence of questions I walk teams through.

Start with the honest freshness requirement, stated as a number with a stakeholder's name attached. Most requests for "real time" dissolve under this question into "within a few minutes," which is wonderful news, because the minutes band is where the cheap, boring, reliable options live: Kafka Connect sink, Spark on a comfortable trigger, Firehose, broker-native tables, managed materialization. Pick whichever aligns with the platforms you already run, and spend the savings on maintenance and monitoring.

If the requirement genuinely lands in the seconds band, you are choosing between Flink, aggressively tuned Spark, or a specialist managed pipeline, and you are signing up for the small-file consequences. Decide who operates the engine, you or a vendor, and stand up the compaction pipeline the same week as the ingestion pipeline.

If the requirement is sub-second, stop trying to make the table do it. Choose a hot-layer pattern: real-time OLAP over the hot window with Iceberg as history, a streaming database serving derived views and sinking to Iceberg, or, where your stack aligns and your risk tolerance allows, an emerging federation design. Keep Iceberg as the system of record in every variant.

And in all three bands, ask every option the same three physics questions: who buffers, who commits and how often, and who maintains. Any product that answers all three crisply is worth evaluating. Any product that answers with a latency number and a smile is asking you to discover the answers in production.

Questions I Hear Most Often

The same questions surface every time I present on this topic, and answering them here rounds out the map.

Why not just commit to Iceberg every second and skip the hot layer? Run the arithmetic and the answer states itself. One commit per second is 86,400 snapshots a day, each writing fresh metadata against object storage, each contending at the catalog, each adding files sized by one second of traffic. Within days the table carries hundreds of thousands of files, planning slows, and compaction cannot merge files as fast as ingestion mints them. You would pay warehouse prices in maintenance compute to deliver freshness the read path still cannot honor once caching is counted. The commit floor is not a product limitation to shop around. It is the current cost structure of coordinating snapshots on object storage, and the v4 work is the honest path to lowering it.

Is the small file problem really that serious? It is the number one operational failure mode I encounter in streaming lakehouse deployments, ahead of everything else combined. The insidious part is the timeline: the pipeline demos beautifully, the first week is fine, and the degradation compounds quietly until a month later queries take ten times longer and nobody changed anything. By then the backlog is large enough that the first compaction run is itself a heavy job. The fix is cultural as much as technical. Treat file count and average file size as service health metrics from day one, alert on them, and never sign off on an ingestion pipeline whose maintenance pipeline is a to-do item.

Should I use equality deletes or avoid them? Use them for what they are: a write-side deferral mechanism for streams that cannot afford to locate rows at write time, which mostly means Flink CDC. Then treat the backlog as a liability with a burn-down schedule. Reads pay for every unresolved equality delete, so the operational goal is a short half-life: frequent maintenance that resolves them into deletion vectors and rewritten files. The ecosystem is moving to shorten that half-life automatically, with active work on converting equality deletes to deletion vectors near the point of write. If your pipeline is append-only, the entire question disappears, which is one more reason to model streams as append-plus-derived-tables when the semantics allow it.

Do I still need Kafka at all, or can sources write straight to Iceberg? Direct writes are possible, plenty of pipelines run Spark or a vendor tool from source to table with no broker, and for pure batch-shaped flows that is fine. What the broker buys in a streaming architecture is decoupling and replay: many consumers off one stream, backpressure absorption, the ability to rebuild a table from history after a bug, and a natural feed for the hot layer patterns in this article. Notice that the industry is answering this question in an interesting way: rather than removing Kafka in favor of Iceberg, it is fusing them, with brokers that materialize Iceberg natively and tiered designs that use the table format as the broker's own cold storage. The stream and the table are becoming two temperatures of one system rather than two systems.

How does the choice of catalog affect streaming? More than most evaluations account for. Every commit is a catalog interaction, so a streaming table's commit rate becomes the catalog's write load, and a catalog that handles commits slowly or serializes them poorly becomes the pipeline's bottleneck regardless of engine tuning. The REST catalog protocol has become the meeting point of the ecosystem, with implementations like Apache Polaris, and the practical advice is to load-test your catalog at your intended commit rate across all streaming tables combined, not per table. Multi-table streaming estates concentrate surprising write pressure on this one small service.

What about streaming reads from Iceberg, not just writes? A real and useful capability that deserves its own article. Engines can consume an Iceberg table incrementally, processing new snapshots as they commit, which turns the table into a replayable, governed stream for downstream jobs. The v3 row lineage feature strengthens this by giving rows stable identity across commits, making change feeds more trustworthy. Freshness of a streaming read is bounded by the upstream commit cadence, so everything in this article about the write side sets the floor for the read side. Teams increasingly chain these: stream into a raw table, stream out of it into derived tables, with Iceberg as the durable seam between stages.

Will Iceberg v4 change the advice in this article? It will move the boundaries and keep the structure. Single-file commits and the adaptive metadata tree attack the commit floor directly, which should pull sustainable commit cadences down and make the seconds band cheaper to operate, and the maintenance story shifts as metadata rebalancing joins the housekeeping roster. What v4 does not change is the shape of the reasoning: visibility still arrives at commits, hot serving still wants purpose-built layers, and maintenance still balances writers against readers. When v4 lands and vendors reprice their latency claims, re-run the three physics questions and the map will redraw itself correctly.

What is the single most common mistake you see? Chasing a freshness number nobody actually needs. The most expensive words in streaming architecture are "real time" spoken without a number attached. Teams build Flink estates, hot layers, and compaction fleets to hit seconds when their consumers act on minutes, and the carrying cost of that gap compounds monthly. Interrogate the requirement first, with the stakeholder in the room and a specific decision or experience on the table. The best streaming architecture is the slowest one that meets the honest requirement.

How to Evaluate the Next Entrant

One more tool before the closing, because this market is not done producing products. Roughly once a month a new offering claims to have solved streaming into Iceberg, and you will be asked to evaluate one. Here is the interrogation I run, in order.

First, ask where the data lives during the gap between arrival and commit, and who can query it there. This single question separates the honest architectures from the redefined ones. If the answer is a buffer nobody can query, the product's freshness is its commit cadence, full stop. If the answer is a queryable hot tier, you are looking at a hybrid architecture wearing a product name, which is fine, but then evaluate it as a hybrid: what engine queries the hot tier, what guarantees span the seam, and what happens to in-flight data when a node dies.

Second, ask for the commit cadence at your volume and the file sizes it produces, then ask who compacts them and on whose compute bill. Vendors quote latency at the cadence that flatters them and file health at the cadence that flatters them, and those are rarely the same cadence. Getting both numbers for one configuration tells you what you will actually run.

Third, ask whether the tables it writes are fully standard. Can Spark, Dremio, Trino, and Flink read them with no vendor library. Can an external engine run compaction and snapshot expiration, or does maintenance route exclusively through the vendor. Can you evolve the schema from outside. Tables that pass all three are assets. Tables that fail any of them are a rental with an Iceberg logo.

Fourth, ask what happens during the bad hour: a schema change upstream, a poison message, a catalog outage, a rebalance under peak load. Streaming pipelines earn their keep in the bad hour, and the maturity gap between a two-year-old product and a battle-tested one is invisible in every demo and vivid in every incident.

A product with good answers to all four is worth a pilot regardless of how new it is. A product that dodges any of them has answered anyway.

Closing Thoughts

Streaming to Iceberg in July 2026 is a solved problem with unsolved edges. The middle of the market, minutes-fresh tables fed from Kafka or CDC, is genuinely commoditized, with a dozen good answers spanning open source and managed. The seconds band is achievable and operationally demanding, with Flink still the standard-bearer and the maintenance pipeline as the entry fee. The sub-second band belongs to hybrid architectures where Iceberg anchors history while a hot layer serves the moment, and the most interesting engineering of the next two years, in v4's commit redesign and in stream-table federation, is aimed at narrowing exactly that gap.

If you take one model from this article, take the three physics facts: nothing is visible until committed, commits have a floor, and frequent commits demand maintenance. Every product in this crowded space is a different negotiation with those three constraints, and now you can read the negotiations yourself.

If this way of working through architecture is useful to you, it is the same approach I take at book length. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, and I have written additional titles on lakehouse architecture, data engineering, and agentic analytics, all built to take you from the mental models to running systems.

Browse the full collection of my books on data and AI at books.alexmerced.com.

The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter

Alex Merced — Wed, 08 Jul 2026 22:49:29 +0000

If you want to know where Apache Iceberg is headed, do not read the press releases. Read the dev mailing list.

I say that as someone who reads it every week. Vendor blogs will tell you a feature is coming. Conference keynotes will tell you a feature is exciting. The dev list tells you what is actually settled, what is genuinely contested, and which design questions keep smart people arguing at 11pm across time zones. The gap between the marketing version of Iceberg v4 and the mailing list version of Iceberg v4 is exactly the gap between a story and the truth.

So this article is my mid-2026 checkpoint on v4, grounded in what the community has voted on, what it is actively debating, and what has quietly changed since Iceberg Summit in April put v4 on every conference slide. I wrote a broad v4 overview in June. This piece goes deeper on where things stand right now, in July, with the dev list as the primary source. My goal, as always, is to make the logic of these proposals accessible. You should finish this article able to explain not just what v4 proposes, but why each piece exists and what problem forced it into existence.

First, the Honest Status

Let me set expectations before we get into the fun parts.

Iceberg v4 is not released. There is no v4 spec you can set on a table today. The current stable line is the 1.11 release from May 2026, and that release sits firmly in the v3 era, with v3 features like variant shredding and deletion vectors as the production-grade capabilities you should actually be building on. The practical guidance I gave in June has not changed one bit: treat v3 as the production target and v4 as the horizon worth watching.

What has changed is how concrete the horizon has become. V4 stopped being a wish list some time ago, and over the past two months it crossed another threshold: pieces of it are now formally ratified spec text. In May 2026, the community voted to add relative path support to the v4 spec. In the same window, a vote passed adding the new Content Stats representation to v4, the typed and structured replacement for the old column statistics maps. In early June, the community voted to add a draft spec for a new compact bitmap format to the repository. These are not blog posts or design docs anymore. They are the spec, or drafts formally adopted into the spec process, decided in public votes with recorded results.

So the accurate picture of v4 in July 2026 is a spectrum. On one end sit ratified pieces like relative paths and content stats. In the middle sit heavily developed proposals with active design syncs, like the adaptive metadata tree and single-file commits. On the other end sit live arguments that could still go several directions, like the exact fate of the partition tuple and where column-level updates should live. Walking that spectrum, from settled to contested, is the plan for the rest of this article.

How a Proposal Becomes Spec: Reading the Process Itself

Before touring the proposals, it helps to know the machinery they move through, because the machinery is how you judge maturity. Iceberg design work follows a recognizable lifecycle, and once you can spot the stages, the dev list stops looking like noise and starts looking like a status board.

Ideas enter as a DISCUSS thread, usually paired with a design document and often a GitHub issue or an Iceberg Enhancement Proposal. This stage can run weeks or months, and message volume here signals contested design space, not trouble. The efficient column updates thread has drawn more than fifty messages precisely because the question is worth arguing.

Serious proposals then grow supporting structure. Recurring community syncs get scheduled, with recordings and written summaries posted back to the list so nothing is decided in a room. The single-file commits work runs on this cadence today, with syncs organized and summarized publicly, appendices added to the design doc as options get mapped, and specific contributors owning specific open questions. When you see meeting summaries and change-detection diagrams landing on a thread, you are watching a proposal in its engineering phase.

The finish line is a VOTE thread. A committer proposes specific spec text, the community votes in public over a defined window, and the result gets recorded. That is what happened with relative paths in May, with the content stats representation the same month, and with the draft bitmap spec in June. Vote threads are short and undramatic on purpose. By the time text reaches a vote, the arguments already happened.

Two reading tips follow from this. First, weigh spec text and vote results above everything else, including design docs, because docs describe intentions and votes record decisions. Second, calibrate on who is in a thread. When you see the same design question drawing sustained engagement from engineers at half a dozen competing companies, the outcome will bind all of them, which is exactly what makes it trustworthy. A quiet thread with one company's engineers talking to themselves is a signal of a different kind.

With the machinery in view, the tour that follows is organized by lifecycle stage: ratified first, engineering phase second, open arguments third.

A Ninety-Second Refresher on the Metadata Tree

Every v4 proposal is a change to Iceberg's metadata, so we need the current model in our heads. If you know it cold, skip ahead.

An Iceberg table is a tree of files. At the bottom sit data files, usually Parquet, holding the rows. Above them sit manifest files, which list groups of data files along with per-file statistics like row counts and column min and max values. Above those sits a manifest list, one per snapshot, collecting all the manifests that make up one consistent version of the table. At the top sits a metadata file written as JSON, which records the schema, partition specs, sort orders, snapshot history, and a pointer to the current snapshot. A catalog holds the pointer to the current metadata file, and commits happen by atomically swapping that pointer.

This tree is why Iceberg works. Query planning reads metadata instead of listing storage. Statistics let engines skip files that cannot match a filter. Snapshots give you time travel and isolation. The design has carried tables to tens of petabytes.

The design also carries assumptions from the era that produced it. It assumes commits are relatively infrequent, so it is acceptable that every commit writes a new metadata JSON, a new manifest list, and new manifests. It assumes tables are modest in width, so it is acceptable that statistics live in generic maps inside row-oriented Avro files. It assumes tables rarely move, so it is acceptable that every file reference is an absolute URI with the bucket baked in. Streaming workloads broke the first assumption. AI feature tables broke the second. Multi-region operations broke the third. V4 is the format renegotiating all three assumptions at once, and the dev list is where the negotiation happens.

What Is Already Decided: Relative Paths

Start with the most settled item, since it shows what the finish line looks like.

Iceberg has always stored absolute paths. Every manifest and metadata file embeds full URIs down to the bucket and region. This was a defensible early choice that prevented ambiguity on eventually consistent object stores. It also meant that moving a table, for disaster recovery, region migration, or cloning into a test environment, required rewriting every path in the metadata tree. For big tables that rewrite was a project, not an operation.

The v4 answer stores paths relative to a table root, with the catalog supplying the root location. Move the directory tree, update the catalog, done. The internal relationships never change, so nothing needs rewriting. Absolute paths remain legal for references that genuinely live outside the table root.

The dev list history here is instructive. The proposal thread ran for months, working through edge cases like mixed absolute and relative references, how imported files behave, and what writers must guarantee. Then Daniel Weeks brought it to a formal vote in mid-May 2026, and it passed. The spec's language around table location already reads in terms of "v4 and later." When people ask me how Apache governance actually functions, this thread is my answer: a proposal, months of public argument, a refined design, a vote, and then spec text that every vendor implements from the same document.

Relative paths will be one of those features nobody writes excited blog posts about in three years, because it will have quietly deleted a whole category of operational pain. Those are often the best features.

Also Decided: Content Stats, the New Shape of Statistics

The second ratified piece is less visible and more consequential. In May 2026, Eduard Tudenhöfner brought the Content Stats representation to a vote, and the community adopted it into v4.

Here is the problem it solves. Today, per-file column statistics live in generic maps keyed by field ID: one map for lower bounds, one for upper bounds, one for null counts, and so on. The values are serialized binary blobs. This design has three weaknesses that get worse every year. Wide tables carry enormous maps, and row-oriented Avro forces engines to deserialize all of it even when planning needs bounds for exactly one column. Serialization loses type context, which creates subtle hazards as schemas evolve. And the map structure cannot express anything richer than a scalar per column, which locks out entire categories of useful metadata.

Content Stats replaces the maps with a typed, structured representation. Statistics become real structured data, with logical and physical types preserved, projectable field by field. You can read the lower bounds for three columns without touching stats for the other two hundred. The v3 spec already gestured in this direction for special cases, storing variant bounds as structured objects keyed by JSON path and geospatial bounds in dedicated structs. V4 makes that the general model rather than the exception.

The reason I call this consequential is what it makes possible downstream. A structured, extensible stats model can carry per-field metrics that the old maps never could: bounds for fields nested inside a variant, bounding boxes for geometry, and eventually the kinds of sketches and index structures that approximate nearest neighbor search needs. The dev list already carries a follow-on thread on aggregate column stats for v4, exploring table-level and manifest-level aggregations layered on the new representation, so engines can answer questions like "roughly how many distinct values does this column hold" without scanning file-level stats at all. Statistics are becoming first-class data, and first-class data grows capabilities.

There is a companion proposal worth mentioning in the same breath: delta-encoded schemas. Every schema change in Iceberg today appends a complete copy of the schema to the metadata file. A wide table with a long evolution history drags hundreds of full schema copies around in its metadata JSON forever. The proposal stores schema changes as deltas against prior versions instead. It is a small idea aimed at the same disease: metadata bloat that grows with table width and table age. The thread is younger than the stats work, but it points the same direction.

The Headline Fight: Single-File Commits and the Adaptive Metadata Tree

Now we get to the proposal that anchors v4, and the one where the July 2026 dev list is liveliest.

The problem statement is easy to feel. Every Iceberg commit today writes a new metadata JSON, a new manifest list, and at least one new manifest, even when the commit adds a single small data file. For hourly batch jobs, nobody notices. For a streaming job committing every few seconds, the metadata writing dominates the work, small files pile up against storage prefixes until the object store throttles, and compaction jobs end up fighting the ingestion they exist to support. Write amplification is the tax, and streaming pays it at a punishing rate.

The proposed cure restructures the tree around a Root Manifest that replaces the manifest list and becomes the single entry point for a snapshot. The hierarchy flattens. And the root gains a critical new ability: small changes can be inlined directly into it. A tiny streaming commit writes one file, the new root, rather than a cascade of metadata files. As inlined entries accumulate, background maintenance rebalances them down into leaf manifests, restoring the layered structure that makes planning fast on huge tables. The design is called adaptive because the tree's shape follows the workload: hot streaming tables keep recent writes near the root for cheap commits, batch tables settle into the classic layered shape.

That is the elevator version. The dev list version, as of this summer, is a set of hard, specific engineering questions being worked in public, with recurring design syncs that Amogh Jahagirdar has been organizing and summarizing back to the list.

The question that has generated the most traffic lately is deceptively narrow: what happens to the partition tuple? In the current format, every manifest entry carries a tuple of partition values, and every manifest is bound to a single partition spec. The v4 tree wants to break that coupling, so a root can reference files spanning multiple partition specs and metadata can cluster in whatever way serves reads best. Ryan Blue opened a dedicated thread on partition tuples in v4, and the discussion has explored a striking option: eliminating the stored tuple entirely and reconstructing partition information from column bounds, using the property that a file is effectively partitioned on a field when that field's lower and upper bounds are equal.

The complication, and the reason the thread runs long, is the edge cases. String and binary columns can have truncated bounds in stats, which breaks the reconstruction trick for identity-partitioned string columns unless writers are required to keep exact bounds for them. And the main consumer of partition tuples turns out to be equality delete matching, which raises a further option discussed in the thread: make v4 equality deletes global and scope their application by stats rather than by partition. Every option trades something. Keeping the tuple keeps coupling. Reconstructing from bounds demands exactness guarantees. Going global on equality deletes changes delete semantics. This is exactly the kind of question that looks tiny from the outside and determines the shape of the format from the inside.

The other pressure point is read cost. Inlined entries near the root are wonderful for writers and a new burden for readers, who must scan them during planning. List participants have pushed hard on this: does the spec accept a linear scan of inlined entries as the price of write throughput, and what happens to a REST catalog that has to partially decode hundreds of inlined entries per planning request under high concurrency? Push too much work to the catalog and you risk turning it into a small query engine. Flush entries to leaves too eagerly and you resurrect the small-file storm that single-file commits were built to end. The change-detection work, mapping how incremental readers detect what changed between snapshots in the new tree across a variety of write scenarios, has its own section in the design doc with diagrams contributed by Steven Wu, and the group is verifying those cases methodically.

None of this reads like a proposal in trouble. It reads like a proposal being taken seriously. The write path and the read path are being priced against each other in the open, workload by workload, before anyone votes. The result will be better for the arguing.

The Quieter Metadata Question: What Happens to metadata.json?

Alongside the tree redesign runs a related conversation that has picked up real momentum since spring: what should become of the metadata JSON file itself?

Two threads carry it. One asks the blunt question directly, "metadata.json in v4?", probing whether the top-level file should shrink, change format, or delegate most of its contents elsewhere. The other, "Offloading Snapshots from Metadata.json", attacks the file's biggest source of bloat. Today the metadata file carries the table's snapshot history inline, and for tables with long histories and frequent commits, that log grows until the file is megabytes of JSON that every reader parses and every writer rewrites on every commit. Streaming tables, which commit constantly, feel this worst. The offloading proposal moves snapshot history out of the top-level file into separate structures, so the file every commit rewrites stays small and stable in size.

The thread drew sustained engagement through the spring from a wide slice of the community, Ryan Blue, Yufei Gu, Steven Wu, Jean-Baptiste Onofré, Péter Váry, and others, working through how readers discover offloaded history, how time travel resolves against it, and how the change interacts with the root manifest design. I group these threads with single-file commits as one campaign: every layer of metadata that today gets rewritten wholesale on every commit is being reexamined, and the test each layer must pass is "does the cost of a commit scale with the size of the change, or with the size of the table?" V4's answer, layer by layer, is the change, not the table.

Add the long-standing companion idea of storing manifests in Parquet rather than Avro, so metadata reads get column pruning like data reads, and you can see the full shape: a metadata layer that is columnar, structured, incrementally committed, and slim at the top. Each proposal is votable on its own. Together they are one redesign.

The Hottest Thread of the Season: Efficient Column Updates

If you sort this spring's dev list by message volume, one discussion towers over the rest: efficient column updates in Iceberg. It has drawn more replies than any v4 thread, and the reason is that it sits on a genuinely contested design boundary.

The problem is the wide-table workload that AI teams live in. Picture a feature table with hundreds of columns or an embedding table with large vectors, updated by jobs that recompute a handful of columns and leave the rest alone. Iceberg today offers no way to update a column without rewriting entire rows, which means entire files. Refresh one embedding column and you rewrite two hundred columns of untouched data alongside it. At scale, the write amplification is ruinous, and teams route around Iceberg entirely for these tables.

The proposal introduces column update files: write only the changed columns to new files, leave base files untouched, and stitch the two together at read time to materialize full rows. Related design work on the thread covers the file representation and the metadata representation for these column-level artifacts, and the current scope focuses on updates that touch a column across all rows, leaving row-subset partial updates for later.

The reason the thread runs to dozens of messages is a real architectural argument, not bikeshedding. One camp asks whether this belongs in Iceberg at all or whether Parquet should solve it, perhaps through logical files that map columns to physical files, especially since Parquet is separately working on cheaper footers. The counterargument is that a column-to-file mapping inside Parquet duplicates what manifests already do, putting a second bookkeeping system inside the first. Contributors have brought comparative notes on how Lance, Hudi, and Paimon approach column groups and partial updates, and there is a compelling side observation that independently updatable column families would also cut commit conflicts, since writers updating disjoint families stop contending on the same rows. Gábor Kaszab, Péter Váry, Russell Spitzer, Gang Wu, Anurag Mantripragada, and a rotating cast of others have kept this discussion rigorous for months.

My read as of July: the workload pressure is undeniable and the community clearly intends to serve it, but the layering question, Iceberg versus Parquet versus both, is the least settled major design question in v4. Watch this thread above all others if wide tables are your life.

Deletes Keep Evolving: Compact Bitmaps and the Equality Delete Endgame

V3's deletion vectors were a big win, and I have written a full deep dive on them. The v4 conversation shows the community already sharpening that work along two lines.

The first is the compact bitmap format. Ryan Blue opened a discussion proposing a new bitmap serialization, and by early June the community voted to add a draft bitmap spec to the repository. The motivation is to make the bitmap structures behind deletion vectors leaner and more broadly useful, with an eye toward the new metadata tree, where compact bitmap-like structures can serve more jobs than row deletion. The thread drew detailed technical review from contributors across several companies, the kind of scrutiny you want on a byte-level format that every engine will implement.

The second is the slow squeeze on equality deletes. Equality deletes, which mark rows dead by value rather than position, remain the write-cheap tool that streaming CDC writers rely on, and they remain expensive for readers until maintenance resolves them. A thread on Flink converting equality deletes to deletion vectors has been working through how the streaming engine that produces most equality deletes in the wild can compact them into position-based vectors closer to write time, shrinking the window where readers pay the equality-matching tax. Combine that with the v4 partition tuple discussion, where one option on the table is redefining equality delete scoping entirely, and the direction is visible: the community is engineering equality deletes into a narrower and more disciplined role, a short-lived buffer rather than a resting state. I would not be surprised if the v4-era guidance eventually treats unresolved equality deletes the way we now treat uncompacted small files, as a condition to monitor and burn down.

The Long Tail: Row Timestamps, Tags, and Capability Signaling

A few smaller threads round out the July picture, and they are worth knowing because small spec changes often deliver outsized quality-of-life gains.

The row timestamp proposal would give rows a spec-level notion of when they were written, building on the row lineage foundation that v3 established. Row lineage gave rows persistent identity across commits, which made change data capture and incremental processing tractable. A trustworthy per-row timestamp extends that story for auditing, temporal queries, and downstream CDC consumers, and the thread has been actively working through semantics with Steven Wu and Micah Kornfield among the participants. Getting time semantics right in a format spec is famously subtle, which is why this one earns its long discussion.

The tags field proposal would add a general-purpose tagging mechanism to v4 metadata structures, giving engines and tools a sanctioned place to attach small annotations without abusing properties maps. And at the REST layer, a discussion about adding a client capabilities header to the catalog protocol addresses a problem v4 itself is creating: as the format grows more optional and more adaptive, catalogs and clients need a clean way to declare what they each understand, so a mixed fleet of engines at different support levels can share tables safely. That last one is the connective tissue that makes a multi-engine v4 rollout survivable, and it is exactly the kind of unglamorous work that determines whether a spec transition goes smoothly.

The Convergence Question, Six Weeks Later

I covered the Databricks convergence announcement at length in June, so here I will just update the temperature.

The pitch, for anyone catching up: Databricks proposed that Delta Lake 5.0 adopt the Iceberg v4 metadata tree as its native content metadata, producing one on-disk structure that both Delta and Iceberg clients read and write directly, with no translation layer. The technical claim is that the two formats already converged on the same ideas, columnar metadata, manifest-style trees, deletion vectors, and maintaining two encodings of the same ideas serves nobody.

What the past six weeks have clarified is the relationship between that narrative and the actual work. The convergence story rides on top of the v4 design process. It does not drive it. The threads I have walked through in this article, partition tuples, root manifest scan costs, snapshot offloading, bitmap formats, are being argued on their engineering merits by contributors from many companies, with Databricks engineers as participants among peers rather than authors of a fait accompli. Whether Delta 5.0 ultimately adopts the resulting tree is a Delta decision. Whether the tree is good is an Iceberg community decision, and it is being made the slow way, in public, one thread at a time. That ordering is healthy, and so far it is holding.

My advice from June stands: treat convergence as a direction to watch, not a plan to build on. The thing to actually build on is the v4 work itself, which will benefit Iceberg users regardless of what Delta does.

Questions I Hear Most Often About v4

Every time I present on v4, at meetups, on the podcast, or in customer conversations, a familiar set of questions comes back. Answering them here fills in the edges, and each answer doubles as a review of the ideas above.

When will v4 ship? Nobody knows, and anyone who gives you a confident date is guessing. What I can offer is the shape of the process. V3 offers the template: the spec was ratified in mid-2025, and engine support then rolled out across the ecosystem over the following year. V4 will follow the same arc, ratification after the open questions resolve, then a staggered implementation wave. The ratified pieces, relative paths and content stats, tell you the process is moving. The open arguments, partition tuples and column update layering, tell you it is not done. My honest read in July 2026 is that the direction is locked and the timeline is not, and that is the correct order for those two things to settle.

Will I have to migrate my tables? Not on anyone's schedule but your own. Format version upgrades in Iceberg are opt-in, table by table, through a property change. Your v2 and v3 tables keep working indefinitely, and every engine that adds v4 support keeps reading older formats. The pattern from past transitions holds: upgrade a table when a specific new capability pays for the change, not because a version number exists. The one planning item worth doing early is an inventory of every engine, tool, and script that touches your tables, since a table upgraded to v4 becomes invisible to a reader that never learned v4. The capabilities-header work on the REST catalog side exists precisely to make that fleet coordination less painful.

Does v4 make v3 adoption a waste? The opposite. Several v4 proposals are direct extensions of v3 features, and running v3 now is how you position for them. Deletion vectors are the foundation the compact bitmap work refines. Row lineage is the foundation the row timestamp proposal builds on. The variant type is a major consumer of the new content stats representation. Teams that adopt v3 features today are learning the operational patterns that v4 assumes as background. Waiting on v4 while ignoring v3 gets you the worst of both timelines.

Is v4 mostly about streaming? Streaming is the loudest driver, but I count three, and the balance matters. The commit economics work serves streaming. The stats and metadata-as-data work serves AI and extreme-width tables. Relative paths and the operational threads serve platform teams running Iceberg across regions and disaster recovery boundaries. If your workloads are classic batch analytics on stable tables, v4 will still reach you through faster planning and cheaper maintenance, just less dramatically. The format is widening its coverage, not pivoting.

Should I be worried about the Databricks influence? Attention is warranted. Worry, based on what the list actually shows, is not. The observable facts as of July: proposals get argued by contributors across Google, Apple, Snowflake, Netflix, Microsoft, LinkedIn, Starburst, and many independents, votes happen in public with recorded results, and the two ratified v4 pieces so far went through exactly that gauntlet. The governance test is not whether a large vendor proposes things. Large vendors always propose things, and their production pain makes their proposals valuable. The test is whether a proposal from anyone can pass without community consent, and nothing this year suggests it can. Keep watching, and keep judging by the votes rather than the press cycle.

What happens to my tooling for compaction and maintenance? It evolves rather than disappears, and this is worth internalizing early. The adaptive tree does not eliminate maintenance, it relocates it: rebalancing inlined entries from the root into leaf manifests becomes a maintenance activity in its own right, joining compaction and snapshot expiration on the schedule. Snapshot offloading changes what expiration touches. Column update files, if they land, will bring a stitching cost that maintenance can reduce by materializing updated columns back into base files. The through-line of the whole spec's philosophy holds in v4: writers get speed, readers get stability, and background maintenance is the pressure valve between them. Budget for maintenance being different, not smaller.

How do the Parquet and Arrow projects fit into this? More tightly than most coverage acknowledges. Several v4 questions turn on what the file format beneath the table format can do. The column update debate hinges partly on whether Parquet grows a logical-file concept. Columnar metadata leans on Parquet footer improvements to keep small metadata reads fast. The variant and geo types already live as joint efforts across the Iceberg and Parquet specs, and that co-evolution continues. When I tell people to follow the Iceberg dev list, I increasingly add the Parquet list to the assignment, since the layers are moving together and decisions ricochet between them. My weekly Apache newsletter tracks both for exactly this reason.

What is the single most important thread to watch for the rest of 2026? If I had to pick one, it is the single-file commits sync track, including the partition tuple question, because it is the keystone. The root manifest design determines what snapshot offloading offloads into, what the bitmap structures index, what change detection walks, and what commit costs look like for every workload. Most of the other proposals flex to fit whatever shape it settles into. Second place goes to efficient column updates, because its layering question, Iceberg or Parquet or both, is the most genuinely undecided architecture call on the board, and its resolution will say a lot about how the two projects divide responsibility for the next decade.

Where do I follow all this without making it a part-time job? Three tiers, by effort. Lowest effort: follow recaps from people who read the source, and check the official spec page occasionally for language that says "v4 and later," since spec text is ground truth. Medium effort: skim the dev list archives monthly at lists.apache.org, subjects only, and open anything tagged DISCUSS or VOTE that touches your workload. Highest effort: subscribe to the list and read the design documents linked from the big threads, which is where the diagrams and cost analyses live. The community does all of this in the open specifically so that you can. Take them up on it.

Why This All Coheres

Step back from the individual threads and v4 resolves into one picture with three panels.

The first panel is commit economics. Single-file commits, the root manifest, snapshot offloading, and delta-encoded schemas all attack the same invariant: today, the cost of committing scales with the size of the table's metadata rather than the size of the change. Every one of these proposals rewrites that invariant so streaming-rate commits stop being self-harm.

The second panel is metadata as data. Content stats, aggregate stats, Parquet manifests, and the compact bitmap work all treat metadata as structured, typed, columnar information that deserves the same query optimization machinery as the data itself. Richer metadata that stays cheap to read is what makes planning possible at extreme width and, eventually, what opens the door to the index structures that AI retrieval workloads need.

The third panel is granularity of change. Deletion vectors gave us row-level change without file rewrites in v3. Column update files aim to give us column-level change without row rewrites in v4. Relative paths give us table-level relocation without metadata rewrites. In each case the format learns to express a smaller unit of change, and every smaller unit multiplies what workloads the lakehouse can hold.

Streaming, AI, and operations at scale. Those are the three forces, and every thread on the list maps to one of them. The format is being reshaped by the people who run it hardest, which is the best possible source of design pressure.

What Practitioners Should Do in July 2026

Let me close the technical tour with grounded guidance, updated for exactly where things stand.

Run v3 and actually use it. The 1.11 line is current, and v3 capabilities like deletion vectors, the variant type with shredding, row lineage, and the geo types are where the immediate wins live. I still meet teams debating v4 timelines who have not turned on deletion vectors. Harvest the fruit that is ripe.

Track the threads that map to your pain, not all of them. Streaming teams should follow single-file commits, snapshot offloading, and the Flink equality delete conversion. AI platform teams should follow efficient column updates and the stats work. Multi-region operators should note that relative paths are now ratified and start planning for the day their engines support v4 tables. Reading everything is my job. Reading what matters to your workload is yours.

Learn to read the list yourself. The archives at lists.apache.org are public and searchable, and the JSON API behind them makes it easy to track thread activity programmatically, which is exactly how I keep my newsletters current. An hour a month skimming subjects will keep you ahead of every secondhand summary, including mine.

Discount predictions about timing. Nobody serious has committed a v4 ship date, and the honest tell is the state of the arguments: partition tuples and column update layering are genuinely open. Specs ship after the arguments resolve. What you can bank on is the direction, because the direction is now backed by votes, not just enthusiasm.

And hold the meta-lesson. The reason any of this is worth your attention is that it happens in the open, under governance no single vendor controls, with ratification by public vote. That is why an Iceberg table you write today will be readable by engines that do not exist yet, and it is why the arguments are slow. The slowness is the feature. It is what makes the result safe to build a decade on.

Go Deeper

Everything in this article came from following the project at the source: the dev list threads, the design documents, the votes, and the spec text. If you want to build the kind of foundation that makes those sources readable, from metadata internals through operating real lakehouse systems and the AI workloads now reshaping them, that is what my books are for. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, along with further titles on lakehouse architecture, data engineering, and agentic analytics.

Browse the full collection of my books on data and AI at books.alexmerced.com and turn the horizon into working knowledge before it arrives.

Delete Files vs Deletion Vectors in Apache Iceberg: How V3 Rewrote the Economics of Changing Data

Alex Merced — Wed, 08 Jul 2026 22:36:45 +0000

Here is a fact that surprises almost everyone the first time they hear it: in a data lakehouse, deleting a single row is one of the hardest things you can do.

Inserting a billion rows? Easy. Scanning a petabyte? Routine. But run DELETE FROM orders WHERE order_id = 12345 against a table built on files in object storage, and you have asked the system to do something its foundations actively resist. The files that hold your data cannot be edited. Object stores like Amazon S3 do not let you open a file and change byte 4,000,017. Files get written once, read many times, and eventually removed. That is the deal.

Apache Iceberg's answer to this constraint has evolved across versions of its table format spec, and the evolution tells a great engineering story. Version 2 introduced delete files, which made row-level changes practical. Version 3 introduced deletion vectors, which made them fast and stable under pressure. The difference between the two sounds like a storage detail, but it reshapes how reads behave, how updates scale, and what kinds of workloads a lakehouse can honestly support.

This article is a deep dive into both mechanisms, written for the reader who wants the logic to actually make sense rather than just memorizing feature names. We will build up from first principles: why immutability forces strange designs, how v2 delete files work and where they hurt, what deletion vectors change, and why the access patterns improve so dramatically. No prior knowledge of the spec required. Some patience for analogies strongly recommended, because I intend to use several.

The Problem: Editing Books in a Library That Forbids Pens

Let me set up the mental model we will use for the whole article.

Picture a vast library. Every book in it is printed, bound, and sealed. The library's one absolute rule is that nobody may write in a book, tear out a page, or alter a volume in any way. You may add new books, and you may remove entire books from the shelves, but the books themselves are frozen the moment they arrive.

This library is an Iceberg table. The books are data files, typically Parquet. The rule is the immutability of object storage. And the catalog at the front desk, which tracks exactly which books belong to the current collection, is Iceberg's metadata layer. Every "edition" of the library, meaning every snapshot of the table, is just a list of which books count.

Appending data fits this world perfectly. New data becomes new books, and the catalog adds them to the list. Reading fits perfectly too. But now a patron walks in and says: paragraph three on page 212 of one specific book is wrong, remove it.

You have exactly two honest strategies.

Strategy one: reprint the book. Take the whole volume, typeset a new copy identical except for the offending paragraph, add the new book to the catalog, and drop the old book from the list. The library stays clean and simple. Every book on the shelf is fully correct, and readers just read. The cost lands entirely on the writer, who reprinted hundreds of pages to remove one paragraph. In Iceberg terms, this is copy-on-write. Deleting one row rewrites the whole data file that contains it.

Strategy two: publish errata. Leave the sealed book alone. Instead, print a slim companion pamphlet that says "in book X, ignore paragraph three on page 212," and file it in the catalog next to the book. Writing is now nearly free. The cost moves to every future reader, who must check for pamphlets before trusting any page. In Iceberg terms, this is merge-on-read, and the pamphlets are delete files.

Neither strategy is wrong. They trade the same total work between writers and readers. Copy-on-write suits tables that change rarely and get read constantly. Merge-on-read suits tables that change constantly, like anything fed by change data capture from an operational database, streaming updates, or frequent GDPR-style targeted deletes.

Iceberg v2 made merge-on-read a first-class citizen. And the entire story of this article is about what those pamphlets look like, because it turns out the format of an erratum matters enormously once you have millions of them.

Iceberg V2 Delete Files: Two Kinds of Pamphlets

The v2 spec defines two kinds of delete files, and they answer the "which rows are dead" question in genuinely different ways. Understanding both is worth your time, because the difference explains a lot of real-world engine behavior.

Position delete files identify a dead row by its address: the path of the data file it lives in, plus its row number within that file. A position delete file is itself a data file, usually Parquet, whose rows are pairs like "file s3://bucket/data/00042.parquet, position 1387." The pamphlet says: in this exact book, ignore line 1,387.

Position deletes are precise. A reader holding a data file and its matching position deletes knows exactly which rows to skip, with zero ambiguity and no value comparisons. The catch is that the writer must know positions. To write "position 1387 is deleted," something first had to read file 00042 and find that the target row sits at position 1387. The writer pays a lookup cost to produce a cheap-to-apply delete.

Equality delete files identify dead rows by their values instead: "any row where order_id equals 12345 is deleted." No file paths, no positions. The pamphlet says: wherever you see this sentence in any book, ignore it.

Equality deletes flip the trade. The writer pays almost nothing. A streaming pipeline receiving "order 12345 was deleted" from a source database can write that fact immediately without scanning anything. This is why engines like Apache Flink lean on equality deletes for high-velocity change data capture. But the reader inherits the search. Every scan of potentially affected data must compare row values against the equality conditions to decide what survives. The delete is cheap to declare and expensive to apply, on every read, until compaction cleans it up.

So v2 gave the ecosystem a legitimate toolkit. Targeted engine-driven deletes and updates typically produced position deletes. Streaming CDC produced equality deletes. Compaction jobs periodically folded the pamphlets back into reprinted books. Merge-on-read on an open table format became real, and it is hard to overstate how important that was for bringing warehouse-style workloads onto data lakes.

Then people used it at scale, and the cracks showed.

Where V2 Position Deletes Hurt: A Story of Too Many Pamphlets

The problems with v2 position deletes were not correctness problems. They were problems of accumulation and shape. Let me walk through them one at a time, because each one motivates a specific design choice in v3.

Problem one: pamphlets multiply. Every delete operation writes new delete files. Run a small targeted delete every five minutes against the same hot data, and you do not get one growing erratum per book. You get a fresh stack of tiny pamphlets with every commit. A data file that suffers frequent updates might have dozens or hundreds of position delete files referencing it across the table's current snapshot.

Now think about what a reader must do. To scan one data file correctly, the engine must find every delete file that might apply to it, open each one, read the positions, and merge them all into a single picture of which rows are dead. That means many small reads against object storage, where every request carries latency and cost. It means memory spent holding and merging position lists. The AWS analytics team described this exact failure mode when explaining the motivation for v3: many small delete files placing a heavy burden on engines through numerous file reads and costly in-memory conversions.

The library version: before reading one book, the librarian must gather forty pamphlets from different drawers, cross-reference all of them, and compile a master list of lines to skip. The reading itself was never the slow part.

Problem two: pamphlets go stale and pile up. Position delete files in v2 were not consolidated on write. New deletes did not merge with old deletes against the same data file. They just accumulated as additional files until a maintenance job compacted things. Between maintenance runs, read performance degraded steadily as the pamphlet stacks grew. Tables under constant modification needed aggressive, expensive compaction schedules just to hold query latency steady. Skipping maintenance for a busy week could mean noticeably slower dashboards by Friday.

Problem three: the granularity dilemma. Engines writing position deletes in v2 faced an awkward choice about how to scope delete files. Write one delete file per affected data file, and a commit touching ten thousand data files produces ten thousand new small files, an operational headache all its own. Write broader delete files scoped to a partition, and readers scanning any one data file must open delete files that mostly describe other data files, reading and filtering irrelevant positions. Neither option was good. Engines picked their poison, and users inherited whichever downside their engine chose.

Problem four: the pamphlets were books. This one is subtle but real. Position delete files in v2 were themselves Parquet files, with schemas, columns for the file path and position, headers, footers, and all the machinery of a general-purpose columnar format. Parquet is built for large analytical datasets, and it is wonderful at that job. Using it to store what is logically a set of integers is like using a shipping container to mail a postcard. Every read of a delete file paid format overhead to extract a small amount of very simple information, and file path strings repeated over and over inside them.

Add these up and you get the v2 experience under heavy churn: correct results, mounting read amplification, a growing metadata sprawl of tiny files, and a permanent tax paid to maintenance jobs to keep the whole thing acceptable. For moderate workloads it was fine. For the workloads people increasingly wanted, meaning near-real-time CDC mirroring of operational databases into the lakehouse, it strained.

The v3 designers looked at all four problems and noticed something: every one of them traces back to the decision to represent deletes as an open-ended pile of files that readers must discover and reconcile. Fix the representation, and the whole pile of problems collapses.

Enter Deletion Vectors: One Card Per Book

Iceberg v3 replaces position delete files with deletion vectors, and the core idea fits in a sentence: every data file gets at most one compact, binary record of exactly which of its rows are deleted.

Not a stack of pamphlets. One card, kept current, per book.

Let me unpack the three design decisions packed into that sentence, because each one kills a specific v2 problem.

Decision one: at most one deletion vector per data file per snapshot. This is a hard rule in the spec, not a suggestion. When a writer deletes more rows from a data file that already has a deletion vector, it cannot just add another record. It must read the existing vector, merge the new positions into it, and write the result as the file's new single vector. The spec goes further for migration: if any position delete files still exist for a data file from its v2 days, a writer updating that file's deletes must fold them into the vector too, so readers holding a vector can safely ignore old position delete files entirely.

Notice what this does. The reconciliation work that v2 pushed onto every reader now happens once, at write time, performed by the party who is already writing anyway. Readers never merge anything. For any data file, there is one authoritative answer to "which rows are dead," and it is exactly one lookup away. Problem one and problem two die together. Delete information stops accumulating into stacks because the stack can never exceed height one.

Decision two: the vector is a bitmap, not a list. A deletion vector represents deleted positions as a Roaring bitmap, a compressed bitmap structure. I will explain bitmaps properly in the next section, because they are genuinely delightful, but the immediate point is compactness and speed. Checking whether row 1,387 is deleted becomes a bitmap membership test, one of the cheapest operations computers do, rather than a search through merged lists. Storage shrinks dramatically compared to Parquet files enumerating positions. Problem four dies here: the postcard finally travels as a postcard.

Decision three: vectors live in Puffin files. Puffin is a file format from the Iceberg project designed for exactly this category of thing: compact binary blobs of statistics and auxiliary structures that ride alongside data files. A single Puffin file can hold many deletion vectors for many different data files, and Iceberg's metadata records, for each data file, which Puffin file holds its vector plus the exact byte offset and length of the blob within it.

That last detail deserves a pause, because it solves problem three, the granularity dilemma, with real elegance. In v2 you chose between one delete file per data file (too many files) or broad delete files (irrelevant reads). In v3, writers get file-level granularity and file-count efficiency at the same time. A commit deleting rows across a thousand data files can pack a thousand deletion vectors into one Puffin file. Readers needing the vector for one specific data file seek directly to its offset and read only its bytes. Nobody reads anything irrelevant, and nobody floods the object store with thousands of tiny files. The trade-off simply no longer exists.

There is one more clever wrinkle: writers are not required to rewrite Puffin files that contain replaced vectors. If a vector inside a shared Puffin file gets superseded by a new merged vector elsewhere, the old bytes just become dead weight until maintenance reclaims them. That keeps the write path fast, and it is a classic Iceberg move, trading a little garbage for a lot of speed and letting cleanup happen asynchronously.

Roaring Bitmaps, Explained Like You Are Human

I promised an accessible explanation of the bitmap, so let us earn the word "accessible."

Start with the plain idea. A data file holds rows at positions 0, 1, 2, and so on. Imagine a row of light switches, one per position. Switch on means deleted, switch off means alive. That row of switches is a bitmap. To ask "is row 1,387 deleted," you look at switch 1,387. No searching, no comparing, just a direct look. One bit of storage per row.

One bit per row is already tiny. A data file with a million rows needs a raw bitmap of one million bits, about 122 kilobytes, to describe any possible pattern of deletions across all of them. Compare that with a Parquet position delete file spelling out a long file path string and a big integer for every single deleted row.

But real deletion patterns let us do far better than raw bitmaps, because real patterns are not random. Deletes cluster. A batch job removes a contiguous chunk of rows that arrived together. A GDPR request touches a handful of scattered rows. Most files have either very few deletes or big dense runs of them. Roaring bitmaps are a widely used structure built to exploit exactly this, and they show up across serious data systems for good reason.

The intuition behind Roaring is neighborhood-level bookkeeping. The bitmap divides the full range of positions into fixed-size neighborhoods and picks a different representation for each neighborhood depending on how many switches are on there. A neighborhood with just three deleted rows does not deserve a full grid of switches. It stores a short list: "3, 41, 907." A neighborhood where deletion is heavy flips to the actual bit grid, which is more compact than a long list once membership gets dense. Runs of consecutive deletions can be recorded as ranges: "everything from 20,000 to 45,000 is deleted" is one compact fact rather than 25,000 entries.

Each neighborhood independently picks whichever representation is smallest for its own situation, and the structure adapts as deletes accumulate. The result stays small whether a file has five deleted rows, five million, or a solid block in the middle. Membership tests stay fast in every representation. And merging two Roaring bitmaps, exactly the operation writers perform when folding new deletes into an existing vector, is fast and well-trodden, since the format has years of production hardening across the industry behind it. Iceberg's spec builds on this foundation, using 64-bit row positions with the standard 32-bit Roaring machinery covering the ranges that practically occur.

Back to the library one more time. The v2 pamphlet stack made the librarian collect and cross-reference errata before every reading. The v3 card is a single laminated sheet clipped inside the book's front cover, marked up in a shorthand that compresses "ignore lines 20,000 through 45,000" into a single stroke. The librarian glances at the card and reads. That is the whole ceremony now.

How the Access Patterns Actually Change

Formats are means. Access patterns are the ends. Let us walk through the moments in a table's life and watch what changed, because this is where the design decisions become felt experience.

The read path. In v2, planning a scan over a data file meant consulting metadata for all position delete files whose scope might cover it, fetching those files from object storage, decoding Parquet, filtering out entries about other data files, and merging everything into an in-memory structure before the actual data scan could apply it. The cost scaled with delete history: the more modification a file had suffered since its last compaction, the more work every subsequent query performed, over and over.

In v3, planning finds one metadata entry per data file pointing at one blob. The engine issues one ranged read at a known offset, gets a Roaring bitmap, and streams the data file while testing each row position against the bitmap. The cost is flat. It does not matter whether the file endured one delete commit or one thousand since compaction, because writers merged history into a single current vector as they went. Query latency stops degrading between maintenance runs, which the community has repeatedly called out as the headline benefit: performance no longer decays as deletions accumulate, and cost spikes from delete-file sprawl stop appearing.

The write path. Writers took on the merge obligation, so are writes worse? Barely, and the work is proportionate. A delete commit identifies affected data files, computes new dead positions, loads each affected file's existing vector if one exists, unions the bitmaps, and writes fresh vectors packed into a new Puffin file. Bitmap unions are cheap, and the writer was already doing per-file work to find the positions. Compare that honestly against v2's alternative: v2 writes were only cheaper because they quietly deferred reconciliation onto every future reader, forever, until compaction. V3 moves a small, bounded cost to the one moment where it is paid exactly once. This is simply better accounting.

Change data capture and streaming. This is the workload that motivated so much of the design, and it is where the improvement compounds. A CDC pipeline mirroring an operational database delivers a relentless drizzle of small updates and deletes. Under v2, that drizzle became continuous growth in delete file count, which became read amplification, which became a compaction treadmill you could never step off. Under v3, the same drizzle becomes in-place refinement of per-file vectors. Community benchmarking of merge-on-read under v3 has shown filtered reads seeing dramatic speedups under high churn, with the advantages growing as change volume scales. The AWS teams that benchmarked v3 deletion vectors against v2 position deletes on EMR and elsewhere frame the same conclusion: stable query performance and reduced fragmentation over time, in heavy-update scenarios that previously required constant babysitting.

Maintenance. Compaction does not disappear in v3, and nobody should tell you it does. Data files still accumulate deleted rows that occupy space until a rewrite physically drops them, and heavily deleted files still deserve rewriting for scan efficiency. What changes is the pressure. In v2, compaction defended query latency itself, so falling behind hurt immediately and visibly. In v3, queries hold steady on their own, and compaction returns to its proper job of reclaiming storage and right-sizing files on a relaxed schedule. Fewer emergency maintenance windows, more boring Tuesdays. In data infrastructure, boring is the highest compliment.

Concurrency. A quieter benefit worth naming: the one-vector-per-file rule gives concurrent writers a crisp conflict model. Two commits deleting rows from the same data file visibly contend on that file's vector, and Iceberg's optimistic concurrency handles retry and merge cleanly. In v2, overlapping delete commits could both succeed by each adding pamphlets to the pile, papering over contention by making readers pay for it later. V3 surfaces the conflict at the moment it happens and resolves it once.

What About Equality Deletes?

A careful reader will have noticed that everything above concerns position deletes. So what happened to equality deletes, the write-cheap pamphlets that streaming engines love?

They survive in v3. Deletion vectors are positional by nature. A bitmap of row positions can only be built by something that knows positions, which means something that has located the target rows. Equality deletes exist precisely for writers who refuse to pay that lookup at write time, so a bitmap cannot replace them without destroying their reason to exist.

The practical pattern in the ecosystem is a division of labor across time. Streaming writers land equality deletes for immediate, cheap durability of change events. Maintenance and compaction processes then convert that backlog, resolving equality conditions into concrete row positions and folding the results into deletion vectors, restoring the fast stable read path. Equality deletes work as a short-term buffer, and deletion vectors work as the settled steady state. Meanwhile the position delete file, the v2 workhorse, is formally deprecated in v3. New tables on the v3 format produce deletion vectors for positional deletes by default, and the spec requires that when updating deletes for a data file, any lingering position deletes get absorbed into its vector.

The deprecation is worth dwelling on for a moment, because table format specs rarely remove things. Deprecating position delete files was the community saying, with unusual clarity, that the v2 representation was a dead end at scale and the ecosystem should converge on the vector model. That kind of decisive pruning keeps a spec healthy. It is also a small window into how Iceberg evolves: real workloads exposed real limits, the community absorbed lessons from across the industry, including similar vector designs proven elsewhere, and the format moved. Open standards improve in public, and this feature is one of the cleaner examples I can point to.

A Worked Example: One Table, One Week, Both Worlds

Abstractions settle best with a story, so let us run the same week twice.

The table is customer_orders, merge-on-read, fed by CDC from a production database. It holds 2,000 data files. Business is steady: all week, order corrections and cancellations trickle in, touching a few thousand rows spread across roughly 400 of those files, in small commits landing every few minutes.

The week on v2. Each commit writes position delete files covering the rows it touched. By Wednesday, hot data files each have fifteen or twenty small delete files pointing at them, and the table has accumulated thousands of new delete files overall. The nightly dashboard queries scan wide ranges of the table, and each scanned data file drags its personal pile of pamphlets into memory first. Object storage request counts swell. Latency climbs a little each day, the way it always does, and the platform team's compaction job on Thursday night brings it back down, the way it always does. Everyone has stopped questioning this rhythm. It is just what the lakehouse costs.

The week on v3. Each commit computes positions for the rows it touched, merges them into the existing vectors for the affected data files, and writes the updated vectors packed into one new Puffin file per commit. A hot data file that gets touched thirty times during the week still has exactly one deletion vector on Friday, reflecting all thirty commits. Dashboard queries fetch one small blob per scanned file, all week, at flat cost. Thursday's compaction still runs, but it is reclaiming space from deleted rows at leisure, not rescuing query latency. Nobody notices anything, which is the point. The dramatic version of this story is that there is no dramatic version anymore.

Same table, same business events, same total information. The only thing that changed is the shape of the bookkeeping, and the shape turned a weekly performance sawtooth into a flat line.

Practical Guidance for Adopting Deletion Vectors

Some grounded advice for putting this into practice, drawn from where the ecosystem stands today.

Check your engine versions before you leap. Deletion vectors require Iceberg format version 3, and v3 support has been rolling across the ecosystem since the spec's ratification in mid-2025. Apache Spark support arrived through recent Iceberg releases, engines like Dremio and Trino and the major cloud services have been shipping v3 capabilities, and vectors are produced by default once a table is on format version 3 in current implementations. Verify every engine that touches a shared table, readers included, since a v3 table with vectors is not legible to a reader that only speaks v2.

Upgrade deliberately. Moving a table to v3 is a metadata operation, a table property change setting the format version, and existing data files stay valid. Existing v2 position delete files also remain readable, and the spec's migration rule handles convergence: as writers touch files, old position deletes get folded into new vectors. A table under active modification therefore migrates itself gradually. Running a compaction after upgrading accelerates the convergence and gets you to the clean steady state sooner.

Revisit assumptions that v2 taught you. If your platform runs aggressive compaction schedules purely to defend read latency against delete file sprawl, v3 likely lets you relax the frequency and spend that compute elsewhere. If you steered workloads toward copy-on-write specifically because merge-on-read reads degraded too fast, that calculus deserves a rerun, since merge-on-read under vectors holds up far better. Copy-on-write still wins for read-hot, rarely modified tables. The gap for update-heavy tables just narrowed a lot.

And keep expectations honest. Deletion vectors do not make deleted rows free. The dead rows still sit inside data files consuming storage until a rewrite drops them, and a file that is 90 percent deleted still deserves compaction. Vectors fix the cost of knowing what is deleted, which was the part that scaled badly. Physical cleanup remains a maintenance concern, just a calmer one.

Questions I Hear Most Often

When I cover this topic at meetups and on my podcast, a familiar set of questions comes back from the audience. Working through them here fills in edges the main narrative skipped, and each answer is a chance to reinforce the core model.

Do deletion vectors change my query results in any way? No. Vectors and delete files are alternative encodings of the same logical fact, namely the set of rows that no longer belong to the current snapshot. A query against a v3 table with vectors returns exactly what the equivalent v2 table would return. What changes is the cost profile of producing that answer, not the answer. Correctness was never v2's problem.

How do deletion vectors interact with time travel? Beautifully, and this trips people up in a good way. Iceberg snapshots are immutable, and each snapshot references the specific vectors that were current when it was committed. When a writer merges new deletes into a file's vector, it writes a new vector for the new snapshot. The old snapshot still points at the old vector. Query the table as of last Tuesday and you get last Tuesday's deletion state, applied through last Tuesday's vectors. Nothing about the vector model weakens the format's history guarantees, since vectors participate in snapshots exactly like data files always have.

Does a delete now require reading data files to find row positions? For positional deletes, yes, something has to locate the target rows, and that was equally true in v2 for position delete files. The engine scans candidate files, identifies matching rows, and records their positions. Iceberg's metadata makes this cheaper than it sounds, since partition pruning and column statistics narrow the candidate files before any scanning starts. And for writers that genuinely cannot afford the lookup, equality deletes remain available as the deferred option, as covered earlier.

What happens if a data file has both an old position delete file and a new deletion vector? The spec resolves this cleanly in favor of the vector. When a writer creates a vector for a data file, it must absorb all previously written position deletes for that file, and from that point readers holding the vector can ignore matching position delete files entirely. There is never a situation where a reader must combine both representations for one file. One card per book, and once the card exists, the pamphlets for that book are void.

Can multiple deletion vectors share a Puffin file, and does that cause coupling problems? Many vectors can share one Puffin file, and no, coupling stays loose. Each vector is addressed by its own offset and length within the file, so readers touch only the bytes for the data file they care about. When one vector in a shared Puffin file gets superseded, the others remain perfectly valid where they sit, and the stale bytes wait for cleanup. Writers gain the file-count efficiency of packing without readers inheriting any cross-file entanglement.

Is this the same as Delta Lake's deletion vectors? The family resemblance is real and acknowledged. Delta Lake shipped a deletion vector feature earlier, and the broader industry, including systems well outside the table format world, converged on compressed bitmaps for row invalidation because the approach genuinely works. Iceberg's version is its own design within the Iceberg metadata model, with the one-vector-per-file rule, Puffin storage, and spec-mandated migration behavior. I take the convergence as a healthy sign. When independent communities land on the same shape of answer, the shape is probably right.

Does merge-on-read now beat copy-on-write everywhere? No, and beware anyone selling that conclusion. Copy-on-write still produces the purest read path there is, plain data files with nothing to check, and for tables that are read constantly and modified rarely, rewriting the occasional file remains a great bargain. What v3 changed is the slope of the trade. Merge-on-read used to degrade under churn badly enough that teams avoided it even for workloads it suited. Now it holds steady, so the decision returns to the honest fundamentals: modification frequency, read patterns, and latency requirements, rather than fear of pamphlet sprawl.

Do I still need to run compaction and snapshot expiration? Yes, on both counts, and it bears repeating because "vectors fix deletes" is easy to over-read. Deleted rows still physically occupy space inside data files until compaction rewrites them out. Old snapshots still pin old files, vectors included, until expiration releases them. Vectors removed the emergency from maintenance, not the need for it. Think of v3 as converting maintenance from a performance defense into routine housekeeping.

How does this affect my object storage bill? Generally favorably, through two channels. Request counts drop, since readers fetch one ranged blob per data file instead of opening piles of small delete files, and request charges on high-traffic tables are a real line item. Storage for delete information drops too, since compressed bitmaps are far smaller than Parquet files enumerating paths and positions row by row. Offsetting this slightly, superseded vectors linger in shared Puffin files until cleanup. Net effect across published tests and field reports points the right direction: less storage, fewer requests, cheaper scans.

Where should I go deeper after this article? The Iceberg spec's sections on row-level deletes and the Puffin format are more readable than most specs, and the engine documentation for whatever you run, whether Dremio, Spark, Trino, or a managed service, will cover the version knobs and defaults. The benchmark posts from AWS and community authors on v2 versus v3 delete performance are worth your time for concrete numbers on workloads resembling yours. And the dev list archives show the design discussions themselves, which I find is where the deepest understanding lives.

The Bigger Lesson Hiding in a Small Feature

Zoom out with me, because I think deletion vectors teach something beyond Iceberg.

Every durable storage system that supports modification eventually faces the same question: when you cannot change the past, where do you record the corrections? Accountants faced it centuries ago and invented adjusting entries rather than erasing ledgers. Version control systems faced it and chose immutable commits with evolving references. Iceberg v2 and v3 are two answers to the same question, and the difference between them is a lesson in where to spend work.

V2 let corrections accumulate as an open set of records and asked readers to assemble the truth. V3 requires writers to maintain the assembled truth continuously, one compact structure per data file, so readers just look it up. The total information is identical. The difference is that v3 does the assembly once, at the moment of change, instead of on every read forever after. Almost every scaling problem in data systems eventually yields to some version of this move: find the work being repeated implicitly, and do it once explicitly.

It also shows why representation is destiny. Position delete files and deletion vectors encode the same facts. Yet one representation produced file sprawl, read amplification, and a maintenance treadmill, and the other produced flat lookups and bounded state, purely through choices about granularity, format, and ownership of the merge. When someone tells you a format war is bikeshedding, remember this pair.

If you take one model away from this article, take the library. V2 answered "which rows are deleted" with a stack of pamphlets the reader must reconcile. V3 answers it with one laminated card per book, kept current by whoever last made a change, written in a shorthand built for exactly this job. Everything else, the Puffin packing, the Roaring compression, the one-per-file rule, the deprecation of the old way, is engineering in service of that single clean idea.

The lakehouse promise has always been warehouse capabilities on open, ownable storage. Row-level change was the capability where that promise strained hardest, and deletion vectors are the moment it stopped straining. Tables that mutate constantly now behave like tables, not like archives with apology notes attached.

If explanations like this one work for you, this is how I write books too. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, and I have written additional titles on lakehouse architecture, data engineering, and AI, all built around making the underlying logic of these systems genuinely understandable. You can browse the full collection of my books on data and AI at books.alexmerced.com.

The Variant Type in Apache Iceberg: How Shredding Turns Messy JSON Into Fast Analytics

Alex Merced — Wed, 08 Jul 2026 22:25:57 +0000

Every data engineer I talk to has the same story. Somewhere in their company, there is a table with a column full of JSON strings. Maybe it holds event payloads from a mobile app. Maybe it holds sensor readings from a fleet of devices. Maybe it holds the raw output of some third party API that changes its shape every quarter. Whatever the source, that column is both the most valuable and the most painful part of the table.

It is valuable because it holds the raw truth. It is painful because every query that touches it pays a tax. The engine has to read the whole string, parse it, walk the structure, and pull out the one field the query actually wanted. Multiply that by billions of rows and you get slow dashboards, angry analysts, and cloud bills that make finance teams nervous.

Apache Iceberg version 3 of the table format spec introduces the Variant type to fix this problem. Variant gives you the flexibility of JSON with performance that gets close to regular typed columns. The trick that makes this possible is called shredding, and shredding is what I want to explain in this article.

My goal here is not to walk you through the spec line by line. The spec exists and you can read it. My goal is to make the logic of Variant and shredding click for you. I want you to finish this article and think, "oh, of course that is how it works." Once the mental model lands, the spec details, the engine documentation, and the benchmark numbers all become easy to reason about.

The Two Bad Options We Lived With Before

To appreciate Variant, you need to feel the pain of the two options that came before it. For years, anyone storing semi-structured data in an analytic table picked their poison from a menu of two.

Option one: store the JSON as a string. You declare a column of type string or varchar, and you dump the raw JSON text into it. This is the path of least resistance. Ingestion is trivial. Nothing ever breaks on write, since any valid text fits in a string column.

The cost shows up at read time. Say you have a query that wants the device_id field out of a payload with two hundred fields. The engine cannot reach in and grab just that field. Text is opaque to a columnar engine. It has to load the full string for every row, run a JSON parser over it, build some internal representation of the document, and then extract the one value it wanted. That parsing work happens on every query, every time, forever. You pay full price for the whole document to read one field.

There is a storage cost too. JSON text is verbose. Field names repeat on every single row. The number 42 takes two bytes as a string plus quoting and structure, when a binary integer could hold far larger values in a fixed, compact encoding. Compression helps, but you are compressing waste rather than avoiding it.

Option two: flatten the JSON into real columns. You look at your payloads, decide which fields matter, and create a wide table where each field becomes a typed column. Now queries are fast. The engine reads only the columns it needs, the values are properly typed, and statistics let it skip files that cannot match a filter.

The cost here shows up in the schema. Semi-structured data earns the name because its structure is not stable. Devices get firmware updates and start sending new fields. Different event types carry different attributes. One team nests an object where another sends a flat value. Every one of those changes becomes a schema migration. Your table sprouts hundreds of mostly null columns. New fields arriving in production either break the pipeline or silently vanish before anyone maps them. You have traded query pain for operations pain.

Most real teams ended up running both patterns at once. A raw string column for completeness, plus a curated flattened table for the fields people query most, plus a pipeline keeping the two in sync. That is three things to maintain in exchange for zero new information.

Variant exists to collapse this menu. One column, flexible on write, fast on read.

What Variant Actually Is

At the level of the Iceberg spec, Variant is a data type for values whose structure can differ from row to row. One row might hold an object with ten fields. The next row might hold an object with thirty different fields, or an array, or a plain number. The table schema does not care. The column is just typed as Variant, and each row carries whatever shape it carries.

That sounds like JSON, and conceptually it is close. Variant supports objects, arrays, strings, booleans, and nulls, just like JSON. It goes further on the primitive side. Variant values can hold real dates, timestamps with and without time zones, binary data, and exact decimals. JSON forces all of those into strings or lossy floating point numbers. Variant keeps them as first class typed values, which matters a lot for analytics where a timestamp should behave like a timestamp.

The key design decision is that Variant values are not stored as text. They are stored in a binary encoding defined in the Apache Parquet project. Iceberg v3 adopted this encoding rather than inventing its own, which was a smart move. It means the same physical representation works across Parquet, Iceberg, Spark, and every other engine that speaks the standard. No translation layers, no vendor lock-in at the file level.

The binary encoding splits every Variant value into two pieces: a metadata section and a value section.

The metadata section is essentially a dictionary of field names. Take a JSON document with fields like user_id, event_type, and timestamp. In text form, those names are spelled out in full on every row. In the Variant encoding, the names get collected into a dictionary once, and the actual values refer to them by a small integer ID. Think of it like a legend on a map. Instead of writing "hospital" next to every hospital, the map prints a small symbol and defines it once in the corner.

The value section holds the actual data, with each primitive stored in an efficient typed form. An integer is stored as an integer, not as digit characters. A timestamp is stored as a number, not as a formatted string. Objects and arrays are laid out with offsets, so an engine can jump directly to a specific field without scanning everything before it.

Stop and appreciate what that last part buys you. With JSON text, finding the device_id field means parsing characters from the start of the document until you happen upon it. With the Variant binary encoding, the engine looks up the field ID in the dictionary, follows an offset, and lands directly on the value. It is the difference between reading a book page by page to find a chapter and using the table of contents.

Binary encoding alone already beats string storage. Parsing is cheaper, storage is smaller, and typed values behave correctly. But binary encoding alone does not get you to columnar performance. For that, we need shredding.

Why Binary Blobs Still Fight the File Format

Here is the tension. Parquet, the file format underneath most Iceberg tables, is columnar. Its entire performance story rests on one idea: store each column's values together, so a query reading three columns out of two hundred touches only those three. Column-by-column storage also enables great compression, since similar values sit next to each other, and it enables statistics, since the format can record the minimum and maximum value of each column in each chunk of the file.

Those statistics power one of the most important optimizations in analytics: pruning. Suppose a file's footer says the event_date column in this file ranges from March 1 to March 31. A query filtering for June dates never needs to open that file at all. At scale, pruning routinely eliminates the vast majority of data before any real work begins. Queries feel fast not because engines scan fast, but because they avoid scanning.

Now put a Variant value into a Parquet file as a single binary column. From Parquet's point of view, each row holds one opaque blob of bytes. Parquet cannot see inside it. There are no per-field statistics, because Parquet does not know fields exist. There is no way to read just the device_id bytes, because they are interleaved with everything else inside each row's blob.

So a query filtering on a field inside the Variant is back to brute force. Read every blob in every file, decode each one, extract the field, evaluate the filter. The decoding is much cheaper than JSON parsing, so this is still a win over strings. But all the pruning magic and column-skipping magic that makes columnar analytics fly is switched off.

The structure inside the blob is invisible to the format, and invisible structure cannot be optimized. Shredding is the answer to that. Shredding makes the structure visible.

Shredding: The Core Idea

Here is shredding in one sentence: at write time, the engine looks at the Variant values it is about to store, notices which fields show up consistently, and stores those fields as real, separate, typed Parquet columns alongside the binary encoding.

Let me make that concrete. Imagine you are ingesting clickstream events. Nearly every event carries user_id as a string, event_type as a string, and ts as a timestamp. Beyond those, events carry a grab bag of extra fields that vary by event type. A page view has a url. A purchase has an amount and a currency. An error event has a stack_trace.

Without shredding, every event becomes one binary blob in one Parquet column. With shredding, the writer notices the common fields and pulls them out. Inside the Parquet file, the Variant column physically becomes a group of columns: one typed column for user_id, one for event_type, one for ts, and a residual binary column holding whatever did not get pulled out.

The word "shredding" describes exactly this. The document gets shredded into pieces, and the pieces that occur often enough get filed into their own columns.

I like to explain it with a mailroom analogy. Picture a mailroom that receives thousands of envelopes a day, each stuffed with a different mix of documents. The lazy approach is to shelve the sealed envelopes and open them whenever someone asks a question. That is the binary blob approach. The smart mailroom clerk notices that almost every envelope contains an invoice, a shipping label, and a receipt. So the clerk opens envelopes on arrival, files invoices in the invoice drawer, labels in the label drawer, receipts in the receipt drawer, and keeps the leftover odds and ends in the original envelope on the shelf. Now when someone asks "what was the total of all invoices in March," nobody opens a single envelope. They go straight to the invoice drawer.

The drawers are shredded columns. The envelope on the shelf is the residual binary. And the property that matters most is that nothing was thrown away. Every document is still findable. Common questions just got dramatically faster.

The Mechanics: value and typed_value Pairs

Let us go one level deeper, because the mechanics are elegant and understanding them helps you predict performance.

The Parquet Variant Shredding spec defines how a shredded Variant is laid out. For each field the writer decides to shred, the file stores a pair of columns, conventionally called typed_value and value.

The typed_value column holds the field when it matches the expected type. If the writer decided user_id shreds as a string, then every row where user_id is actually a string lands in the typed column, as a plain Parquet string with all the usual columnar benefits.

The value column is the fallback. Semi-structured data does not sign contracts. Some rogue row might send user_id as a number, or omit it entirely. Rows that do not fit the expected type keep that field in binary Variant form in the fallback column. The pair together always represents the truth: for any given row and field, the data lives in exactly one of the two, and readers know how to check.

This pairing is what lets shredding coexist with chaos. The writer makes a bet on each field's type based on the data it observes. When the bet pays off, which is most of the time for genuinely common fields, the value sits in a fast typed column. When the bet misses, correctness is preserved through the fallback. No write ever fails because a row disagreed with the shredding scheme.

Nesting works the same way, recursively. If most payloads contain an address object with city and zip inside it, the writer can shred address into a group, and inside that group shred city and zip into their own typed columns. A query filtering on city reads one narrow string column, even though city lives two levels deep in the original documents.

Fields that appear rarely never get shredded at all. They stay in the residual binary alongside the row. That is the right call. A field that appears in one row per million would waste space as a dedicated column, since Parquet still has to track the nulls for the other 999,999 rows. Shredding concentrates its effort where repetition creates payoff.

One more mechanical detail worth knowing: the shredding scheme is decided per file at write time, not fixed in the table schema. The Iceberg table schema just says the column is a Variant. Which fields got shredded, and to what types, is recorded inside each Parquet file. Files written in January can shred different fields than files written in June as the data evolves. Readers discover each file's layout from the file itself. Flexibility survives all the way down.

How the Writer Decides What to Shred

A fair question at this point: how does the writer know which fields deserve columns? Nobody declared a schema. That was the whole point.

Engines handle this through inference at write time, and implementations vary in the details. A common pattern is buffering. The writer holds a batch of incoming rows in memory, scans their structure, and tallies which fields appear, how often, and with what types. Fields that clear a frequency bar with a consistent type get picked for shredding. Then the writer flushes the batch into a Parquet file laid out according to that decision.

In Apache Spark's Iceberg integration, this behavior sits behind table properties. Setting write.parquet.shred-variants to true turns inference on, and a companion property controls how many rows the writer buffers to make its decision. A larger buffer means smarter decisions at the price of more memory during writes. Dremio's implementation applies shredding by default on write to Iceberg v3 tables and offers per-table control over the behavior. Other engines make their own choices, but the shape is the same everywhere: observe, decide, lay out.

I want to be honest about the cost, because there is one. Inference and restructuring are real work. Published community benchmarks on this are useful for calibration. One set of experiments on GitHub event data found that enabling shredding added roughly 35 percent to write time across repeated append runs. Another benchmark on EMR with Iceberg 1.11 measured writes at about 2.7 times slower with shredding on, in exchange for reads that averaged about 34 percent faster across 21 filter and aggregation tests, with the shredded table winning 20 of the 21 patterns. Storage grew about 20 percent in that test, since the file carried both the shredded columns and residual data, though results vary a great deal with data shape.

Do not treat those exact numbers as gospel for your workload. Treat them as evidence of the trade shape: pay once at write time, collect on every read for the life of the file. For the standard analytic pattern of write once and query thousands of times, that trade is excellent. For a write-heavy stream that almost nobody queries by field, it might not be. I will come back to this when we talk about when to use what.

Statistics and Pruning: Where the Real Speed Comes From

Faster field extraction is nice. But the biggest wins from shredding come from something quieter: statistics.

Remember that once a field is shredded, it is a genuine Parquet column. Genuine Parquet columns get min and max values, null counts, and all the other metadata that Parquet records per column chunk. Iceberg's own metadata layer participates too. The v3 spec allows Variant columns to carry lower and upper bounds for fields within the variant, keyed by normalized JSON path expressions, so field-level bounds can flow up into Iceberg manifests where scan planning happens.

Connect the dots and you get pruning on JSON fields. Say your telemetry table holds a Variant payload with a shredded severity field, and a query asks for rows where severity equals "critical". The planner checks the bounds. Any file whose severity values range only from "debug" to "info" is skipped without being opened. Any row group inside a surviving file whose stats rule out "critical" is skipped too. The query might touch two percent of the physical data.

This is the exact optimization that made structured columnar analytics fast in the first place, now applied to fields buried inside semi-structured documents. Before Variant shredding, that entire class of optimization was simply unavailable to JSON data. Every query was a full scan wearing different clothes. After shredding, semi-structured fields play by the same rules as ordinary columns.

It helps to tally the layers of savings on a filtered query against a shredded field. First, file pruning skips whole files using Iceberg metadata. Second, row group pruning skips chunks inside files using Parquet stats. Third, column projection reads only the shredded column for the field, not the residual binary and not the other shredded fields. Fourth, the values arrive already typed, so there is no per-row decoding of documents. Each layer multiplies with the others. That compounding is why benchmark deltas on filter-heavy workloads look so large, and why vendors are competing on Variant scan performance. Snowflake, for instance, published an eleven-query Iceberg v3 Variant benchmark showing per-workload speedups like 9.47x on full-object retrieval, precisely because implementation quality on shredded layouts differs.

The Read Path: variant_get and Transparent Fallback

Let us follow a query through the system, because the read path is where all the design choices pay off or do not.

The standard way to pull a field out of a Variant is a function most engines call variant_get. You hand it the column, a JSON path like $.device.firmware.version, and optionally the type you want back. Spark exposes it this way, Dremio provides VARIANT_GET, and other engines follow similar conventions.

Here is the important part: you write the same query whether or not the field is shredded. The path expression describes the logical document. What happens physically is the engine's job.

When the engine plans the query, it inspects the Parquet files in play and checks each file's shredding layout. In files where $.device.firmware.version was shredded, the engine reads that one narrow typed column and is done. In files where it was not, the engine falls back to reading the binary Variant and extracting the field from the encoding. A single query can mix both behaviors across files, since layout is a per-file decision. The user never sees the seam.

This transparency matters more than it might seem. It means shredding is purely an optimization, never a contract. Your queries do not break when the writer's inference changes its mind between files. Your pipelines do not need to know which fields made the cut. Old files written before you enabled shredding keep working next to new files written after. The logical model stays simple while the physical model does whatever is fastest.

There is a practical wrinkle worth flagging. Since shredding decisions live inside data files, it is not obvious from a SQL prompt which paths in your table actually got shredded. DESCRIBE TABLE shows one Variant column either way. Community tooling has started to appear that audits Parquet files and reports which paths are fully shredded, partially shredded, or left in binary, so you can check whether your hot filter paths are getting the columnar treatment. If a critical query filters on a path that inference did not shred, that is a signal to look at your write settings or your data distribution.

Interoperability: Why the Parquet Spec Matters So Much

I spend a lot of my time in the Apache Iceberg community, and one thing I have watched closely is how the Variant work got structured across projects. The type semantics live in the Iceberg spec. The binary encoding and the shredding layout live in the Parquet project. That split was deliberate, and it is the reason Variant is more than a feature. It is a standard.

Because the physical layout is defined at the Parquet level, any engine that implements the spec can read any other engine's shredded files. Spark 4.1 can write shredded Variant data into a shared Iceberg v3 table, and Dremio reads it and uses the shredded layout transparently on its read path. A team can run ingestion on one engine and analytics on another with no coordination beyond both conforming to the published specs. Snowflake has brought its decade of production Variant experience to the same open layout, and DuckDB has introduced native Variant support in the same family of encoding. The Iceberg v3 spec was ratified in June 2025, and since then engine support has been landing across Spark, Flink, and the commercial platforms at a steady clip.

Compare this to how semi-structured support used to work. Every warehouse had its own proprietary internal representation. Your JSON was fast inside one vendor's walls and inert everywhere else. Moving engines meant re-ingesting and re-optimizing everything. With Variant on Iceberg, the optimized representation itself is portable. The shredded columns, the statistics, the residual encoding, all of it sits in open files on your own object storage, readable by whatever engine comes next.

For anyone building a lakehouse, this is the property to care about. Performance features come and go. Formats that multiple competing vendors implement against a shared spec tend to stick around.

When to Use Variant, and When Not To

New capabilities invite overuse, so let me offer some judgment about where Variant fits.

Variant shines when structure genuinely varies or genuinely evolves. Application logs, where every service logs its own fields. Event telemetry, where the schema changes with every app release. IoT payloads, where a fleet of devices runs a mix of firmware versions, each emitting slightly different JSON. API responses from systems you do not control. Configuration snapshots and user profiles with sparse optional attributes. In all of these, the flexible column absorbs change that would otherwise become schema migrations and pipeline breakage.

Variant is the wrong tool when your data has a stable, known structure. If every row has the same twelve fields and always will, declare twelve columns. Plain columns are still simpler and faster than shredded Variant fields, with no inference step and no residual overhead. Do not wrap structured data in a flexibility layer it does not need. I have started seeing tables where someone made every column a Variant "to be safe," and that is a mistake. You pay flexibility costs for rigidity you already had.

The shredding toggle deserves its own judgment call, and the write benchmarks give us the frame. Read-heavy tables with field-level filters and aggregations should shred. That is the classic analytics profile, and the write penalty amortizes across thousands of queries. High-frequency streaming ingestion with tight latency budgets might leave shredding off, or shred later during compaction and maintenance, if the engine supports rewriting files with a different layout. Workloads that always retrieve full documents rather than individual fields get less from shredding, since the residual read happens anyway.

A few operational notes from the field. Variant requires format version 3, so tables on v1 or v2 need to migrate first, and adding a Variant column to an older-format table is not supported by the spec. Engine support is real but still maturing, so check the exact versions in your stack. Spark's mainstream support arrived in the 4.x line with Iceberg 1.10 and later. And some surrounding features lag: fine-grained access control on Variant columns, for example, is not yet supported in certain catalog and governance integrations. Test your specific combination before betting production on it.

A Worked Example, End to End

Let me tie the whole model together with a small scenario you can hold in your head.

You run analytics for a company with a fleet of delivery vehicles. Each vehicle reports telemetry every few seconds as JSON. Core fields appear on nearly every message: vehicle_id, ts, speed, fuel_pct, and a location object with lat and lon. Beyond that, messages vary. Refrigerated trucks report cargo_temp. Newer models report a battery object. Diagnostic events attach fault codes that older firmware formats differently than newer firmware.

You create an Iceberg v3 table with an id column and a Variant column named payload, with shredding enabled on write. Ingestion parses each JSON message into the Variant binary encoding and appends in batches.

At write time, the engine buffers rows and tallies structure. It sees vehicle_id, ts, speed, fuel_pct, location.lat, and location.lon on nearly every row with stable types, so those become typed Parquet columns inside the Variant group. It sees cargo_temp on 20 percent of rows. Depending on thresholds, that may shred too. The long tail of fault codes and firmware quirks stays in the residual binary. Each Parquet file records its own layout, and each file's footer carries min and max stats for every shredded column, with field bounds flowing into Iceberg manifests.

Now the queries arrive. An analyst asks for average speed by hour for one vehicle last Tuesday. The planner prunes to files whose ts bounds overlap Tuesday and whose vehicle_id bounds include the target. Inside surviving files, it reads exactly two narrow typed columns. No JSON is parsed anywhere. The query runs like it would on a fully structured table, because for these fields, it effectively is one.

An operations engineer asks a rarer question: show me the raw fault payloads for refrigerated trucks that reported cargo temperature above threshold. The temperature filter runs against a shredded column with stats, pruning hard. For the surviving rows, the engine pulls the fault details out of the residual binary. Slower per row than a typed column, but the pruning already shrank the row count so much that nobody cares.

Three months later, a firmware update adds a tire_pressure object to new messages. Nothing breaks. No migration runs. New files start shredding the new fields once they become common. Old files keep their old layout. Queries spanning both eras read each file according to its own footer.

That is the promise, delivered: JSON-grade flexibility at write time, columnar-grade behavior at read time, and evolution absorbed without ceremony.

The Deeper Pattern Worth Noticing

Step back from the mechanics for a second, because Variant shredding is an instance of a design pattern that shows up all over great data systems, and recognizing the pattern will serve you beyond this one feature.

The pattern: keep the logical model simple and let the physical layer be clever. Users see one flexible column and one extraction function. Underneath, writers infer structure, split values across typed and fallback storage, record per-file layouts, and publish statistics. Readers stitch it all back together invisibly. Complexity gets pushed to the layer that can automate it, and simplicity is preserved at the layer humans touch.

Iceberg does this everywhere. Hidden partitioning lets users query natural columns while the format manages partition values. Snapshot metadata lets users time travel with one clause while the format manages manifest trees. Variant shredding extends the same philosophy to the shape of the data itself. The schema-on-read versus schema-on-write debate that consumed a decade of data architecture arguments quietly dissolves, since shredding gives you schema discovery on write with schema flexibility preserved.

It also tells you where the ecosystem is heading. The v3 spec allows Variant bounds in metadata but leaves plenty of room for engines to compete on inference quality, shredding policy, and scan implementation. Expect maintenance procedures that re-shred old files based on observed query patterns. Expect smarter inference that watches which paths queries actually filter on. The spec defines the contract, and the innovation happens inside it. That is exactly how open standards are supposed to work.

Questions I Hear Most Often

Whenever I present on Variant at conferences or community meetups, the same handful of questions come up. Answering them here rounds out the picture, and each answer reinforces some part of the mental model we built above.

Does shredding change what my data means? No. Shredding is a physical layout decision, invisible to the logical model. The set of documents in the table is identical whether shredding is on or off. A full retrieval of any row reconstructs the exact same value either way. This is why you can flip the setting between writes without breaking anything. Think of it like reorganizing a warehouse. The inventory did not change, only where things sit on the shelves.

What happens when a field has mixed types across rows? This is the case the value and typed_value pairing exists for. Suppose status_code arrives as an integer on 95 percent of rows and as a string on the rest, thanks to one misbehaving service. The writer might shred it as an integer. The integer rows land in the typed column and enjoy full columnar treatment. The string rows fall back to the binary encoding for that field. Queries still see every row correctly. Filters against the field evaluate both storage locations. You lose some performance on the messy rows, and that is the honest, proportionate cost of messy data.

Can I control which fields get shredded instead of relying on inference? This depends on the engine, and it is an area of active development across the ecosystem. Inference is the default posture in current implementations, with table properties controlling whether shredding happens and how much data the writer samples. As implementations mature, expect finer controls, since teams with well-understood hot paths will want to pin them. The spec itself does not care how the decision gets made. It only defines how a decision, once made, is recorded in the file.

How does Variant interact with compaction and table maintenance? Cleanly, and sometimes to your advantage. Compaction jobs rewrite small files into larger ones, and a rewrite is a fresh chance to make shredding decisions with more data in view. A stream of tiny files written under latency pressure without shredding can be compacted later into large, well-shredded files. This gives you a nice division of labor: the ingestion path optimizes for write speed, and the maintenance path optimizes the layout for reads. Several engines are moving in exactly this direction.

Should I still extract truly critical fields into top-level table columns? Often, yes. Variant does not forbid promotion, it just makes promotion optional rather than mandatory. If tenant_id drives your partitioning, or a field participates in join keys and access policies, giving it a real top-level column keeps it visible to every part of the system, including layers that do not yet understand Variant internals. A sensible pattern is a handful of promoted structural columns plus one Variant column for the evolving payload. You get governance and partitioning on the stable spine, and flexibility on everything else.

Is Variant just for JSON? JSON is the headline use case, but the type is broader than the format. Anything that parses into objects, arrays, and primitives can flow into a Variant, and the primitive set exceeds what JSON text can express. Data arriving from Avro sources, protocol buffers, or engine-native structs can be converted with type fidelity. Dremio, for example, offers TO_VARIANT for converting SQL-typed values while preserving their types, alongside PARSE_JSON for raw text. A date that enters through TO_VARIANT stays a real date inside the Variant, which JSON alone could never promise.

What is the catch? I have covered the write penalty and the storage overhead, so let me name the softer catch: observability. Shredding decisions are made by machines, per file, based on sampled data. When a hot query path underperforms, the reason may be that inference never shredded the path it filters on, and nothing at the SQL layer will tell you that directly. Until engines surface layout information natively, auditing tools that inspect Parquet footers fill the gap. Build the habit of verifying that your important paths actually got the columnar treatment, especially after changing write settings or data sources.

Where does this leave the old string column pattern? Retirement, gradually. There is no scenario where a fresh design should store JSON as varchar in an Iceberg v3 table. The one legitimate reason to keep raw text around is byte-exact archival of the original message for audit or replay, and even then, the text column should sit next to a Variant column rather than replace it. For existing tables, the migration is a rewrite: parse the string column into a Variant column, backfill, then repoint queries. It is real work, but every query on that data pays the parsing tax until you do it.

Closing Thoughts

Semi-structured data stopped being an edge case a long time ago. Logs, events, telemetry, and API payloads make up an enormous share of what modern organizations actually collect, and the tooling for it inside open table formats had lagged behind the warehouses for years. The Variant type in Apache Iceberg v3 closes that gap, and shredding is the mechanism that makes it more than a convenience.

If you take one mental model away from this article, take this one. Variant stores every document in a compact binary form so nothing is ever lost, and shredding notices the structure your data repeats and quietly promotes it into real columns with real statistics. Common questions hit fast typed storage. Rare questions fall back to the binary. The engine decides per file, the reader adapts per file, and you write the same query either way.

The mailroom clerk files the invoices in the invoice drawer and keeps the envelope on the shelf. That is the whole idea. Everything else is careful engineering to make that idea safe, portable, and fast across an open ecosystem of engines.

If you found this style of explanation useful, this is exactly how I approach the books I write on data architecture and AI. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, along with other titles on lakehouse architecture and agentic analytics, all aimed at making complex systems make sense. You can browse the full collection of my books on data and AI at books.alexmerced.com.

The State of Agentic AI Standards in 2026: MCP, A2A, WebMCP, OSI, and the Protocol Stack Taking Shape

Alex Merced — Tue, 07 Jul 2026 19:13:26 +0000

In 2023, an AI agent was a demo. In 2024, it was a framework. In 2025, it was a hundred incompatible frameworks. And in 2026, something genuinely new is happening: the agent world is growing a protocol stack, a set of open standards that determine how agents reach tools, talk to websites, talk to each other, understand business meaning, pay for things, and show their work to humans.

I watch this space from two vantage points. My weekly AI newsletter tracks the standards and protocol developments as they happen, and my day job in the lakehouse world puts me on the receiving end of them, because the single biggest consumer of agent standards is turning out to be data platforms. The agents everyone is building want, more than anything else, to query, analyze, and act on enterprise data, which means every protocol in this article eventually terminates at something I write about the rest of the week: a catalog, a table, a semantic definition.

So this is my mid-2026 map of the agentic standards territory. What each protocol actually does, stated plainly. Where each one stands right now: shipped, versioned, previewed, or aspirational. How they layer rather than compete. The honest open problems, governance, security, and sheer protocol sprawl. And a decision framework for builders who need to ship something this quarter without betting on the wrong horse. The history of data infrastructure teaches one big lesson about moments like this, and I will lean on it throughout: standards wars end, the winners are the ones that stay neutral and layered, and the people who understood the layering early built the durable things.

Why Standards, and Why Now: The N Times M Problem

Every protocol in this article exists to kill the same monster, so let me introduce the monster properly.

Call it the N times M problem. You have N agents, assistants, copilots, and autonomous workflows. You have M things they need to touch: databases, SaaS applications, websites, payment systems, other agents, and human interfaces. Without standards, every pairing is a custom integration, N times M pieces of glue code, each with its own authentication, its own data shapes, its own failure modes, each breaking whenever either side changes. That is not an inconvenience, it is a scaling wall: the glue grows quadratically while the value grows linearly, and by late 2024 every serious builder had hit it.

Standards convert N times M into N plus M. Each agent implements a protocol once. Each tool or counterparty implements it once. Any agent can then reach any tool, and the glue evaporates. This is the oldest trick in computing: it is what TCP did for networks, what HTTP did for documents, what SQL did for queries, what USB did for peripherals, and, in my corner of the world, what Arrow did for in-memory data, Parquet for files, Iceberg for tables, and the Iceberg REST protocol for catalogs. The agent ecosystem is simply the newest domain to rediscover that agreements beat features.

What makes 2026 the turning point is that the agreements stopped being proposals and started being versioned, governed, shipped standards with adoption numbers. The consensus that has emerged is a layered stack, with each layer answering a different question, and the fastest way to understand the space is to walk the layers. That is the structure of everything that follows: tools, then the web, then agent-to-agent, then semantics, then payments, then the human interface, with the cross-cutting problems of identity and security woven through.

Three Years in Fast Forward: How We Got Here

A compressed timeline sets the stage, because the speed of this standardization is itself the story, and each date below marks a layer locking into place.

Late 2024: Anthropic introduces the Model Context Protocol, and the initial reaction is polite interest, one vendor's integration scheme among many. The design choice that ages best is the humility: MCP standardizes plumbing and stays out of opinions about agent architecture, which is exactly what lets everyone adopt it.

Early 2025: MCP adoption crosses the tipping point as the other model providers and the major frameworks embrace it, the moment the ecosystem realizes a tool built once can serve every agent. Server counts go vertical, and "MCP server" enters the vocabulary of teams that had never shipped an integration before.

April 2025: Google introduces A2A with dozens of partners, explicitly positioned as MCP's complement rather than competitor, tools versus peers, and the layered-stack framing enters the discourse.

Mid to late 2025: the institutional phase. Google donates A2A to the Linux Foundation within months of launch. IBM's rival agent-communication effort merges into A2A rather than fighting it. Snowflake launches the Open Semantic Interchange with the BI and data tooling world aboard, naming semantics as the missing layer for trustworthy data agents. The payments flags plant in a rush: mandates, checkout protocols, machine-native micropayments. And enterprises stop piloting and start deploying, which converts every standards question from theoretical to budgetary.

Early 2026: the shipping phase. The OSI specification goes live under Apache 2.0 in January with a public repository and working group. Google ships the WebMCP preview in Chrome Canary in February, developed with Microsoft on a W3C track, extending the stack to the browser. Enterprise supporters of the layered stack pass the hundred mark.

April to June 2026: the maturity markers. A2A reaches version 1.0 with signed Agent Cards, 150-plus production organizations, and SDKs across five languages. Data platforms operationalize the stack, managed MCP servers, MCP governance acquisitions, semantic layer roadmaps, and the Apache Polaris community votes to bring an OSI-aligned semantic specification into the open catalog world. The stack stops being a diagram and becomes a deployment.

Eighteen months from one vendor's protocol to a multi-foundation, multi-layer standards stack with version numbers and adoption counts. For comparison, the web took most of a decade to travel the equivalent distance. Whatever else is true of the agent era, its standardization clock runs fast, which raises the stakes of getting the layering right the first time.

MCP: The Tool Layer That Won

Start at the foundation, because one layer of this stack is no longer contested. The Model Context Protocol, introduced by Anthropic in late 2024, answers the first question every agent asks: how do I reach tools and data? And in the eighteen months since, it has become the closest thing the agent world has to a settled standard.

The design is deliberately boring, which is a compliment. An MCP server wraps some capability, a database, a file system, a SaaS API, a search index, and exposes it through a standard interface of tools the agent can invoke, resources it can read, and prompts it can use. An MCP client, living inside the agent or its host application, discovers what a server offers and calls it. JSON-RPC carries the messages, with a streamable HTTP transport for remote servers and OAuth-based flows handling authorization. One protocol, and any compliant agent can use any compliant server. The USB-C analogy the ecosystem adopted early remains the right one: a universal port for AI capabilities, dull by design, transformative in aggregate.

The 2026 status is broad, deep, and institutional. The specification has continued to revise on a steady cadence, with the late-2025 revision now the widely supported baseline across enterprise implementations. Every major model provider and agent framework speaks it. The server ecosystem runs to the thousands, spanning every database, developer tool, and SaaS platform that matters, with registries and directories maturing from lists into infrastructure. And the clearest adoption signal is who now operates managed MCP surfaces: the data platforms. Snowflake runs managed MCP servers exposing governed query and search tools, and acquired a company specifically for enterprise MCP governance, agent identities, permissions, and audit trails across tools. Dremio exposes its lakehouse through MCP so agents query governed Iceberg data with the same access controls humans get. When the most conservative buyers in software, enterprise data platforms, operationalize a protocol and build governance products around it, the standardization argument is over.

The honest critiques have shifted accordingly, from "will it win" to the problems of winning. Context bloat, where agents drown in tool definitions from too many connected servers, is driving work on better discovery and selective loading. Security researchers keep demonstrating that a tool interface is an injection surface, more on that below. And governance remains the structural question: MCP is an open specification with open process, but stewardship still centers on Anthropic, and the ecosystem periodically debates whether the tool layer's constitution should live at a neutral foundation the way the agent-to-agent layer's now does. Watch that conversation, because the precedents from my world, formats thriving after donation to neutral homes, all point one direction.

A2A: Agents Talking to Agents, Now at 1.0

One layer up sits the question MCP deliberately does not answer: how do agents find and work with each other? That is the Agent-to-Agent protocol, A2A, and its past year is the fastest maturation story in this article.

The design mirrors how the web solved service discovery. Every agent publishes an Agent Card, a machine-readable description of its capabilities, supported task types, input formats, and authentication requirements. A client agent discovers a remote agent through its card, then delegates tasks over a JSON-and-HTTP protocol with support for long-running work, streaming results, and multi-turn exchanges. The framing the community settled on is exactly right: MCP connects agents to tools, A2A connects agents to peers. A tool is invoked and returns. A peer is delegated to, and negotiates.

The institutional trajectory is the story. Google introduced A2A in April 2025 with dozens of launch partners, then donated it to the Linux Foundation within months, placing it under neutral governance with founding participation from Amazon, Cisco, Google, Microsoft, Salesforce, SAP, and ServiceNow. IBM's competing Agent Communication Protocol merged into A2A in August 2025, consolidating the layer instead of fragmenting it. And in April 2026, A2A reached version 1.0, a stable production standard, shipping alongside signed Agent Cards for verifiable identity. The adoption numbers at the one-year mark: over 150 organizations running it in production, SDKs across five languages, native support in essentially every major agent framework, LangGraph, CrewAI, LlamaIndex, Semantic Kernel, AutoGen, Google's ADK, Microsoft's, and general availability inside the big cloud agent platforms.

My assessment: A2A executed the open-standard playbook almost flawlessly, propose broadly, donate quickly, absorb rivals, version deliberately, and it is now the presumptive answer at its layer for enterprise multi-agent systems. The unsettled edges are real but peripheral: a decentralized alternative vision built on W3C decentralized identifiers continues to develop for those who find A2A's model too web-conventional, and the deeper question of what agents should say to each other, the semantics of delegation, remains above the protocol, which standardizes the envelope rather than the meaning. Envelope first is the right order. It is also not the finish line, which brings us to semantics shortly.

WebMCP and the Agentic Web: Teaching Sites to Speak Agent

The third layer addresses the messiest surface agents touch: the web itself. Agents have been using websites the hard way, screenshotting pages and simulating clicks, an approach that is brittle, slow, and adversarial to sites that never consented. The 2026 development is the emergence of a consensual, structured alternative.

The headline is WebMCP. In February 2026, Google shipped an early preview in Chrome Canary of a protocol for structured agent interaction with websites, developed jointly with Microsoft through the W3C. The design has two halves: a declarative API through which ordinary HTML forms and page elements become agent-usable capabilities, and an imperative JavaScript API through which sites expose dynamic functionality as callable tools. The name is the strategy: it extends the MCP mental model to the browser, making a website, in effect, an MCP server that any visiting agent can discover and use, with the site rather than the agent defining what is offered. For site owners, that flips the agent relationship from scraping-by-force to capability-by-consent. For the stack, it fills the web-shaped hole between MCP's APIs and the human-shaped web.

Around WebMCP sits a supporting cast of smaller conventions maturing in parallel. The llms.txt convention gives sites a way to publish agent-oriented content maps, agents.md is emerging as guidance for AI coding assistants within repositories, a convention now reaching even the Apache data projects I cover, and Web Bot Auth work in the identity space aims to let legitimate agents authenticate themselves to sites rather than masquerading as browsers. None of these is finished. Together they sketch the agentic web's social contract: sites declare what agents may do, agents identify themselves honestly, and the DOM-scraping arms race gets replaced by an interface both sides maintain on purpose.

Status honesty: this is the youngest layer in the stack. WebMCP is a preview in one browser channel with a standards-track ambition, not a deployed norm, and the economic questions, what happens to sites whose business model assumed human eyeballs, are unresolved and larger than the technology. But the direction has the right participants, the browser vendors and the W3C, and the right shape, capability declaration over interface simulation. I expect this layer to look as inevitable in 2028 as MCP looks today.

OSI and the Semantic Layer: The Standard Closest to My Heart

Now the layer where the agent world and my world collide head-on: semantics. And here I get to connect this article to the rest of my series, because the Open Semantic Interchange is the standard that runs through the lakehouse.

The problem it attacks is the one every data leader discovered the moment they pointed an agent at their warehouse. Agents can find tables, columns, and joins. What they cannot find is meaning: what counts as an active customer, how revenue is recognized, which of the four margin calculations is the official one. Those definitions exist, but they live scattered and duplicated across BI tools, semantic layers, dbt projects, and tribal memory, each in a proprietary dialect. An agent without them does not merely lack context, it confidently computes the wrong number, and confident wrongness is the failure mode that kills enterprise AI projects. Text-to-SQL was never the hard part. Text-to-the-right-SQL is a semantics problem.

OSI is the ecosystem's answer: a vendor-neutral specification for semantic definitions, metrics, dimensions, entities, relationships, and their business meaning, so that semantics defined once can travel across BI tools, semantic layers, catalogs, and agents. Snowflake launched the initiative in September 2025 with founding partners spanning the BI and data tooling world, Salesforce and Tableau, dbt Labs, and a fast-growing roster. The execution since has been genuinely encouraging: the specification went live under the Apache 2.0 license in January 2026 with a public repository and working group, the partner list has compounded monthly through the catalog, governance, and analytics vendors, and this June's Snowflake Summit put OSI at the center of its semantic governance story rather than the footnotes.

The development that convinces me most, though, is the one I reported in my Polaris article: the Apache Polaris community voted this season to accept an OSI-aligned semantic model API specification into its orbit, alongside its active semantic layer design work. Read the layering there. The semantic standard defines the language of business meaning, and the open catalog becomes a governed home where that meaning lives next to the tables it describes, discoverable and permissioned like any other asset, served to agents through the same MCP surfaces the platforms already run. That is the full stack clicking together: an agent reaches the platform through MCP, coordinates with peers through A2A, and gets its definitions from an OSI-shaped semantic layer governed in an open catalog. Every layer neutral, every layer swappable. It is the lakehouse philosophy applied to meaning, and it is why I keep telling data teams that the agent era makes their semantic debt the most expensive debt they hold.

Sober status notes: the specification is young, translation from a dozen incumbent semantic dialects is genuinely hard, and initiative-led standards must still prove multi-vendor governance durability the way A2A did. But of everything in this article, OSI is the standard whose success or failure most directly determines whether enterprise agents produce trustworthy answers. I am rooting for it with receipts attached.

Payments and Commerce: The Most Crowded Layer

If the semantic layer is the most important to my readers, the payments layer is the most crowded, and it needs a map more than an evangelist.

The question is real: agents that book, buy, and subscribe need a way to pay that does not involve handing them a credit card number and hoping. The answers, as of mid-2026, come in overlapping flavors. AP2, the Agent Payments Protocol that emerged alongside the A2A ecosystem with dozens of payment-industry partners, centers on cryptographic mandates, verifiable, user-signed authorizations that scope what an agent may spend, on what, within what bounds, giving every party a proof trail. The Agentic Commerce Protocol from OpenAI and Stripe standardizes in-conversation checkout between agents and merchants. x402 from Coinbase revives a dormant HTTP status code for machine-native micropayments in stablecoins, aimed at agents paying per-call for APIs and content. Stripe's machine-payments work and Visa's trusted-agent efforts add rails-side identity and settlement pieces. And commerce-flow proposals stitch the shopping journey itself.

The layering lens sorts the crowd better than a bracket does: identity and authorization mandates, checkout semantics, and settlement rails are different problems, and several of these can compose rather than compete. But I will be more candid here than the press releases are: this layer has too many flags planted, the major backers map suspiciously well onto incumbent payment interests, and history says consolidation, mergers, absorptions, or quiet abandonments, is coming before maturity arrives. Builders should treat agent payments as a capability to pilot behind an abstraction, not a foundation to marry. The A2A-and-ACP merger showed this ecosystem can consolidate well. The payments layer is where that muscle gets tested next.

The Human Layer and the Cross-Cutting Problems

Three shorter fronts complete the map, and each carries the same lesson: the stack is real, and its seams are where the risk lives.

The human interface layer standardizes how agent work reaches people: protocols like AG-UI for streaming agent output into applications and companion efforts for richer agent-driven interfaces. This matters more than it sounds, because the alternative is every product reinventing the chat-plus-artifacts pattern incompatibly, and because human oversight, approvals, interruptions, visibility into agent reasoning, is a protocol problem before it is a UX problem. It is early, framework-led rather than foundation-governed, and worth watching without betting on.

Identity and trust cut across every layer, and 2026's progress is real but partial: signed Agent Cards give A2A verifiable identities, Web Bot Auth points the same direction for the web, payment mandates carry cryptographic accountability, and enterprise MCP governance products supply the audit trails compliance demands. What does not yet exist is the unified answer, one way for an agent to carry who it is, who it acts for, and what it may do across all the layers at once. Expect that to be the defining standards fight of 2027.

And security is the cross-cutting problem that keeps me honest about all of it. OWASP now publishes a top-ten for agentic applications, and the sobering theme is that every protocol in this article widens the attack surface it standardizes: tool descriptions and web content become prompt-injection vectors, agent-to-agent delegation becomes confused-deputy risk at machine speed, and payment autonomy raises the stakes of every upstream compromise. The standards are responding, scoped authorizations, signed identities, human-in-the-loop checkpoints, but the honest state is that agentic security practice trails agentic capability by a distance every practitioner should respect. The protocols made agents composable. Composability is exactly what attackers compose.

A Worked Example: One Request Through the Whole Stack

Layer diagrams convince nobody, so let me run a single realistic request through the full stack and name every standard as it fires. The request, typed by a sales director into her company's assistant: "How did the spring campaign perform in Germany, and order a refreshed report for the regional review."

The assistant, an orchestrating agent, starts by deciding it needs two specialists: the analytics agent the data team operates, and the document agent the operations team runs. It discovers both through their Agent Cards, verifying the signatures introduced with A2A 1.0 so it knows it is delegating to the genuine articles, and hands the analysis task to the analytics agent over A2A, a delegation with streaming progress rather than a fire-and-forget call, since the analysis may take a minute.

The analytics agent now needs data, which means the tool layer. Through MCP, it connects to the lakehouse platform's managed server, authenticating as its own principal, not as a borrowed human account. Behind that MCP surface, the catalog, Polaris in my telling, evaluates the agent's role grants, approves access to the marketing and orders tables, and vends short-lived, scoped storage credentials, so the agent's reach is bounded by policy and every touch lands in an audit log against its verified identity. The governance machinery I described in my Polaris article is doing its job here without a human in sight, which was always the point.

Before writing a query, the agent asks the question that separates 2026 from 2024: what does "campaign performance" mean here? It pulls the semantic definitions, the OSI-shaped metrics the analytics team defined once, revenue recognized net of returns, attribution windows, the official margin calculation, from the semantic layer the catalog governs. The SQL it generates now encodes the company's definitions rather than the model's guesses, and the numbers it returns will match the CFO's dashboard, because they came from the same definitions. The result set itself, a few hundred thousand rows aggregated down, travels back not as bloated JSON but as columnar Arrow over the platform's high-speed interface, the transport argument from my Arrow article playing its small quiet part.

The analysis returns to the orchestrator through the A2A task stream, and the human layer takes over: results render into the director's application through the emerging agent-UI conventions, charts, a summary, and, importantly, the lineage of which definitions were used, so a skeptical reviewer can trace every number. She approves the second half of the request, a human checkpoint deliberately placed before anything irreversible.

The document agent now needs the updated brand template from the design vendor's portal. Rather than screen-scraping, it uses the portal's WebMCP-declared capabilities, the site itself exposing "download current template" as a structured tool, and identifies itself honestly through the agent-authentication conventions rather than masquerading as a browser. And when the vendor's premium asset requires payment, the agent presents a cryptographic mandate, an authorization the company pre-signed scoping what this agent may spend and on what, so the transaction clears with a proof trail instead of a shared credit card number.

Total standards fired in one mundane business request: A2A for discovery and delegation, signed cards for agent identity, MCP for tool access, catalog RBAC and credential vending for governed data reach, OSI-shaped semantics for meaning, Arrow for the payload, agent-UI conventions for the human, WebMCP and agent auth for the web, and a payment mandate for the purchase. Two years ago, every one of those junctions was custom glue or an unmanaged risk. That is what a protocol stack is for, and it is also the honest security lesson: nine junctions is nine boundaries to defend, which is why the governance and audit story threaded through every step is not decoration. It is the product.

What the Stack Means for Data People

Let me close the tour by planting the flag where I always plant it, because the agent standards story and the open data story are converging into one story, and my readers sit at the junction.

Ask what agents actually do all day in an enterprise, and the answer is overwhelmingly: work with data. Query it, summarize it, reconcile it, act on it. Which means the protocol stack in this article, in practice, terminates at the data platform, and every layer maps to a piece of the open lakehouse I spend the rest of my week writing about. MCP is how agents reach the catalog and the engine, and the serious platforms now ship governed MCP surfaces natively. The semantic layer, OSI-shaped, catalog-governed, is how agents stop guessing at meaning. Catalog governance, Polaris in my telling, is how agent principals get the same RBAC, credential vending, and audit trails as humans, which is the only agent-governance story that survives a compliance review. Even the transport question loops back: when agents start moving real analytical payloads rather than chat snippets, JSON gives way to Arrow over Flight, the argument I made in my Arrow piece.

So my advice to data teams is unglamorous and urgent: the best preparation for the agent era is aggressively finishing the open lakehouse era. Get the tables into open formats, the catalog into a governed open standard, the semantics defined once and exportably, the access model principal-based. Every one of those was already best practice. Agents just turned best practice into prerequisite, because agents amplify whatever they land on: good governance becomes a multiplier, and ambiguity becomes confident wrongness at scale.

What Is Deliberately Not Standardized, and Why That Matters

One more analytical lens before the guidance, because knowing what the stack refuses to standardize tells you as much as knowing what it covers, and it predicts where the next flags will plant.

Nothing in the stack standardizes how an agent thinks. Reasoning strategies, planning loops, memory architectures, model choice, and orchestration patterns all remain deliberately outside every specification, and that restraint is a feature. MCP does not care whether the caller is a single model or a swarm. A2A standardizes the envelope of delegation while staying silent on the intelligence inside either party. The protocols learned the web's lesson: HTTP never standardized how a server generates a page, and that silence is exactly what let the application layer innovate for thirty years. The competitive frontier for agent products stays wide open above the plumbing, which is why every vendor could afford to adopt the plumbing.

Nothing yet standardizes evaluation and trust in results. There is no protocol for "how confident is this answer," no standard for attaching provenance and evaluation scores to agent outputs, no shared way to express "this analysis used these definitions against these snapshots of these tables." The pieces exist in fragments, the semantic layer supplies definitional provenance, table formats supply data versioning, audit logs supply the trail, but the assembled artifact, a verifiable answer, has no spec. I suspect this becomes a serious standards conversation by 2027, and the data world will supply much of its raw material, because lineage and reproducibility are problems we have been solving for a decade.

And nothing standardizes the economics. When agent traffic replaces human traffic on a website, when agents comparison-shop at machine speed, when a vendor's API becomes a line item in a thousand agents' budgets, the technical protocols are ready and the business models are not. WebMCP can express what a site permits, and llms.txt can express what it offers, but neither expresses what the site gets in return, and the tension between agent-friendly and revenue-sustaining is unresolved across the industry. Payment protocols are a partial answer for explicit transactions. The implicit economy of attention and advertising has no agent-era equivalent yet, and that vacuum, more than any technical gap, is what makes the agentic web layer the hardest to predict.

The pattern across all three gaps: the stack standardized coordination first and judgment not at all, which is the historically correct order, and it means the interesting fights of the next two years happen above the protocols this article mapped. Plan for the plumbing to be settled and the judgment layer to be contested, and allocate your attention accordingly.

A Builder's Decision Framework

For the builders who need to ship this quarter, the map compressed into guidance.

Adopt MCP without hesitation. It is the settled layer, the skills transfer everywhere, and wrapping your internal capabilities as MCP servers is the highest-return integration work available in 2026. Treat server design as API design, minimal, well-described tools beat sprawling ones, and treat every tool as a security boundary from day one.

Adopt A2A when you genuinely have multiple agents that must interoperate across teams, frameworks, or organizations, which is a later stage than most projects admit. Single-agent products dressed as multi-agent systems buy complexity without buying capability. When you do adopt it, 1.0 and signed cards make this the year it is safe to.

Track WebMCP and the agentic web conventions if you own web properties, because publishing agent-facing capability, even just llms.txt today, positions you for the consensual-agent-traffic world coming, and it costs almost nothing.

Engage OSI now if you are a data team, by inventorying where your semantic definitions live and getting them into exportable shape, regardless of which tools you run. The standard is young, but semantic debt compounds daily and the paydown is valuable under every future.

Abstract payments, pilot narrowly, and wait for consolidation. And across every layer, hold the two tests this whole series keeps applying to standards: is the governance neutral enough that competitors co-invest, and can you exit without rewriting your product? Layers that pass both are infrastructure. Layers that fail either are features wearing a standards costume.

Questions I Hear Most Often

The recurring questions, answered plainly.

Is MCP versus A2A a real choice I have to make? No, and the framing misleads. They answer different questions at different layers, tools versus peers, and mature agent systems use both, the way a web service uses both a database driver and HTTP without choosing between them. The real choices are within layers, and even those are thinner than the discourse suggests: MCP has no serious rival at its layer, and A2A absorbed its main one.

Aren't there just too many protocols? At the payments and interface layers, yes, and I said so above. But distinguish sprawl from layering: six protocols answering six different questions is architecture, six answering the same question is a war. The core stack, MCP, A2A, WebMCP, OSI, is the former. The test I apply to every new announcement: does this answer a question no existing layer answers? Most new entrants fail it, which is itself useful information.

Which of these will still exist in 2030? Applying the neutrality-and-layering lens: MCP's function is permanent whatever its governance evolution, A2A has the institutional shape of a survivor, the agentic-web layer will exist in some form because browser vendors and the W3C are committed, and semantics will be standardized because the alternative is agents that cannot be trusted with numbers, whether under the OSI banner or its successor. Payments will consolidate to fewer names than today. The safest prediction in the set: the layered architecture itself outlives every individual flag.

How do I secure any of this? Start from the OWASP agentic top ten, then apply the boring disciplines that always work: least-privilege credentials per agent via scoped, short-lived tokens, which is exactly what catalog credential vending provides on the data side, human approval gates on irreversible actions, treating all retrieved content, tool outputs included, as untrusted input, and audit logging of every agent action against a real identity. The protocols increasingly carry the primitives, signed cards, mandates, OAuth scopes. The assembly is still your job.

Do I need a semantic layer before I deploy data agents? You need semantic definitions before you can trust data agents, and a governed layer is the honest way to have them. Teams that skip this ship agents that demo beautifully and then produce three different revenue numbers in production, which burns organizational trust that takes years to rebuild. Start smaller than a platform project: pick the twenty metrics that matter, define them once, make them machine-readable, and grow from there. That is also, not coincidentally, the on-ramp OSI is paving.

Where do I follow all this without it becoming a job? The specifications themselves are public, MCP's site, A2A at the Linux Foundation, the W3C explainers, OSI's repository and updates page, and reading a spec beats reading ten takes about it. For the weekly digest, this is exactly what my AI newsletter exists for, and the cross-project view, watching the agent standards and the data standards converge, is precisely the beat I cover across both newsletters.

Closing Thoughts

The state of agentic AI standards in mid-2026 is a stack solidifying from the bottom up. The tool layer is won. The agent-to-agent layer just reached 1.0 under neutral governance with real production adoption. The web layer has its first browser preview and its social contract sketched. The semantic layer, the one that decides whether agents can be trusted with meaning, has a live specification and, tellingly, a home forming inside the open data catalog world. Above and around them, payments jostle toward consolidation, interfaces and identity mature unevenly, and security races to keep up, which is to say: it looks exactly like every previous protocol era looked at this stage, the web included.

The through-line of this whole series has been that open, neutral, layered standards are how infrastructure wins, told through Arrow and Parquet and Iceberg and Polaris. The agent world is now running that same movie at four times speed, and the ending is predictable in shape if not in names: the glue code dies, the layers stabilize, the value moves to what you build on top, and the organizations that prepared their foundations, data foundations most of all, compound fastest when it does.

If you want to build that foundation properly, from the open table formats through the catalogs and semantics that agents depend on, that is what my books are for. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, and my other titles cover lakehouse architecture, data engineering, and the agentic analytics wave this article maps.

Browse the full collection of my books on data and AI at books.alexmerced.com.

Meet Apache Ossie: The Open Semantic Interchange Finds Its Home at the ASF

Alex Merced — Tue, 07 Jul 2026 18:46:30 +0000

In June 2026, a project quietly entered the Apache Incubator that I believe will matter as much to the next decade of data as Iceberg mattered to the last one. It is called Apache Ossie, it was formerly known as the Open Semantic Interchange, and its job is to standardize something the data industry has never standardized: what our data actually means.

I have personal stakes to declare, as always. The proposal's champion, Jean-Baptiste Onofré, is a colleague of mine at Dremio, and Dremio is one of the three companies named as core developers of the project alongside Snowflake and dbt Labs. I have been writing about the Open Semantic Interchange since its launch, I flagged its trajectory in my recent articles on Apache Polaris and on agentic AI standards, and watching it arrive at the Apache Software Foundation feels like watching a prediction come true faster than I predicted it. So no, I am not neutral. What I can offer instead is what I always offer: the receipts, the honest caveats, and an explanation built so that anyone can follow it.

That last part is the real mission of this article. Ossie lives in a corner of the data world, semantic layers, metrics definitions, ontologies, that even seasoned engineers find foggy, and most coverage of it assumes you already speak the jargon. I want to do the opposite. By the end of this piece, you should understand what problem Ossie exists to solve, what the project actually contains, why it needed a vendor-neutral home, what incubation at the ASF means, and why the timing, in the middle of the AI agent wave, is not a coincidence. No prior semantic-layer knowledge required.

The Problem, Told as a Story Everyone Has Lived

Forget technology for a moment and consider a Monday morning at a perfectly ordinary company.

The head of marketing walks into the leadership meeting with a slide: monthly active users are up 12 percent. The head of product follows with her dashboard: monthly active users are flat. The CFO, working from the finance team's warehouse queries, has them down 3 percent. Three smart teams, three careful analyses, three different numbers for what sounds like one simple thing. The next forty minutes of the meeting are spent not making decisions but arguing about whose number is real.

Here is the uncomfortable truth: nobody in that room is wrong. Marketing counts anyone who opened the app. Product counts anyone who performed a meaningful action, and excludes internal test accounts. Finance counts paying seats, and their pipeline lags a week. Each definition is defensible. The problem is that the company never wrote down which one is the definition, in a form all their tools could share, so every tool and team quietly invented its own. The industry has a name for this disease: semantic drift. The same business concept, defined inconsistently across an organization's systems, drifting further apart with every new tool, every new hire, every new dashboard.

Every data professional reading this has lived some version of that meeting. What is new in 2026 is who else attends it. Companies are now pointing AI agents at their data and asking questions in plain English: what is our churn rate, how did the campaign perform, which customers are at risk. An agent asked to calculate churn looks at the warehouse and finds three tables that might be relevant, several plausible formulas, and no way to know which one the business considers true. A human analyst in that situation walks down the hall and asks someone. The agent just picks one, confidently, and the result is a plausible-looking number computed with logic no one endorsed. Multiply that by every question every agent answers every day, and semantic drift graduates from a chronic annoyance to an acute liability.

The old costs of drift, the meeting arguments, the weeks engineers spend manually reconciling definitions between systems, the migration projects that stall because business logic is trapped inside one vendor's tool, were bad enough that the industry tolerated them for decades anyway. The AI cost is the one that finally forced the issue. That is the context in which the Open Semantic Interchange was born, and it is the context in which it just became Apache Ossie.

What a Semantic Model Actually Is, in Plain Language

Before I can explain Ossie, I need to demystify the thing it standardizes, because "semantic model" is one of those phrases that sounds far more exotic than it is.

A semantic model is simply the company's data dictionary, written precisely enough that software can use it. It captures a few kinds of things. Metrics: the named numbers the business runs on, revenue, churn rate, monthly active users, each with its exact formula, the filters that apply, and the grain it is measured at. Dimensions: the ways you slice those numbers, by region, by month, by product line, by customer segment. Entities and relationships: the nouns of the business, customers, orders, subscriptions, and how they connect, including the unglamorous but vital facts like "the sales system's Customer_Code and the billing system's Account_UID refer to the same customer."

Concretely, a semantic model entry for our troubled metric might say, in a structured, machine-readable form: monthly active users is defined as the count of distinct users who performed at least one qualifying action in the trailing thirty days, excluding internal accounts, where a qualifying action is any event in this named list, measured against the events table in the analytics schema. Written down like that, the definition stops being folklore. A BI tool can read it and build the dashboard from it. A data engineer can test against it. An AI agent can quote it and compute with it. And when the business decides to change the definition, it changes in one place and every consumer follows.

None of this is a new idea. BI vendors have shipped semantic layers for decades, and tools like dbt made metric definitions a first-class artifact for the modern stack. The problem was never that semantic models did not exist. The problem is that every tool spoke its own dialect. The definitions in your BI tool's semantic layer, your dbt project, your catalog's glossary, and your CRM's configuration are written in incompatible formats, cannot read each other, and therefore drift apart, which lands us right back in the Monday meeting. Worse, because your business logic is trapped in each tool's proprietary format, leaving any tool means rewriting your company's brain from scratch, which is a form of lock-in more binding than any data format ever was.

If this shape of problem sounds familiar to readers of my other work, it should. Data files had this problem before Parquet. Tables had it before Iceberg. Catalogs had it before the Iceberg REST protocol and Polaris. In every case the cure was the same: stop standardizing the tools, standardize the interchange. Ossie is that cure applied to meaning itself.

What Apache Ossie Actually Is

Now the project itself, concretely, because Ossie is refreshingly tangible for something with "semantic" in its description. Open the repository at github.com/apache/ossie and here is what you find.

At the center sits the core specification: a vendor-neutral format, expressed in YAML and JSON, for writing down semantic models, the metrics, dimensions, entities, and relationships we just discussed, in a way any tool can read and write without loss of meaning. The spec ships as human-readable documentation alongside machine-readable schemas, so a tool builder can validate that a model conforms. This is the heart of the project: not software you run, but an agreement about how meaning is written down, the same way Parquet is fundamentally an agreement about how bytes are laid out.

Around the spec sit the pieces that make an agreement practical. A converters directory holds reference translators between Ossie and the existing dialects, with converters for dbt, GoodData, Salesforce, and Apache Polaris already in the tree, because a standard nobody can migrate to is a whitepaper, and the entire adoption path runs through translating what companies already have. A validation directory holds tooling to check models against the schema. An ontology directory holds the standardized vocabulary work, the shared upper-level concepts that let one company's model be intelligible to another's tools. An examples directory includes a complete semantic model for the TPC-DS benchmark schema, which I would point to as the fastest way for a practitioner to get the feel of the format: a familiar retail-flavored schema, fully described in Ossie terms. And a Python package provides the core types for programmatic work.

The design philosophy threaded through all of it deserves a sentence of appreciation. Ossie's architecture is decentralized: systems read semantic metadata from the source that owns it rather than depending on point-to-point field mappings between every pair of tools. The proposal states the contrast plainly, and it echoes everything I have written about the N times M problem in the agent standards world. Manual mapping between systems grows quadratically and breaks whenever a schema changes. Self-describing data, where the meaning travels attached to the source and every consumer reads the same description, converts that quadratic mess into a linear one. The dream, stated simply: define once, understood everywhere.

One more plain-language clarification, because it trips people up. Ossie does not store your data, query your data, or replace your warehouse, lakehouse, BI tool, or semantic layer product. It is the interchange format those systems use to agree with each other, the way a PDF does not replace your word processor. Your tools keep their engines and their interfaces. What changes is that the definitions inside them stop being trapped.

What an Ossie Model Looks Like, Without the Jargon

Since the specification is the heart of the project, let me walk through what an Ossie model conceptually contains, in plain terms, so the format stops being abstract. I will describe rather than reproduce, because the spec is young and evolving, but the shape is stable enough to internalize.

An Ossie model is a structured document, YAML or JSON, that a human can read and a machine can validate. Think of it as having four kinds of entries, layered from concrete to abstract.

At the bottom are the pointers to physical reality: which datasets the model describes, the tables or views in your warehouse or lakehouse, and how the logical names in the model map to physical columns. This grounding layer is what lets a definition be executable rather than merely aspirational, and it is why the catalog integration matters so much: the catalog already knows where the tables live and who may touch them, so a model registered there inherits that grounding.

Above that sit the entities and relationships: the declaration that a customer is a thing, identified by this key, appearing in these datasets under these different column names, related to orders one-to-many, related to subscriptions through this join. This is the layer that kills the "third system" problem the proposal describes, where the knowledge that Customer_Code and Account_UID mean the same person lived only in an engineer's head or a stale wiki page. In an Ossie model, that equivalence is a declared, versioned, machine-readable fact.

Above that sit the dimensions and measures: the attributes you slice by, time at various grains, geography, segment, and the raw quantifiable facts, order amounts, event counts, that metrics are built from. And at the top sit the metrics themselves: named, documented calculations with their formulas, filters, exclusions, and grains, the monthly active users and churn rates that the Monday meeting argues about, now written with the precision of code and the readability of documentation. Each metric can carry its human description alongside its machine logic, which matters enormously for the AI use case: an agent gets both the formula to compute with and the prose to explain itself with.

Two properties of the format deserve emphasis because they carry the philosophy. First, models are plain text files, which means they live in version control, changes arrive as reviewable diffs, history is preserved, and the workflows every engineering team already trusts, pull requests, approvals, CI validation against the schema, apply to business definitions with no new machinery. The industry spent a decade learning that analytics logic deserves software engineering discipline. Ossie extends that lesson to meaning itself. Second, models are composable and referenceable rather than monolithic: the ambition, being worked through in the composability working group, is that a finance domain model and a product domain model can each own their pieces and reference each other, so a large organization's semantics can be federated the way its data already is, rather than requiring one impossible galactic model maintained by one overwhelmed team.

If you want to see all of this concretely, the examples directory in the repository contains a complete model for the TPC-DS schema, the industry's standard retail benchmark. Reading it is the fastest education available: familiar tables, store sales, customers, items, described entity by entity and metric by metric in the actual format. An afternoon with that file will teach you more than any article, this one included.

The Road Here: From Initiative to Incubator in Nine Months

The institutional story is short and fast, and the speed is part of the story.

In September 2025, Snowflake launched the Open Semantic Interchange initiative together with Salesforce, dbt Labs, and a founding coalition of BI and data tooling companies, naming semantic fragmentation as the shared enemy and a vendor-neutral standard as the goal. I covered the launch with hopeful skepticism at the time, because vendor-led standards initiatives have a mixed history: some become Iceberg, and some become press releases.

This one shipped. The repository opened in November 2025. In January 2026, the v0.1 specification went live under the Apache 2.0 license, giving the industry its first neutral, YAML-based standard for metric and dimension definitions. The partner roster compounded from seventeen launch organizations to more than fifty, pulling in Databricks, ThoughtSpot, Collibra, AtScale, Atlan, and a long tail of the analytics ecosystem, and, notably, pulling in competitors of the founders, which is always the tell that a standard is real. Working groups formed with dedicated leads across five fronts: the metric language itself, composability, catalog integration, ontology, and a synchronization API. By the time of the incubation proposal, the project counted more than a hundred commits, dozens of merged pull requests from contributors across at least a dozen companies, and over seventy active design discussions in the open.

Then, in June 2026, Jean-Baptiste Onofré brought the proposal to the Apache Incubator: Ossie, a data semantic specification and framework. The discussion thread ran on the public incubator list, the formal vote followed, and it passed with binding support from across the Incubator PMC. The resources spun up within weeks, the apache/ossie repository, the dev mailing list, the podling page, and the announcement JB shared publicly marked the moment: the Open Semantic Interchange had become Apache Ossie, incubating.

Nine months from launch to the Incubator is fast by any standard, and it reflects a deliberate strategy visible in the proposal itself: the initiative adopted Apache-style governance from the beginning, the Apache 2.0 license, a public specification process with discussion windows and votes, merit-based contribution, precisely so that the eventual donation would be a formality rather than a renovation. The people involved knew the playbook because they had run it before, which brings us to who they are.

Why "Ossie," and Why the ASF

Two questions everyone asks first, answered honestly.

The name is the easy one. "Open Semantic Interchange" and its acronym OSI were always headed for trouble: the acronym collides with the Open Source Initiative, one of the most established names in all of open source, not to mention decades of networking engineers for whom OSI means a seven-layer model. The proposal addresses this directly, and the community chose Ossie, keeping a phonetic echo of the original while clearing the trademark and confusion risks. Expect the phrase "Apache Ossie, formerly known as the Open Semantic Interchange" to do heavy lifting in the ecosystem's vocabulary for a year or so, this article included.

The deeper question is why a working initiative with fifty partners needed the Apache Software Foundation at all, and the answer is the same argument I have now made across this entire series of articles. A standard's entire value is neutrality. Companies will only pour their business logic into an interchange format if they are certain the format cannot be tilted, stalled, or captured by any single vendor, including and especially the vendor that started it. Snowflake launching OSI was necessary and commendable, and it was also, inevitably, a reason for some rivals to hesitate. Donating the project to the ASF converts "trust Snowflake and friends" into "trust a twenty-five-year-old foundation whose entire constitution is preventing capture." The proposal says this in its own words: the interest is governance alignment, not brand decoration, because a vendor-neutral interoperability standard requires consensus-based, multi-stakeholder governance, which is what the ASF provides.

For readers newer to the Apache world, here is what incubation practically means. Ossie is now a podling: a project admitted to the Apache Incubator, operating under ASF rules, with experienced mentors assigned, in Ossie's case JB Onofré, Russell Spitzer, Holden Karau, and Zili Chen, names that between them span the Iceberg, Polaris, Spark, and broader Apache data ecosystems. Over the coming incubation period, the project must demonstrate the things the ASF certifies: that its intellectual property is clean, that its releases follow foundation policy, and above all that its community is diverse and self-governing enough to outlive any founder's interest. Graduation to a Top-Level Project, if and when it comes, is the ASF's audit stamp saying exactly that. Readers of my Polaris article have seen this movie: Polaris entered the Incubator in 2024 amid similar "is it just the vendors' project" skepticism and graduated eighteen months later with a contributor base that answered the question. Ossie's proposal explicitly names the Polaris playbook, community sprints, curated first issues, onboarding workshops, as its model for the same journey.

Who Is Behind It, and Why That Roster Matters

Read an incubation proposal closely and the roster tells you more than the prose. Ossie's is unusually revealing.

The core development today comes from three companies: Snowflake, Dremio, and dbt Labs, with Salesforce alongside in the initial governance. Look at that list through the lens of my previous articles and something remarkable is hiding in plain sight: Snowflake and Dremio are the two companies that co-created Apache Polaris, and here they are again, joined by the company whose semantic layer work in dbt largely defined the modern metrics conversation. The initial project management committee spans all four companies, the mentors are veterans of the exact Apache data projects Ossie must integrate with, and the champion, JB, is a long-time ASF member who helped shepherd Polaris through its own incubation. The initial committer list reaches beyond the founders too, including independent contributors and names like Jochen Christ, known for the data contract movement, which signals the project understands its neighbors.

The contributor flow reaches wider still: the proposal documents merged work from Snowflake, Salesforce, Databricks, dbt Labs, AtScale, Atlan, RelationalAI, ThoughtSpot, GoodData, Honeydew, and Hex. Pause on that list. It contains direct competitors at nearly every layer, warehouses that compete with lakehouses, BI tools that compete with each other, semantic layer products whose entire commercial moat was, until now, the proprietary format Ossie replaces. When companies co-invest in dissolving their own lock-in, it is because they have concluded the market demands it, and their customers, who have spent years asking why business definitions cannot travel between tools, supplied that demand.

I will state my house's position plainly, since I opened by declaring it. Dremio's involvement in Ossie is the same bet it made co-creating Arrow, championing Iceberg, and co-creating Polaris: that open standards at every layer of the stack grow the whole market and let vendors compete on their engines and experiences rather than on their customers' inability to leave. Semantics was the last major layer without such a standard. It is entirely consistent that the same crowd showed up to build one.

How Ossie Snaps Into the Stack You Already Know

For readers following my series, this is where the threads tie together, because Ossie is not arriving into a vacuum. It is arriving into a stack that has been preparing a socket for it.

Start with the catalog. In my Polaris article, I reported that the Polaris community had been running a high-traffic design discussion on semantic layer support and had formally voted to accept an OSI-aligned semantic model API specification. Now look at Ossie's side of the handshake: a Polaris converter already merged in the repository, catalog integration named as a dedicated working group, and deepening Polaris integration named explicitly in the proposal's future plans. The architecture taking shape is the one I sketched then: Ossie defines the language in which business meaning is written, and the open catalog gives that meaning a governed home next to the tables it describes, discoverable, permissioned, and versioned like any other asset. Definitions stop living in a wiki nobody trusts and start living where the data lives.

Then the modeling and BI layer. The dbt converters are merged, and dbt's MetricFlow can already consume Ossie models directly as an alternative to its native configuration, which makes the promise concrete: a metric defined once in the neutral format, executed by the tool a team already uses. Converters for GoodData and Salesforce sit alongside, with the working groups grinding through the genuinely hard translation questions, time semantics, dialect differences, composability, in public design discussions.

Then the engines. A Spark converter is in review, an expression-language proposal is under discussion in the Iceberg community, and the proposal's long-term vision names collaboration across Spark, Flink, Impala, Iceberg, and Polaris: a future where a semantic query, "give me monthly active users by region," can be interpreted consistently by different engines because the definition and eventually the query specification are standard. That is the roadmap's most ambitious line, a semantic query standard with reference engine implementations, and I will believe it when I see it while being delighted the community is aiming there.

And then, of course, the agents, because everything in 2026 ends there. In my agentic standards article I described the emerging stack: MCP as the tool layer, A2A between agents, and a semantic standard as the layer that decides whether agents can be trusted with meaning. Ossie is that layer's candidate, now with neutral governance to match the layers around it. The end-to-end picture writes itself: an agent reaches the lakehouse through a governed MCP surface, the catalog vends it scoped credentials and, alongside the tables, the Ossie-formatted definitions of the metrics it is about to compute. The agent's SQL encodes the company's definition of churn rather than the model's best guess, and its answer matches the CFO's dashboard because both flowed from the same source of truth. Deterministic meaning is the phrase the proposal uses, and it is exactly the missing ingredient every enterprise AI post-mortem has been naming for two years.

A Worked Example: One Metric's Journey

Let me make the whole thing concrete with a single metric traveling the pipeline, because abstractions about semantics are exactly what this project exists to end.

A subscription business decides, after one too many Monday meetings, to fix churn. The analytics lead convenes finance, product, and the data team, and they hammer out the definition: churn rate is the count of subscriptions canceled in the period, excluding involuntary payment failures that recover within seven days, divided by subscriptions active at the period start, measured monthly. Today, that hard-won agreement would be enshrined in a slide, and each tool would reimplement it slightly differently within a quarter.

Instead, the team writes it once as an Ossie model: the churn metric with its formula and exclusions, the subscription entity it depends on, the relationship between the billing system's account identifier and the product database's user identifier, the month dimension it is measured over. The model is validated against the schema, checked into version control like code, reviewed like code, and registered in the company's Polaris catalog, where it lives next to the tables it describes, governed by the same access controls.

Now watch it travel. The BI platform reads the model through a converter and builds its dashboards from it, no re-definition, no drift. The dbt project consumes the same model through MetricFlow, so the transformation layer and the dashboard layer are provably computing the same thing. When the data science team's notebook and the finance team's spreadsheet plugin pull churn, they pull the definition, not a rumor of it. When an executive asks the company's AI assistant how churn is trending, the agent retrieves the Ossie definition from the catalog through the same MCP surface it uses for the data, computes with the endorsed formula, and can cite the definition in its answer, including the payment-failure exclusion a guessing model would never have known about.

Six months later, the business decides recovered payment failures should count within fourteen days, not seven. The change is a pull request against one file. It is discussed, approved, versioned, and every consumer, dashboards, transformations, notebooks, agents, follows automatically, with the change history preserved for the auditor who will one day ask why the March number moved. That is the entire pitch of Apache Ossie compressed into one metric's life: define once, govern once, and let every tool and every agent read the same truth.

Honest Caveats, Because Every New Project Deserves Them

I am enthusiastic about Ossie, and this series has a rule: enthusiasm travels with the caveats attached.

The specification is young. Version 0.1 shipped in January, the metrics language is still being formalized, and the hardest semantic problems, time and windowing logic, dialect-specific expressions, composability of models across domains, are exactly the ones the working groups are still designing in the open. Companies evaluating Ossie today should read it as a direction to align with and contribute to, not a finished contract to bet a migration on this quarter. The proposal itself is candid that the roadmap belongs to the community now.

Translation is genuinely hard. The converter strategy is the right adoption path, and it collides with reality: existing semantic dialects encode subtly different assumptions, and lossless round-tripping between a decade of proprietary formats and a young neutral one will take years of grind. The measure of Ossie's success will be boring converter release notes, not launch announcements, and I mean that as guidance for what to watch.

The contributor base is concentrated. Three companies dominate core development today, the proposal names this as its top incubation risk, and the ASF's diversity requirement exists precisely to force the issue before graduation. The Polaris precedent says it can be done. The precedent is a plan, not a guarantee, and the new-contributor pipeline, the sprints and first issues and workshops the proposal promises, is where that plan succeeds or fails.

And standards can lose. Adjacent efforts, proprietary semantic layers with enormous installed bases, catalog vendors with their own business-semantics ambitions, could fragment the territory Ossie means to unify, and enterprise inertia is the strongest force in software. My grounds for optimism are the roster of competitors already inside the tent and the structural tailwind: the AI agent wave punishes semantic fragmentation more brutally every quarter, and no proprietary format can be the industry-wide answer by definition. But I have watched enough standards efforts to say it plainly: the next eighteen months of community growth will decide this, not the elegance of the specification.

What This Means for You, by Role

Let me close the analysis with the practical translations, because a project like this touches very different readers differently.

If you lead data or analytics teams, the strategic move is to start treating your semantic definitions as an exportable asset now, whatever tools you run. Inventory where your metric definitions actually live, get the twenty that matter written down precisely, put them under version control, and watch Ossie's converter coverage for your stack. Every hour spent paying down semantic debt appreciates under any future, and it appreciates fastest under the one where your agents need it.

If you are a data engineer or analytics engineer, the on-ramp is delightfully concrete: clone apache/ossie, read the core spec, and walk the TPC-DS example, which will teach you the format in an afternoon. If your company runs dbt, the MetricFlow path means you can experiment with consuming a neutral model today. And if you have ever cursed a semantic translation problem between two specific tools, the converters directory is where that scar tissue becomes a valued contribution.

If you build data tools or work at a vendor, the calculus is the one Iceberg taught: implementing the standard early is how you inherit the ecosystem rather than fight it, and the working groups are open, with the composability and sync API efforts particularly hungry for implementer perspectives.

And if you are looking for an open source project to grow with, podlings are the best entry point the Apache world offers. Small enough that individual contributions are visible, young enough that committership is genuinely earnable, and Ossie specifically has committed to the curated-first-issue, onboarding-sprint playbook. The dev list is dev@ossie.apache.org, the discussions and Slack are linked from the repository, and the community is at the stage where showing up consistently is the whole secret. I said it about Polaris two years ago and it came true for several people who acted on it. I am saying it again.

Questions I Hear Most Often

The early questions, answered directly, since a new project generates the same handful everywhere.

Is Ossie a semantic layer product I install? No. It is a specification plus converters and tooling: the interchange format that semantic layer products, catalogs, BI tools, and agents read and write. Your tools remain your tools. Ossie is how they agree.

How is this different from dbt's metrics, or my BI tool's semantic model? Those are dialects, excellent ones, bound to their tools. Ossie is the neutral format between them, and the tell is that dbt Labs is a core developer building the converters itself. The relationship is Parquet-to-databases, not competitor-to-competitor.

Why should I trust this will not just serve Snowflake and friends? Because that is precisely the question ASF incubation exists to answer with receipts rather than assurances: public governance, merit-based committership, and a graduation bar that requires contributor diversity. Watch the archives, not the press releases, and hold it to the standard Polaris was held to.

Does Ossie handle the actual querying of metrics? Not yet, by design. Today it standardizes definitions. The roadmap's semantic query standard and reference engines aim at consistent execution across engines, and that is the long game, with the appropriately long timeline.

What is the relationship to Apache Polaris? Complementary by design: Ossie defines the format of semantic models, Polaris is building the governed catalog home where they live and are served, and the converter plus the accepted API specification are the handshake. My Polaris article covers the catalog side of the story.

When can I use it in production? The honest answer: the specification is v0.1 and incubating, first implementations are real, and the right 2026 posture for most teams is align, experiment, and contribute rather than migrate. If the community executes, that answer changes within a couple of release cycles, and my newsletters will be tracking exactly that.

Closing Thoughts

Step back far enough and the arc of open data infrastructure is one sentence repeated at ascending layers: we standardized the bytes, then the files, then the tables, then the catalogs, and each time, competition moved up and users won. Apache Ossie is that sentence reaching the top of the stack, the layer where data becomes meaning, arriving at exactly the moment AI agents made ungoverned meaning too expensive to tolerate, carried by many of the same people who standardized the layers below, into the same foundation that made those standards trustworthy.

It is a young podling with a v0.1 specification, a concentrated contributor base, and everything left to prove, and I would not have written five thousand words about it if I thought those caveats were the story. The story is that the last proprietary stronghold in the data stack, the definitions themselves, now has an open, neutral challenger with the right architecture, the right roster, and the right home. Semantic drift has been the quiet tax on every data team's credibility for as long as data teams have existed. For the first time, there is a serious, community-governed plan to end it.

The project is at github.com/apache/ossie, the site is ossie.apache.org, and the dev list is open to anyone. As for me, I will be covering Ossie's incubation the way I cover Iceberg, Polaris, Arrow, and Parquet, weekly, from the source, in my newsletters.

And if you want the deep foundation this whole stack rests on, from table formats through catalogs to the semantics and agents now arriving on top, that is what my books are for. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, with further titles on lakehouse architecture, data engineering, and agentic analytics.

Browse the full collection of my books on data and AI at books.alexmerced.com.

Apache Data Lakehouse Weekly: June 24 to July 1, 2026

Alex Merced — Wed, 01 Jul 2026 15:02:24 +0000

The open lakehouse runs on a small stack of Apache projects, and this was a week where those projects spent most of their energy on the boring work that makes software trustworthy. Iceberg voted to lock down the meaning of expressions and named identities for functions, then turned around and asked a harder question: when five different codebases all claim to follow the same spec, how do we prove they agree? Polaris worked through the plumbing of running one catalog on many databases without a rebuild, welcomed a new committer, and failed a release vote for the right reasons. Parquet dug into what a version number even means once features ship faster than releases. Arrow rebuilt its benchmarking service, partly with an AI agent doing the typing. DataFusion cut a clean release of its Python bindings. Below is what the community built and argued about, and why each thread matters for anyone running data on open formats.

If you are new to this stack, here is the quick map. Parquet is the file format that stores your data as columns on cheap object storage. Arrow is the in-memory format that moves those columns between tools fast. Iceberg is the table format that turns a pile of Parquet files into a real table with schema changes, time travel, and safe concurrent writes. Polaris is the catalog that keeps track of which tables exist and who can read them. DataFusion is a query engine that runs SQL over all of it. These five projects fit together into what people call the open lakehouse, a way to run analytics and AI on open standards instead of a single vendor's closed system. When these projects agree on a spec, your data stays portable. When they drift, you get locked in by accident. That is why the correctness work below matters as much as any flashy feature.

Apache Iceberg

The headline this week was a vote, and it was a big one. Ryan Blue opened a vote to adopt the new expressions spec, a document that defines the minimal structure and behavior of expressions in Iceberg. An expression is the part of a query that filters or transforms data, the piece that says "where the date is after January first" or "where the region equals west." For years each Iceberg implementation carried its own idea of how expressions should behave. Writing that behavior down in a shared spec sounds dull until you realize what it unblocks. Once every engine agrees on exactly what an expression means, features that depend on precise filtering can move forward without each team guessing at the details. The vote drew strong support fast. Steven Wu and Szehon Ho gave binding plus-ones, with Szehon calling the definition elegant after reading it through. Anoop Johnson, Manu Zhang, and Andrei Tserakhau added non-binding support, and Manu said plainly that he was excited about the use cases the spec opens up. Thirty-two messages later, the thread stood as one of the most active of the week across every project on this list.

Right behind it, Szehon Ho started a vote to add a specific-name field to the UDF spec. A UDF is a user-defined function, a bit of custom logic a user writes and then calls by name inside SQL. The problem Szehon set out to fix is a familiar one from the SQL standard. A single function name can have several versions that each take different inputs, and a catalog needs a way to point at one exact version rather than the whole family. The specific-name field gives Iceberg that pointer, matching the concept from the SQL spec directly. Yufei Gu, huaxin gao, Yuya Ebihara, Ryan Blue, Manish Malhotra, Prashant Singh, and Russell Spitzer all weighed in. Taken together with the expressions vote, the week showed a project methodically pinning down the small pieces of behavior that a mature standard needs before the fancy features on top of it can be built safely.

The most interesting design conversation was not a vote at all. Neelesh Salian opened a discussion on cross-implementation conformance testing, and it struck a nerve. Iceberg now has five separate codebases in five languages: Java, Python, Rust, Go, and C++. Each one ships its own tests. What none of them share is a way to check that a table written by one gets read the same way by another. Neelesh framed the gap clearly, and Matt Topol jumped in to say this is something he had wanted for a long time, citing case after case where implementations quietly disagreed. Tanmay Rauth put the real risk into words that stuck: the hardest problems are not the outright bugs, they are the cases where two implementations both look correct and still produce different results. Danny Jones said his team had already been building similar test sets and welcomed a shared reference. The value of a physical reference artifact, a real table that everyone tests against, came up again and again.

That thread did not stand alone. Sung Yun proposed a shared cross-language test fixtures repository called iceberg-testing in the same window, driving at the same problem from a slightly different angle. He named the same five language implementations and the same slow drift in how each one reads the spec. Anurag Mantripragada connected the two threads directly and asked whether they were really one effort. Sung agreed the proposals overlapped and said the two of them had synced offline to converge. This is healthy community behavior worth pointing out. Two contributors saw the same gap, wrote it up independently, noticed the overlap, and started merging their work rather than competing. The result should be a single shared test suite and a set of reference tables that every Iceberg implementation checks itself against. For teams that run mixed engines, a Rust reader here and a Java writer there, that guarantee is the difference between trusting your data and hoping it lines up.

A meatier spec debate ran through the column update file representation thread. The question was how to store updates to individual columns, and the choice came down to dense versus sparse layouts. Steven Wu argued that supporting both options forces every engine to implement the more complex sparse read path, which raises the cost for everyone. Gábor Kaszab agreed that the case against a dense-only layout was not strong and that dense is more straightforward to implement across languages. Andrei Tserakhau made the sharpest point in favor of picking one and mandating it. Dense, he noted, is just a special case of sparse, so the two are not symmetric. Allowing both means every reader carries the heavier code even when the data never needs it. The thread leaned toward mandating a single dense representation now, with room to add column families later for teams that want separate files per group of columns. This is the kind of decision that never makes a headline and shapes performance for years.

Housekeeping got real attention too. Kevin Liu proposed cutting continuous integration time by running the JDK 21 test suite only on the main and nightly branches, keeping JDK 17 on every pull request. Continuous integration is the automated system that runs the full test suite on each proposed change. Running two Java versions on every pull request burns a lot of shared compute, and Iceberg has been watching its usage of the ASF pool of GitHub-hosted runners. Russell Spitzer asked the practical question of who gets alerted when a nightly build breaks on Java 21, and Kevin pointed to the GitHub interface and the continuous integration notification list. Danny Jones floated GitHub merge queues as an alternative, and Russell said turning off the extra run is simply cheaper, since a change that breaks only Java 21 and not Java 17 is rare. Ajantha Bhat tied it back to the broader Iceberg consumption of ASF shared runners thread and confirmed a first merged step toward using one Java version for pull request checks. Small change, real savings, and a sign of a project that has grown large enough to care about its compute footprint.

On the release front, Danny Jones called a vote to release Apache Iceberg Rust 0.10.0 RC1, with Manu Zhang, Rich Bowen, and L. C. Hsieh among those checking the candidate. The Rust implementation keeps shipping at a steady clip, and its progress is part of why teams that want Iceberg without a Java runtime now have a real path. Kevin Liu and Amogh Jahagirdar also sorted out whether to cut 1.11.1 and 1.10.3 patch releases on the Java side, working through which fixes belong in which milestone so the production branches stay clean while newer work continues on the main line.

Several forward-looking proposals landed that point at where Iceberg is headed. Talat Uyarer opened a discussion on a FileRef type for unstructured objects, aimed at letting tables reference images, video, and machine learning artifacts through catalog-brokered access rather than raw paths. Jean-Baptiste Onofré welcomed the idea and pointed to a parallel discussion already running in Polaris, a reminder that the two projects share a lot of surface. On the transactional side, Matt Butrovich pointed a new contributor toward the ongoing work on first-class primary key tables, part of a wider push to add constraint support, including primary keys, to a format that started life as an append-friendly analytics store. huaxin gao summarized the latest index support sync and shared a decision that reads like a small philosophy statement: an index is not a table, it is its own kind of object that reuses table machinery under the hood. William Hyun proposed file-level access delegation in the REST catalog spec, since delegated access today is scoped to whole tables and some workloads need finer control during scan planning. Walaa Eldin Moustafa cross-posted to the Iceberg and Spark lists to gather input on how Spark should route queries against Iceberg materialized views. Andrei Tserakhau moved the collation support discussion from talk to code with a spec-change pull request and reference implementations in both Go and Java. Sunmin Lee raised a geospatial design question about declaring row-level bounding-box covering columns that mirrors the GeoParquet bbox pattern. And Tomohiro Tanaka asked for feedback on a table_properties_log metadata table that exposes the history of table property changes. Read as a group, these threads show a format stretching in three directions at once: toward unstructured data, toward transactional guarantees, and toward richer types, all while the votes above keep the core precise.

Apache Polaris

Polaris had the busiest mailing list of the week by raw volume, and the through-line was a project learning to say no to complexity. Dmitri Bourlatchkov opened a discussion on modular design for new features that set the tone. Polaris has been drawing a flood of interesting proposals, which is good news for a young project. The flip side is that every new feature bolted into the core makes the whole system harder to keep stable and simple to run. Dmitri asked the community to think about how to add capabilities without turning the codebase into a tangle. Russell Spitzer agreed that features should not be coupled into core and runtime in ways that are hard to unwind. Dmitri then pushed back on his own framing, saying he was not convinced every new proposal needs its own isolated Gradle module and staged rollout, a nice example of a maintainer arguing against overcorrection. Anand Kumar Sankaran brought a real-world angle from Workday, which consumes Polaris as a set of Maven dependencies and layers custom authentication and listeners on top. Yufei Gu and Robert Stupp joined the debate over when a feature earns its own module and when that is just proliferation for its own sake. The conclusion trended toward judgment over blanket rules, which is the right answer even if it is harder to enforce.

That governance thread was not abstract. It was the backdrop for the semantic layer support discussion, one of the week's richest at eleven messages. The plan is to store Open Semantic Interchange data as Polaris entities, giving the catalog a home for the business definitions that sit above raw tables. Dmitri worried about hard dependencies running from the runtime and service layers into the new semantic API implementation. Yufei Gu asked why an empty HTTP layer for a disabled feature is a problem, suggesting a plain 404 or 501 response when the feature is off. Alexandre Dutra said he is not a fan of gating an entire API behind a feature flag, and raised a security angle: if a vulnerability shows up in code that ships unconditionally to every user, everyone is exposed even if they never turn the feature on. Romain Manni-Bucau split the question into code modularity, where he saw little debate, and the harder question of what the default Docker image should contain. Yufei countered that a security fix generally lands against the whole project regardless of where an API lives, so the vulnerability argument does not cleanly favor modules. The debate stayed civil and specific, and it fed straight into a vote. Yufei Gu called a vote to accept the OSI Semantic Model API specification, which introduces the initial scaffolding for that semantic model work.

Two threads dug into how Polaris talks to catalogs and where its boundaries sit. Alexandre Dutra led a discussion on non-IRC endpoints in IRC config responses, asking whether the Iceberg REST config endpoint should double as a universal capability discovery tool for Polaris. He and Dmitri agreed that repurposing the config endpoint for everything looks like a misuse of its intent. Dmitri added a technical caveat about the endpoints logically sitting under the catalog base URI at /api/catalog, and the two worked through how policy endpoints and generic table endpoints fit that structure. Yufei Gu proposed extracting the config implementation out of the Iceberg catalog handler so it can serve broader needs. This is the sort of boundary work that keeps a catalog from turning into a junk drawer of unrelated APIs.

On the operational side, Alexandre Dutra opened a discussion on supporting multiple datasources with runtime activation. The goal is simple to state and useful in practice: let an operator switch the backing database, say from PostgreSQL to MySQL, without rebuilding Polaris from source. Yufei Gu asked whether Polaris should manage its own connection pools with a tool like HikariCP. Dmitri framed Alexandre's pull request as an incremental technical improvement to the Quarkus server that does not force any redesign but opens the door to more flexible deployments later. Russell Spitzer asked for clarity on exactly what the change delivers, and Alexandre confirmed the assessment and separated his work from a parallel MySQL effort so the two do not collide. This connects to a longer cleanup arc. In a related thread, Alexandre also moved forward on deprecating the TreeMap-based metastore and its companions for eventual removal, part of a push toward a cleaner default persistence story. Robert Stupp separately revived the discussion on replacing MinIO for S3-compatible storage on the test side, weighing which backend best serves getting-started examples versus test suites.

Polaris also spent time on a real infrastructure question that touches every Java project in the ecosystem. Robert Stupp opened a discussion on Jackson 3 readiness, the widely used library for reading and writing JSON. Robert was careful to say this is not about jumping to Quarkus 4 right now, since Polaris still runs on Quarkus 3. The aim is to prepare for Jackson 3 gradually so the project avoids one giant risky upgrade later. Alexandre Dutra liked the incremental path and asked to understand the impact better. Romain Manni-Bucau suggested a different direction: lean on the JSON-P and JSON-B standards as the API so any vendor can supply the implementation, rather than binding deeply to one JSON library. Jean-Baptiste Onofré agreed the two are related efforts and saw value in the standards approach. Robert wanted to nail down the Polaris-specific impact first, and noted the Iceberg side will need the same conversation eventually. Boring on the surface, load-bearing underneath: choices like this decide how painful the next five years of upgrades will be.

The release story taught a small lesson in doing things right. Jean-Baptiste Onofré voted minus one, binding, on the Apache Polaris 1.6.0 rc0 candidate after finding a missing Spark bundle artifact and some LICENSE issues in the source distribution. EJ Wang had cut the candidate, and the vote did not pass. A failed release vote is not a failure of the project. It is the process working. The checks caught real problems before they reached users. With EJ heading out on vacation, Jean-Baptiste volunteered to prepare the 1.6.0 rc1 candidate himself, and the release target sat around late June. That kind of hand-off, one contributor picking up another's work without ceremony, is what keeps a community project moving when any single person steps away.

There was good news for the people behind the code too. Jean-Baptiste Onofré announced Nandor Kollar as a new Polaris committer, and the congratulations poured in from Robert Stupp, Alexandre Dutra, Ajantha Bhat, Adam Christian, Dmitri Bourlatchkov, Yufei Gu, and Kevin Liu. New committers matter more than they seem to. Each one widens the group of people trusted to review and merge code, which spreads the load and speeds up the whole project. In the same warm register, Jean-Baptiste let the list know he was back after several weeks of travel and planned to return to his usual pace, drawing friendly replies from Kevin Liu, Danica Fine, and Keith Chapman.

A cluster of smaller design threads rounded out the week and showed the catalog maturing at the edges. Yufei Gu and Dmitri worked through a proposal for REST endpoints exposing table metrics and events, with Yufei flagging that the current query API shape is too tied to the example metrics in the Iceberg REST spec. Grace Chen requested reviews for the first phase of entity-level filtering for list operations, part of a visibility filtering proposal that decides which users see which entities. Dmitri argued for not exposing authorization denial details in 403 messages, preferring a random reference ID over leaking why access was denied, a sound security instinct. huaxin gao and Dmitri converged on Model B in the idempotency-key design for the Iceberg REST catalog, which stamps a key into the entity to make repeated requests safe. Two community threads also stood out: Rich Bowen reached out about recording a PlusOne interview, the ASF's short conversation series about project communities, and Kevin Liu flagged that the Slack invite link had expired again and needed a refresh, a tiny recurring friction that every growing community knows well. Underneath the STS token and vended credentials questions raised by evaluators testing federated catalogs, the pattern was consistent: real users are kicking the tires on Polaris federation and reporting back what is unclear.

Apache Arrow

Arrow's list was quieter this week, but two of its threads carry outsized significance for where data tooling is going. Wes McKinney posted an update on the Arrow conbench data and the conbench open source project, pointing the community to a rebuilt version at conbench-v2.arrow-dev.org. Conbench is the service that tracks Arrow's performance benchmarks over time and flags when a change makes something slower. The detail that makes this thread notable is how the rebuild happened. Wes said the summary of the work was written by Codex, the AI coding agent, and that much of the development ran unattended against a mandate to rebuild the backend. Rok Mihevc gave the kind of feedback that keeps a rebuild honest. He appreciated the darker color palette for late-night viewing but preferred the old interactive graph, which classified data points and drew a trendline with confidence bands. Wes agreed to bring the chart back in line with the old one and admitted he had not spent much effort on the interface yet. Antoine Pitrou listed what he cares about most in conbench: regression detection, which he called quite solid after a lot of tuning to the algorithm, and readable benchmark results. This thread is a small window into a larger shift. An AI agent rebuilt a core piece of an Apache project's infrastructure, and the humans reviewed it, pushed back on the parts that lost value, and merged the parts that worked. That is the collaboration model taking shape across the ecosystem, and it is worth watching closely.

It is worth sitting with why benchmarking infrastructure is the kind of thing a project guards so carefully. Arrow is the in-memory format that a huge slice of the data world uses to move columns of data between tools without copying and reformatting them. When a change makes Arrow even a few percent slower, that cost multiplies across every system that depends on it. Conbench is the early-warning system that catches those slowdowns before they ship. So when Wes handed much of the rebuild to Codex, he was handing an agent responsibility for a piece of infrastructure that protects the performance promises of the whole project. The fact that it worked, and that the review caught the places where the new interface lost useful detail like the classified points and confidence bands Rok wanted back, is a real data point about where agent-assisted maintenance stands today. The agent did the heavy lifting on an unglamorous backend rebuild, and the experienced maintainers decided what was good enough to keep. Neither replaced the other.

The second standout came from outside the usual contributor group. Sam Arch, a PhD student at Carnegie Mellon co-advised by Andy Pavlo and Jignesh Patel, announced an ADBC extension for DuckDB. ADBC is the Arrow Database Connectivity standard, a way for tools to move Arrow data in and out of databases without slow row-by-row conversion. Getting ADBC into DuckDB, the popular in-process analytics database, connects two fast-moving corners of the data world. Rusty Conover congratulated Sam on the release, praised the connection-profile integration, and said he plans to add support for it to his adbc_scanner project. Aldrin weighed in on the framing of a hand-coded claim in the announcement. The thread is short, but it signals healthy cross-pollination. Academic database research and the Arrow standards are meeting in a widely used tool, and the maintainers are already talking about how their projects connect.

The rest of Arrow's week was community maintenance, which matters more than it sounds. Robert Thomson reported on the project's use of ASF shared GitHub-hosted runners, noting that Arrow had dropped to twelfth in minutes consumed over the prior seven days, a real improvement, while framing runner usage as an ongoing discipline rather than a one-time fix. That echoes the same continuous-integration cost conversation happening in Iceberg, a shared pressure across the whole foundation. Nic Crane asked for extra help closing out old issues, describing automation that flags stale bug reports and the manual work she, Alenka, and Rok have been doing to clear the backlog. Ian Cook announced the biweekly Arrow community meeting for July 1 at 16:00 UTC. And Zehua Zou raised a cross-format question about allowing the VARIANT value field to be omitted, noting that parquet-format does not allow it while Arrow's documentation does. That last thread is a good bridge into Parquet, where the VARIANT type drew heavy attention this week.

Apache Parquet

Parquet spent the week wrestling with a question that sounds simple and is not: what should a version number mean? Daniel Weeks drove a long discussion on the future of Parquet versioning, the busiest thread on any list this week at seventeen messages. It grew out of a proposed change to how paths in the schema get handled, and it quickly became a debate about process. Daniel argued that the community should coordinate and agree on what belongs in a major version bump rather than forcing a new version every time an incompatible change lands. Micah Kornfield and Ed Seidl worked through the specifics, with Ed noting that his path-in-schema change was partly a test of the documented process for handling forward-incompatible changes. Andrew Lamb captured the core tension in a way worth repeating in plain terms. One option is to keep using version numbers, which people understand and which the rest of the industry uses, but which are a blunt instrument, since touching any single feature of a new version can seem to require the whole version. The other option leans on per-feature signaling, which is precise but less familiar. Antoine Pitrou and Ryan Blue added their perspectives, and the thread did not fully resolve, which is fine. Deciding what a version promises is the kind of question a format needs to answer carefully once, because everyone downstream lives with the answer.

That versioning debate had two direct offshoots. Micah Kornfield proposed moving parquet-format releases to semantic versioning, with the concrete idea of a major version bump every time a release includes an incompatible change. And Andrew Lamb reported progress on documenting which features live in which versions of Parquet, announcing that a merged pull request put a new explanatory page live on the website and that a second pull request renders the feature table automatically. This pairing is the practical answer to the abstract debate. If you cannot easily tell which features a version contains, the version number carries less meaning, so writing that mapping down and keeping it current is real progress.

The release process moved forward on a specific spec change. The vote on GH-583 to define ordering for INT96 timestamps, started by Divjot Arora, gathered binding plus-ones from Ryan Blue, Micah Kornfield, Gang Wu, and Daniel Weeks, with Andrew Lamb adding support. INT96 is an old ninety-six-bit timestamp type that Parquet inherited from its early days. It has been deprecated for years, but real files in the wild still use it, so pinning down exactly how those timestamps should sort matters for anyone reading legacy data. Micah noted that Ryan's support implied comfort with the Java implementation, whose review was still in progress, and that Ed Seidl was handling the Rust side. Andrew added that older versions of arrow-rs that panic on this data can get patched releases if the problem shows up in practice. Ryan was careful to separate his vote on the spec direction from the still-open Java code review. This is a good example of how Apache projects split the question of what to build from the question of whether a specific implementation is ready.

The most exciting announcement came from Gunnar Morling, who shared the release of Hardwood 1.0, a new Parquet reader for the JVM. Hardwood is built from the ground up to keep external dependencies to a minimum, with a writer planned to follow. Fewer dependencies means fewer transitive security vulnerabilities to chase, a point Steve Loughran made right away in his congratulations. Steve, who has an open pull request to harden variant parsing, asked how Hardwood handles the VARIANT type. Gunnar ran Steve's test fixtures through Hardwood and reported that it rejects the malformed cases except for one depth case that lacks a guard so far. Pritam Pan asked whether Hardwood integrates with Apache Spark down the road, and Gunnar said he was not aware of any such discussion and did not want to speak for the Spark side. A fresh, lean reader for one of the most widely used file formats in data is good for the whole ecosystem, since it gives teams another well-built option and keeps the incumbents honest.

VARIANT hardening was a running theme. Steve Loughran opened a discussion on how deep a realistic variant depth is, tied to his pull request that adds shallow validation of variant inputs in parquet-java. The VARIANT type stores flexible, semi-structured data, which is powerful and also a place where malformed input can cause trouble if a reader trusts it blindly. Kurtis Wright asked a sharp question: is Parquet the right layer to build reader guardrails that writers can choose to ignore, or does that belong somewhere else? Gunnar Morling argued that a parser should be able to reject malformed payloads on its own, since not all Parquet use runs through Iceberg, so building the guardrails into the parser makes sure they apply everywhere. This connects to Kevin Liu and Micah Kornfield's separate thread on how older parquet-java readers should behave when they hit VARIANT columns, where version 1.15.x fails on a newer logical type. Together these threads show a format working out how to add powerful new types without breaking the readers already deployed across thousands of systems.

Several more threads filled out a productive week. Rok Mihevc moved to introduce a FIXED_SIZE_LIST logical type based on benchmarks and design-doc feedback, giving Parquet a native way to describe lists with a fixed number of elements. Alkis Evlogimenos reported that the FILE proposal is in good shape, with path, size, offset, and etag in place and an active thread on adding content_type. Micah Kornfield nudged forward an AI tooling policy for Parquet, suggesting the community open a pull request with the current draft, a sign that Parquet, like others, is writing down how it wants AI-generated contributions handled. Russell Spitzer supported adopting AssertJ for test assertions as a gradual improvement for consistency. Jiayi Wang shared written recaps from the Parquet Footer Working Group's third session. And the community looked for a facilitator for the July 1 Parquet sync when Julien Le Dem flagged he will be out. For a format that many people think of as finished, Parquet is very much still evolving.

Apache DataFusion

DataFusion kept it focused this week with a clean release effort. Tim Saucer called a vote to release the DataFusion Python bindings, version 54.0.0, on release candidate two. DataFusion is a fast query engine written in Rust, and its Python bindings let data scientists and engineers drive that engine from Python without touching Rust. The vote passed with binding support from Matt Butrovich, who verified on macOS, Andrew Lamb, who ran the checks on an M3 Mac and reviewed the changelog, L. C. Hsieh, who verified on an M4 Mac, and Adrian Garcia Badaracco, who tested on an M4 MacBook Pro. Renato Marroquín Mogrovejo added non-binding support. Andrew also thanked a contributor named Nuno for helping review many of the pull requests in the release.

A short thread, but a healthy one, and it is worth explaining what the ceremony is doing for readers new to Apache. A release candidate is a proposed final build that has not shipped yet. Before it becomes official, members of the project download it, verify the cryptographic signatures and checksums, check the license files, and run the build on their own hardware. A binding vote comes from a project committer whose plus-one counts toward the official tally, while a non-binding vote is a welcome check from anyone else in the community. The reason four people ran the same checks on four different Macs is that a release only earns trust when independent people confirm it builds and passes on machines the release manager does not control. This is the same discipline that caused Polaris to reject its 1.6.0 rc0 candidate over a missing artifact. The vote is not a rubber stamp. It is the community putting its name on the build.

The steady cadence of DataFusion releases is part of why the Rust data stack keeps gaining ground, and the Python bindings are the on-ramp that brings that speed to the analysts who never leave their notebooks. It also lines up with a quiet cross-project pattern this week. Iceberg cut a Rust release candidate, DataFusion cut a Rust-backed Python release, and the Arrow ecosystem moved ADBC into DuckDB. Rust keeps showing up as the language teams reach for when they want the speed of native code without a Java runtime, and Python keeps showing up as the surface those teams expose to their users. For anyone deciding what to build a data platform on, the message is that the fast open engines and the friendly high-level interface are no longer a trade-off. You can have both, and the release votes this week are the receipts that the pieces are production-ready.

Cross-Project Themes

Three patterns ran across the lists this week, and each one says something about where the open lakehouse is heading.

The first is a shift from building features to proving correctness. Iceberg's expressions and UDF votes, its conformance testing and shared test fixtures threads, and Parquet's fight over what a version number means all point the same direction. These projects have enough implementations and enough production users that the community can no longer trust everyone to interpret the spec the same way. So the work turns toward writing behavior down precisely, then building shared tests that prove the implementations agree. This is what maturity looks like in open standards. The exciting phase of adding capabilities gives way to the harder phase of guaranteeing that a table written anywhere reads correctly everywhere. For anyone betting a business on open formats, that guarantee is the whole point.

The second theme is the quiet arrival of AI agents inside the projects themselves. Wes McKinney rebuilt Arrow's conbench service with heavy help from Codex and said so openly, then let human reviewers push back on the parts that lost value. Parquet started drafting an AI tooling policy. These are not press releases about AI. They are working engineers folding agents into real maintenance and being transparent about it, while keeping human judgment in the loop. The lakehouse community is also the community building the data layer that agents run on, so it makes sense that these projects are among the first to work out the norms for agent-assisted contribution. Expect more projects to write down how they want AI-generated code reviewed and merged.

The third theme is shared infrastructure discipline. Iceberg and Arrow both spent real energy this week on trimming their use of the ASF pool of GitHub-hosted continuous-integration runners. Iceberg moved to run fewer Java versions on pull requests. Arrow celebrated dropping to twelfth in minutes consumed. This is a foundation-wide pressure, since every Apache project draws from the same limited pool, and it shows these communities acting like responsible neighbors rather than maximizing their own convenience. The VARIANT type also crossed project lines all week, showing up in Arrow, in Parquet's hardening and compatibility threads, and in the format-level question of whether a value field can be omitted. When a single type sparks parallel discussions on three lists, it is a sign that semi-structured data is becoming a first-class citizen of the lakehouse, and the community is working out the rules together.

Looking Ahead

Watch the Iceberg conformance testing and iceberg-testing threads to see whether the two converged proposals produce a real shared test suite, since that is a meaningful step toward guaranteed cross-engine correctness. Keep an eye on Polaris 1.6.0, where a fresh release candidate should follow the failed rc0 vote, and on the semantic layer work now that the OSI Semantic Model API specification is up for a vote. Parquet's versioning debate is unlikely to end quickly, so the semantic-versioning proposal and the feature-documentation page are the concrete pieces to track. And the VARIANT hardening work across Parquet and Arrow is worth following for any team that stores semi-structured data, since the guardrails being designed now will shape how safely that data moves between tools. The unifying story is a set of projects growing up together, trading raw feature velocity for the correctness and stability that production workloads demand.

Resources & Further Learning

Get Started with Dremio

Try Dremio Free. Build your lakehouse on Iceberg with a free trial
Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow. Learn how Dremio brings the open lakehouse stack together

Free Downloads

Apache Iceberg: The Definitive Guide. O'Reilly book, free download
Apache Polaris: The Definitive Guide. O'Reilly book, free download

Books by Alex Merced