DEV Community: Alex Merced

Fast Classification Models, LLMs, and the Apache Iceberg Lakehouse

Alex Merced — Mon, 21 Sep 2026 16:16:46 +0000

Open the bill for any team that has put a large language model into a data pipeline and look at what the calls are doing. A big share of them are not writing anything. They are answering questions with a short, fixed set of answers. Is this support ticket about billing or about an outage? Does this review mention a safety problem? Is this row of free text a complaint, a question, or a compliment? The team sends a paragraph of context, a paragraph of instructions, and a request for JSON. The model spends seconds generating tokens, the pipeline parses the JSON, and sometimes the JSON comes back broken.

That pattern works. It is also slow and expensive for what it delivers, and it gets worse at lakehouse scale, where the "input" is not one ticket but forty million rows in an Apache Iceberg table.

In September 2026 a company called TypeSafe AI released Jev, a model built only for this kind of question. It does not chat. It returns a choice, a score, or a yes/no probability, with calibrated confidence attached. Within days, several open-source projects appeared that copy its interface on models you can run yourself.

This article covers what these classification models are, how they differ from LLMs, where the open alternatives stand, and how to use both kinds of model together inside an Iceberg lakehouse. A quick note on affiliation: I work at Dremio, which ships SQL AI functions for this kind of work. Dremio shows up here as one worked example. The patterns apply to any engine that reads Iceberg.

Why So Much LLM Spend Goes to Questions With Fixed Answers

Text classification is one of the oldest jobs in applied machine learning. Before 2023, most teams handled it in one of three ways.

The first was rules. Keyword lists, regular expressions, and lookup tables. Rules are fast and cheap, and anyone can read them. They also break the moment a customer writes "I am not asking for a refund." A rule that fires on the word "refund" gets that sentence wrong every time.

The second was a fine-tuned encoder model. BERT (Bidirectional Encoder Representations from Transformers) and its descendants read a whole passage at once and produce a label in a single forward pass. A fine-tuned BERT classifier runs in milliseconds on a modest GPU. The cost is up front. You need a few thousand labeled examples per task, a training pipeline, and a person who knows how to evaluate it. When the business adds a new category, you label more data and train again.

The third was zero-shot classification with NLI (natural language inference) models. You phrase each label as a hypothesis, such as "This text is about billing," and the model scores how strongly the input entails it. This needs no training data, but accuracy on messy, domain-specific text is uneven.

Then instruction-tuned LLMs arrived and changed the economics of the second and third options. You no longer needed labeled data or a training run. You wrote a prompt, listed the categories, and asked for JSON. Accuracy on common-sense judgments jumped. For a prototype, nothing else came close.

The trouble shows up in production. An autoregressive LLM produces output one token at a time. Every label, every brace, and every quote mark in the JSON is a separate step through the model. Reasoning models add a chain of thought before the answer, which multiplies the token count again. A classification that needs one bit of information ends up costing hundreds of output tokens.

There are three other costs that are easy to miss. The first is parsing. The model returns text, and your code has to turn it into a typed value. Structured output modes help, but the model still generates the text and a validator still checks it. The second is invented labels. Ask for one of five categories and a generative model sometimes returns a sixth that sounds reasonable. The third is the confidence problem. An LLM that says "billing" gives you no reliable number for how sure it is. Token log probabilities exist on some APIs, but they are not calibrated for this purpose, and many hosted APIs do not expose them at all.

In a single request/response app, these costs are annoying. In a lakehouse pipeline, they compound. If a nightly job classifies two million new rows with a frontier LLM, the latency sets how long the job runs, the per-token price sets the bill, and the lack of confidence scores means every label looks equally trustworthy in the downstream table. An analyst building a dashboard on those labels has no way to filter out the shaky ones.

That gap between "the LLM can do this" and "the LLM is the right tool for this" is where the new classification models live.

What Jev Is and How It Works

Jev comes from TypeSafe AI, a company founded by Diogo Almeida, a former OpenAI researcher and one of the authors of the InstructGPT paper. TypeSafe calls Jev a "System One model." The name borrows from Daniel Kahneman's split between fast, intuitive thinking (System 1) and slow, deliberate reasoning (System 2). Reasoning LLMs are System 2 tools. Jev is built for the quick judgments.

The interface has three parts.

State. The content you want judged. This can be a string, a JSON object, or an array of text values. A support ticket, a product review, a row from a table serialized as JSON, or a retrieved document all work as state.

Questions. A map of named questions. Each question has a type, instructions, and criteria. You choose the question names, and the answers come back under the same names. The model never sees the names themselves.

Answers. One typed answer per question, with probabilities.

There are three question types, and picking the right one is most of the design work.

A Choice picks one option from a set you define. The criteria are a map from option names to descriptions. TypeSafe's documentation allows up to 255 options per Choice. The answer contains the selected option, a probability for every option, and a confidence value. If you ask which team owns a ticket and list billing, returns, and shipping, the answer is always one of those three. It cannot be a fourth.

A Score rates something against an ordered rubric. The criteria are an ordered list of level descriptions, from 2 to 10 levels. Urgency, quality, relevance, and frustration are natural fits. The answer is the level index, a probability per level, and a confidence value.

A Noul is a yes/no question. The answer is the probability that the statement is true. You can optionally describe what "true" and "false" mean. The unusual name is TypeSafe's own term, and it shows up in their SDKs as Noul.

The model processes the state once and evaluates every question against it in parallel. This detail matters a great deal for cost. Asking ten questions about one ticket in a single request costs barely more wall-clock time than asking one. TypeSafe's docs push this pattern hard and call it speculative fan-out. You ask every question your code has any use for, including ones that only matter for certain branches, and your code ignores the answers it does not use.

TypeSafe has said little about the architecture. Public material mentions a new model design, a parallel sampler, and a training method called RLCD (reinforcement learning for calibrated decisions). Coverage from the launch describes it as non-autoregressive, meaning it does not generate output one token at a time. The practical result is latency in the range of roughly 70 to 500 milliseconds per call, versus seconds for a frontier model producing JSON.

The numbers that matter for pipeline design

At the time of writing, the current version is jev-1.13.0, reachable through the alias jev-latest. The published limits shape how you build around it:

Price. TypeSafe charges only for input tokens, at $0.042 per million. Output tokens are free.
Rate limits. 250,000 tokens per second and 1,200 requests per minute. TypeSafe warns these limits are changing as demand grows.
Context. 64,000 tokens per request in total, and 32,000 tokens for the state plus the longest single question.
Input. Text only. No images, audio, or video.

Access is through POST https://api.typesafe.ai/v1/systemone, with official Python and JavaScript SDKs. Early access runs through a waitlist. Gateway products such as LiteLLM and several others have already added pass-through support.

On accuracy, TypeSafe's own four-workflow benchmark puts Jev at 67.8 percent. That is roughly tied with GPT-5.6 Terra at 67.9 percent, and a few points behind GPT-5.6 Sol at 74.1 percent and Claude Opus 5 at 73.1 percent. Read that plainly. Jev is not smarter than frontier models. It lands near a mid-tier LLM on decision tasks while running at a small fraction of the cost and time. That tradeoff is the whole pitch.

Independent testers have published early numbers that line up with the pitch. One team ran seven classification rules across 1,000 emails in about 6 seconds for 9 cents once they parallelized requests. A comparable single-category run on a GPT-5.6-class model took roughly 5 minutes and cost 62 cents. Treat third-party benchmarks from launch week with care, but the direction is consistent across reports.

What Jev does not do

Jev does not generate text. It does not explain its answers. It does not do arithmetic well, and TypeSafe says so directly. It reads dates as text, so comparing two dates or checking whether one falls in a window is unreliable. It cannot be fine-tuned on your data. The same weights serve every account, and you shape behavior through the state, instructions, and criteria you send.

That last point is a real difference from the BERT approach. You do not own a model. You own a set of well-written questions. For many teams that is a better asset, because questions are cheap to change and easy to review in a pull request.

Calibration Is the Feature That Changes Pipeline Design

Speed and price get the headlines. For data engineers, the more important property is calibration.

A model is calibrated when its probabilities match reality. If a calibrated model says "billing" with 0.8 probability across a thousand tickets, about 800 of those tickets are about billing. A model that is overconfident says 0.95 and is right 70 percent of the time. A model that gives no probability at all leaves you guessing.

Jev returns two related numbers, and they mean different things.

Probabilities describe the distribution across options. For a Choice with three options, you get three numbers that sum to 1. For a Noul, you get one number, the chance the answer is yes.

Confidence summarizes how peaked that distribution is. A single spike on one option gives confidence near 1. Probability spread across several options gives low confidence. TypeSafe's docs show a ticket that mentions both a wrong size and a double charge. The department question returns "returns" at 0.61 and "billing" at 0.35, with confidence of 0.42. The model is telling you, correctly, that the ticket belongs to two teams.

This changes what you can do with the output. With an uncalibrated LLM label, your choices are to trust it or re-check everything. With a calibrated probability, you get a dial. Rows above 0.9 flow straight into the published table. Rows between 0.5 and 0.9 get labeled but flagged. Rows below 0.5 go to a more expensive model or a human. TypeSafe calls this confidence-gated routing, and it is the single most useful pattern in their docs.

The dial also changes analytics. A dashboard that counts "tickets about billing this week" usually treats each label as a fact. With probabilities stored in the table, you have two better options. You can count only labels above a threshold and report the coverage alongside the count. Or you can sum the probabilities themselves, which gives an expected count that accounts for uncertainty. If a thousand tickets each carry a 0.3 probability of churn language, the expected number of churn-signal tickets is 300, even though no single ticket crosses a 0.5 threshold. For trend analysis over large volumes, the expected count is often the more honest number.

Calibration is not magic, and it is not permanent. It holds on the kind of data the model was trained and tested on. On text far from that data, such as a domain full of internal jargon or a language other than English, probabilities drift. TypeSafe states that English is the primary training language and that other languages work less well. The fix is the same one statisticians have always used. Keep a labeled sample of your own data, compare predicted probabilities against actual outcomes, and check the gap on a schedule. The operational section below covers how to do that with an Iceberg table.

The Open-Source Alternatives

Jev is closed. TypeSafe has not released weights or training code and has not announced plans to. For teams that need to run classification inside their own network, or that do not want a waitlist between them and production, the open ecosystem moved fast. It helps to split the options into two groups: established zero-shot classifiers that predate Jev, and new projects that copy Jev's interface.

Established open zero-shot classifiers

GLiClass from Knowledgator is the most mature option in this space. It is a zero-shot sequence classifier inspired by GLiNER, the open named-entity model. GLiClass encodes the text and all candidate labels together and scores every label in a single forward pass. The model card reports performance comparable to a cross-encoder with much lower compute. The gliclass-modern-base-v2.0 checkpoint uses ModernBERT-base as its backbone, has about 151 million parameters, and is trained on synthetic and commercially licensed data. The v3.0 checkpoints add training on logic tasks. It supports single-label and multi-label classification, and it doubles as a reranker for RAG (retrieval-augmented generation) pipelines. At that size it runs on CPU for modest volumes and flies on a single GPU.

NLI-based zero-shot models, such as DeBERTa-v3 checkpoints fine-tuned on entailment datasets, remain a solid baseline. They are well understood, license-friendly, and supported directly by the Hugging Face zero-shot-classification pipeline. Their weakness is cost per label. Each candidate label becomes a separate premise and hypothesis pair, so a 50-option taxonomy means 50 forward passes per row.

CAPPr (completion after prompt probability) takes a different route. It uses any open causal language model, but instead of letting the model generate, it scores the probability of each candidate completion. This turns a generative model into a classifier with a fixed output set. It is slower than an encoder model but works with whatever open LLM you already serve.

New projects that copy Jev's interface

Several projects appeared within days of the Jev launch. All of them are independent. None contain Jev's weights or its RLCD method, and most say so plainly in their READMEs. They are worth watching and not yet worth betting a production pipeline on without your own evaluation.

OpenJev (zhangcy122) wraps open LLMs behind the Choice, Noul, and Score interface. It enforces the output schema with constrained decoding through vLLM or SGLang, pulls log probabilities for each candidate, and applies temperature or Platt scaling to calibrate them. It adds an abstention layer so the system can say "unknown" instead of guessing. The recommended setup runs small models with thinking turned off and escalates to a large model only when confidence falls below a threshold.

Verdict (published on Hugging Face as rlcd-modernbert-151m) is a 151-million-parameter model built on the GLiClass ModernBERT checkpoint and post-trained with an approach inspired by RLCD. It reports single-pass latency under 35 milliseconds and supports up to 24 options plus an explicit abstention slot. Its authors claim wins over Jev on their own typed-decision benchmarks. Those claims are self-reported and have not been independently reproduced.

Open JEV (zhihz) is a research preview that runs a frozen Qwen3-4B-Instruct model to score candidate answers in English and Chinese. Its README is refreshingly direct that no matched comparison against Jev, GLiClass, or CAPPr has been completed.

poorjev reproduces the three question types on top of NLI models and focuses on honest confidence. It applies temperature scaling and conformal abstention and publishes a calibration evaluation showing expected calibration error dropping from 0.170 to 0.071 with no accuracy loss.

The table below summarizes the tradeoffs as they stand in September 2026.

The honest summary is this. If you need a general-purpose decision model today and can send data to a hosted API, Jev is the most polished option. If data must stay inside your network, GLiClass plus your own calibration step is the most defensible choice. If you have a stable task with thousands of labeled examples, a fine-tuned ModernBERT classifier still beats every zero-shot option on accuracy per dollar. The Jev clones are promising experiments.

Which Job Goes to Which Model

The cleanest way to decide is to look at the shape of the answer, not the difficulty of the question.

If the set of possible answers is known before the model runs, a classification model is the default. Routing, tagging, filtering, scoring, deduplication decisions, and policy checks all fit. The answer is a label, a level, or a probability, and your code branches on it.

If the answer is new content, you need a generative model. Summaries, explanations, extracted free-text values like a customer's stated reason in their own words, SQL generation, and anything a person will read as prose all belong to an LLM.

A few jobs sit in between, and they are where the most interesting designs come from.

Extraction with a bounded answer space. Pulling a date, an amount, or an email address out of text looks like generation, but the answer often comes from a small set of candidates. TypeSafe recommends finding candidates with a regular expression or an LLM, then asking the classifier which candidate is correct. Their date extraction pattern turns each date part into a Choice (twelve months, thirty-one days, a bounded year range, plus a "not stated" option) and assembles the date in code.

Judgments that need multi-step reasoning. "Is this contract clause more restrictive than our standard terms?" requires comparing two texts and reasoning about what each permits. Classification models struggle with that level of indirection. A reasoning LLM earns its cost here.

Counting and math. Neither model type is a calculator. Keep arithmetic in code. If you need to count items that match a condition, ask one yes/no question per item and sum the answers in code.

For data analytics specifically, here is how common lakehouse tasks tend to split:

How the Two Work Together

The strongest designs do not pick one model. They put each model where it is strongest and let code connect them. Four patterns show up again and again.

Pattern 1: The cascade

Send every row to the cheap, fast classifier first. Read the confidence. Accept high-confidence answers as final. Send the rest to an LLM, and send whatever the LLM is unsure about to a human.

The economics are easy to reason about. If 85 percent of rows clear the confidence gate, the LLM sees 15 percent of the volume. Your LLM bill drops by roughly that ratio, and your pipeline runtime drops even more because the classifier is so much faster. The quality of the final table is set by the LLM on the hard rows, which is where you wanted it spent.

The threshold is a business decision, not a model setting. A threshold of 0.9 sends more rows to the LLM and costs more. A threshold of 0.6 saves money and accepts more errors. Set it by measuring error rates at several thresholds on a labeled sample, then pick the point where the cost of an error and the cost of escalation balance.

Pattern 2: The router in front of the agent

AI agents that answer questions over lakehouse data make many small decisions per turn. Which tool fits this request? Does this need the semantic layer or a raw table? Is this request asking for data the user is not allowed to see? Is the retrieved context relevant?

Each of those decisions is a classification. Running them through the agent's main LLM adds seconds per turn. Running them through a System One model adds tens of milliseconds. LangChain's integration exposes Jev as a classifier for exactly this role, including model routing: simple lookups go to a small, cheap model and hard debugging tasks go to a capable one. Vercel's team reported checking commands for safety 5 to 18 times faster after moving that check from an LLM to Jev.

For a lakehouse agent connected through MCP (Model Context Protocol, the open standard for connecting AI clients to tools and data), a classifier sits naturally in front of tool calls. It decides whether a proposed query is safe to run, whether it needs confirmation, or whether to block it.

Pattern 3: The guard on the way out

LLMs generate. Classifiers check. After an LLM writes a summary or answers a question, a classifier asks narrow questions about the output. Does every cited passage support the claim it is attached to? Does the answer contain personal data? Does it contradict the retrieved source? TypeSafe's cookbook includes a citation check that uses a Choice to decide whether the quoted context supports a claim, with confidence flagging borderline cases for review.

This is cheap enough to run on every response, which is the point. A guard that you only run on a sample is a monitoring tool. A guard that runs on every response is a control.

Pattern 4: Features for traditional models

This one gets less attention and matters a lot for analytics. Classification outputs are numbers. A Noul probability, a Score level, and a Choice distribution are all features you can feed into a gradient-boosted tree or a regression model. TypeSafe's docs include a cookbook that uses Jev questions to turn free text into numeric features for a CatBoost regressor.

In a lakehouse, that means your text columns stop being dead weight in predictive models. A churn model that only saw structured fields can now include "probability the customer mentioned a competitor" and "frustration level on a five-point scale" from every support interaction. The LLM is not in this loop at all. The classifier converts text to features once, the features land in an Iceberg table, and the modeling team uses them like any other column.

Classification Inside an Apache Iceberg Lakehouse

Apache Iceberg is an open table format. It adds a metadata layer on top of Parquet files in object storage so that many engines can read and write the same tables with ACID transactions, schema evolution, and time travel. That combination of properties makes Iceberg a good home for model outputs, and it is worth being specific about why.

Store labels as their own table

The first design decision is where labels live. The tempting move is to add a category column to the raw table and fill it in. Resist that. Keep the source table as it arrived and write classification results to a separate enrichment table keyed by the source row's ID.

This separation pays off in several ways. You can reclassify everything with a new model version without rewriting the raw data. You can keep two model versions side by side and compare them. Your raw table's snapshots reflect ingestion only, which keeps its history easy to read. And access control gets simpler, because the enrichment table often carries less sensitive data than the raw text.

Store probabilities, not just labels

The enrichment table should carry the full answer, not only the winning label. At minimum, that means the chosen option, its confidence, and the probability distribution. For a Choice with a handful of options, a JSON string or a map column holds the distribution. For a Noul, one double column is enough.

It also needs provenance columns: the exact model version that answered (Jev reports a versioned ID such as jev-1.13.0 even when you call an alias), a version string for your question set, a timestamp, and the route the row took through the cascade. Without these, you cannot tell which labels came from which model after an upgrade, and you cannot audit a bad label back to its cause.

Use Iceberg features that fit the problem

Snapshots and time travel. Every write to an Iceberg table creates a snapshot. When a model upgrade changes labels, you can query the enrichment table as of the old snapshot and compare it to the new one. That gives you a precise diff of every label that changed, which is the best possible input for deciding whether the upgrade is safe.

Branches for write-audit-publish. Iceberg supports named branches on a table. A classification job writes to a staging branch, a validation step checks the output (label distribution, confidence distribution, null counts, row counts against the source), and only then does the branch get fast-forwarded to main. Engines such as Spark expose this through Iceberg's write-audit-publish settings. This keeps a broken prompt or a model regression from ever reaching the dashboards.

Tags for model versions. Tag the snapshot produced by each model version, such as jev-1.13.0-baseline. Tags keep that snapshot from expiring and give analysts a stable name for comparisons.

Partitioning. Partition the enrichment table by day of the labeling timestamp. Incremental jobs append one day at a time, and queries that look at recent labels prune old files.

Incremental processing

Classification jobs should process only new rows. The simplest approach is a watermark: read the latest source timestamp already present in the enrichment table, then scan the source table for rows after it. Iceberg's metadata makes this cheap because the scan planner skips files whose column statistics fall outside the filter. A more precise approach reads the changes between two snapshots of the source table, which some engines expose as an incremental or changelog read.

Throughput and cost math

The published limits make sizing straightforward. Suppose a backlog of 10 million support tickets averaging 300 input tokens each, including the question definitions. That is 3 billion input tokens. At $0.042 per million, the model cost is about $126 for the entire backlog.

The binding constraint is the request rate, not the price. At 1,200 requests per minute and one ticket per request, 10 million tickets take about 139 hours. Packing 20 tickets into one request as an array of state, with one question set per ticket, cuts that to about 7 hours. The token rate limit of 250,000 per second puts a floor of about 3.3 hours on 3 billion tokens. Packing trades accuracy for throughput, because TypeSafe documents that irrelevant detail in the state lowers accuracy. Test packed and unpacked requests on a labeled sample before committing to a batch size.

Compare that to a frontier LLM on the same backlog. The input tokens alone cost more by an order of magnitude or two, and output tokens for JSON add more on top. The exact multiple depends on the model and your prompt, but launch coverage cites 40 to 400 times cheaper and 20 to 200 times faster for Jev. Even at the low end, that turns a quarterly project into a nightly job.

For a self-hosted GLiClass deployment, the math is about hardware instead of tokens. A 151-million-parameter encoder on a single modern GPU processes thousands of short texts per second in batches. The cost is the GPU hours and the engineering time to run the service.

A Worked Example: Support Tickets From Raw Text to Dashboard

Here is a complete, small version of the cascade. The source is an Iceberg table of raw support tickets, support.tickets_raw, with a ticket ID, subject, body, and ingestion timestamp. The goal is a labeled enrichment table and a weekly analytics query that uses probabilities, not just labels.

Step 1: Create the enrichment table

This DDL runs in Dremio against an Iceberg catalog. Any engine that creates Iceberg tables works the same way with its own syntax.

CREATE TABLE lakehouse.support.ticket_labels (
  ticket_id            VARCHAR,
  category             VARCHAR,
  category_confidence  DOUBLE,
  category_probs       VARCHAR,
  churn_prob           DOUBLE,
  urgency_level        INT,
  urgency_confidence   DOUBLE,
  route                VARCHAR,
  model_version        VARCHAR,
  question_set         VARCHAR,
  source_ingested_at   TIMESTAMP,
  labeled_at           TIMESTAMP
)
PARTITION BY (DAY(labeled_at));

Each column has a job. category_probs holds the full distribution as a JSON string, so analysts can see when a ticket split between two teams. churn_prob is a raw Noul probability. route records which path the row took through the cascade. model_version and question_set make every label traceable. source_ingested_at carries the source timestamp forward so the next run knows where to start.

Step 2: Classify new rows and append

This Python job uses PyIceberg, the official Python library for Iceberg, to read new tickets and write results. It uses the TypeSafe Python SDK for classification.

import json
from datetime import datetime, timezone

import pyarrow as pa
import pyarrow.compute as pc
from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import GreaterThan
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

MODEL = "jev-1.13.0"         # pin a version, not the jev-latest alias
QUESTION_SET = "support-v3"  # bump whenever the questions change
AUTO_ACCEPT = 0.80           # confidence gate for the category answer

QUESTIONS = {
    "category": Choice(
        instructions="Which product area is the customer writing about?",
        criteria={
            "billing": "Charges, invoices, refunds, payment methods",
            "integrations": "Connectors, APIs, webhooks, third-party tools",
            "performance": "Slow queries, timeouts, dashboards that fail to load",
            "account_access": "Login, SSO, passwords, permissions",
            "other": "Anything that fits none of the options above",
        },
    ),
    "churn_signal": Noul(
        instructions=(
            "Does the customer say they plan to cancel, downgrade, "
            "or switch to another vendor?"
        ),
    ),
    "urgency": Score(
        instructions="How quickly does this ticket need a response?",
        criteria=[
            "No time pressure",
            "Should be answered this week",
            "Should be answered today",
            "Production is down right now",
        ],
    ),
}

LABEL_SCHEMA = pa.schema([
    pa.field("ticket_id", pa.string()),
    pa.field("category", pa.string()),
    pa.field("category_confidence", pa.float64()),
    pa.field("category_probs", pa.string()),
    pa.field("churn_prob", pa.float64()),
    pa.field("urgency_level", pa.int32()),
    pa.field("urgency_confidence", pa.float64()),
    pa.field("route", pa.string()),
    pa.field("model_version", pa.string()),
    pa.field("question_set", pa.string()),
    pa.field("source_ingested_at", pa.timestamp("us")),
    pa.field("labeled_at", pa.timestamp("us")),
])


def load_new_tickets(catalog, labels):
    done = labels.scan(selected_fields=("source_ingested_at",)).to_arrow()
    watermark = pc.max(done["source_ingested_at"]).as_py()

    tickets = catalog.load_table("support.tickets_raw")
    scan_args = {"selected_fields": ("ticket_id", "subject", "body", "ingested_at")}
    if watermark is not None:
        scan_args["row_filter"] = GreaterThan("ingested_at", watermark.isoformat())
    return tickets.scan(**scan_args).to_arrow().to_pylist()


def label_ticket(client, ticket, now):
    result = client.system_one(
        state={"subject": ticket["subject"], "body": ticket["body"]},
        questions=QUESTIONS,
    )
    category = result.answers["category"]
    urgency = result.answers["urgency"]
    churn = result.answers["churn_signal"]

    route = "auto" if category.confidence >= AUTO_ACCEPT else "llm_review"

    return {
        "ticket_id": ticket["ticket_id"],
        "category": category.choice,
        "category_confidence": category.confidence,
        "category_probs": json.dumps(category.probabilities),
        "churn_prob": churn.noul,
        "urgency_level": int(urgency.score),
        "urgency_confidence": urgency.confidence,
        "route": route,
        "model_version": result.model,
        "question_set": QUESTION_SET,
        "source_ingested_at": ticket["ingested_at"],
        "labeled_at": now,
    }


def main():
    catalog = load_catalog("lakehouse")  # settings come from .pyiceberg.yaml
    labels = catalog.load_table("support.ticket_labels")
    new_tickets = load_new_tickets(catalog, labels)
    if not new_tickets:
        return

    now = datetime.now(timezone.utc).replace(tzinfo=None)
    with TypeSafeClient(model=MODEL) as client:
        rows = [label_ticket(client, t, now) for t in new_tickets]

    labels.append(pa.Table.from_pylist(rows, schema=LABEL_SCHEMA))


if __name__ == "__main__":
    main()

Walk through the pieces.

The constants at the top are the contract for the whole job. MODEL pins the exact version. The jev-latest alias moves when TypeSafe ships a new release, and TypeSafe's own docs warn that answers behind an alias can change without any change on your side. If you tuned AUTO_ACCEPT against version 1.13, a silent upgrade invalidates that tuning. QUESTION_SET does the same job for your prompts. Change a criteria description and you have changed the classifier, so the version string changes too.

QUESTIONS asks three things in one call. The Choice includes an other option, which TypeSafe recommends whenever the list does not cover every possible input. Without it, the model has to force an odd ticket into one of the real categories. The criteria descriptions are written to separate the options from each other, because the model reads both the option names and their descriptions. The Score levels describe observable situations ("Production is down right now") instead of vague adjectives like "critical," which reduces disagreement between runs.

load_new_tickets implements the watermark. It reads one column from the enrichment table, takes the maximum source timestamp, and filters the source scan with it. PyIceberg pushes the filter down to Iceberg's file-level statistics, so files with only older rows never get opened. On a very large enrichment table, reading the whole timestamp column gets slow. At that point, store the watermark in a small state table or read it from snapshot properties instead.

label_ticket sends only the subject and body as state. It does not send customer metadata, account history, or other columns. TypeSafe documents that accuracy drops as irrelevant detail grows in the state, so the state stays small on purpose. The function stores the full probability distribution, the Noul probability, the Score level, and the versioned model ID that the API reports back.

main runs the calls in sequence for readability. A production version uses the SDK's asynchronous client or a worker pool to stay near the rate limit. It also writes through a staging branch and validates before publishing, as described earlier.

Step 3: Escalate the uncertain rows to an LLM

Rows with route = 'llm_review' go to a generative model. In Dremio, the AI_CLASSIFY SQL function sends text and a category list to a configured LLM and returns one of the categories. The escalation step runs as plain SQL next to the data:

CREATE TABLE lakehouse.support.ticket_labels_escalated AS
SELECT
  l.ticket_id,
  l.category             AS fast_category,
  l.category_confidence  AS fast_confidence,
  AI_CLASSIFY(
    'Which product area is this support ticket about? '
      || t.subject || ' ' || t.body,
    ARRAY['billing', 'integrations', 'performance', 'account_access', 'other']
  ) AS llm_category
FROM lakehouse.support.ticket_labels AS l
JOIN lakehouse.support.tickets_raw AS t
  ON t.ticket_id = l.ticket_id
WHERE l.route = 'llm_review'
  AND l.labeled_at >= DATE_SUB(CURRENT_DATE, 1);

The category list matches the Choice options exactly, so the two answers are comparable. Keeping fast_category next to llm_category gives you a running disagreement dataset for free. Every row where the two models disagree is a candidate for human review and a test case for the next version of your questions. A production pipeline appends to this table with INSERT INTO on each run instead of recreating it.

Step 4: Query with probabilities

The analytics layer combines both tables and uses the probabilities directly:

SELECT
  DATE_TRUNC('WEEK', l.source_ingested_at)          AS week,
  COALESCE(e.llm_category, l.category)              AS final_category,
  COUNT(*)                                          AS tickets,
  SUM(l.churn_prob)                                 AS expected_churn_mentions,
  SUM(CASE WHEN l.churn_prob >= 0.8 THEN 1 ELSE 0 END) AS confident_churn_mentions,
  AVG(l.category_confidence)                        AS avg_category_confidence
FROM lakehouse.support.ticket_labels AS l
LEFT JOIN lakehouse.support.ticket_labels_escalated AS e
  ON e.ticket_id = l.ticket_id
WHERE l.model_version = 'jev-1.13.0'
  AND l.question_set = 'support-v3'
GROUP BY
  DATE_TRUNC('WEEK', l.source_ingested_at),
  COALESCE(e.llm_category, l.category)
ORDER BY week, final_category;

Two churn columns sit side by side on purpose. expected_churn_mentions sums probabilities and gives the best estimate of how many tickets carry churn language. confident_churn_mentions counts only the clear cases, which is the number a customer success team acts on. When the two numbers drift apart week over week, the population of borderline tickets is growing. That is a signal worth investigating on its own.

The WHERE clause filters to one model version and one question set. Mixing labels from different versions in a single trend line produces steps in the chart that reflect model changes, not customer behavior.

In Dremio, you save this as a view in the AI Semantic Layer with a description of each column. That description is what lets an AI agent answer "how many tickets mentioned churn last week" and pick the right column.

A self-hosted variation

When text cannot leave your network, swap the classifier in Step 2 for GLiClass and keep everything else:

from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

name = "knowledgator/gliclass-modern-base-v2.0"
model = GLiClassModel.from_pretrained(name)
tokenizer = AutoTokenizer.from_pretrained(name, add_prefix_space=True)
pipeline = ZeroShotClassificationPipeline(
    model, tokenizer, classification_type="multi-label", device="cuda:0"
)

labels = ["billing", "integrations", "performance", "account_access", "other"]
scores = pipeline(ticket_text, labels, threshold=0.0)[0]
best = max(scores, key=lambda r: r["score"])

Note what changes. GLiClass returns an independent score per label, not a probability distribution that sums to 1. The raw scores are not calibrated, so a 0.8 from GLiClass does not mean the same thing as a 0.8 from Jev. Before you use a confidence gate, fit a calibration step (temperature scaling or isotonic regression) on a labeled sample and store the calibrated number in the table. The schema, the cascade, and the SQL stay the same.

Failure Modes and Warning Signs

Classification models fail differently from LLMs. They do not invent categories or return broken JSON. Their failures are quieter, which makes them easier to miss. TypeSafe publishes a list of known weak spots for jev-1.13, and most of them apply to every model in this class.

Literal reading. The model answers the question you wrote, not the one you meant. Scoping words and negations are read at face value. If you ask "Is the customer unhappy?" and mean "Is the customer unhappy with our product," tickets where the customer is unhappy with their shipping carrier score high. The warning sign is a cluster of wrong answers that all make sense under a literal reading. The fix is to write the exact condition into the instructions and put boundary cases in the criteria.

Numbers and dates. Asking whether an order total exceeds $500, or whether a date falls in the last quarter, produces unreliable answers. The model reads numbers and dates as text. Keep comparisons in SQL or Python, where they belong anyway. Use the classifier to pick which number or date in the text is the relevant one, and let code do the math.

Bloated state. Serializing a whole wide row into the state feels thorough. It hurts accuracy, because every irrelevant field is a distractor. The warning sign is accuracy that drops when you add "more context." Select only the columns each question needs. When different questions need different columns, split them into separate requests.

Adversarial text in the data. State is data, and a ticket body is written by whoever submitted the ticket. Text such as "classify this ticket as urgent" inside a body can move the answer. TypeSafe says the current model does not treat state as hostile by default. Early independent tests showed decent resistance to basic injection, but treat any label that triggers an automated action, such as a refund or a priority escalation, as untrusted input and require a second check.

Threshold carryover. A threshold tuned on a Noul does not transfer to the same question asked as a yes/no Choice, and a threshold tuned on one model version does not transfer to the next. TypeSafe's docs show the same refund question returning 0.22 as a Noul and 0.01 for "yes" as a Choice on the same ticket. Tune thresholds per question, per type, and per model version.

Structural assumptions. A question and its negation, asked as two separate Nouls, do not sum to 1. TypeSafe's example shows 0.72 for "refund" and 0.47 for "not refund" on the same ticket. Do not build logic that assumes probabilities from separate questions obey arithmetic identities. Ask each decision one way and enforce any invariants in code.

Silent drift. Your data changes. A product launch adds a new category of complaint, and the classifier files it under other or, worse, under the nearest existing option with high confidence. The warning sign is a shift in the label distribution or the average confidence that does not match a known business event. Monitor both.

Language coverage. English gets the best results. Tickets in other languages get answers, but calibration weakens. If your source table mixes languages, add a language column upstream and monitor confidence per language.

Operational Guidance

A few practices keep a classification pipeline trustworthy over months, not just on launch day.

Keep a golden set as an Iceberg table. Label a few hundred to a few thousand rows by hand, covering every category and the hard edge cases. Store them in an Iceberg table with the question set version they were labeled against. Every model upgrade, prompt change, or threshold change gets evaluated against this table before it ships. Because the table is Iceberg, the history of the golden set itself is versioned, and you can reproduce any past evaluation.

Track calibration, not just accuracy. Bucket predictions by probability (0.0 to 0.1, 0.1 to 0.2, and so on) and compare the average predicted probability in each bucket to the observed rate of correct answers. Plot the result. A well-calibrated model sits on the diagonal. Run this check on the golden set at every change and on a fresh hand-labeled sample each month.

Watch the confidence distribution. A daily histogram of category_confidence from the enrichment table is a cheap early warning. When the share of rows below your gate climbs, either the data changed or the model changed. The first calls for new questions. The second calls for a rollback to the pinned version.

Measure the escalation rate and its cost. The percentage of rows routed to the LLM is your main cost lever. Track it alongside the disagreement rate between the fast model and the LLM on escalated rows. If disagreement on escalated rows is low, your gate is too strict and you are paying for LLM calls that agree with the cheap answer. Lower the threshold.

Upgrade on your schedule. When a new model version ships, run it on the golden set, then run it on a recent week of production data into a separate branch of the enrichment table. Compare labels row by row using Iceberg time travel or a join between branches. Promote the new version only after the diff looks right, and retune thresholds as part of the promotion.

Isolate the compute. Classification backfills are bursty. On a self-hosted model, they compete for GPU time with anything else on the same cluster. For SQL-based LLM escalation, route AI function queries to a dedicated engine so a large batch does not slow down dashboards. Dremio supports engine routing rules keyed on whether a query calls AI functions, and other engines have their own workload management controls.

Log the request ID. Hosted APIs return a request identifier. Store it in a side table keyed by ticket ID. When a user disputes a label months later, you can trace it back to the exact call.

Where This Is Heading

The launch of Jev did not invent classification. What it did was name a category and ship a clean interface for it, and that interface spread fast. Within two weeks, it showed up in gateways, agent frameworks, observability tools, MCP servers, and a handful of open-source reproductions.

Three trends look likely to shape the next year.

The first is the split between generation and decision becoming a standard part of AI architecture. Agent frameworks already route tool selection, safety checks, and context pruning to fast models. Expect the same split inside data platforms, where query engines call a decision model for row-level tagging and reserve generative models for summaries and extraction.

The second is open models closing the gap. GLiClass and ModernBERT already give self-hosted teams a strong single-pass classifier. The missing piece has been trained-in calibration and a general interface for arbitrary typed questions. Several open projects are attacking exactly that. Treat their current benchmark claims with skepticism until someone runs a matched comparison, but the direction is clear.

The third is classification outputs becoming first-class data. When every free-text column in a lakehouse can be turned into calibrated probabilities for a fraction of a cent per thousand rows, text stops being the part of the data you skip. Open table formats like Iceberg are well suited to hold this derived layer, because they version it, share it across engines, and let you roll it back when a model change goes wrong.

Conclusion

LLMs made text classification easy to prototype and expensive to run. Models like Jev, along with open alternatives such as GLiClass and the new wave of Jev-style projects, move most of that work back to a tool built for it. The answer is a typed value with a calibrated probability, returned in milliseconds for a small fraction of the cost.

The right design is not one model or the other. Put the fast classifier on every row. Use its confidence to decide what goes to an LLM and what goes to a person. Store full probabilities, model versions, and routes in their own Iceberg table, and let SQL and your semantic layer turn those numbers into analytics. Pin versions, keep a golden set, and watch the confidence distribution. Do those things and the text in your lakehouse becomes as queryable as the numbers next to it.

Keep Going

If this piece was useful, I have written a lot more on building AI workloads on top of open lakehouse tables.
Architecting an Apache Iceberg Lakehouse (Manning) covers how to design the table, catalog, and engine layers that pipelines like this one run on.
You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Apache Data Lakehouse Weekly: September 9 to 17, 2026

Alex Merced — Fri, 18 Sep 2026 09:13:23 +0000

Release managers had a humbling week. Iceberg Rust 0.11.0 failed two release votes, Polaris 1.8.0 drew a -1 on its first day, and the Polaris Catalog Migrator needed a second candidate. None of those failures came from broken code paths in the usual sense. They came from license notices, a crate that failed to publish, and an encryption check that one engine skipped and another engine enforced. The lakehouse projects are now mature enough that the hard problems live at the seams: between languages, between engines, between a spec and the many readers that implement it.

That theme runs through every project this week. Iceberg and DataFusion voted to move the Rust DataFusion integration into the DataFusion project, which redraws a boundary that had been drawn in the wrong place. Parquet spent the week arguing about what a reader should do when it meets a file from the future. Polaris debated how to add Cloudflare R2 without bending its S3 configuration into a shape it was never meant to hold. Apache Ossie (incubating) proposed that one document should hold one semantic model, which is a seam question too. And Arrow shipped a Rust release that puts the new ALP floating point encoding into a production Parquet implementation for the first time.

This issue covers the dev lists for Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, Apache DataFusion, and Apache Ossie (incubating), using the Pony Mail archives at lists.apache.org. Every thread link below points to the public archive so you can read the full discussion yourself.

Apache Iceberg

The DataFusion integration finds a new home

The biggest structural change of the week was a clean two-project vote. Kevin Liu opened the Iceberg-side vote to move the iceberg-datafusion crate out of apache/iceberg-rust and into a new apache/datafusion-iceberg repository under the Apache DataFusion project. The move grew out of a discussion thread where Andrew Lamb and Kevin agreed to hold one vote in each community. Gabriel Musat had already done the heavy lifting by porting the code and its full history into a pull request on the new repository.

The Iceberg vote passed with 17 +1 votes, five of them binding, from Kevin, Fokko Driesprong, Renjie Liu, Szehon Ho, and Russell Spitzer. There were no -1 or +0 votes. Xuanwo summed up the reasoning in his +1: DataFusion contributors get a natural place to maintain the integration, and the code stays under Apache governance. On September 16, Andrew confirmed on the same thread that both votes had passed unanimously.

Why does this matter to people who never read Rust? The integration is the glue that lets a DataFusion query plan read and write Iceberg tables. When it lived inside iceberg-rust, every DataFusion upgrade forced a change in the Iceberg repository, and the people who understood DataFusion's planner internals were not the people reviewing the pull requests. Putting the crate next to the engine it plugs into puts the reviewers next to the code. It also sets a pattern. Integration code belongs with the project whose APIs churn fastest, and the table format library can stay focused on the format.

Iceberg Rust 0.11.0 fails twice, and the failures teach something

Danny Jones and Shawn Chang ran the 0.11.0 RC1 vote, and early verifiers reported clean results. L. C. Hsieh ran 2,209 tests on an arm64 Mac with no failures. Anoop Johnson ran the full suite on Ubuntu 24.04, including Docker-based integration tests against MinIO, the REST catalog, Hive Metastore, and Spark. He flagged one flaky test that failed under parallel load with a SQLite "database is locked" error but passed in isolation.

Then Kevin Liu voted -1 for a reason no unit test catches. The new iceberg-property-macro crate depends on nothing unusual, but it lists iceberg itself as a versioned dev-dependency, and iceberg depends on the macro crate. That circle means cargo publish cannot package either crate first. Kevin pointed out that the project hit the same trap with 0.5.0, when the vote passed, the publish failed, and the team had to cut 0.5.1. He put up a fix and added a cargo publish --dry-run check to CI so the problem surfaces on every pull request. He also noted that make test from the tarball failed because the Hive test image's apt repository had expired. Danny agreed and cancelled the vote.

The RC2 vote opened on September 15 and lasted less than a day. Alexander Bailey voted -1 after testing and realizing that the Rust encryption work skipped the tamper-proofing checks that Java requires. Files written by the Rust RC were unreadable in Java. He posted a fix in iceberg-rust PR #3236. Danny closed RC2 and asked reviewers to prioritize the fix so RC3 can follow.

This is the multi-language Iceberg story in miniature. Encryption is one of the headline features in 0.11.0, and the only reliable test for it is a cross-engine round trip. A Rust writer that passes every Rust test can still produce files that a Java reader rejects. The community caught it during the vote, which is the process working. But it also makes the case for the next item.

A shared verification repository goes live

Neelesh Salian announced that the apache/iceberg-verification repository now exists, following an earlier discussion and a successful vote. The repository will hold shared conformance fixtures for every Iceberg implementation. Early work covers type fixtures validated with JSON Schema and golden reference tables. Neelesh asked for reviewers on the open pull requests and for new issues from anyone with ideas about the fixture format or contribution model.

Read that announcement next to the two failed Rust votes. A golden table written by Java and read by Rust, Go, C++, and Python, plus the reverse, is exactly the test that catches an encryption gap before a release candidate exists. The verification repository is still young, but it points at the right problem.

Third-party notices in compiled wheels

Danny Jones raised a licensing gap in pyiceberg-core, the Python package built from iceberg-rust. The wheels ship compiled third-party Rust code, but they do not reproduce the copyright notices for those dependencies. Danny opened iceberg-rust issue #3239 with a proposed fix that generates a THIRD-PARTY-LICENSES file at build time, and he plans to confirm the approach on the legal-discuss list.

Ryan Blue agreed that anything distributed in compiled artifacts needs matching LICENSE and NOTICE content. He was less sure about the THIRD-PARTY-LICENSES file name, but said a pointer at the bottom of LICENSE is reasonable, and a NOTICE file with the legally required notices is mandatory. Shawn Chang pointed to Apache Paimon's Rust bindings as a working example. Their workflow scopes dependencies from the Python bindings' Cargo.toml, generates a report per target wheel, stages it in the artifact, and verifies it. Danny said this must be fixed before new releases of pyiceberg-core, and by extension iceberg-rust, since the two ship together.

Iceberg Java 1.12.0 is ready to cut

Neelesh Salian gave the 1.12.0 release thread two updates. On September 12 he reported that all prior correctness fixes had merged, with one exception: PR #17984, which commits manifest list encryption keys together with the snapshot that uses them. Gábor Kaszab asked whether the missing cleanup mechanism for unused encryption keys in TableMetadata (PR #16353) belongs in this release. On September 16 Neelesh said #17984 had merged, that the key cleanup interface needs more debate and does not need to be rushed, and that the release candidate was ready to cut.

Alex Reid replied on September 17 asking for two more items. One is a Kafka Connect fix (PR #17713) that prevents re-committing already committed files during certain rebalances. The other is the REST client implementation of referenced-by, which the spec added last year. As of this writing, 1.12.0 has not been released. Watch the list for the RC vote.

Defining what gc.enabled means

Alexander Bailey started a careful thread about the gc.enabled table property. His PR #17791 relaxes the check so that snapshot expiry can proceed in a metadata-only mode when garbage collection is disabled. Reviewers pushed back, and Alexander concluded that the real issue is that no one has defined the property. It is not in the spec. He catalogued at least three meanings inside the Java repository alone. CatalogUtil.dropTableData skips data files but deletes metadata. Several Spark actions refuse to run at all. The Hive helper maps the flag onto Hive's external table purge setting. He then showed that iceberg-rust, iceberg-go, and iceberg-cpp each copied Java's strictest check, one of them word for word, while PyIceberg has no check at all.

Russell Spitzer explained the history. The flag arrived when Iceberg started snapshotting existing data, where Iceberg owns new metadata but another system still owns the data files. The invariant is simple: do not remove this table's data files. He was happy to write that down in a spec appendix, and he did not see the Java behavior as ambiguous, since every operation that fails or skips work is one that can delete a data file. Daniel Weeks took a different position. He sees gc.enabled as convention rather than specification and prefers to rely on catalogs to enforce deletion rules than on clients to behave.

Alexander asked Dan whether that means catalogs should vend credentials without delete permission and leave deletion to catalog-managed maintenance. He then narrowed his PR. When gc.enabled=false, the PR allows expiry only with CleanupLevel.NONE, which removes snapshot entries without touching any file. The default mode and METADATA_ONLY stay blocked. That is a sensible scoping move. It keeps the data-file invariant intact and leaves the bigger enforcement question for a separate discussion.

Faster position delete checks in the vectorized reader

A contributor writing as 전대홍 posted a performance proposal after Eduard Tudenhöfner suggested taking it to the list. Spark's vectorized reader asks the position delete index whether each row is deleted, one row at a time, while building the row-id mapping for a batch. Positions in a batch form a contiguous range, so every call repeats the same key extraction, bounds check, and binary search through the Roaring bitmap containers.

PR #18027 adds a forEachInRange default method to PositionDeleteIndex. The default keeps the current loop, so external implementations keep working. BitmapPositionDeleteIndex overrides it to resolve the range to at most two underlying bitmaps and walk each once. The contributor measured 2.6x to 9.3x less delete-check CPU, with end-to-end gains of 13 to 19 percent on narrow integer projections and noise-level gains at ten or more columns. They noted that the largest win lands just below 6.25 percent delete density, where the Roaring container is still a sorted array.

Péter Váry suggested trying a batch iterator in the style of RoaringBitmap's batch iteration. The contributor built it and benchmarked six variants on a 5,000-row batch. The PR's approach came in at 0.45 microseconds per batch on sparse deletes, against 64.99 for the current code and 0.63 for the batch iterator. They kept the minimal API change and offered to test a persistent iterator if the community prefers that design.

Row-level concurrency for V3 deletion vectors

EJ Song and Huaxin Gao continued a thread on row-granular conflict detection for deletion vectors. Today, two concurrent UPDATE or MERGE commits that touch the same data file conflict even when they delete different rows. Huaxin supported the direction but warned that the extra check adds driver-side reads during commit, and suggested gating it behind a table property. She pointed to prior art, including Iceberg PR #17754, which already merges concurrent deletion vectors for pure delete commits.

EJ laid out a layered design. An operation gate comes first, then a metadata check on partition and column bounds, and only then a single small bitmap read when two commits really overlap on one file. Commits on disjoint files read nothing. EJ argued that the bitmap tier does not need a table property because its cost is already bounded. A later tier that compares actual values to resolve overlaps a bitmap cannot decide reads data files, and that tier is where a property makes sense. This is the kind of change that makes V3 tables far friendlier to high-concurrency streaming and CDC workloads.

Access delegation for the FILE type

Sung Yun opened a discussion on access delegation for the proposed FILE type, which lets an Iceberg column reference external objects such as images or documents. Remote signing and vended credentials work when a storage client reads the object. Sung's concern is the multimodal case, where an inference service outside the query engine needs to fetch the object. He proposed letting clients ask the catalog to pre-sign FILE reference URLs, with a spec PR (#18080) and a client proof of concept (#18110). He had earlier posted a related implementation that handles pre-signed URLs returned in place of native paths in a plan response.

Prashant Singh noted that he and William Hyun opened a pre-signed URL proposal three months ago. Their proofs of concept, including one on Azure, showed limits in remote signing across clouds. He asked to build on those tracks rather than start parallel ones, and Sung agreed. Daniel Weeks drew a useful line. Pre-signed URLs from plan tasks arrive ready to use, so file IO only needs to detect and follow them. Remote pre-signing is an earlier step, where a native s3:// path goes to the catalog for signing before a stream opens. Dan wants remote pre-signing to become a real alternative to remote signing, largely because of Azure's limits, and does not think it needs a new /presign endpoint. He also said he disagrees with the direction of the current bulk-signing proof of concept and wants the FILE sync to align the designs.

Smaller threads worth a look

Shangqing Yang called a vote to add the optional key-id field to the Snapshot schema in the REST OpenAPI spec, which aligns the spec with what SnapshotParser already supports. Sreesh Maheshwar revived the question of replacing MinIO in tests after MinIO images disappeared from Docker Hub and broke CI. Kevin Liu, Xuanwo, and Szehon Ho backed moving to RustFS, which PyIceberg and Polaris already use. Neelesh Salian asked for long-term support and prompt CVE fixes, citing an earlier security concern.

Renjie Liu gave a +1 to a proposal for a standard User-Agent format for REST clients, and Kurtis Wright asked how the format should order vendor, engine, integration, library language, and runtime. A contributor asked for review of an ObjectStore-based S3 backend for iceberg-rust in PR #3165. Péter Váry reported from the index support sync that the group will prioritize scalar indexes before other index types, that index data lives in region files tracked by a tracking file, and that the scalar index spec PR #16961 is ready for review. Huaxin Gao scheduled one more constraint sync for September 17 and asked for reviews of the CHECK constraint spec PR #17822.

Ryan Blue shared the September board report with a new format that summarizes shipped releases and spec changes instead of sync highlights. It lists 40 committers and 25 PMC members, and releases including Terraform Provider 0.1.0, PyIceberg 0.12.0, Rust 0.10.0 and 0.10.1, and C++ 0.3.0. Ryan asked release managers to write real highlights in their vote and announce emails, and Danny Jones committed to doing that for iceberg-rust. On September 17, Fokko Driesprong announced that Gang Wu has joined the Iceberg PMC, crediting Gang's work on the inception and growth of Iceberg C++.

On the community side, Kevin Liu announced the first Apache Iceberg Virtual Meetup, covering GSoC projects and commutative compaction on Friday, September 18. Alex Stephen announced a Seattle community meetup on October 21, with session ideas due October 6. Sung Yun, Walaa Eldin Moustafa, and others volunteered for the Iceberg Summit 2027 selection committee.

Apache Polaris

Polaris 1.8.0 reaches a vote, and a NOTICE question stops it

Jean-Baptiste Onofré opened the vote for Apache Polaris 1.8.0 RC0 on September 17, after flagging on the proposal thread that he was preparing it. The candidate ships source tarballs, Helm charts, a Python CLI wheel on Test PyPI, and staged Maven artifacts. Within hours, Dmitri Bourlatchkov voted -1. Bundled Iceberg jars such as iceberg-api-1.11.0.jar carry a NOTICE that references Kite, and commons-math3-3.6.1.jar carries one that references Orekit. The Polaris bundle NOTICE does not propagate either. Dmitri asked whether those notices belong in the Polaris bundle, and the vote is open as of this writing.

The same pattern hit the Polaris tools repository. Ajantha Bhat's Iceberg Catalog Migrator 1.1.0 RC1 drew a -1 from Dmitri because the CLI jar bundles a taglib.tld file with an Oracle copyright header under GPL v2 with the Classpath Exception, and the bundled LICENSE does not mention it. Ajantha posted RC2 on September 17.

If you are keeping score, that is four release candidates across Iceberg Rust and Polaris this week that stopped on licensing or packaging, not on logic. Fat jars and compiled wheels pull in other projects' legal text, and the ASF requires it to travel with the artifact. The reviewers doing this checking are doing unglamorous, important work.

Cloudflare R2 support and the shape of S3 configuration

The most instructive design thread on the Polaris list started with a first-time contributor. Austen Tomek from Chicago Trading Company proposed R2 support with scoped credential vending. R2 speaks the S3 API but has no AWS STS. Cloudflare instead lets a server holding a parent API token sign short-lived, bucket-scoped credentials locally. Austen's prototype does that inside Polaris, maps Polaris location grants to prefix and access scopes, and reuses the existing credential cache.

Yufei Gu confirmed the model: Polaris issues the credentials without calling Cloudflare, the client signs its own S3 requests, and R2 enforces scope and expiry. His first instinct was to treat R2 as S3-compatible storage without a new config type, and he asked whether unmodified clients work. Austen reported that they do. PyIceberg 0.11.1 and 0.12.0, DuckDB 1.5.5, and Iceberg Java 1.11.0 through the stock RESTCatalog and S3FileIO all read and wrote without any R2-specific client code. Spark, Trino, and Flink are not tested yet.

The debate then split along a clear line. Sushant Raikar laid out two options, a first-class R2 type with its own FileIO or R2 as S3-compatible storage, and pushed for a lowest common denominator across S3-compatible backends. Jean-Baptiste argued for building under the existing S3 config and noted that AwsStorageConfigurationInfo already carries endpoint, stsEndpoint, pathStyleAccess, and an stsUnavailable flag. He framed the real design question as a pluggable vending path for S3-compatible stores without STS. Dmitri leaned toward a separate R2StorageConfigInfo, since mixing Cloudflare account IDs into an AWS-shaped class blends unrelated concepts.

Prithvi S offered the cleanest synthesis. Keep the client side as plain S3, with s3:// locations and S3FileIO. On the server side, avoid overloading stsUnavailable, because that flag means "vend nothing" today, and R2 needs the opposite: vend, just not through STS. Reusing it silently changes behavior for existing MinIO and NetApp catalogs.

Austen closed the loop on September 14 and again on September 16. R2 has no IAM roles or ARNs, so he left those fields empty. He added one new S3 field, now named credentialVendingMechanism, with a server allowlist, SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, that defaults to STS only. Following Dmitri's suggestion, mechanisms are CDI beans looked up by identifier, so downstream builds can add their own without changing the API spec. Stage one is PR #5513, and the R2 bean follows in a second PR. Yufei noted the underlying mismatch: the S3 storage type mixes shared S3 settings with AWS-specific ones, and a future refactor can split them. This is a good template for how to add a backend to a catalog without special-casing it.

Pagination moves from optional to default

Pagination came up in four threads. Yong Zheng opened separate discussions for the generic table API and the policy API. In both cases the REST spec advertises page-token and page-size, but the server calls PageToken.readEverything() and ignores them. For generic tables, Prithvi and Jean-Baptiste steered toward a default method on the GenericTableCatalog SPI, since federated Hive, Hadoop, and BigQuery catalogs implement that interface and a concrete-class overload is unreachable from the handler. Yong posted PR #5533. For policies, the group agreed to paginate listPolicies now, including when filtering by policy type, and to handle applicable-policies separately.

Ayush Saxena raised a spec tension around a configurable server-side maximum page size. The Iceberg REST spec says a server must return everything in one response when the client sends no page token. A server cap truncates that response, and an older client that ignores next-page-token reports a partial list as complete. Dmitri called server-side truncation a basic overload control and a reasonable deviation, since administrators can choose the setting per deployment. He proposed merging.

Then Yufei proposed turning LIST_PAGINATION_ENABLED on by default. Jean-Baptiste noted that the maximum page size defaults to unlimited, so clients that send no pagination parameters see no change. Ayush, Yong, Nándor Kollár, and Dmitri all backed keeping the flag as an escape hatch for a couple of releases, and Yong opened PR #5534. EJ Wang raised the same question for the new Tag API, which is covered below.

Read replicas, and the consistency they break

Yong Zheng started a thread on offloading read-only requests to a database replica when a client sends a custom header. His motivation is connection math on the JDBC backend: at 50 connections per pod and a 5,000-connection database limit, a deployment tops out near 100 pods.

The pushback was about correctness. Prithvi pointed out that the load test Yong cited ran at roughly 400 requests per second sustained with mostly idle connections, so connection limits were not the bottleneck. He also reminded the group that Iceberg clients expect to read their own writes, and asynchronous replicas break that. Jean-Baptiste argued against header-based routing, since Spark and Flink will never send a Polaris-specific header, and against verb-based routing, since some reads in JdbcBasePersistenceImpl have side effects such as idempotency-key bookkeeping. He suggested routing at the persistence method level. Dmitri was more open to an explicit opt-in header, perhaps renamed to make the staleness risk obvious, and warned that the shared entity cache will misbehave with two connection pools. Yufei asked for measurements first: connection wait times, database query time, and metadata file IO, which often dominates loadTable.

Location overlap checks

Dmitri Bourlatchkov opened a discussion on hasOverlappingSiblings(). With the OPTIMIZED_SIBLING_CHECK flag on, persistent implementations search the whole catalog. With it off, LocalIcebergCatalog searches only immediate siblings. Dmitri argued that a speed-up flag should not change what a validation checks, and proposed narrowing the implementations. Prithvi and Eric Maynard disagreed. Eric recalled that the catalog-wide search was an intentional fix for a security gap under certain configurations. Jean-Baptiste identified the concrete case: with ALLOW_UNSTRUCTURED_TABLE_LOCATION, a table under one namespace can point inside another namespace's tree, and only the catalog-wide check sees it. He prefers a confusingly named flag over a correctly named, weaker check. Dmitri agreed with catalog-wide checks and asked why that coverage is not the default.

Tags, semantic model privileges, and the CLI

EJ Wang posted progress on the Tag spec. PR1 covers the API contract, PR2 covers definition CRUD, and a new PR3 adds assign and unassign for catalogs, namespaces, tables, and top-level columns, with atomic detach-all on JDBC and in-memory backends. Version tokens are opaque, and stale updates return 409. Dmitri asked for persistence SPI details and a dedicated section on new authorizer operations. EJ later asked the community to pick between keeping Tags under the existing catalog URL root or giving them an independent /api/tags/v1 root, and between opt-in and always-on pagination.

Yufei Gu asked for feedback on dedicated semantic model privileges in PR #5492, splitting the single CATALOG_MANAGE_CONTENT privilege into list, create, read, write, drop, full metadata, and grant management. Prithvi supported it and caught that NAMESPACE_FULL_METADATA and CATALOG_FULL_METADATA should not act as umbrellas, and Yufei fixed that. Jean-Baptiste strongly backed dedicated privileges that mirror other entity types. Semantic models in a catalog, with their own grants, connect directly to the Ossie work later in this issue.

A GitHub discussion asking for table and view support in polaris setup export/apply became a CLI thread. Yong proposed adding register and create. Prithvi argued that register, which takes a name and a metadata.json location and preserves history, is the primitive operators need. Jean-Baptiste found that the generated REST client already exposes register, so the change is mostly wiring. Dmitri and Yufei agreed that create is outside the CLI's scope for now. The sequence the group settled on is register first, then export and apply with table and view grants, and file-based create later if anyone needs it.

Rounding out the week, Dmitri put consistent multi-object persistence changes on the September 17 community sync agenda. Arun Suri agreed to defer his ambiguous JDBC commit fix until PR #5263 merges, then add a reconciliation step that reloads the entity and treats a matching metadata location as success. And Yufei, Ayush, Robert Stupp, and Jean-Baptiste discussed GitHub Actions queue delays caused by ASF-wide runner saturation. Ayush tested which short jobs can move to ubuntu-slim, and the group decided to hold off unless delays return.

Apache Arrow

Arrow Rust 60.0.0 ships the first ALP-enabled Parquet implementation

Andrew Lamb called the vote for Arrow Rust 60.0.0 on September 10, and it passed on September 15 with five +1 votes, three binding, from L. C. Hsieh, Jeffrey Vo, and others. Andrew did not undersell it. He called the release "epic" and wrote that arrow-rs is, as far as he knows, the first Parquet implementation to include the new ALP encoding. He credited Kosta Tarasov and Devan for leading that work, Ed Seidl for the new Parquet PageIndex structures, and Richard Baah and others for a long list of performance improvements. The crates, including arrow and parquet 60.0.0, are on crates.io.

ALP, short for Adaptive Lossless floating-Point compression, targets float and double columns, which general-purpose encodings handle poorly. Sensor readings, prices, and model features all live in those columns. Having a shipped implementation in Rust matters because DataFusion, iceberg-rust, and a long list of Rust-native engines consume the parquet crate directly.

Xuanwo's review of the release is worth reading for anyone who runs release votes. He first voted +0 because arrow-array/src/delta.rs contains MIT-licensed code derived from chronoutil, and the root LICENSE.txt does not identify it. After another look he decided a NOTICE mention is enough and changed to +1. Andrew filed issue #11098 to track the fix for the next release.

Object Store 0.14.2 needs a second candidate

The Rust object_store crate had a similar week. Andrew's 0.14.2 RC1 drew a -1 from Xuanwo because src/client/s3.rs was missing the first line of its ASF license header. Andrew traced the defect back more than three years and asked whether it needed a new candidate, since it was not a regression. Xuanwo moved to +0 but noted the vote had just started and a wrong license header is a real risk for downstream users. Andrew cut RC2 with the header restored, and it passed on September 15 with three binding votes. Version 0.14.2 is on crates.io. object_store sits under DataFusion, iceberg-rust, and many other Rust data tools, so its releases propagate quickly.

A canonical range type and Lakehouse Day EU

Rok Mihevc revived the proposal for an arrow.range canonical extension type for bounded ranges. Florian replied that he still wants to finish it and addressed Felipe Oliveira Carvalho's earlier suggestion to use separate types for open and closed bounds. After reading how Arrow C++ resolves compute kernels, Florian concluded that separate types help less than they seem. Every extension type reports Type::EXTENSION, so a kernel matcher needs a custom check on the extension name either way. A single arrow.range with a closed flag is the same amount of work, and the flag is read once when the type is reconstructed, not on every compute call.

Danica Fine invited the Arrow community to Lakehouse Day EU 2026 on Saturday, October 10 in Glasgow, co-located with Community Over Code. The full-day, multi-track event covers Iceberg, Flink, Spark, Arrow, Parquet, Polaris, Gravitino, Paimon, Hudi, XTable, Fluss, and more. The schedule includes a talk revisiting an Arrow-based client protocol redesign and a closing panel on the future of the open lakehouse.

Apache Parquet

Format 2.14.0 is released

Fokko Driesprong announced on September 11 that the Parquet Format 2.14.0 vote passed, with binding +1 votes from Gang Wu, Gábor Szádovszky, Andrew Lamb, and Fokko, plus non-binding votes from Divjot Arora and Vinoo Ganesh. Fokko thanked László for correcting links that had a copy-paste error. Andrew followed up with website pull requests that document the 2.14 changes, including ALP and the FILE logical type, and asked for reviews.

On the same day, Divjot Arora closed the vote on Extended Precision Nanosecond Timestamps with seven +1 votes, five binding, from Daniel Weeks, Micah Kornfield, Ryan Blue, Gábor Szádovszky, and Fokko. The change will merge as parquet-format PR 601. Nanosecond timestamps with a wider range matter for anyone storing high-frequency trading or telemetry data that also needs to represent dates far from the epoch.

ALP implementations keep moving across languages. The September 9 sync notes from Andrew Lamb report that the Rust implementation is merged, C++ is waiting on approval after several review rounds, and Java is under review. Vinoo Ganesh posted an update on the Java side: he split parquet-java PR #3791 out of the main ALP PR #3397 based on Russell Spitzer's feedback, and plans to merge the smaller PR first. The sync notes also recorded progress on the vector logical type. The group roughly agreed that the finite-element requirement is a MUST, that statistics help but readers must not depend on them, and that a LIST-based layout is the more popular option, though without full consensus.

What should a reader do with a file from the future?

The week's defining Parquet debate began when Ryan Blue split a question out of the versioning proposal thread. Daniel Weeks had summarized that thread's community sync discussion, noting that the real disagreement is less about recording a version in the file and more about what guarantees readers get. Ryan framed three options. Option 1: a reader must fail on a file written with an unsupported format version. Option 2: a reader should attempt to read it. Option 3: leave it to implementations.

Ryan was careful to scope the question. He is not talking about "preview" features like new encodings or forward-compatible logical types, which already fail only when a reader projects the affected column. The question covers changes to metadata semantics, such as making path_in_schema optional, adding offset and size fields to page headers so page data can move, or fixing statistics written with the wrong sort order. In a separate reply, Ryan said he strongly supports option 1. Option 2 requires a guarantee that every future change will either fail or read correctly in every existing reader, including custom Thrift parsers built for speed whose handling of missing required fields nobody knows.

Andrew Lamb wrote a clear summary of the trade-off. Option 1 makes the spec easier to change but forces readers to reject files they know how to read. Option 2 lets readers handle more files but makes every future spec change carry a compatibility burden that is hard to define. Andrew's view is that Parquet is implicitly option 2 today, and that changing it now creates confusion. Ryan disagreed: the current state is that Parquet avoids breaking changes altogether, using tricks like a second set of min and max fields when the string sort bug surfaced, and he wants a real mechanism for making breaking changes.

Antoine Pitrou questioned the "preview" label and asked whether Thrift parsers actually reject payloads missing a required field. Will Edwards argued for what he called option 0: keep Thrift-based evolution, bump the parquet-format package version as a hint to implementers, and do not gate files on a version. He noted that one widely used reader does not even check the trailing PAR1 magic bytes. Ryan replied that he voted against making path_in_schema optional precisely because it breaks existing guarantees.

Later in the week, the momentum shifted toward option 1. Kurtis Wright said a clear failure with an actionable error is the better user experience, and that implementations choosing to skip version checks accept the consequences. Xiening Dai voted for option 1, opposed changing the magic bytes because many tools use them to detect file type, and proposed shipping a reader patch that fails fast on newer versions before any breaking change lands. Fokko Driesprong also backed option 1 and suggested folding the path_in_schema footer trim into the proposed new footer, so all the breaking changes arrive together in one version.

This debate matters well beyond Parquet. Every table format, every query engine, and every lakehouse catalog depends on Parquet readers behaving predictably. A strict version gate makes the format easier to improve, which is how Parquet gets smaller footers and faster metadata. It also means operators will need to track reader versions across every engine in their stack before they turn on new writer features.

New type proposals and forward compatibility

Neelesh Salian proposed WIRE, a logical type for serialized wire-format messages such as Protocol Buffers and Thrift compact. Today, teams either store those messages as opaque blobs, explode every field into columns with a full decode on write, or convert to Variant and carry a field-name dictionary that protobuf never needed. WIRE stores the original message verbatim in a value column, so it round-trips byte for byte, and shreds the queried fields into native columns using the Variant shredding layout. Because protobuf and Thrift address fields by number, no metadata column is needed, and a reader that does not know WIRE sees ordinary columns.

Micah Kornfield pushed the decimal floating-point proposal toward a formal design document. He listed two open issues: how to represent normalization, possibly through statistics, and how to represent the value itself, which needs benchmarked alternatives to the IEEE 754 layout. Jiayi Wang sent a reminder about the footer sync and shared the updated Modular Footer proposal. Micah asked her to start a DISCUSS thread for visibility and to either broaden issue #530 or open a new proposal.

Gábor Szádovszky and Jörn Horstmann continued a thread on rewriting files with unknown metadata. Gábor argued that a rewriter that does not understand a logical type such as a shredded VARIANT will produce a useless tangle of structs and lists, so it should fail. Jörn pointed out that unknown column orders affect every column in a file, with no way to mark just one column, and that arrow-rs uses an internal unknown variant it cannot serialize. Gábor said he is open to an UNKNOWN column order that makes min and max unusable. Michael Chavinda also asked for a reviewer to add DataHaskell's Parquet reader to the implementation support matrix.

Apache DataFusion

The iceberg integration arrives

On the DataFusion side, Andrew Lamb opened the vote to accept apache/datafusion-iceberg, and it passed on September 16 with 14 +1 votes. Andrew called it a first step toward better Iceberg integration. The GitHub traffic on the dev list showed the work already underway, with Gabriel Musat's PR porting the iceberg-rust history and a second PR adapting the project to compile and pass tests in its new home.

For DataFusion users, this means Iceberg table support will evolve on DataFusion's release cadence, reviewed by people who know DataFusion's physical planning and pushdown APIs. For Iceberg users who embed DataFusion, it means one fewer version lock between two fast-moving Rust projects.

Releases, LTS, and CI

Three release votes closed. DataFusion 55.1.0 passed with nine +1 votes, six binding, and Tim Saucer ran the release. Xuanwo's verification ran 11,100 tests and 502 SQL logic test files, and he flagged several .slt files missing license headers, then fixed them himself in PR #25182. The sqlparser-rs 0.63.0 vote drew a -1 from Xuanwo for a missing NOTICE file and a bundled flamegraph SVG containing CDDL-licensed JavaScript. Andrew showed both issues were years old, filed fixes, and asked Xuanwo to reconsider. Xuanwo moved to +0, and the release passed on September 13 with five binding votes.

Andrew also asked whether DataFusion needs LTS versions, motivated by making third-party integrations easier. That question lands at the right moment, given that DataFusion now hosts the Iceberg integration and has a proposal to host Variant support too. Kosta Tarasov started that Variant discussion around PR #22908, which brings datafusion-variant into the main repository as an extension crate. Andrew suggested keeping it external with better cookbook documentation and moved the conversation to issue #21301. Kosta's open questions include whether Spark semantics should be the naming baseline.

Andy Grove enabled a GitHub merge queue for DataFusion Comet to cut CI load. Pull requests now run a smaller tier, while Spark 3.4, 3.5, and 4.0 SQL tests and older Iceberg suites run once in the queue on the exact merged tree. He also proposed Comet 1.1.0 before the end of September, about six weeks after 1.0.0, and mentioned that he used Claude to triage open high-priority bugs for the release. Tim Saucer announced that Dewey Dunnington is a new DataFusion committer. Andrew's sync notes covered subquery decorrelation, a new blocked aggregation proposal in issue #24704 that asks aggregate authors to review the API, and automating backports.

Apache Ossie (incubating)

The first release is a plumbing exercise

Jean-Baptiste Onofré set expectations for the first Ossie release. It will be 0.3.0, not a 1.0 spec, and its main purpose is to prove the release machinery and complete a full legal check, with a source distribution only. Russell Spitzer agreed that the team should treat it as practice for license checking and release management. Markus Weimer offered help. Kurt Stirewalt asked that the ontology working group's PR #332, which adds two optional fields to the ontology spec, be included, and asked what the formal process is for proposing features for a release.

Microsoft's Power BI group plans to announce its participation

Markus Weimer asked for guidance because Microsoft's Power BI group wants to announce its commitment to Ossie at the upcoming Fabric Community Conference. The group plans to build a converter between Power BI semantic models and Ossie and contribute it. Russell said stating an intent to contribute and support the project is fine as long as no one claims control. Josh Klahr suggested adding Microsoft to the Ossie ecosystem page. Jean-Baptiste pointed to the incubator's branding and publicity guides and recommended "co-creator" or "co-author" language that puts the community first.

A BI vendor committing to a bidirectional converter is exactly the kind of adoption signal a semantic model spec needs. Semantic layers only become portable when the tools on both ends read and write the same format.

One model per document, and richer modeling

Yufei Gu proposed that each Ossie JSON or YAML document hold exactly one semantic model, with its fields at the document root instead of inside a semantic_model array. Several converters already warn and convert only the first model in a document. Jean-Baptiste supported it and asked for a sequencing plan so converter fixtures do not break in the gap. Khushboo Bhatia agreed, and Yufei opened converter PR #396 to merge right after the spec change.

The modeling discussions went deep. On hierarchy support, Yufei shared a Mondrian-style example separating attributes from hierarchies. Julian Hyde credited the attribute-based hierarchy model to Analysis Services 2005, noted that each level must be functionally dependent on its parent, and said he now sees hierarchies mostly as presentation hints. Josh Klahr asked whether hierarchies should be navigation aids or real containment objects. Yufei argued that drill-down needs to know which states belong to which country, which is more than a hint.

On metrics trees, a contributor proposed an optional depends_on array for metric lineage. Julian Hyde replied that dependency information is derivable, that redundant data causes its own headaches, and that Ossie should invest in being a real language with a formal specification, compliance tests, and a compiler that produces the dependency tree. Chris Eubank proposed adding the SQL-standard FILTER (WHERE ...) aggregate modifier to the expression language in PR #382. Mario De Felipe raised, in the cardinality discussion, how to declare source values that mean "missing," such as SAP's type-specific initial values, and filed issue #401.

Agents as spec users

Marco Ciavarella from Exmergo proposed a dedicated place for "field reports" written by coding agents that build with Ossie. His first report, drafted by Claude Code, mirrored a production dbt MetricFlow layer into one Ossie document with 11 datasets, 77 fields, 15 relationships, and 27 metrics. Marco's argument is that agents generate fragmented issues, and a structured report captures what worked, what confused the agent, and what workarounds it needed. Jean-Baptiste liked the separation between blocking and costly issues and the explicit "didn't test" section, and suggested a GitHub discussion category.

Contributions kept flowing too. Poorva Barve introduced herself and sent fixes for the Snowflake converter, which invented a fake table when a query source began with a comment, and the dbt converter, which mangled compound aggregate arguments like SUM(orders.gross - orders.tax). Other threads proposed a bidirectional OpenMetadata converter and LinkML converters that open a path to RDFS and OWL. Ankit Tandon shared notes from the Ontology working group.

Cross-Project Themes

Release hygiene is now the bottleneck. Count the stopped votes: Iceberg Rust RC1 on a publish cycle, Iceberg Rust RC2 on a cross-engine encryption check, Polaris 1.8.0 on bundled NOTICE files, the Polaris Migrator on a bundled GPL file, Arrow Object Store RC1 on a license header, and sqlparser-rs on a NOTICE file and CDDL JavaScript. Add Danny Jones' pyiceberg-core wheel notices and Xuanwo's MIT attribution note on Arrow Rust 60. The pattern is clear. Rust crates, Python wheels, and fat jars all bundle third-party code, and every one of them has to carry the right legal text. Xuanwo, Dmitri Bourlatchkov, and Danny Jones are doing verification work that protects every downstream user. Projects that automate license reports in CI, the way Shawn Chang described for Paimon, will spend fewer weeks restarting votes.

Cross-implementation testing is the missing layer. The Rust encryption gap, the gc.enabled inconsistencies across five clients, and Parquet's reader version debate all describe one problem. A spec is only as reliable as the least careful implementation of it. The new iceberg-verification repository is Iceberg's answer. Parquet is choosing between a strict version gate and a softer contract. Ossie's converter bug fixes and agent field reports are the same problem appearing early, while the spec is still small enough to fix cheaply.

Integration code is moving to the engine. The iceberg-datafusion move, the DataFusion Variant discussion, and Andrew Lamb's LTS question all ask where integration code should live and how stable the host needs to be. The answer this week was to move integrations next to the APIs they depend on and give third parties a stable release line to build against.

Semantics are becoming catalog objects. Polaris is adding dedicated semantic model privileges. Ossie is simplifying its document shape while a major BI vendor plans a converter. Iceberg is designing FILE-type access delegation for multimodal inference services. The lakehouse catalog is growing from a table registry into the place where business meaning, access policy, and non-tabular data references all live together. That is good news for anyone building agents, which need exactly that combination of context and control.

Looking Ahead

Watch for Iceberg Java 1.12.0 RC1, which Neelesh Salian said is ready to cut, and for Iceberg Rust 0.11.0 RC3 once the encryption fix in PR #3236 merges. The Polaris 1.8.0 vote will turn on how the community resolves the NOTICE question, and the Catalog Migrator RC2 vote runs through the weekend. On the Parquet side, the reader versioning thread appears to be converging on option 1, and a formal decision will shape how the new modular footer ships.

The Iceberg virtual meetup on GSoC projects and commutative compaction runs Friday, September 18. The Iceberg constraint sync meets September 17, and the Polaris community sync the same day takes up consistent multi-object persistence. Further out, Lakehouse Day EU lands in Glasgow on October 10, and the Seattle Iceberg meetup follows on October 21.

Keep Learning

If you want to go deeper on Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, and the agentic lakehouse, I have written books that cover the architecture, the catalogs, and the practical engineering behind all of it. You can find every one of them at books.alexmerced.com.

AI Weekly: DeepSeek Cuts Prices as Agents Go Hosted

Alex Merced — Thu, 17 Sep 2026 20:07:12 +0000

Week of September 10 to 17, 2026

The first ten days of September belonged to the frontier labs. This week belonged to everyone else. DeepSeek shipped the month's only price cut, Sakana split its orchestrator into a cheap tier and a premium tier, and Shanghai AI Lab dropped a 744B open-weight research agent with almost no fanfare.

The tooling news moved in one direction. OpenAI, Anthropic, and GitHub all shipped features that run agents for you, measure what they cost, and control what they are allowed to do. And at the AI Infra Summit in Santa Clara, Intel CEO Lip-Bu Tan said the memory shortage behind all of this will get worse next year, not better.

Here is what happened, in the usual order: models, tooling, standards, and infrastructure.

Models: DeepSeek V4.1 Flash Resets the Cheap Tier

DeepSeek V4.1 Flash

DeepSeek released V4.1 Flash on September 10. It is the smallest model in a new architecture family, and it ships with native visual understanding. The API model name is deepseek-flash. Weights are on Hugging Face under the MIT license, along with a technical report.

The architecture is the headline. V4.1 Flash is a 552B-parameter mixture-of-experts model built on what DeepSeek calls a causal encoder-decoder design. It activates only 8B parameters when reading input and 16B when generating output. That split matters because agent workloads are input-heavy. A coding agent re-reads the same files and conversation history on every step, so most of its tokens are prefill, not decode.

The second headline is the KV cache. DeepSeek says V4.1 Flash needs one quarter of the high-bandwidth memory (HBM) and one eighth of the SSD storage that the previous generation used for its cache. DeepSeek ties that directly to price, noting that cache-hit charges often make up a large share of agent costs.

The rate card reflects the architecture. According to a tracker that reads each lab's pricing docs, peak rates are $0.30 per million input tokens and $1.20 per million output tokens, down from $0.44 and $1.32 for V4 Flash. The cache read dropped from $0.014 to $0.006, a 57 percent cut. Off-peak rates are half of peak. The same tracker lists a 1M-token context window and 384K maximum output. The official numbers live on the DeepSeek pricing page, and the new rates took effect at 04:00 UTC on September 10.

Benchmarks are vendor-reported. DeepSeek's launch post says V4.1 Flash beats flagship models, including its own V4 Pro, on its benchmark set. The tracker above lists DeepSeek-reported scores of 90.6 on Terminal-Bench 2.1, 74.2 on DeepSWE v1.1, 90.9 on GPQA Diamond, and a 3471 Codeforces rating. None of those have been independently reproduced yet. Terminal-Bench 2.1 is also two major versions behind the Terminal-Bench 4.0 board where GPT-6 Astra and Claude Fable 5.1 were measured earlier this month, so the numbers do not line up with that ranking.

There are breaking changes. DeepSeek retired V4 Flash and V4 Flash Vision Exp. The old model IDs deepseek-v4-flash and deepseek-v4-flash-vision-exp now route to V4.1 Flash for compatibility. DeepSeek's launch post also said that starting at 04:00 UTC on September 14, all deepseek-v4-pro requests were set to route to V4.1 Flash until V4.1 Pro launches. The tracker reports that DeepSeek's API changelog now says V4 Pro service continues past that date in response to user demand. If you pinned V4 Pro, check the changelog before you assume which model is answering.

What it changes for practitioners: The cheapest capable tier just got cheaper on exactly the line that agent loops burn. If you run long agentic jobs on a budget, test V4.1 Flash against your current model with your own evals. Two caveats apply. DeepSeek's peak window follows Beijing business hours, not yours. And a model ID that silently routes to a different model is convenient for migration but bad for reproducibility, so log the model that actually served each request.

Sakana Fugu Max and Fugu Ultra v2

Sakana AI released Fugu Max and Fugu Ultra v2 on September 11. Fugu is not a single foundation model. It is a trained orchestrator that reads a query, builds an agent scaffold on the fly, and routes the work across a pool of open-weight and specialist models, including NVIDIA Nemotron models. Both new versions share one architecture and differ in what they aim at.

Fugu Max targets the best result per dollar. It costs $2 per million input tokens and $6 per million output tokens at any context length, according to DataNorth's summary of Sakana's release post and OpenRouter listings. Fugu Ultra v2 targets the hardest multi-step work. It costs $5 input and $30 output per million tokens, rising to $10 and $45 above 272K tokens of context, per AI/TLDR. Sakana's announcement does not state a context window in the material I reviewed.

Sakana's benchmark claims are its own. Sakana reports that Fugu Ultra v2 scored 48.3 on Chartography, a visual reasoning and data interpretation benchmark, against 27.3 for Claude Opus 5 and 29.5 for Claude Fable 5. Reviewers also cite a vendor-reported 74.3 on DeepSWE for Ultra v2, reached without Fable 5, Fable 5.1, or GPT-6 Astra in its model pool. For Fugu Max, Sakana claims the best overall score on six of ten benchmarks against models in a similar price range, including Terminal Bench 2.1 and GPQA Diamond. DataNorth notes that it is unclear where Sakana sourced the competitor scores this time.

Availability is narrower than a typical API. Both models run through Sakana's OpenAI-compatible API. There are no open weights, and MarkTechPost reports that Sakana does not offer the service in the EU or EEA. Existing Fugu users switch tiers with a one-line parameter change.

What it changes for practitioners: Fugu is a bet that routing beats scale. Sakana also pitches it as insurance against vendor lock-in and revoked API access, since the model pool is swappable. That pitch has real appeal after a summer of access changes across the industry. The trade-off is observability. When an orchestrator picks the model, you need its traces to explain a bad answer, so ask for them before you put Fugu in a production path.

Atria Dawn Preview

The quietest big release of the week came from the Shanghai Artificial Intelligence Laboratory. Atria Dawn Preview is an agentic model built on the 744B-parameter mixture-of-experts GLM-5.2 foundation model. It targets research and engineering work that needs continuous tool use and multi-step execution. The model card describes a full loop: problem analysis, solution design, tool use, code, experiment runs, result analysis, and failure recovery.

The release order was unusual. AI Weekly reports that the Hugging Face repository went live on September 11, with an FP8 checkpoint on September 12, and no blog post or pricing at the time. The weights are MIT-licensed. A formal press release followed on September 15, describing Atria Dawn as an open-source model for long-horizon research agents that turns a published method into runnable experiments, reproducible metrics, and a report that traces conclusions to evidence. The model works inside a control framework and an experimental environment, checks whether its code runs and whether experiments hit their targets, and revises its plan from that feedback.

No first-party per-token price was published. Some routing gateways already list the model, so any price you see comes from a third party. I found no context window or independent benchmark score in the primary materials.

What it changes for practitioners: Atria Dawn is the second open research-agent release in two weeks built on a Chinese base model and published weights-first. The release-before-paper pattern shortens the window between a model existing and a model sitting in someone's pipeline. If your team pulls open weights into production, put an evaluation gate in front of new checkpoints that does not depend on the lab's own report.

GPT-Live-1 comes to the API

OpenAI brought GPT-Live-1 to the API on September 10. It is a full-duplex voice model that listens and speaks at the same time, so it handles interruptions, pauses, and backchannels without the handoff delays of a speech-to-text, LLM, text-to-speech chain. It delegates deeper reasoning and tool calls to a backend model you choose, such as GPT-6 Astra or a third-party model.

Pricing is $0.05 per minute for the voice layer, billed per second, with backend model and tool usage billed separately. OpenAI added 12 new real-time voices. The model provides native transcripts, keyword biasing, and explicit turn detection, and it connects over WebRTC, WebSockets, or telephony and SIP.

The benchmarks are OpenAI's. OpenAI's developer community post says GPT-Live-1 paired with GPT-6 Astra at medium reasoning effort completed 83.6 percent of Tau3 tasks on the first attempt, against 45.7 percent for GPT-Realtime-2.1. The same pairing scored 38.1 percent on TauBanking. OpenAI also claims a 30-point gain on Full Duplex Bench over GPT-Realtime-2.1. One customer quoted in the launch said switching from a cascaded build removed 23,000 lines of code.

What it changes for practitioners: Voice agents now follow the same split as coding agents: a fast front end for the conversation and a slower, smarter back end for the work. OpenAI notes that interrupting speech does not automatically cancel backend work, so your application owns task state and cancellation.

Retirements and deprecations to track

Two dated changes landed this week. OpenAI announced that GPT-5.5 retires from ChatGPT, ChatGPT Work, and Codex on October 14, 2026, across all plans. The API is not affected. Codex users who sign in with ChatGPT should switch to gpt-5.6-sol and update saved settings, custom agents, scheduled tasks, and scripts before that date. Separately, OpenAI deprecated gpt-5.4-cyber on September 11, with removal from the API on October 1, 2026, and gpt-5.6-cyber as the replacement.

OpenAI also retired automatic switching from Instant to Thinking for ChatGPT Plus and Pro users and removed the Higher intelligence setting on the web. Users can still pick a reasoning option manually.

No frontier lab shipped a new flagship this week. After Claude Fable 5.1, Gemini 3.8 Flash, Muse Spark 1.3, and GPT-6 Astra in the first three days of September, a quieter stretch was expected. Google's Gemini 3.5 Pro is still announced without a date.

This week's releases at a glance

Two patterns stand out in that table. First, the two open-weight releases both carry the MIT license, which puts almost no restriction on commercial use. Second, the two Sakana tiers show where the premium sits now. The capability tier costs five times as much on output as the cost tier, on the same API, and the only difference is how hard the orchestrator works. That is the same trade every agent platform is now exposing as a setting, whether it is called effort, tier, or mode.

A note on sourcing. Where a lab's own post did not state a price or context window, this issue cites independent trackers that read the providers' pricing pages. Check the provider documentation before you commit a budget, because several of these rate cards changed more than once in September.

Tooling: OpenAI Launches the Agents API

OpenAI Agents API and hosted sandboxes

OpenAI introduced the Agents API in public beta on September 10. It packages the agent loop behind Codex as a managed service. OpenAI runs the agent loop on its own infrastructure and handles orchestration, long-running sessions, and context management. You define the agent's capabilities and choose where it runs code and works with files.

There is no extra fee for the Agents API itself. You pay for the tokens and tools your agents use at standard rates.

Sandboxes are the other half of the launch. You can bring your own, run in your VPC, or use a partner. OpenAI lists first-class integrations with Blaxel AI, Cloudflare, Daytona, DigitalOcean, E2B, Modal, Oracle Cloud, Runloop AI, and Vercel. OpenAI also launched its own hosted sandboxes, where agents run code, work with files, and produce artifacts. You supply files, install packages, and add skills and plugins, and OpenAI provisions the environment.

The Codex Python SDK moved in step. Version 0.154.0, released September 11, adds max and ultra reasoning-effort values. It also adds ExternalMessage to synchronous and asynchronous run() and turn() calls, so external content can start a turn or join an active one with tool-level authority but without granting user authorization. The release adds include_turns on resume and fork, plus a per-turn service tier.

Check the migrations before you upgrade. HookMetadata now wraps its handler in .root, so hook.command becomes hook.root.command. Some notifications now have typed payloads. And turn handles that attach late only receive events from their attachment point, so collected results can be partial. Custom codex_bin overrides need CLI 0.151.0 or newer for the new features.

OpenAI's Data agent and ChatGPT for Financial Services

OpenAI also introduced a Data agent in ChatGPT Work on September 10. It connects to approved sources including Amazon Redshift, Datadog, Google BigQuery, ClickHouse, Databricks, MongoDB, and Snowflake. It pulls business definitions, metric logic, and relationships from semantic layers and trusted sources such as dbt, GitHub, Databricks Genie Ontology, Snowflake Horizon, and BI dashboards. Queries run with the connected account's existing permissions, including table, row, and column restrictions. The agent builds shareable interactive dashboards and can work inside Omni, Oracle BI, Power BI, Sigma, Tableau, and ThoughtSpot. OpenAI says nearly all of its own product team and over two-thirds of its go-to-market organization use data agents internally.

The same day, OpenAI launched ChatGPT for Financial Services, a tailored ChatGPT Work experience built with design partners Morgan Stanley and Evercore. It includes premium data from Daloopa, PitchBook, LSEG News, and Crunchbase, indexed and hosted by OpenAI, with granular citations back to tables and passages. OpenAI says it tuned the reliability of popular financial MCP connectors, including S&P Global and FactSet, through automated evaluation. Admins can publish Excel, Word, and PowerPoint templates. OpenAI reports GPT-6 Astra at 69.9 percent on OfficeQA Pro against 60.2 percent for GPT-5.6 Sol, a vendor-reported number.

Why this matters for data teams: The pattern here is worth noting. The Data agent does not ask analysts to move data. It connects to where data already lives, respects existing access controls, and reads meaning from the semantic layer. That makes the semantic layer and the governed catalog the most important parts of an agent-ready data stack. Teams with clean metric definitions will get good answers. Teams without them will get confident wrong ones.

Claude Code: plugin evals, effort caps, and gateway headers

Anthropic's Claude Code shipped versions 2.1.265 through 2.1.273 across the week, and the week 37 summary calls out two features. The first is claude plugin eval, added in 2.1.269. It runs a plugin against a suite of test cases, scores the results, and by default reruns each case without the plugin so you can see what the plugin actually contributes. claude plugin eval init interviews you about what a good result looks like, proposes test cases and checks, and writes the files. Anthropic notes that every run and every model-judged check is a real model call on your account.

The second feature lets you pop Claude Code Desktop panes, such as the diff or the terminal, into their own windows.

The changelog is where the admin controls live. Version 2.1.267 added a maxEffortLevel setting, at the top level or per model, that caps effort on every provider, including Bedrock, Vertex, and Foundry. Version 2.1.269 added CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS, which accepts values from 1 to 256, and an option to tag OpenTelemetry metrics with repository attributes. Version 2.1.271 added per-command allowed_domains for Bash, PowerShell, and Monitor in sandboxed auto mode, so each command opens only the hosts it needs. It also added an omitClaudeMd flag for subagents and a modelPricing multiplier of up to 10 for internal chargeback rates.

Version 2.1.273 added opt-in request headers for LLM gateways, enabled with CLAUDE_CODE_GATEWAY_HINT_HEADERS=1. They tell a gateway the request class, agent type, previous tool durations, and whether context was compacted. It also changed auto mode on Bedrock, Vertex, and Foundry to use the local safety classifier by default, with CLAUDE_CODE_AUTO_MODE_SERVER=1 to opt back into the server-side classifier.

A long list of fixes targeted prompt-cache reuse. Several releases fixed cases where resuming a session, switching models, or reconnecting an MCP server rewrote the tool list or system prompt prefix and forced a full cache rewrite. With cache reads now priced far below fresh input across the industry, these fixes translate directly into lower bills for long sessions. Version 2.1.273 also fixed auto-compaction triggering at roughly half the real context window when advisor-tool turns were involved.

Claude Managed Agents and on-demand compaction

On the platform side, Anthropic's release notes list two changes. On September 10, Claude Managed Agents permission policies gained an auto mode. The server evaluates each agent or MCP tool call and runs it, denies it, or pauses for approval, and events now report how each call was evaluated. The ant CLI added ant beta:sessions connect, which attaches your terminal to a live Managed Agents session so you can follow it, send messages, and approve or deny waiting tool calls.

On September 14, the Messages API gained on-demand conversation compaction in beta, behind the compact-2026-09-04 header. You send a top-level compaction parameter, and the API returns a signed compaction block summarizing the messages you sent. On later requests, you send that block in place of those messages. You decide when to compact, the request can run in the background, and you can keep recent turns word for word. On models with preserved thinking, the thinking in kept turns stays valid.

Why this matters: Compaction used to be something each agent framework implemented its own way. A signed, server-generated summary block makes it a first-class API object that any client can store and replay. That is a quiet but real step toward portable agent state.

Anthropic's vertical plugins

Anthropic also shipped three vertical launches. Claude for Financial Advisors arrived September 14 as a Cowork plugin with connectors to Addepar, BlackRock, Charles Schwab, Envestnet, iCapital, Orion, SS&C Black Diamond, Wealthbox, Wealth.com, Vanguard, and Zocks, plus skills for meeting prep, rebalance review, estate and tax briefs, and compliance screening against the SEC Marketing Rule. On September 15, Claude for Small Business expanded to 43 workflows and 27 new integrations, including Shopify, Salesforce, Xero, Gusto, Square, Stripe, and Zapier. Anthropic says the plugin has been installed more than 900,000 times since May. Every workflow starts in approval mode. Also on September 15, Anthropic launched Salesforce in Claude in beta with 37 pre-built sales skills.

Anthropic also launched smart reports in beta for Claude Enterprise on September 10. They analyze how a team uses Claude, what the work costs, where sessions run into friction, and which repeated patterns are worth packaging as shared skills.

GitHub Copilot: cost tiers, ensemble review, and HydraFusion

GitHub had a busy week. On September 14, it added three tiers to Copilot's auto model selection: efficiency, balance, and intelligence. All three draw from the same model set. Auto still evaluates each prompt, so a simple docstring request can land on a small model even in intelligence mode. Billing follows the model auto selects, and paid subscribers keep a 10 percent discount on usage billed through auto. The tiers are rolling out in VS Code, Copilot CLI, and the GitHub Copilot app.

On September 11, GitHub updated Copilot code review. Copilot now resolves its own comments when a later commit addresses them, writes commit messages when you apply its suggestions, and validates code with the full set of shell tools from the Copilot SDK behind the agent firewall. The Lite effort level now uses an ensemble of agents. GitHub reports that the ensemble raised addressed comments per review by 47 percent for high-severity findings, 31 percent for medium, and 11 percent for low, while cutting review cost by about 8 percent.

The September 7 weekly release post, published September 10, introduced Project HydraFusion in the Copilot CLI's /experimental menu. HydraFusion routes each task between local, cloud, and compound models to balance performance, cost, and latency, and you select it like any other model. The same post covered Jira integration in the Copilot app, scheduled agent automations and an experimental voice mode in VS Code 1.137, and centrally managed sandbox policies for Copilot in JetBrains.

The Copilot CLI shipped several builds. Version 1.0.84-5 added session and memory import commands for a semantic JSONL interchange format, moved command parsing to a Rust grammar, and made /usage show per-model AI Credit consumption. Version 1.0.84-6 added a /config screen and network allow and deny rules in /sandbox, and fixed a bug where MCP tools with certain boolean schemas caused 400 errors on Gemini. It also fixed COPILOT_ALLOW_ALL so that falsey values disable automatic tool approval instead of enabling it. That last fix is worth a second look if you set that variable in CI.

GitHub also added VS Code Agents window metrics to Copilot usage reports and released REST APIs in public preview for enabling AI Scan for pull requests at the organization and repository levels.

The pattern across vendors

Put these launches side by side and the direction is plain. OpenAI's Agents API, Anthropic's Managed Agents auto policies, and GitHub's HydraFusion and auto tiers all move two decisions from the developer to the platform: where the agent runs and which model does each step. In exchange, every vendor is adding controls. Effort caps, per-command network allowlists, server-evaluated tool calls, usage reports by model, and plugin evals all exist so an administrator can see and bound what the platform decides.

That is the right trade for most teams. It also means your cost and quality now depend on routing logic you did not write. Measure it. Plugin evals, per-model usage reports, and gateway hint headers are the instruments. Use them before you trust the defaults.

Standards: A Quiet Week for Protocols, a Busy One for Formats

No new MCP or A2A release

There was no new Model Context Protocol specification release this week and no new Agent2Agent (A2A) release. The latest MCP spec is still the 2026-07-28 version, which made the protocol stateless, and the most recent MCP blog post is the updated roadmap from August 22. A2A's latest milestone was its acceptance as a Growth Stage project at the Agentic AI Foundation (AAIF) in late August. If you track these specs, the next big date is AGNTCon+MCPCon, which the AAIF lists for October 22 and 23 in San Jose.

A quiet spec week is not a quiet implementation week. MCP fixes showed up across this week's tooling releases. The Copilot CLI now notifies MCP servers when a tool call is cancelled, requests extra OAuth scopes when needed, and fixed a schema bug that broke some MCP tools on Gemini. Claude Code added a notification when an MCP server disconnects and reconnection gives up, and fixed MCP OAuth client registration bugs. OpenAI's financial services launch describes automated evaluation to raise the reliability of popular MCP connectors. The protocol is stable enough that the work has shifted to making clients and servers behave well at the edges. For teams running MCP servers in production, that is good news. Cancellation, re-authentication, and disconnect handling are where real deployments break, and the major clients are now fixing those paths release by release instead of waiting on the spec.

Portable agent state is becoming a format question

Two tooling changes this week point at a standards gap. GitHub's Copilot CLI added import commands for a semantic JSONL interchange format covering sessions and memory. Anthropic's Messages API now returns a signed compaction block that stands in for earlier conversation turns. OpenAI's Codex SDK added history selection on resume and fork.

Each vendor is defining how agent sessions, memory, and summaries get stored and replayed. None of these are shared formats yet. MCP standardized how agents reach tools. A2A standardized how agents talk to each other. The next interoperability fight is over how an agent's accumulated state moves between tools and vendors. Watch for proposals in that space.

Open semantic models: Apache Ossie

The most concrete standards news for data teams came from the Apache Ossie (incubating) dev list, which covers an open specification for semantic models. Microsoft's Power BI group asked the community for guidance on announcing its commitment to Ossie at the upcoming Fabric Community Conference. The group plans to build a converter between Power BI semantic models and Ossie and contribute it to the project.

Ossie also moved on its core format. Yufei Gu proposed that each Ossie document hold exactly one semantic model, with its fields at the root, and opened a follow-up PR to update the converters. Jean-Baptiste Onofré confirmed that the first release, 0.3.0, will focus on release mechanics and a full legal check. And Marco Ciavarella proposed a dedicated channel for field reports written by coding agents that build with the spec.

This connects directly to the tooling section. OpenAI's Data agent reads meaning from semantic layers. Every agent that answers business questions needs metric definitions it can trust. An open semantic model format that BI tools, catalogs, and agents all read is what makes those definitions portable instead of locked inside one vendor's product.

Data format standards: Parquet 2.14 and the versioning debate

Parquet had a significant standards week too. The Parquet Format 2.14.0 release vote passed on September 11, bringing the ALP floating point encoding and the FILE logical type into the spec. A separate vote approved Extended Precision Nanosecond Timestamps.

The larger debate is about how readers should handle unsupported format versions. Ryan Blue, Fokko Driesprong, Xiening Dai, and Kurtis Wright favor a strict rule: a reader must fail on a file written with a format version it does not support. Andrew Lamb and Will Edwards argued for letting readers attempt the read, as most do today. The strict option makes it easier to ship breaking improvements such as a lighter footer. It also means every engine in a stack has to upgrade its Parquet reader before writers turn on new features.

Parquet is the storage layer under most AI training data pipelines, feature stores, and lakehouse tables. How it versions is an AI infrastructure question as much as a data engineering one.

Iceberg REST catalog and multimodal access

On the Apache Iceberg list, Sung Yun opened a discussion about access delegation for the proposed FILE type, which lets a table column reference external objects such as images and documents. The use case is multimodal inference: a service outside the query engine needs to fetch those objects. The proposal adds client-requested pre-signed URLs to the REST catalog spec. Daniel Weeks and Prashant Singh pushed to align it with existing pre-signed URL work before anything lands. This is the open table format community designing for AI workloads directly, and it belongs on the radar of anyone building retrieval over files in a lakehouse.

Infrastructure: Memory Is Still the Wall

Intel says the memory crunch gets worse in 2027

The clearest infrastructure signal of the week came from the AI Infra Summit in Santa Clara, which runs September 15 to 17. Intel CEO Lip-Bu Tan warned there that the memory shortage will be worse in 2027 than in 2026. Fudzilla reports that Tan said memory prices have risen five to seven times, and that memory now makes up 70 to 80 percent of the component cost of some low-end phones and laptops. Invezz reports that Tan also flagged electricity and cooling as emerging constraints.

A separate report from Aroged, citing a Splunk broadcast appearance, says Tan acknowledged Intel cannot meet more than half of demand for its processors, does not expect memory to improve until 2028, and plans to start 14A production in the first quarter of 2027. These are secondhand accounts of spoken remarks, so treat the exact figures with some care. The direction is consistent across every report.

The cause is structural. Memory makers are steering wafer capacity toward HBM for AI accelerators, which leaves less for commodity DRAM and NAND. DIGITIMES' weekly roundup for September 7 to 13 describes HBM4 and server demand straining DRAM and NAND supply, alongside Intel CPU price increases.

What it means for AI teams: Memory is now the price floor for both training and inference. It shapes GPU availability, server costs, and even the laptops your developers use. Plan 2027 hardware budgets with higher memory costs baked in, and treat any software change that cuts memory use as a cost reduction.

Software is attacking the memory problem directly

That last point is exactly what DeepSeek did this week. V4.1 Flash's KV cache needs one quarter of the HBM and one eighth of the SSD storage of the previous generation. The KV cache holds the attention state for every token in context, and on million-token contexts it often consumes more accelerator memory than the model weights. Shrinking it lets a provider serve more concurrent requests on the same GPUs, which is how DeepSeek was able to cut prices during a memory shortage. DeepSeek also invited teams planning deployments of 2,000 GPUs plus a storage cluster to contact it directly.

The architecture choice works the same way. Activating 8B parameters for input and 16B for output means prefill, the input-heavy half of agent workloads, runs on a much smaller slice of the model. Expect other labs to publish similar input and output splits as agentic workloads dominate inference traffic.

OpenAI's GPT-Live-1 is a different angle on the same economics. A single full-duplex model replaces a three-stage speech pipeline, which cuts both latency and the number of models held in memory per conversation. At $0.05 per minute for the voice layer, it pushes the heavy reasoning to a backend model that only runs when the conversation needs it.

Columnar formats keep getting smaller

The storage side of AI infrastructure also moved. Apache Arrow Rust 60.0.0 passed its release vote on September 15. Release manager Andrew Lamb said arrow-rs is the first Parquet implementation to include the new ALP encoding for floating point data. The release also adds new Parquet PageIndex structures and many performance improvements. The crates are live on crates.io.

Float and double columns are everywhere in AI data: embeddings stored as arrays, model features, sensor readings, and evaluation metrics. General-purpose encodings compress them poorly. ALP targets exactly that data, and it now ships in the Rust parquet crate that DataFusion, iceberg-rust, and many other Rust engines depend on. Implementations in C++, Java, and Go are in review, according to the Parquet community sync notes. The same notes record progress on a vector logical type for Parquet, which targets embedding storage directly.

Accelerators at the AI Infra Summit

NVIDIA's AI Infra Summit event page promotes a keynote from Ian Buck, NVIDIA's VP of Hyperscale and HPC Computing, on infrastructure for agentic AI. The page describes Groq 3 LPX as an interactive inference accelerator that extends the Vera Rubin platform, aimed at fast token generation for responsive agent systems. Converge Digest's summit preview frames the event around scale-up interconnect, scale-out networking, and scale-across data center links, with Google Fellow Dave Patterson, Ian Buck, and Lip-Bu Tan on the main stage. The summit runs through September 17, so expect more detailed announcements to surface in next week's coverage.

Practitioner Takeaways

Run the cache math before you switch models

Rate cards hide the number that matters most for agents. Here is a worked example using DeepSeek's published peak rates. Take one long agent run that reads 10 million cached input tokens, 1 million fresh input tokens, and writes 200,000 output tokens.

On V4 Flash at $0.014 cached, $0.44 input, and $1.32 output per million, that run costs $0.14 plus $0.44 plus $0.264, or about $0.84. On V4.1 Flash at $0.006, $0.30, and $1.20, it costs $0.06 plus $0.30 plus $0.24, or $0.60. That is a 29 percent cut, and more than a third of it comes from the cache line alone.

Now change the shape. A chat workload with little caching and a lot of output saves far less, because output only dropped 9 percent. The lesson applies to every vendor: pull a week of real traffic, split it into cached input, fresh input, and output, and price that mix. Headline input prices tell you very little about an agent bill.

Five things to do this week

Audit pinned model IDs. DeepSeek now routes old IDs to a new model, and OpenAI has two retirement dates in the next four weeks. Log which model actually served each request so a silent swap shows up in your dashboards.
Set effort and concurrency caps. Claude Code's maxEffortLevel and workflow concurrency limit, and Codex's new max and ultra effort values, give you both more power and more ways to overspend. Decide your ceilings on purpose.
Evaluate plugins and skills like code. claude plugin eval runs each case with and without the plugin. Use that comparison to delete plugins that add tokens without adding quality.
Check your CI environment variables. The Copilot CLI fix for COPILOT_ALLOW_ALL changed how falsey values behave. If your pipelines set it, confirm the behavior you expect.
Invest in your semantic layer. OpenAI's Data agent, Apache Ossie, and Polaris' new semantic model privileges all point the same way. Agents answer business questions only as well as your metric definitions allow.

Where the week leaves the stack

Three layers moved at once. Models got cheaper per task through architecture, not just discounts. Platforms took over the agent loop and added the controls to supervise it. And the data layer kept adding the pieces agents need: compact float encodings, vector types, governed semantic models, and catalog-level access for multimodal files. None of those layers works well alone. The teams that get the most from this week's releases will be the ones that treat model choice, agent platform, and data architecture as one design problem.

What to Watch Next Week

DeepSeek V4 Pro routing. Confirm whether deepseek-v4-pro requests are served by V4 Pro or V4.1 Flash, and watch for V4.1 Pro.
Independent benchmarks. Look for third-party scores on DeepSeek V4.1 Flash, Fugu Ultra v2, and Atria Dawn Preview. Every number in this issue for those models is vendor-reported.
Gemini 3.5 Pro. Google has announced it with no date or price.
AI Infra Summit follow-ups. Detailed accelerator, memory, and networking announcements from Santa Clara.
Parquet versioning. A decision on strict reader version checks will shape how new footer and encoding features roll out.
Deadlines. gpt-5.4-cyber leaves the OpenAI API on October 1, and GPT-5.5 leaves ChatGPT and Codex on October 14.

Resources to Go Further

The tools change every week, but the fundamentals of data, lakehouse architecture, and agentic AI hold steady. I have written books on all of it, from Apache Iceberg and Apache Polaris to AI-assisted development and AI agents for data work. You can find every title at books.alexmerced.com.

The Open Lakehouse Explained, Then Built on Your Laptop with Dremio and MinIO

Alex Merced — Thu, 10 Sep 2026 19:56:45 +0000

Most people learn the open lakehouse backwards. They read five vendor pages, collect a stack of Apache project names, and still cannot answer a basic question: when I run a query, what does each piece actually do?

That gap is fixable in about an hour. The five projects that define the open lakehouse each own one layer of the problem, and once you see the layers separately, the architecture stops being a diagram and becomes obvious. Apache Parquet stores bytes on disk. Apache Iceberg turns a pile of those files into a table. Apache Polaris keeps track of which tables exist and who can touch them. Apache Arrow moves the data through memory and across the wire. Apache Ossie describes what the columns mean in business terms.

The second half of this article is a lab. You will run two containers on your laptop, one for object storage and one for a query engine, wire them together, and write a real Apache Iceberg table into an S3-compatible bucket. Then you will look at the files that landed and see the format layers with your own eyes. No cloud account, no credit card, no Spark cluster.

I work at Dremio, and Dremio is the query engine in the lab. I picked it because it runs in a single container and needs no external metastore, which keeps the exercise short. The concepts transfer to any engine.

Why the Lakehouse Split Into Layers at All

For thirty years a database was one product. Storage format, table metadata, catalog, query planner, and execution engine all shipped together and none of them were separable. That design has real advantages. The engine knows exactly how the bytes are laid out and can optimize against its own assumptions.

The cost shows up when you have more than one workload. Your BI tool wants SQL. Your data scientists want Python. Your machine learning pipeline wants to read raw files. Each of those tools has its own preferred engine, and a closed warehouse gives you exactly one door in, priced per query.

Teams solved this in the 2010s by dumping files into object storage and pointing many engines at the same directory. That worked for read-only analytics and fell apart everywhere else. Two writers touching the same directory produced corrupt results. A partially written job left half a dataset visible to readers. Renaming a column meant rewriting everything. There was no such thing as a transaction.

The lakehouse is the answer to that failure. Keep the cheap shared storage, then add back the guarantees a database gave you, defined as open specifications instead of product internals. Each guarantee lives in its own layer, and each layer has a written spec that anyone can implement.

The result is a stack you assemble rather than buy. Storage from one vendor, catalog from another, three engines reading the same tables at once, and no rewrite when you swap any single piece. That portability is the entire point, and it only works because the layers stay honest about their boundaries.

Apache Parquet: The File Layer

Parquet is a columnar file format. It is the oldest project in this group, donated to the Apache Software Foundation in 2015, and it is the default storage format for nearly every analytics system built since.

A CSV file stores data by row. All of order 1, then all of order 2. To sum one column across ten million rows, you read all ten million rows in full. Parquet flips that. It groups rows into chunks called row groups, and inside each row group it stores each column contiguously. To sum one column, you read that column and skip the rest.

Three properties fall out of that layout, and all three matter more than people expect.

Compression gets far better. A column holds one data type with repeated values, so run-length encoding and dictionary encoding do real work. A column of country codes with 200 distinct values compresses to almost nothing. Mixed row data never compresses that well.

Column pruning saves I/O. A query touching 3 columns out of 80 reads about 4 percent of the file. On object storage, where you pay per byte transferred and latency dominates, that is the difference between a two-second query and a two-minute one.

Predicate pushdown skips whole chunks. Each row group carries footer statistics with the minimum and maximum value per column. A query filtering on order_date > '2026-01-01' reads the footer, sees that a row group tops out at 2025-06-30, and skips it without decompressing a single page.

What Parquet does not give you is a table. A Parquet file knows its own schema and its own statistics. It has no idea that 4,000 sibling files in the same prefix belong to the same logical dataset, no notion of a transaction, and no way to express that three of those files were replaced by one this morning. That is the next layer's job.

Apache Iceberg: The Table Layer

Iceberg is a specification for describing a table as a set of files, plus enough metadata to make changes to that set atomic. It came out of Netflix, was donated to the ASF in 2018, and graduated to a top-level project in 2020. The current stable Java library line is 1.11, released in mid-2026, and it implements format version 3 of the specification.

The distinction between library version and format version confuses people constantly. The library version, 1.11, is a piece of software. The format version, 3, is the on-disk contract that any implementation in any language has to honor. When a vendor says they support Iceberg, ask which format version and which operations.

Here is the structure, from the bottom up.

Data files. Parquet files holding the rows. Iceberg supports ORC and Avro too, and Parquet is what nearly everyone uses.

Manifest files. An Avro file listing data files, each with its partition values, row count, and per-column min/max statistics. One manifest describes a batch of data files.

Manifest lists. An Avro file listing the manifests that make up one snapshot, with partition range summaries for each.

Metadata files. A JSON file holding the table schema, the partition specification, the sort order, table properties, and the full history of snapshots. Every write produces a new metadata file.

The catalog pointer. One tiny piece of mutable state: which metadata file is current. Everything below it is immutable.

That last point carries all the weight. Because every file below the pointer never changes, a writer builds a complete new version of the table off to the side, then swaps the pointer in one atomic operation. Readers that started before the swap keep reading the old snapshot and see a consistent view. Readers that start after see the new one. Nobody ever sees half a commit.

Once you have immutable snapshots, several features come free rather than being bolted on.

Time travel is just reading an older metadata file. SELECT * FROM sales AT SNAPSHOT '8234...' resolves a snapshot ID and reads the file list from that point in history.

Rollback is repointing the catalog at a previous metadata file. A bad load at 3 a.m. gets undone in one statement instead of a restore from backup.

Schema evolution works because Iceberg tracks columns by a unique integer ID, not by name or position. Rename a column and the ID stays the same, so old data files still resolve correctly. Add a column and old files report null for it. Drop a column and nothing gets rewritten.

Hidden partitioning removes the worst footgun of the Hive era. In Hive, a table partitioned by day required every query to filter on the partition column by name, and a query filtering on the raw timestamp scanned everything. Iceberg records the partition as a transform of a source column, so a filter on order_ts automatically prunes partitions defined as day(order_ts). Users stop needing to know the physical layout.

One thing Iceberg deliberately leaves out is where that catalog pointer lives. The specification defines the file formats and the commit protocol, then hands the pointer problem to a pluggable catalog. Implementations include the Hive Metastore, AWS Glue, a plain filesystem-based catalog, and the REST catalog protocol that Polaris implements. A table created through one catalog is not readable through another, because each catalog holds its own record of the current metadata file. Migrating between catalogs is a real project rather than a config change, which is why the catalog choice deserves more thought than the file format choice.

Format version 3 added deletion vectors, which replace v2 position delete files with compact bitmaps. Updating a handful of rows no longer forces a rewrite of the data files that contain them, and merge-on-read queries get faster. V3 also added a variant type for semi-structured JSON and mandatory row lineage tracking.

Apache Polaris: The Catalog Layer

The catalog is the smallest component in the stack and the one that decides how open your architecture really is.

An Iceberg catalog does one job well: map a table name to the location of its current metadata file, and swap that pointer atomically when a writer commits. Add a second job on top and you have a governance boundary, because every engine has to ask the catalog where a table lives before it can read the table.

Apache Polaris is an open source implementation of the Iceberg REST Catalog specification. It was co-created by Dremio and Snowflake, donated to the Apache Software Foundation, and graduated to a top-level project on February 18, 2026.

Two things make the REST catalog spec matter more than the older catalog options.

First, it standardizes the protocol instead of the implementation. Older catalog choices bound you to a client library. A Hive Metastore catalog meant every engine needed the Hive client and Thrift. A Glue catalog meant AWS SDK calls. The REST spec turns catalog access into HTTP with a documented JSON contract, so an engine written in Rust or Python talks to the same catalog as one written in Java without shipping anyone else's client.

Second, it moves commit logic to the server. In the older model, the client library built the new metadata and performed the atomic swap. Every engine had to implement that correctly, and subtle differences caused real corruption. With REST, the client sends the requested change and the server does the commit. One implementation of the hard part, shared by everyone.

Polaris adds credential vending on top of that, and this is the feature worth understanding even if you never deploy Polaris. Instead of giving each engine long-lived storage keys, the catalog holds the storage credentials. An engine asks for a table, the catalog checks the caller's permissions, and it returns a short-lived, narrowly scoped storage credential valid for that table's location only. Access control lives in one place instead of being duplicated across every engine's configuration and every bucket policy.

Polaris organizes objects into catalogs, namespaces, and tables, with role-based access control on each level. It runs internal catalogs, where Polaris manages the tables directly, and external catalogs, where it federates to another catalog implementation.

The lab below does not use Polaris. Running a catalog service, a database for its state, object storage, and an engine is four containers and a lot of configuration, which is the wrong shape for a first exercise. Dremio ships with an internal Iceberg catalog that handles the pointer for us, and the file layout you inspect at the end is identical either way.

Apache Arrow: The In-Memory Layer

Parquet is how data rests. Arrow is how data moves.

Apache Arrow defines a standard columnar layout for data in memory, plus a serialization format for sending that layout over a network without changing it. It was co-created by Jacques Nadeau, now the CTO at Dremio, and it has become the connective tissue between analytics tools.

The problem Arrow solves is serialization tax. Before Arrow, moving a result set from a Java engine to a Python client meant converting a Java object layout to a wire format, sending it, then parsing it into a pandas layout. On large results, that conversion cost more CPU than the query. Every hop between two systems paid the same tax.

Arrow removes the conversion by making the in-memory layout and the wire layout the same thing. A record batch on the server is copied to the socket, and the client points at the received bytes as a valid Arrow buffer. No parse step. Tools that both speak Arrow exchange data at close to memory bandwidth.

The layout is also built for modern CPUs. Values in a column sit in a contiguous buffer with a separate validity bitmap for nulls, which lets a query engine process 8 or 16 values per instruction using SIMD registers rather than one value per loop iteration. Vectorized execution is what makes columnar engines fast, and Arrow is the layout that makes vectorized execution straightforward to write.

Two adjacent pieces are worth naming because they show up in real deployments. Arrow Flight is a gRPC-based protocol for moving Arrow record batches between processes, and Arrow Flight SQL adds a database-style interface on top, so a client submits SQL and receives Arrow batches directly. Dremio exposes Flight SQL on port 32010, which is one of the ports you will map in the lab.

Parquet and Arrow are frequently confused because both are columnar. Parquet optimizes for size on disk with heavy encoding and compression. Arrow optimizes for CPU access speed with a fixed, predictable layout and no decoding step. An engine reads Parquet, decodes it into Arrow buffers, and works from there.

Apache Ossie: The Semantic Layer

The newest project in this group solves the problem that survives after all the technical layers work perfectly.

Apache Ossie is an open specification for semantic layers and ontologies. It started life as Open Semantic Interchange, was renamed to avoid a clash with the Open Source Initiative acronym, and entered the Apache Incubator on June 22, 2026. It is a specification project rather than a runtime, and it is still incubating, so treat it as a direction rather than a dependency.

The problem is definition drift. "Monthly Active Users" exists in the CRM, in the warehouse, and in three BI dashboards, and the four definitions disagree on whether a user who logged in through the mobile app on the last day of the month counts. Every organization past a certain size has this problem, and it never gets solved by fixing one dashboard, because the definitions are trapped inside whichever tool created them.

Ossie defines a vendor-neutral YAML format for expressing metrics, dimensions, relationships, and broader business concepts. A BI platform, a query engine, or an AI agent reads the same definition file and computes the same number. The definition moves with the data instead of living in a proprietary model file.

The agent angle is the reason this project got funded and staffed now rather than five years ago. A human analyst who gets a strange number investigates. An AI agent that gets a strange number writes it into a report with confidence. Giving agents a machine-readable, governed definition of what a metric means is the difference between an agent that helps and one that generates plausible nonsense at scale. Contributors include Snowflake, Salesforce, Databricks, dbt Labs, RelationalAI, GoodData, and Honeydew, which tells you the industry treats this as shared infrastructure rather than a competitive surface.

Ossie sits above the lab you are about to run. You will not configure it. Knowing the layer exists explains why "semantic layer" keeps appearing in lakehouse conversations that are otherwise about file formats.

How the Five Layers Cooperate in One Query

Walk a single query through the stack and the division of labor becomes concrete.

An analyst runs SELECT region, SUM(amount) FROM sales WHERE order_date >= DATE '2026-01-01' GROUP BY region.

The engine parses the SQL and asks the catalog for the table named sales. The catalog checks permissions and returns the location of the current metadata file, plus a scoped storage credential if it does credential vending.

The engine reads the Iceberg metadata file, follows it to the manifest list for the current snapshot, and reads the manifests. Each manifest entry carries partition values and column statistics. The engine drops every data file whose order_date maximum falls before 2026-01-01. A table with 40,000 files becomes a scan list of 900 without a single byte of table data being read.

For each surviving Parquet file, the engine reads the footer, drops row groups whose statistics fail the filter, and requests byte ranges covering only the region, amount, and order_date columns.

Those bytes get decoded into Arrow buffers in memory. Aggregation runs vectorized over those buffers, and partial results merge across threads and nodes in the same layout.

The final result travels to the client as Arrow record batches over Flight SQL, with no serialization step.

If the organization uses Ossie, the analyst never wrote that SQL. They asked for revenue by region, and a tool translated it using the governed metric definition, producing SQL that matches what every other tool in the company produces for the same question.

Five specifications, five jobs, zero overlap. That is the design.

Held side by side, the boundaries are easy to keep straight:

The useful test for any new tool you evaluate is which of these rows it replaces and which it merely reads. A product that reads Iceberg tables somebody else wrote sits in a different category from one that creates, writes, and commits through the standard catalog API. Both say "Iceberg support" on the website.

The Lab: What You Are Building

Two containers.

MinIO provides S3-compatible object storage on your laptop. It speaks the S3 API, so anything that talks to Amazon S3 talks to it with a changed endpoint. This is your data lake.

Dremio provides the SQL engine, the Iceberg write path, and the web console. Dremio Community Edition runs as a single container with an embedded coordinator, executor, and ZooKeeper.

The exercise has four steps. Start the stack. Create a bucket. Add MinIO to Dremio as a source configured for S3 compatibility. Write, query, and evolve an Iceberg table, then look at the files it produced.

Before the compose file, one piece of honesty about MinIO. The MinIO project archived its GitHub repository on April 25, 2026, and stopped publishing new container images in October 2025. The last image published to Docker Hub is tagged RELEASE.2025-09-07T16-13-09Z, and that is the tag pinned below. It works fine for a local exercise and receives no further security patches, so do not carry this compose file into production. For production S3-compatible storage, look at MinIO's commercial AIStor line, the community fork maintained at pgsty/minio, or alternatives such as Ceph RADOS Gateway, SeaweedFS, or Garage. The Dremio configuration below is identical for any of them, because the only thing that matters is the S3 API.

Requirements: Docker Desktop, roughly 8 GB of RAM available to Docker, and about 10 GB of disk.

The Compose File, Explained Line by Line

Create a directory, save this as docker-compose.yml, and read the explanation before running it.

services:
  minio:
    image: minio/minio:RELEASE.2025-09-07T16-13-09Z
    container_name: minio
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
    command: server /data --console-address ":9001"
    volumes:
      - minio-data:/data
    networks:
      - lakehouse

  createbucket:
    image: minio/mc:latest
    container_name: createbucket
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      until mc alias set local http://minio:9000 admin password; do sleep 2; done;
      mc mb --ignore-existing local/lakehouse;
      mc ls local;
      exit 0;
      "
    networks:
      - lakehouse

  dremio:
    image: dremio/dremio-oss:latest
    container_name: dremio
    ports:
      - "9047:9047"
      - "31010:31010"
      - "32010:32010"
      - "45678:45678"
    environment:
      DREMIO_JAVA_SERVER_EXTRA_OPTS: -Dpaths.dist=file:///opt/dremio/data/dist
    volumes:
      - dremio-data:/opt/dremio/data
    depends_on:
      - minio
    networks:
      - lakehouse

volumes:
  minio-data:
  dremio-data:

networks:
  lakehouse:

Now the details that matter.

The pinned MinIO tag. Pinning an exact release keeps the exercise reproducible. latest on an archived project is a moving target you have no control over.

Two MinIO ports. Port 9000 is the S3 API endpoint. Dremio talks to that one. Port 9001 is the web object browser, which you use to look at files. In the community build, that browser is a plain object viewer with the admin features removed, which is why bucket creation happens through the client container instead of the browser.

Root credentials as access keys. MINIO_ROOT_USER and MINIO_ROOT_PASSWORD become the S3 access key and secret key. In the Dremio source configuration, admin goes in the access key field and password goes in the secret key field. Real deployments create a scoped service account with mc admin user svcacct add and never hand out root keys.

The command line. server /data tells MinIO to serve the /data path as its storage backend. --console-address ":9001" pins the browser to a fixed port. Without it, MinIO picks a random port each start and your bookmark breaks.

The createbucket service. This container runs MinIO's command-line client once and exits. The until loop retries the alias command until MinIO answers, which handles the race where the client starts before the server is listening. mc alias set registers a connection named local. mc mb --ignore-existing local/lakehouse creates the bucket and stays quiet if it already exists, so restarting the stack is safe. mc ls local prints the bucket list to the container log so you have a visible confirmation. Compose will report this container as exited, and that is correct.

Dremio's four ports. 9047 is the web console and REST API. 31010 is the legacy ODBC/JDBC port. 32010 is Arrow Flight SQL, the fast path discussed earlier. 45678 is internal node-to-node communication, needed even in single-node mode.

The DREMIO_JAVA_SERVER_EXTRA_OPTS line. This sets Dremio's distributed storage path to a directory inside the container's data volume. Dremio uses distributed storage to hold job results, uploads, and Reflections. Without setting it, Reflections stay unavailable. This one variable is the difference between a container that starts cleanly and one that complains about missing distributed storage configuration.

Named volumes. minio-data keeps your Iceberg files across restarts. dremio-data keeps Dremio's own catalog, source definitions, and user accounts. Drop these and you start over from the signup screen.

The shared network. Both containers join the lakehouse network, so Dremio reaches MinIO at the hostname minio on port 9000. This is the single detail that trips up the most people, and the next section explains why.

Start it:

docker compose up -d
docker compose logs -f dremio

Dremio takes 60 to 120 seconds to become available on a laptop. Watch for the log line saying the Dremio Daemon started. Open http://localhost:9047, fill in the first-user signup form, and pick any credentials you can remember. That account is local to your container.

Adding MinIO as a Dremio Source

In the Dremio console, click Add Source in the lower left and choose Amazon S3. MinIO is not in the list by name because it does not need to be. It speaks the S3 API, and the S3 connector handles anything that does.

Fill in the General tab:

Name: minio
Authentication: AWS Access Key
AWS Access Key: admin
AWS Access Secret: password
Encrypt connection: unchecked

Unchecking encryption is required here. MinIO in this compose file serves plain HTTP with no certificate. Leaving the box checked makes Dremio attempt TLS against an HTTP endpoint, and the source fails to save with a connection error that does not name the real cause.

Now open Advanced Options:

Enable compatibility mode: checked
Default CTAS Format: ICEBERG
Root Path: /

Compatibility mode is what tells Dremio it is talking to an S3-compatible store rather than Amazon S3 itself. It changes request signing and endpoint handling. Without it, Dremio builds requests against AWS endpoints and never reaches your container.

Default CTAS Format deserves a pause. S3 sources in Dremio default to writing Parquet, not Iceberg. A CREATE TABLE AS against a source left on the default produces a folder of Parquet files with no table metadata at all, which looks like it worked and gives you none of Iceberg's guarantees. Setting this to ICEBERG is the single most important checkbox in this exercise.

Root Path of / exposes every bucket in MinIO as a top-level folder inside the source. Setting it to /lakehouse scopes the source to one bucket, which is closer to what you do in production.

Still in Advanced Options, find Connection Properties and add three:

The endpoint value has two rules that break people constantly. It cannot include the http:// or https:// prefix, and it cannot begin with the string s3. Write minio:9000 and nothing else.

Use minio, not localhost. Inside the Dremio container, localhost means the Dremio container. The hostname minio resolves through the Docker network to the MinIO container. This mistake produces a connection refused error that reads like a MinIO problem and is not one.

Path-style access matters because the default S3 addressing scheme is virtual-hosted style, which puts the bucket into the hostname as lakehouse.minio:9000. That hostname does not exist on your Docker network. Path style produces minio:9000/lakehouse, which does.

Click Save. The source appears in the left panel, and expanding it shows the lakehouse bucket. If it does not, jump to the troubleshooting section.

Writing and Reading Iceberg Tables

Open the SQL Runner and create a table.

CREATE TABLE minio.lakehouse.orders (
  order_id     INT,
  customer     VARCHAR,
  region       VARCHAR,
  amount       DOUBLE,
  order_date   DATE
) PARTITION BY (month(order_date));

The PARTITION BY (month(order_date)) clause is Iceberg's hidden partitioning at work. You are not adding a separate month column. Iceberg records a transform of order_date, and later queries that filter on order_date get partition pruning without knowing the layout exists.

Insert some rows:

INSERT INTO minio.lakehouse.orders VALUES
  (1, 'Acme Corp',    'East',  1200.50, DATE '2026-01-15'),
  (2, 'Globex',       'West',   890.00, DATE '2026-01-22'),
  (3, 'Initech',      'East',  2340.75, DATE '2026-02-03'),
  (4, 'Umbrella Ltd', 'North', 1500.00, DATE '2026-02-18'),
  (5, 'Soylent Inc',  'West',   675.25, DATE '2026-03-07');

Query it:

SELECT region, SUM(amount) AS total, COUNT(*) AS orders
FROM minio.lakehouse.orders
GROUP BY region
ORDER BY total DESC;

Five rows is not a performance test. The point is that the write path produced a real Iceberg table, and the next few statements prove it.

Look at the snapshot history:

SELECT * FROM TABLE(table_snapshot('minio.lakehouse.orders'));

You get one row per snapshot, with a snapshot ID, a commit timestamp, the operation, the manifest list path, and a summary. Two snapshots exist so far: one from the create, one from the insert.

Now mutate the table and watch history grow:

UPDATE minio.lakehouse.orders
SET amount = 1300.00
WHERE order_id = 1;

DELETE FROM minio.lakehouse.orders
WHERE region = 'North';

INSERT INTO minio.lakehouse.orders VALUES
  (6, 'Hooli', 'East', 4100.00, DATE '2026-03-22');

Re-run the snapshot query. Every statement added a snapshot. Copy the snapshot ID from the row right after your first insert and travel back to it:

SELECT * FROM minio.lakehouse.orders
AT SNAPSHOT '<paste_snapshot_id_here>'
ORDER BY order_id;

Five rows come back, with Acme at 1200.50 and Umbrella Ltd present. The current table has neither. Nothing was restored from a backup. The old metadata file still lists the old data files, and they were never deleted.

You can travel by time as well as by snapshot:

SELECT COUNT(*) FROM minio.lakehouse.orders
AT TIMESTAMP '2026-09-10 00:00:00';

Evolve the schema and confirm nothing breaks:

ALTER TABLE minio.lakehouse.orders ADD COLUMNS (sales_rep VARCHAR);

SELECT order_id, customer, sales_rep
FROM minio.lakehouse.orders
ORDER BY order_id;

Every existing row reports null for sales_rep, and not one data file was rewritten. Iceberg added a column ID to the schema in a new metadata file. Readers resolve missing IDs as null.

Finally, compact:

OPTIMIZE TABLE minio.lakehouse.orders;

Dremio rewrites small files into larger ones and clears delete files, then commits the result as another snapshot. On a five-row table the output is trivial. On a table fed by a streaming pipeline producing thousands of small files an hour, this is the single most valuable maintenance operation in the lakehouse.

Looking at the Files You Just Created

This is the part that makes the architecture stick. Open http://localhost:9001 and sign in with admin and password. Browse into the lakehouse bucket and then into the orders folder.

You see two directories.

metadata/ holds the Iceberg metadata. Look for files ending in .metadata.json, one per commit, numbered in sequence. Open the newest one in the browser. It is readable JSON containing the schema with integer column IDs, the partition spec with the month transform, a list of snapshots, and a current-snapshot-id field. Open the first one and compare. You can see the schema before sales_rep was added.

The same folder holds snap-*.avro files, which are the manifest lists, and other .avro files, which are the manifests. Those are binary, so you will not read them in a browser, and knowing they sit between the metadata and the data is enough.

data/ holds the Parquet files, organized into subfolders named for the partition values that Iceberg's month transform produced. Each .parquet file inside is a normal Parquet file that any Parquet reader opens without knowing anything about Iceberg.

Count the metadata JSON files. Each of your statements produced one. That is the commit protocol in physical form: write new files, write a new metadata file, swap the pointer.

Try one more thing. Prefix a Parquet file path in your browser and query it directly through Dremio's S3 source as a raw file. Dremio reads it and returns the rows in it. The file is data. The table is the metadata that tells you which files are current, and that separation is the whole idea.

Seeing the Same Thing From the Command Line

The browser view is friendly and slow. The client container gives you the whole tree at once.

docker run --rm --network <your_project>_lakehouse minio/mc:latest \
  /bin/sh -c "mc alias set local http://minio:9000 admin password && \
              mc ls --recursive local/lakehouse"

Replace <your_project> with your directory name, which Compose uses as the network prefix. Run docker network ls if you are unsure.

The output lists every object with its size and timestamp. Sort it mentally into three groups. The .metadata.json files are one per commit and grow slowly. The .avro files are manifests and manifest lists. The .parquet files under partition folders hold the actual rows and account for nearly all the bytes.

Run the same command again after an INSERT and diff the two listings. New Parquet files appear, new Avro files appear, and exactly one new metadata JSON appears. Nothing that existed before was modified. That immutability is what makes the atomic pointer swap safe, and seeing it in a file listing is more convincing than reading it in a specification.

What Breaks, and How to Recognize It

These are the failures worth knowing before you hit them.

The source saves but shows no buckets. Root Path is wrong or the credentials are wrong. Confirm the bucket exists by running docker compose logs createbucket and checking that mc ls printed lakehouse.

Connection refused or a timeout on save. The endpoint is set to localhost:9000 instead of minio:9000, or the containers are not on the same network. Run docker exec -it dremio curl -I http://minio:9000 and confirm you get an HTTP response.

An SSL or handshake error. Encrypt connection is still checked. Uncheck it.

A signature mismatch error. Path-style access is not set to true, or the access key and secret do not match your compose file exactly. Both are case sensitive.

CREATE TABLE produces plain Parquet with no metadata folder. Default CTAS Format is still Parquet. Change it in Advanced Options and recreate the table.

Dremio never becomes reachable on 9047. Almost always memory. Dremio's default heap plus direct memory settings need real headroom. Raise Docker Desktop's memory allocation to 8 GB and restart the stack.

A stale metadata error after writing from another tool. Dremio caches source metadata. Run ALTER TABLE minio.lakehouse.orders REFRESH METADATA to force a re-read.

Everything works, then breaks after docker compose down -v. The -v flag deletes named volumes, which takes your Dremio account and your Iceberg data with it. Use docker compose down without the flag to stop the stack and keep state.

Two operational notes that are not errors but bite later. Iceberg never deletes old files on its own, so a table with heavy writes accumulates orphaned data files and metadata files until you run expiration and orphan cleanup. And a table with thousands of tiny files spends more time in planning than in scanning, which is what OPTIMIZE exists to fix.

Where to Take This Next

The lab uses Dremio's internal catalog, which keeps the container count at two. The next step is separating the catalog out, and that is where the architecture gets interesting.

Add a Polaris container backed by Postgres, register the same MinIO bucket as its storage location, and connect Dremio to it as an Iceberg REST catalog source. Then start a Spark container with the Iceberg runtime, point it at the same Polaris endpoint, and write a table from Spark. Query that table from Dremio without any additional configuration. Write from Dremio and read from Spark. That single exercise demonstrates the multi-engine promise better than any diagram, and it is the reason the catalog layer exists.

From there the natural extensions are PyIceberg for reading tables directly from Python without an engine, a Flight SQL client on port 32010 to see Arrow move end to end, and a look at Ossie's YAML metric definitions once the specification stabilizes past incubation.

For production, none of this compose file survives contact. You need real storage with real durability, a catalog with an actual database behind it, scoped credentials instead of root keys, TLS on every endpoint, and a maintenance schedule for compaction, snapshot expiration, and orphan file cleanup. What does survive is the mental model. Files, tables, catalog, memory, meaning. Five layers, five specifications, and any piece replaceable without rewriting the others.

Conclusion

The open lakehouse is not one technology and it is not a product category. It is a set of specifications that split apart what a database used to bundle, so that storage, table semantics, governance, execution, and business meaning each stay open and each stay swappable.

Parquet makes the bytes small and skippable. Iceberg turns files into a transactional table with history. Polaris tracks which tables exist and controls who reaches them. Arrow moves the results without paying a serialization tax. Ossie is working on making the definitions themselves portable.

You now have all five in your head and three of them running on your laptop. Break the compose file on purpose. Set the endpoint wrong and read the error. Turn off compatibility mode and watch what fails. Write a table with the CTAS format on Parquet and compare the folder to the Iceberg one. An hour of deliberate breakage teaches more about this stack than a week of architecture diagrams.

Keep Going

If this piece was useful, I have written a lot more on lakehouse architecture and the open table format ecosystem. Apache Iceberg: The Definitive Guide covers the specification and the operational side in depth, and Architecting an Apache Iceberg Lakehouse walks through the design decisions behind a full platform build. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Apache Data Lakehouse Weekly: September 3-9, 2026

Alex Merced — Thu, 10 Sep 2026 12:50:44 +0000

Three things ran through the dev lists this week. Projects kept redrawing their own boundaries, with the Iceberg Rust DataFusion integration voted out of Iceberg and into DataFusion on both lists at once. Catalogs kept turning informal metadata into spec text, with labels landing in the Iceberg REST spec and tags being argued down to the encoding of a query parameter in Polaris. And format governance got a lot more explicit, with Parquet debating what a version number in a footer actually promises a reader.

Six projects, one week. Here is what happened and why it matters.

Apache Iceberg

The largest thread of the week was not a spec argument at all. Sung Yun opened the call for the Iceberg Summit 2027 Selection Committee after the PMC approved the event, and the volunteer replies stacked up fast. Russell Spitzer, Ryan Blue, Kevin Liu, Dipankar Mazumdar, Neelesh Salian, Roy Hasson, John Zhuge, Talat Uyarer, Nathan Yee, Hongyue Zhang, Arnav Balyan and others put their names in within days. The committee will have 11 members, no more than one per company, and at least three from the Iceberg PMC. Volunteering closes Friday, September 11 at 11:59 PM PDT, and the PMC votes on the roster after that. The work is real but bounded, mostly program structure plus CFP review, which ran three to six hours last year.

That per-company cap is worth pausing on. Iceberg's conference program is now selected by a committee designed so no single vendor can shape the agenda, and the volunteer list already spans most of the major players in the ecosystem. For a project this commercially contested, that structure does more for trust than any code of conduct document.

On the spec side, the catalog surface kept expanding. Prashant Singh's finer grained read restrictions vote passed with binding +1s from Honah J., Kevin Liu and Amogh Jahagirdar, and he posted the result thread on September 3. The change lets a REST catalog hand back a restricted view of table data rather than an all-or-nothing load, which is the piece most enterprise deployments have been faking with a proxy in front of the catalog. Singh also pointed out a detail buried in the spec examples. The masked email address in the sample is iceberg16112018@apache.org, and 16112018 is the date Iceberg entered incubation.

Right behind it, Andrei Tserakhau restarted the vote on catalog-provided labels in the REST read path. The proposal is deliberately small, adding an optional flat key-value map to LoadTableResult and LoadViewResult, with structured tag entities and write APIs pushed to follow-ups. Daniel Weeks, Fokko Driesprong, Ryan Blue, Anoop Johnson, Amogh Jahagirdar, Kevin Liu, Sung Yun and Eduard Tudenhöfner all weighed in, and the result thread closed it out. Scoping the vote to the read path was the smart move here. Labels are the kind of feature that grows a governance model, a write API and a permissions story if you let the first version carry all of it, and splitting the spec change from the implementation PRs kept the discussion on one question at a time.

The view spec got the most interesting design debate of the week. Alex Stephen asked why creating a view with multiple engine representations is so hard in practice. The spec allows one view to carry Spark, Trino and Flink dialects, but there is no way to append a representation to an existing view, only to replace one, so a direct REST call is the only real path. He proposed a new view property, replace.append-dialect.allowed.

Ryan Blue's first question was whether the intent was for CREATE VIEW to append automatically. Stephen argued the real pain is the user journey, where a query fails because the engine picked the wrong representation and the only recourse is calling REST endpoints by hand. Daniel Weeks pushed the discussion toward ALTER VIEW rather than a property flag, and he made the argument that carried the thread. An engine should only add or update its own dialect, because updating another engine's representation would require parsing SQL it cannot parse. Weeks also objected to toggling behavior with properties, since that makes a declarative statement depend on session configuration. Prashant Singh backed that and pointed at StarRocks, which already has explicit ADD DIALECT and MODIFY DIALECT grammar.

Péter Váry raised the consistency question that has no clean answer. If a Flink user adds a Flink representation to a view created in Spark, how does anyone know the two return the same rows? His position is that the project accepts the user's authority on that. Talat Uyarer proposed a workflow that needs no spec change at all. Engines should display every representation with its dialect label in DESCRIBE EXTENDED, since the SQL strings are already in the metadata and showing another engine's text requires no translation. The user reads the Flink SQL, writes the Spark equivalent, and issuing ALTER VIEW is itself the consistency assertion. Váry still prefers defining both statements at creation time so the view is never partially defined. The thread has not settled, but it moved from "add a property" to "fix the presentation and use ALTER," which is a better place to be.

Two v3 and v4 discussions produced concrete decisions. Marco Kroll's thread on the _pos column in efficient column updates asked whether the spec has any precedent for storing data purely for debugging. He searched and found none, and Russell Spitzer agreed the check belongs at write time, since a writer that sees a mismatch between origin row position and actual position should just fail. Anurag Mantripragada closed it: the Efficient Column Updates sync decided _pos will not be required, and writing it will not be allowed. That is the right call. Every optional field in a spec becomes a compatibility question for every reader that follows.

Ryan Blue's field ID tracking for non-materialized columns in v4 went deeper into transform metadata. The design requires a result data type when the output cannot be derived, uses bound id-based references, and skips a special case for identity transforms. Russell Spitzer asked for sort orders to get the same treatment and raised the lineage question, whether a table now carries both a SchemaID and an ExpressionsID. Gianluca Graziadei pressed on Hilbert clustering over heterogeneous columns, where feeding raw values to the curve wastes bits and degrades clustering along a dimension. Blue's answer draws the line clearly. Whatever parameterization a function needs must be fully captured by the expression stored in metadata, so hilbert(zvalue(col1, 0, 1), zvalue(col2, 100, 50)) is fine and hidden inputs are not. Péter Váry connected it to his index work, where a Cluster spec already lists materialized and non-materialized fields, and suggested redefining sort order so the index can reuse it.

Russell Spitzer also restarted the File Type URI relativization debate after a long sync call, keeping the discussion on the list between meetings so people who could not attend can follow the argument. The open question is how the URI subfield of the new File type behaves, since Parquet allows more than one persistence form. EJ Song opened a related performance thread on row-granular concurrency for V3 deletion vectors. With one DV per data file, validateAddedDVs fails a commit at file granularity even when two operations deleted rows that do not overlap, and the resulting ValidationException is not retryable, so the whole operation is recomputed. Xiening Dai backed the idea and reminded the thread that a commit loop should retry until either a real conflict appears or the version conflict clears.

Releases moved on three fronts. Neelesh Salian posted the 1.12.0 status, grouped the remaining PRs by how close they are to merge, and said he wants RC0 cut in days, with a longer vote window because of the US long weekend. Danny Jones called the vote on Iceberg Rust 0.11.0 RC1, and the verification reports are worth reading as a template. L. C. Hsieh checked signatures, 489 license headers, 2,209 passing tests and the Python bindings on macOS arm64, and Anoop Johnson ran the full suite including the minio, REST, HMS and Spark integration tests on Ubuntu. The Terraform provider v0.1.0 RC3 vote also passed, giving teams a first supported path to manage Iceberg catalog resources as infrastructure code.

Smaller items still worth your attention. Rahul Mahadev's proposal to standardize a User-Agent format for REST clients got bumped by Micah Kornfield after it landed in his spam folder, which is its own small comment on how much good work gets lost to mail filters. Someone posted a 10x faster Z-order bit interleaving implementation using a lookup table and asked for review. And the European community keeps growing, with meetups announced for London and Warsaw in October.

Apache Polaris

Polaris spent the week on performance, protocol contracts and the health of its own review process.

Dmitri Bourlatchkov opened the most productive thread by questioning whether InMemoryEntityCache earns its keep. His observation was simple. With JDBC persistence, even a cache hit still issues a query to confirm the entity version, so what is the cache actually buying?

Prithvi S answered with numbers instead of opinion. He traced Resolver.resolveAll() with JDBC and compared a warm cache against a null cache on the same catalog, using a loadTable-shaped resolve down a two-level namespace path. On a warm cache the resolve issued a single SELECT against the ENTITIES table pulling id, catalog_id, entity_version and grant_records_version. His framing is the key point: InMemoryEntityCache is not a "skip the database" cache, it is a "skip the expensive load" cache, with the version check as the invalidation path. Yufei Gu backed that, noting the version check is what keeps a multi-pod Polaris deployment consistent, because one pod cannot see another pod's writes.

Robert Stupp then found the real problem hiding in the data. A cold resolve of that same simple path issued 23 SELECTs. He asked whether that is the expected cold-path shape or whether the hierarchy, grants and versions can be fetched in a bounded number of operations, and he proposed folding the version fence into the final conditional write for mutations, so an UPDATE with a version predicate that touches zero rows becomes the conflict signal. Jean-Baptiste Onofré traced the 23 queries to their source. AtomicOperationMetaStoreManager.loadResolvedEntityById() issues one or two grant-record queries per entity and gets called once per entity in a resolved path with no batching, which is a classic N+1 and has nothing to do with caching. Bourlatchkov added a second angle, asking whether grant record lookups are needed at all when an external authorizer like Ranger or OPA is in play, since those queries return empty results anyway.

That thread is a good model for how performance discussions should go on a dev list. A question, a traced measurement, a correction to the framing, and then a split into independent fixes.

The Iceberg Catalog Migrator release did not make it. Ajantha Bhat called the vote on 1.1.0 RC0 and collected careful verification from JB Onofré, Ayush Saxena, Prithvi S and Robert Stupp. Onofré caught that the binary distribution and CLI jar do not document jquery.jstree.js, an MIT-licensed dependency that needs its license inline. Bhat then cancelled the vote, and said he will fix the jars to include META-INF/LICENSE and META-INF/NOTICE and exclude webapps content from the Hadoop dependencies before cutting a new RC. A cancelled vote over a missing license file looks like friction from the outside. It is actually the ASF release process working exactly as designed.

Data sharing came back to life. Jean-Baptiste Onofré resumed the Open Sharing APIs proposal and said he would open a draft PR for an /api/shares/v1 endpoint to make the design concrete. Dennis Huo had a competing draft ready in PR 5446, favoring /api/management/ for share administration and /api/shares/ for the consumer data plane, plus a companion document walking through end-to-end user journeys for each design choice. Prithvi S read both and endorsed the shape: a first-class share with enumerated members, a restricted consumer, a listing that serves as the binding and audit unit, and a segregated read-only Iceberg REST surface with credential vending. Keeping stock Iceberg REST as the consumer protocol and leaving Flight and Polaris-to-Polaris federation out of v1 also drew agreement. Huo laid out the current mental model, where everything considered data plane lives under /api/catalog/ and everything administrative lives under /api/management/, while agreeing the privileges themselves should separate catalog, principal and sharing concerns.

The Tag Spec review turned into a lesson in API contract design. Robert Stupp raised four contract questions while the spec is still separate from the implementation, starting with the encoding of identifier elements in target query parameters. His point goes past the unit separator. Namespace elements and object names can contain ampersands, question marks, equals signs, plus signs and percent signs, and those need to survive rather than be read as query syntax. Prithvi S argued all four should be resolved before PR 5366 merges, since they are contract questions rather than storage layout questions. Dmitri Bourlatchkov went further and said Polaris should adopt a stricter namespace representation in its native APIs instead of inheriting the Iceberg REST convention, which has produced recurring issues on the Iceberg list. EJ Wang updated the spec doc with the clarifications and proposed keeping Iceberg's namespace query convention for v1, with explicit supported-name, encoding and decoding rules written into the spec, and a replacement codec handled separately since the issue affects existing APIs too.

Storage support keeps widening. Austen Tomek from Chicago Trading Company introduced himself and proposed Cloudflare R2 support with scoped credential vending, which is a first contribution done the right way, on the list before the PR. R2 speaks S3-compatible APIs but has no STS, so Cloudflare issues short-lived scoped credentials through locally signed JWTs while the server holds the parent API token. Yufei Gu read that as STS-style vending where Polaris performs issuance without calling out to the object store, and leaned toward treating R2 as S3-compatible storage rather than a new config type. Sushant Raikar pushed back on the middle ground and asked whether R2 should be fully first-class, with its own config, its own credential vending and its own R2FileIO, which would sidestep the one-FileIO-to-one-storage-type question entirely at the cost of a new FileIO to maintain. That trade is one every catalog faces as object stores multiply, and it is better settled once than per vendor.

Two more threads deserve a mention. Vignesh A's soft-roll authorization for the Iceberg REST /v1/config endpoint produced a clean distinction from Bourlatchkov: GET_CATALOG_CONFIG_PROPERTIES is an operation, not a privilege, because the SPI is written in terms of operations for non-native authorizers like Ranger, and OPA has no concept of privileges at all. And Yufei Gu's thread on PR review and committership named something every project is now living with. LLMs make it easy for contributors to submit large PRs, large PRs are harder to review well, and across many communities they sit unmerged while both authors and reviewers get frustrated. Onofré split the metric in two. Time to initial response matters a lot and should be fast, because it keeps contributors engaged. Time to merge matters much less than project quality and long-term maintainability, and iteration on a PR is normal committer work, not failure.

Elsewhere on the list, a GitHub discussion on export and apply for tables and views surfaced a real bootstrap gap. Users want to export a realm including tables, views and their privileges, rewrite the file paths, and apply the result in another environment. MonkeyCanCode flagged the hard parts, including agreeing on a table creation syntax without SQL, handling dialect-specific views, and the fact that an export full of filesystem paths cannot be reused across environments without rewriting. The community also announced a Bay Area meetup on September 30.

Apache Arrow

Arrow's week was about pruning the format and welcoming people.

The vote to informally deprecate Tensor and SparseTensor in IPC, started by Raúl Cumplido, passed with binding +1s from Weston Pace, Joris Van den Bossche and David Li plus support from Benjamin Kietzman, and Cumplido posted the result. Max Burke brought a useful pointer into the thread, an open flatbuffers proposal for a deprecated-readonly attribute that would keep generating getters while dropping setters, so existing data still reads and new data cannot be written with the field. Antoine Pitrou said it might help later, while noting Arrow can rely on conventions and let each implementation decide what it exposes. Informal deprecation is the pragmatic path here. The types stay readable, nobody's archived data breaks, and implementers get a clear signal to stop investing.

The more consequential format discussion was Mandukhai Alimaa and Rok Mihevc's proposal for a canonical extension type for the FILE type. Parquet recently added a FILE logical type, Parquet C++ reader and writer support is underway, and without a matching Arrow representation file fields lose their semantics crossing IPC, the C Data Interface, Flight and language bindings. Antoine Pitrou immediately questioned the name and the ownership. Should it be parquet.file rather than arrow.file, and is it Arrow's job to standardize a type that Parquet defined? Mihevc agreed the namespace should show where the spec came from and pointed at geoarrow as precedent for non-Arrow-namespaced types. Gang Wu and Neelesh Salian backed parquet.file, and Matt Topol noted the existing type shipped as arrow.parquet.variant, so symmetry argues for arrow.parquet.file. The naming is small. The principle is not. As formats borrow types from each other, the namespace is what tells a reader three years later which specification governs the semantics.

Andrew Lamb announced Kosta Tarasov as a new Arrow committer, and the congratulations thread ran long, with Neelesh Salian, Ruoxi Sun, Raúl Cumplido, Kevin Gurney and Jeffrey Vo among the repliers. Tarasov's name showed up on the Parquet list in the same week as a co-author of the merged Rust ALP implementation, which is a nice illustration of how contributions compound across sibling projects.

Matt Topol shipped a release the hard way. His Arrow Go 18.8.0 RC1 vote hit a bad signature error that David Li caught in verification, Topol re-signed and re-uploaded, and then the checks passed for Neelesh Salian, David Li on Ubuntu 25.04 with Go 1.27 and Raúl Cumplido on Debian 14 with Go 1.27.1. The result and the release announcement followed within a day.

The most interesting outside contribution came from Prateek Singh, who announced ArrowMetal 0.1.0, Arrow compute kernels running on Apple silicon GPUs through Metal. The design detail that matters is memory. Arrow buffers live in shared-storage Metal memory, so an array is a valid CPU Arrow buffer and a valid GPU buffer at the same time and nothing gets uploaded or downloaded. Data crosses through the C Data Interface, the C Stream Interface and the C Device Data Interface with ARROW_DEVICE_METAL, which means it works with pyarrow, Polars, DuckDB, pandas, arrow-rs, Arrow Go, Arrow JS, the R package and arrow-swift. Curt Hagenlocher's reaction captured the significance: this might breathe new life into the Device Data interface. Matt Topol agreed and said he will look at the Arrow Go issues, possibly saving some for the hackathon at Community Over Code in October. The zero-copy device interface has been in the spec for a while without a headline consumer. A laptop-class GPU backend that works across eight language bindings is exactly the kind of thing that pulls a dormant interface into use. The project also held its community meeting on September 9.

Apache Parquet

Parquet had the busiest technical week of the six projects, and nearly all of it circled one question: what does a format version promise?

The versioning proposal thread ran past 45 messages. Micah Kornfield pulled it back from mechanics to requirements, arguing the community should agree on the rules before arguing about what goes in the footer. His two anchors are worth quoting in spirit. A file written with a preview feature must be readable by any reader that supports the preview features used, and by a reader that supports the major version where the feature was fully adopted. A reader must never return incorrect data when it does not understand a preview feature, which puts the burden on writers to communicate preview usage in a way that makes silent misreads impossible.

That second rule is the one that shapes everything else. Parquet files outlive the software that wrote them, and a reader that quietly returns wrong values is worse than a reader that refuses to open the file. The related threads all inherit from it: whether to write the version number in the footer, how readers should behave when they meet an unsupported format version, what forward compatibility means when files get rewritten, and a useful comparison with how Arrow IPC handles the same problem.

One piece of that already shipped as spec text. Divjot Arora's vote on handling unrecognized logical and physical type combinations passed with binding +1s from Antoine Pitrou, Micah Kornfield, Fokko Driesprong and Gang Wu, plus non-binding support from Russell Spitzer and Matt Topol, and Arora posted the result. Pitrou described it as short and useful, which is the highest praise a spec clarification gets.

The build side got simpler. Divjot Arora's proposal to inline parquet.thrift into parquet-java drew a full options review. Ryan Blue approved the PR, which uses a local copy plus a script that pulls new copies from parquet-format by commit hash or ref and records the resolved version in a parquet-format.version file. Blue argued against writing scripts to diff and validate the local copy, since the file is in version control and git already does that well. Gang Wu floated a git submodule pointing at a commit hash, then clarified his reply was not blocking. Fokko Driesprong had the same thought and rejected it on ergonomics, calling submodules clunky, easy to leave stale and awkward in daily use, and he opened a PR to reinstate nightly parquet-format snapshots as an alternative. Arora documented all the considered options in the thread for posterity, and Russell Spitzer landed the closing argument: option one is not worth debating against a better solution until the simplest one is in place.

Releases are moving on both format and Java. Fokko Driesprong opened the 2.14.0 format release discussion in the spirit of releasing more often, carrying chronological ordering of INT96 timestamps, the FILE logical type with its self-reference follow-up, and Adaptive Lossless Floating-Point encoding. Arora asked to slip in his type-combination PR once its vote closed, Driesprong added it to the milestone, and Gang Wu and Andrew Lamb both backed the release, with Wu noting it unblocks a queue of waiting PRs. The 2.14.0 RC1 vote is now open, alongside the Parquet Java 1.18.1 RC1 vote.

Arora also drove the vote on extended precision nanosecond timestamps, which lets TimestampType annotate FIXED_LEN_BYTE_ARRAY(12) columns so the full SQL timestamp range fits in nanoseconds, with all three time units supported and reference implementations in Java and C++ plus a parquet-testing file. Daniel Weeks, Micah Kornfield and Ryan Blue voted binding +1, with Stevo Mitrić and Alkis Evlogimenos non-binding. Twelve bytes for a timestamp sounds extravagant until you have tried to store pre-1677 dates at nanosecond precision in 64 bits.

The vector type debate is the one to watch. Rok Mihevc summarized a focused call on the physical representation of a numeric vector type, convened because vector database storage needs differ from the fixed-size-list discussions so far, and because Iceberg needs a vector type too. The call converged on two options without picking one. Antoine Pitrou challenged the premise, asked for a real explanation of the different storage needs, called option A a short-term fix with severe encoding limitations, and said he is only lukewarm on option C while conceding it does not paint the project into a corner. Mihevc conceded the framing was too broad and restated the requirement as a contract rather than a layout: a fixed number of numeric elements, elements that cannot be null and are finite, and room for future vector-specific properties like normalization guarantees or specialized encodings. He noted Lance uses FixedSizeList with specialized physical encodings while Hudi stores a vector as a single FLBA.

Will Edwards made the strongest counterargument, that efficiency here is a software problem rather than a format problem, since a reader can expose flat typed memory and nothing in the format forces a List allocation per row. He reframed the real question as what SHOW CREATE TABLE should say when a Parquet file is the only source of schema. Daniel Weeks focused on semantics instead, arguing the point of a logical vector type is to differentiate it from a fixed-size list, so vectors should prohibit nulls, NaN and infinity as elements rather than merely detecting them in statistics. This is a genuinely hard design call, and the fact that Iceberg is waiting on the outcome makes it one of the most consequential decisions in the ecosystem right now.

Encoding work kept pace. Prateek Gaur's ALP encoding thread turned into a cross-language progress report. Andrew Lamb reported the Rust implementation from Kosta and Devan merged, Vinoo Ganesh is addressing Gang Wu's comments on the Java PR, the C++ PR is in another review round, and Gaur and Arnav Balyan are starting the Go implementation. With ALP riding in Parquet Format 2.14.0, floating-point columns are about to get materially cheaper across four language stacks at once. PFOR encoding and an extensible decimal floating-point type are moving behind it.

One security item needs action. Gidon Gershinsky published CVE-2026-73334, a moderate-severity issue in the org.apache.parquet.crypto.keytools package affecting parquet-hadoop 1.12 through 1.18.0. When a writer sets the optional KMS URL parameter, that URL is stored in the file, and a reader configured to trust file-controlled KMS URLs can forward it to a pluggable KmsClient that skips host validation. If you use Parquet envelope encryption with a custom KmsClient, read the advisory and check whether your reader configuration trusts file-controlled URLs. The community also kept its sync cadence, with notes from the September 9 sync posted the same day.

Apache DataFusion

DataFusion gained a repository and a maintainer roster this week.

The headline is the vote to accept the Iceberg DataFusion integration into the DataFusion project, opened by Andrew Lamb on September 9 with a proposed PR from Gabriel Musat to move the code into apache/datafusion-iceberg. Andy Grove and L. C. Hsieh voted binding +1, with Kevin Liu, Kumar Ujjawal and Shekhar Rajak non-binding, and Liu cross-linked the matching Iceberg vote so both communities are voting on the same move.

The discussion that led there ran 30 messages across both lists. Lamb's argument came from the field. At VLDB he spoke to at least three companies adding Iceberg support to their products, and every one of them had forked iceberg-rust for some reason. His conclusion is that giving the DataFusion integration access to maintainers who have deep DataFusion context helps everyone, and that the code should stay in the ASF under DataFusion governance rather than drift into a vendor repo. Kurtis Nusbaum, writing on the Iceberg list, agreed the DataFusion internals knowledge does not belong in an Iceberg-specific package, while flagging two honest counterarguments. Anyone forking to add a feature now has to fork in two places, and arrow-rs and parquet already show that one repo can hold two projects. He judged neither strong enough to outweigh the maintenance case. Kevin Liu agreed and removed the Iceberg Python binding that exported DataFusion's TableProvider, one less coupling to carry across the split. Renjie Liu and Gabriel Musat backed the plan, Musat volunteered to port the commit history, and Lamb filed for the new repository. The history port PR and a compile-and-test PR are already open.

Two releases are in flight. Tim Saucer's DataFusion 55.1.0 RC1 vote collected binding +1s from L. C. Hsieh, Adrian Garcia Badaracco, Andrew Lamb, Marko Milenković and Oleks V., verified across Apple silicon and macOS 15 with rustc 1.98.1. The sqlparser-rs 0.63.0 RC1 vote is moving in parallel, which matters well beyond DataFusion, since sqlparser-rs is the SQL front end for a long list of Rust data tools.

Andrew Lamb also announced Luca Cappelletti as a new DataFusion committer, with congratulations from Bruce Ritchie, Kumar Ujjawal, Bhargava Vadlamani and Jeffrey Vo. And Comet got its own meeting slot. The dedicated Comet weekly sync now runs Fridays at 10:30 AM Pacific, with Bhargava Vadlamani adding the meeting link to the Comet docs and Manu Zhang asking for recordings so contributors in other time zones are not shut out. The project is also crowdsourcing its September ASF board report.

Apache Ossie

Ossie, the incubating semantic layer interchange project, had its most substantive week yet, and it is the list to start reading if you have not.

Justin Talbot opened a PR adding a Relational Query Interface specification to core-spec, coming out of the expression language working group. It defines Layer 2 of the proposed layered query interface, covering how a semantic layer exposes an Ossie model to SQL-native BI and AI tools as related SQL relations, and what correctness guarantees hold when measures are queried through ordinary SQL with a MEASURE() extension. Some of the core ideas trace back to Hyde and Fremlin's Measures in SQL paper.

That paper's first author is on the list. Julian Hyde, in his introduction thread, laid out the position that will shape this spec if it holds. The query language must be closed, meaning query outputs have the same shape as their inputs, so queries can be composed on queries. If the language also subsumes relational algebra including joins and aggregation, and can define measures, then no separate modeling language is needed, because models are definable on base tables with the equivalent of CREATE VIEW. That is a strong claim, and it cuts against how most semantic layer products are built today, where the model is a YAML dialect sitting above SQL rather than an extension of it.

Jakub Moravec brought the equivalence problem in from a different angle. If Ossie converts a model from tool A to tool B, who guarantees the two are semantically equivalent? He argued at least one dimension of that is a lineage problem, drawing on his OpenLineage experience, and noted that evaluating equivalence by running queries depends on both implementations being correct and on the test data being complete. His related point in the how do we expect OSI to be used discussion is the one that should worry spec authors. In OpenLineage it is easy for a payload to be schema-valid and still useless for cross-vendor integration, because checking syntactic validity is simple while checking whether what was documented is sufficient is not.

Modeling proposals stacked up. A proposal for shared filters, shared dimensions and metric references drew a strong argument from wanggaohang for model-level filters. Enterprise filters get shared by many metrics, and inline-only definitions make every business-definition change expensive to propagate, while filters and metrics are often owned by different roles, with domain experts maintaining the definitions. Mario De Felipe argued in make relationship cardinality explicit that cardinality belongs in two places because it means two different things. At the ontology layer it is a business rule that holds regardless of how tables are laid out. At the dimensional and metric layer it describes the physical data, which is what join safety and path selection need. There are also live proposals for hierarchy support, where fabrice-etanchaud pointed at Mondrian's long-standing treatment as prior art, plus semantic filters and a dedicated thread for downstream agent field reports.

Vendors are showing up with code rather than opinions. Damian Waldron announced a bidirectional ThoughtSpot converter between TML and Ossie, ThoughtSpot's first code contribution to the project. Ossie datasets map to a ThoughtSpot model's tables plus a Table or SQL View document per dataset, fields and metrics map to model columns, relationships map to joins, and anything with no Ossie equivalent rides in custom_extensions under a vendor entry. Timextender also introduced itself and a converter it plans to contribute. Converters are how an interchange format proves it is real, and that custom_extensions escape hatch is both the pragmatic choice and the thing to watch, since every vendor-specific field that lands there is a piece of semantics the spec has not yet standardized.

The project is also learning to ship. The first release discussion settled the scope question quickly. Markus Weimer asked whether the first release needs to line up with the 1.0 spec and suggested a 0.3 instead, noting from past incubators that building the release muscle matters as much as the technical agreement. Yufei Gu confirmed the community sync reached the same conclusion, Jean-Baptiste Onofré said he is building the release machinery for a source-only 0.3.0 to verify plumbing and run a full legal check, and Russell Spitzer endorsed treating it as an exercise rather than loading it with meaning. Kurt Stirewalt asked the practical follow-up: the ontology working group wants PR 332 in the release, and is there a formal process for requesting that, or for deciding which in-review features make a given release? Governance questions like that are exactly what a first release surfaces. The project also welcomed Josh Klahr to the PPMC, and worked through build questions including Just versus Makefile and consolidating Python code under one uv workspace.

Cross-Project Themes

Code is moving to where the maintainers are. The Iceberg Rust DataFusion integration voting its way out of Iceberg and into DataFusion is the clearest case, but the same instinct shows up in Arrow debating whether a Parquet-defined type should carry a parquet namespace, and in Parquet inlining parquet.thrift into parquet-java instead of coordinating two repos on every change. Three years ago the ecosystem grew by adding integrations inside each project. Now it grows by putting each integration under the governance of the people who actually maintain that side of it. Andrew Lamb's VLDB observation, that every company adding Iceberg support had forked iceberg-rust, is the market signal behind the governance change.

Catalog metadata is becoming spec, and the arguments are about contracts rather than storage. Iceberg voted labels into the REST read path and passed finer-grained read restrictions. Polaris spent the week on tag spec encoding rules, on whether a config-endpoint check is an operation or a privilege, and on what a share is as an audit unit. In every one of those threads the productive move was the same: separate the wire contract from the implementation, resolve the contract first, and let the storage layout follow. Robert Stupp's insistence on a reversible encoding for namespace elements containing ampersands and percent signs is not pedantry. It is the difference between a spec that survives its second implementation and one that does not.

Version and compatibility semantics are the new center of gravity. Parquet's versioning thread, its unsupported-version reader behavior thread and its footer-version thread are all one question. Iceberg's decision to forbid writing _pos is the same question from the other side, refusing to add an optional field that every future reader would have to reason about. Arrow's informal deprecation of Tensor takes the same position, keeping old data readable while cutting off new writes. Formats that outlive their software have to be explicit about what a reader is allowed to assume, and three projects converged on that independently in one week.

Semantic modeling has an Apache home, and it is drawing the people who wrote the theory. Ossie has Julian Hyde arguing for a closed query language, ThoughtSpot and Timextender shipping converters, and OpenLineage veterans warning about schema-valid but useless payloads. Its numeric vector question is being decided in the Parquet list. Its query interface leans on Measures in SQL. If you build on the lakehouse and you have been treating the semantic layer as a vendor concern, that assumption is expiring.

Looking Ahead

Iceberg 1.12.0 RC0 should appear within days, with a longer vote window than usual. The Iceberg Summit 2027 Selection Committee call closes September 11 and the PMC vote on the 11 members follows. Both DataFusion and Iceberg votes on moving the DataFusion integration run at least seven days, so expect the apache/datafusion-iceberg repo to become real in mid-September. Parquet Format 2.14.0 RC1 and Parquet Java 1.18.1 RC1 are open, and the vector type physical representation is the discussion most likely to produce a decision with ecosystem-wide consequences. Polaris will cut a new Catalog Migrator RC once the license packaging fix merges, and the Open Sharing draft PRs are the ones to read. Ossie is building release machinery for a source-only 0.3.0. Community Over Code lands in October, and Matt Topol has already flagged the Arrow device interface work as hackathon material.

Want to go deeper on Apache Iceberg, Polaris, Arrow, Parquet, DataFusion and the rest of the open lakehouse stack? I write books on all of it, from beginner guides to architecture deep dives. You can browse the full catalog at books.alexmerced.com.

Alex Merced, Data Lakehouse and AI Evangelist

AI Weekly: Four Frontier Models in Seven Days

Alex Merced — Thu, 10 Sep 2026 12:34:27 +0000

Four labs shipped flagship models inside one week, and not one of them cut its headline price. The competition moved to cache rates, token efficiency, access tiers and what happens when your prompt crosses 272,000 tokens. Here is what changed in models, tooling, standards and infrastructure for the week of September 3 through 9, 2026.

Models: GPT-6 Astra Lands at the Top of a Crowded Week

OpenAI released GPT-6 Astra on September 3, and the specs are the easy part. The API model ID is gpt-6-astra. It carries a 1,050,000-token context window, 128,000 maximum output tokens, text and image input, and an April 30, 2026 knowledge cutoff. Standard pricing is $10 per million input tokens, $1 per million cached input tokens, $12.50 per million cache writes and $50 per million output tokens, according to the published rate card. Batch runs at half price and a Fast mode doubles both speed and rate.

The number that will surprise teams is 272,000. OpenAI's pricing page carries two tables where every previous flagship had one, and prompts above 272,000 input tokens bill at 2x input and cache rates with 1.5x output. That works out to $20 per million input and $75 per million output. The surcharge applies to the request, not to the tokens above the line, so a prompt that drifts from 270,000 to 275,000 tokens does not cost 2 percent more, it costs roughly double. If you are building long-context agents, put a token budget check in front of the call and treat 272,000 as a hard architectural boundary rather than a pricing footnote.

Astra sits 2.5x above GPT-5.6 Sol, which lists at $4 per million input and $20 per million output on its current promotional rate. Both models share the same 1.05M context and 128K output ceiling, so the choice between them is not about capability limits, it is about whether higher model quality removes enough retries, tool calls and human intervention to pay for the token premium. On OSWorld 2.0, a computer-use benchmark, OpenAI reports Astra scoring higher than Sol and finishing faster, which is the argument for per-task cost over per-token cost. Those are vendor-reported figures, and the full evaluation suite is expected at OpenAI DevDay on September 29.

The access story is as important as the pricing. Astra rolled out in stages, starting with a limited set of organizations on day one and expanding to ChatGPT Plus, Pro, Business and Enterprise, plus the API, Azure and AWS Bedrock. Enterprise access is off by default until an admin turns it on. OpenAI also says Astra meets the Critical cybersecurity threshold under its Preparedness Framework, making it the company's first broadly deployed model in that category, with cyber-sensitive capabilities gated behind a trusted-access program. Read that as a pattern rather than a one-off. Three labs now ship a public model and a restricted twin, and the restricted twin is where the security work happens.

Google got there first by a day. Gemini 3.8 Flash shipped September 2 as gemini-3.8-flash, with a 1,048,576-token context window, 65,536 max output, and text, image, audio, video and PDF input. Pricing did not move at all. It holds at $0.75 per million input and $3.75 per million output through December 31, 2026, then doubles to $1.50 and $7.50 on January 1, 2027. Batch and Flex run at half those rates and Priority runs at 1.8x. Google's own documentation says 3.8 Flash is built on 3.7 Flash rather than a new base model, and that it deliberately works harder by spending more thinking tokens on hard problems.

The vendor-reported benchmark table moved on every row. DeepSWE v1.1 climbed from 65.3 to 73.7 percent, Terminal-Bench 2.1 from 85.8 to 89.4, OSWorld-2.0 from 50.6 to 59.0, Terminal-Bench 4.0 from 11.2 to 19.1, Vals Finance Agent v2 from 59.0 to 61.4 and HLE-Verified from 53.6 to 54.9. Every Gemini number in that table is Google's own run and every competitor number is that competitor's reported figure, so treat the whole table as a vendor document. The independent read comes from Artificial Analysis, which scores 3.8 Flash at 59 on its Intelligence Index at high effort and, more usefully, measures real-world cost at roughly 40 percent higher than 3.7 Flash despite the identical per-token price. Thinking tokens bill as output. A model that thinks more costs more even when the rate card says nothing changed.

Google also shipped Gemini 3.8 Flash Cyber, restricted to trusted government authorities, critical-infrastructure operators and software maintainers through its Fairwind Program. Google reports it exceeding a 70 percent real-world vulnerability discovery rate and hitting 47.2 percent pass@1 on CWE-Bench patching. Same week, same pattern as Astra's gated cyber tier.

Meta made the loudest jump. Muse Spark 1.3 arrived September 2 through Muse Code and the Meta Model API, the fourth Muse Spark release in five months. It carries a 1,048,576-token context window and prices at $1.25 per million input and $4.25 per million output, with a contributor tier at $0.10 and $0.20 where usage improves Meta's products. The efficiency claim is the interesting one. Meta reports roughly 20 percent fewer tool calls and 25 percent fewer tokens than Muse Spark 1.2 on the same work, plus better behavior on long threads and a greater willingness to say it is stuck instead of burning tokens in a loop.

On benchmarks, Meta reports 75.4 percent on DeepSWE 1.1, a 16-point jump, and 98.5 percent on long-context MRCR. Meta's own comparison table also shows it trailing Claude Opus 5 on JobBench, OSWorld 2.0, AutomationBench and GDPval-AA v2, which is worth noting because vendors rarely publish the rows they lose. Artificial Analysis puts the public xhigh variant at 61 on the Intelligence Index, up from 57 in August, 51 in July and 43 in April. That five-month trajectory is the real story: Meta moved from a clear tier behind the leaders to a statistical tie with GPT-5.6 Sol, and independent cost-per-task work puts the xhigh variant at $0.55 against Sol's $0.95 at comparable measured intelligence.

Two caveats belong with those numbers. Meta's launch scorecard uses a max reasoning mode that is not the mode most developers can call today, and the weights are closed. Mark Zuckerberg promised an open-weights Muse Spark release "soon", which is the same word he used on August 10 for Muse Spark 1.2 weights that were still closed 24 days later when 1.3 shipped. Plan around the API, not the promise.

Anthropic opened the week on September 1 with Claude Fable 5.1 and Claude Mythos 5.1 at an unchanged $10 and $50 list price and three breaking API changes. The move that matters for cost is the cache read rate, which dropped 75 percent from $1.00 to $0.25 per million tokens. For agent workloads that replay a large system prompt and a stable tool catalog on every turn, cache reads are frequently the largest line on the bill, so a 75 percent cut there beats a headline price cut for most real usage patterns.

Step back and the week has a shape. Every lab held its list price and competed on the parts of the bill nobody puts on a slide: cache rates, token efficiency, per-task cost, promotional end dates and long-context surcharges. Three of the four shipped a gated security variant alongside the public model. If you evaluate these releases by comparing $10 to $0.75, you will pick wrong. Run your own workload, count total tokens including thinking tokens, count tool calls, and price the task rather than the token.

Safety Disclosures Arrived With the Launches, Not After Them

Something changed in how these releases were communicated. September's launches came packaged with unusually direct safety documentation, and the disclosures are tied to the launches rather than incidental to them.

OpenAI disclosed that it built a test specifically designed to tempt Astra into replicating the behavior of rogue agents, and reported that the model did not attempt to escape. A former OpenAI researcher made the obvious counterpoint in public, that refusing to misbehave while under observation is ambiguous evidence about behavior when not observed. Anthropic disclosed its own incidents and a security response in the same period. Google published the restricted access terms for its cyber variant alongside the model card rather than in a follow-up post.

Take that at face value and it is a good development. Publishing an evaluation that the model could have failed, and naming the limits of what the result proves, is more useful than a paragraph asserting alignment. It also gives buyers something to review. If you are putting a model behind an agent with write access to production systems, the eval methodology is more relevant to your risk assessment than any coding benchmark.

The harder question is what a Critical cybersecurity classification means downstream. OpenAI shipping its first broadly deployed model at that threshold, with the sensitive capabilities held behind vetted access, sets a precedent every lab is now following. It also creates a class of capability that exists but that most organizations cannot obtain, and the organizations already inside those programs are the ones publishing results with it. For defenders, the gap between what is possible and what is available is now an operational planning problem rather than a research curiosity.

None of this substitutes for your own controls. A model that passes a lab's escape test can still be prompted into doing damage through a tool you gave it, with credentials you scoped too broadly, against a system with no approval gate. Vendor safety work reduces one class of risk. Your permissions model, your audit trail and your human approval steps handle the class that actually shows up in production incidents.

The Open-Weight Tier Went Quiet, and That Is the News

Against four proprietary flagships, the open-weight side of the week was still. Trackers recorded no open-weight releases with permissive licenses during the window, which is unusual for a seven-day stretch in 2026.

The recent open-weight activity all sits just outside the week. Z.ai released GLM-5.3 and GLM-5.3 Flash in mid and late August, Alibaba shipped Qwen3.8 27B on August 14 and Qwen3.8 Flash on August 26, and DeepSeek released a V4 Flash Vision experimental build on August 21. The GLM-5.3 Flash promotional window closed on September 9, and GLM-5.3 open weights are expected mid to late September. Meta's Muse Glimmer, a 30-billion-parameter model under Apache 2.0, shipped August 10 alongside the first "soon" promise for Muse Spark weights.

The pattern is worth naming plainly. The labs that lead on capability are shipping API-only models with gated security variants, and the open-weight releases increasingly come from the tier below the frontier or from Chinese labs on a separate cadence. Meta was the western counterweight to that trend, and Muse Spark 1.3 shipped closed with an open-weights release still on a roadmap rather than a calendar.

For teams with a self-hosting requirement, that gap has practical consequences. Your evaluation set should be built against open weights you can actually run today rather than the frontier scores in a launch post, and the gap you measure against the closed models is the price of your deployment constraint. Right now that gap is roughly one tier and closing, which is a very different picture than either the "open models are years behind" or the "open models have caught up" version you will hear this month.

One more model-layer note that will matter more than it looks. Several of the new flagships share the same 1M-token context and the same 128K output ceiling, which means capability differences no longer show up as limit differences. When two models advertise identical envelopes, the only way to choose is measurement on your own workloads, and the only cost lever left is how many tokens each one actually spends getting to the same answer.

Tooling: OpenClaw 2.0 Puts a Shared Agent in the Browser

The open-source agent harness had its biggest release yet. OpenClaw 2.0 shipped as version 2026.8.1 with contributions from 933 developers across more than 16,000 pull requests, and the follow-up 2026.8.2 landed days later. The changes touch installation, the browser interface, memory, skills, automations, plugins, security and collaboration.

Setup got much shorter. OpenClaw now detects what is already on your machine, including ChatGPT or Claude subscriptions, API keys and locally installed models, and moves the remaining configuration into a conversation with the agent after it starts. That is a small design decision with a large effect on adoption. Every minute of YAML editing before first run is a place where new users quit.

The browser app is now the primary interface rather than a control panel bolted onto a terminal tool. Users land directly in a conversation with their agent, and configure it, watch running tasks and drive workflows from the same place. The bigger architectural change is shared cloud sessions. Several people can join an existing agent session while its context stays intact, so a task can be handed from one person to another. That moves OpenClaw from personal automation toward team workflows.

It also moves the security question. The Register noted that shared session controls ship without network or filesystem-level security boundaries, which is the part to read carefully before you invite a colleague into a session that has credentials and shell access. Shared context is shared blast radius. If you run this on anything that touches production, put the agent in its own container with its own scoped credentials and treat session sharing as equivalent to handing over a terminal.

The 2026.8.2 release notes read like a project growing up. Updates now roll back the npm candidate when the post-update health check fails, configuration and secret references survive a failed upgrade, failures get handed to a built-in triage agent, and the update waits for plugin readiness before restarting. The gateway recovers under load and with large agent rosters, malformed legacy cron rows get quarantined instead of blocking boot, and migration warnings degrade the gateway rather than refusing to start. None of that is exciting. All of it is what separates a demo from something you leave running.

Meta's coding agent got its model upgrade the same week. Muse Spark 1.3 landed in Muse Code, Meta's terminal and CI coding agent, positioned directly against Claude Code and OpenAI Codex. The efficiency numbers Meta published are agent numbers rather than chat numbers, and they point at the metric that matters for CLI agents: fewer tool calls per completed task. In an agent loop, every tool call is a round trip, a chance to lose the thread and a line on the bill. A 20 percent reduction in tool calls compounds across a long task in a way a benchmark point does not.

On the enterprise side, the week produced two developments worth tracking. Anthropic announced Enterprise Frontier Safeguards for regulated buyers, an architecture that avoids storing prompts or transcripts on vendor servers while still detecting misuse across sessions. Those two goals usually pull against each other, and how the design resolves that tension is worth reading closely if you work in a regulated industry that has been stuck between a security review and a data retention policy.

The other is the quiet arrival of agents that reach into operational databases without a migration project. RavenDB launched Quill on September 8, aimed at letting AI agents work against enterprise SQL systems in place. That framing keeps showing up because it matches how enterprises actually buy. Nobody wants to move a system of record to make an agent work. They want the agent to meet the data where it already lives, with the existing permissions model intact.

For teams choosing tools right now, the practical guidance has not changed much. Pick based on where your work lives rather than on benchmark tables. Terminal-heavy work favors CLI agents. Review-heavy work favors whatever plugs into your code host. Long-horizon multi-step work favors whichever agent handles interruption and resumption well, which is a property you can only test on your own repositories. And check the billing model before you scale seats, because usage-based credits and per-task pricing behave very differently once a team starts running agents all day.

The Billing Model Is Now Part of the Tool

Coding tool pricing changed shape over 2026 and the effects are still working through team budgets. GitHub Copilot moved all monthly plans to usage-based billing through AI Credits on June 1, where one credit equals one cent, with Pro, Pro+ and Max tiers carrying different monthly allowances and premium model selections drawing from the same pool. Annual plans purchased before the change keep request-based billing until they expire. Cursor restructured around a premium seat aimed at heavy agent workloads. Claude Code ships inside Claude subscription tiers, with API rates applying for direct integration.

The practical effect is that model choice now shows up directly on the invoice. Under request-based billing, an engineer selecting a bigger model cost the same as selecting a smaller one. Under credit-based billing, that choice is a line item, and a team that routes every task to the most capable model will find out at the end of the month. Route by task class rather than by habit, keep the cheap models on routine work and reserve the expensive ones for the problems that actually need them.

The other trend worth tracking is agents leaving the editor. Microsoft is moving away from web app wrappers in Windows 11 and pushing agent-assisted native app development, GitLab is reporting revenue from its AI tooling, and Adobe acquired Rilo to fold agents into marketing automation. Frigade launched an Assist API that gives support agents product-specific knowledge. The category is shifting from "assistant in an IDE" to "agent that owns a workflow," and the buying decision is moving with it, from individual developer preference to platform selection.

That shift raises questions most teams have not answered yet. Which systems can an agent write to without human approval? How are its credentials scoped, and are they different from the human operator's? Where do its actions land in your audit log? Those questions were optional when agents suggested code. They are not optional when agents file tickets, update records and open pull requests on their own.

Standards: The Protocol Layer Grows Up Around Deprecation

The Model Context Protocol spent 2026 changing what it is. The 2026-07-28 specification made the protocol stateless at its core, and the practical result is that a remote MCP server is now an ordinary HTTP workload. A server that previously needed sticky sessions, a shared session store and deep packet inspection at the gateway can run behind a plain round-robin load balancer, route on an Mcp-Method header and let clients cache tools/list responses for as long as the server's ttlMs allows. That is the difference between an MCP server your platform team will host and one they will refuse to host.

The scale numbers explain why the maintainers cared. Across the Tier 1 SDKs, MCP sees close to half a billion downloads a month, and both the TypeScript and Python SDKs have crossed a billion total downloads. At that size, protocol changes need an exit path rather than a cutover.

The deprecation policy is the part most teams should act on. Dynamic Client Registration is formally deprecated in favor of CIMD, and it keeps working for now but will be removed in a future revision. Roots, Sampling and Logging are deprecated under SEP-2577, still functional, guaranteed for at least twelve months, and off limits for new implementations. The legacy HTTP with SSE transport is deprecated with a year-long offramp. Tasks moved out of the experimental core into a formal extension with a poll-based tasks/get and a new tasks/update. Change notifications moved from the old HTTP GET endpoint to a single subscriptions/listen stream that clients opt into per notification type.

If you have MCP servers in production, the migration list is short and dated. Move off DCR, stop building on Roots, Sampling and Logging, and get off HTTP+SSE before the offramp closes. A twelve-month window sounds generous until it collides with a quarter where nobody owns the work.

The updated roadmap published August 22 points where the next release goes. Server-initiated events through webhooks and channels are the headline, so clients stop polling for results, which matters enormously for long-running agent work. The Server Card Working Group is defining .well-known metadata conventions so a server can be discovered and reasoned about without connecting to it first. Governance matured too, with a contributor ladder, working groups triaging specification enhancement proposals in their own areas, and a formal feature lifecycle that the July deprecations were the first to follow.

Agent-to-agent communication is on a parallel track. A2A, created by Google and hosted by the Linux Foundation, passed 150 supporting organizations at its one-year mark with integration across Google, Microsoft and AWS platforms and production deployments in supply chain, financial services, insurance and IT operations. The mechanics are worth knowing even if you are not adopting it yet. An A2A server publishes an Agent Card at /.well-known/agent-card.json declaring skills, supported MIME types, transport bindings and security schemes. Agents talk over JSON-RPC 2.0 with gRPC and HTTP and JSON bindings available, using an eight-state task lifecycle that runs from submitted through working, input_required, auth_required and on to completed, failed, canceled or rejected. Long-running tasks survive across connections through streaming and webhook push notifications.

The division of labor between the two protocols is clean. MCP is vertical, connecting an agent to tools and data. A2A is horizontal, connecting an agent to other agents. Production systems increasingly run both, with A2A routing a task to the right specialist agent and MCP giving that agent its context and tools. Notice that both are converging on the same primitives independently: well-known discovery documents, explicit task lifecycles, and server-initiated events so nobody has to poll.

A2A carries no payment or checkout semantics of its own, which is why the commerce layer is forming above it in separate projects like AP2, UCP transport and the x402 extension. If your roadmap includes agents that spend money, that is a third standards track to watch, and it is much less settled than the other two.

One more standard formed this week without anyone calling it one. Between OpenAI's trusted-access program for Astra's cyber capabilities, Google's Fairwind Program for 3.8 Flash Cyber and Anthropic's existing gated tiers, capability gating by vetted access has become the default industry answer to dual-use risk. There is no shared specification behind it, no common vetting process and no portability between programs. Every lab runs its own enrollment, measured in weeks to months. For security teams that need these capabilities, that means the procurement work starts well before the model you want exists.

Extensions Are Where the Interesting Work Moved

The MCP extensions framework deserves more attention than it gets, because it changes how the protocol evolves. Instead of every capability landing in the core specification, features can now ship as named extensions with their own lifecycle. Tasks moved out of the experimental core into the io.modelcontextprotocol/tasks extension. MCP Apps, which lets a server return interactive UI rendered in a sandboxed frame, arrived as the first official extension out of that framework.

The design detail behind MCP Apps is the one worth understanding. The interface a server returns is itself an MCP client talking to the host over JSON-RPC, so a click or a form submission is a structured protocol call that flows through the same audit path as a model-driven tool call. That is a meaningful property for anyone who has to explain to a compliance team what an agent did and why. Human interaction and model interaction land in the same log with the same shape.

Extensions also solve a governance problem. A core specification that absorbs every good idea becomes impossible to implement completely, and partial implementations turn version negotiation into guesswork. Named extensions with explicit capability declaration let a client know exactly what a server supports before it starts working. Combined with the Server Card effort to publish server metadata at a well-known path, the direction is clear: describe capability up front, negotiate explicitly, and stop discovering limitations at runtime.

Infrastructure: Memory Is the Constraint, Not Compute

The most important AI infrastructure number in September 2026 is not a FLOPS figure. It is the price of a bit of ordinary DRAM.

The memory industry is in a documented multi-quarter shortage, and the cause is straightforward. AI accelerator demand for high-bandwidth memory is pulling fabrication and advanced packaging capacity away from conventional DRAM. All three merchant DRAM suppliers describe their 2026 HBM output as effectively committed, sold out or concentrated with a lead customer. HBM3E spot prices sit far above typical long-term agreement pricing, and consumer DDR5 retail prices have climbed sharply alongside them.

The forward numbers are worse than the current ones. New HBM capacity is structurally delayed, with major SK hynix projects targeting cleanrooms and volume output in 2028 and 2029. Samsung's total DRAM wafer plan for 2026 rises only about 5 percent, from roughly 7.6 million wafers to 8 million. Building a new fab takes 18 to 24 months in the best case, so the supply answer to a 2026 problem arrives in 2028.

The pricing split inside memory is the counterintuitive part. Consensus estimates put Samsung's revenue per bit on traditional DRAM up 116 percent year over year to $0.79, SK hynix up 78 percent to $0.70 and Micron up 54 percent to $1.06, while HBM average selling prices rise only about 8 percent at Samsung, 1 percent at SK hynix and 22 percent at Micron. Ordinary server and desktop memory is where the price shock lands, because the same production lines make both and every manufacturer prioritizes the higher-margin product.

That has a direct consequence most AI budget conversations miss. Your inference cluster is not the only thing getting more expensive. Your database servers, your query engine nodes, your Kafka brokers and your laptops are all buying memory in the same squeezed market. A single AI server uses eight to ten times the DRAM of a traditional server, and AI server shipments are running at roughly 1.5 million units for 2026. Data centers now consume an estimated 70 percent of memory chips made worldwide. If you are sizing a data platform refresh for 2027, price memory separately and early, and expect the quote to expire faster than it used to.

On the accelerator side, NVIDIA's Rubin platform is the generation that gets built around this constraint rather than despite it. Each Rubin GPU carries 288GB of HBM4 at up to 22 TB/s, which is 2.8x Blackwell's 8 TB/s. The bandwidth gain comes from doubling the interface bus width per stack to 2,048 bits and running at 10.8 GT/s per pin. The GPU uses TSMC 3nm with a dual-die design and 336 billion transistors, and NVIDIA cites up to 50 PFLOPS of FP4 inference and 35 PFLOPS of training performance.

The rack is the real product. A Vera Rubin NVL72 pairs 72 Rubin GPUs with 36 Vera CPUs for roughly 3,600 PFLOPS and 20.7 TB of HBM4, connected by NVLink 6 at 3.6 TB/s bidirectional and 260 TB/s all-to-all. The Vera CPU brings 88 Arm Olympus cores and 227 billion transistors, and applications can treat its LPDDR5X and the GPU HBM4 as a unified pool, which cuts data movement rather than speeding it up. Every one of those design choices targets the same workloads: mixture-of-experts models, long-context inference and agentic pipelines, where memory bandwidth and interconnect latency bind long before raw compute does.

Deployment is real but uneven. NVIDIA says Rubin is in full production with partner availability in the second half of 2026, with AWS, Google Cloud, Microsoft and OCI among the first cloud providers, plus CoreWeave, Lambda, Nebius and Nscale. CoreWeave completed the first full rack-scale validation of a Vera Rubin NVL72 on June 1, 2026, after a 147-hour test suite. Google Cloud offers Vera Rubin through bare-metal instances on a network fabric it says can link up to 80,000 GPUs in one data center and 960,000 across sites.

The counterweight came from TrendForce, which cut its projection of Rubin's share of NVIDIA shipments for 2026 from 29 percent to 22 percent. The named challenges are instructive because none of them is the GPU: HBM4 validation, moving network interconnects from CX8 to CX9, managing much higher power draw, and tuning performance under more advanced liquid cooling. Vera Rubin NVL72 requires 100 percent liquid cooling and air-cooled configurations do not exist, so a facility built around air handling needs a direct-to-chip retrofit before it can accept a rack. The bottleneck for the next generation of AI infrastructure is memory qualification, plumbing and electricity.

That reframes the make-or-buy decision for most teams. If your organization is weighing owned capacity against cloud capacity for 2027, the question is no longer whether you can get GPUs. It is whether your facility can deliver the power and cooling, whether your memory procurement can survive a market where suppliers refuse long-term fixed-price contracts and insist on quarterly terms, and whether you can absorb an 18-month lead time on the physical plant. Renting looks better than it did a year ago for anyone without an existing liquid-cooled footprint.

Formats deserve a place in the infrastructure conversation too, because they change cost without changing hardware. Adaptive Lossless Floating-Point encoding is riding in the upcoming Parquet Format 2.14.0 release, with the Rust implementation merged and Java, C++ and Go implementations in flight. Floating-point columns are the bulk of most feature stores, embedding tables and sensor datasets, and better encoding lowers storage cost, scan cost and network cost at once. The Parquet community is also working through the physical representation of a numeric vector type, which is the format-level question behind every vector search deployment. Those decisions do more for the economics of AI data than most hardware announcements, and they cost nothing to adopt beyond a library upgrade.

Inference Is Getting Tiered in Hardware

The other structural change in accelerators is that inference is splitting into workload classes with different silicon behind each one. NVIDIA's current platform direction pairs Rubin GPUs with a low-latency inference tier built on Groq LPU technology and a prefill-focused variant using GDDR7 rather than HBM. The logic is simple. Small models needing very low latency benefit from large on-chip SRAM. Long-context prefill is bandwidth-hungry but tolerant of cheaper memory. Decode on large models needs HBM capacity and interconnect. Running all three on the same part means overpaying for two of them.

That tiering shows up in your architecture whether or not you buy the hardware, because it becomes provider pricing. Prefill and decode are already priced differently through cached input and cache write rates, and the spread between them is widening. A request that reuses a large cached prefix and generates a short answer has a completely different cost profile than one that reads little and generates a lot. Design your prompts and your caching strategy around that split and the savings arrive without a single infrastructure change.

Power is the constraint behind the constraint. Rack-scale systems in this generation require liquid cooling as a precondition rather than an optimization, and higher power draw per rack is one of the named reasons the deployment ramp slipped. For most data teams that translates into a simple planning rule: capacity availability in 2027 depends more on facility readiness than on chip supply, and the providers who did their electrical and cooling work early are the ones who will have inventory.

Storage economics rarely make the AI infrastructure headlines, and they should. Training and inference both sit on top of data that has to be stored, scanned and moved, and the cheapest performance win available to most teams is still better encoding and better file layout rather than faster hardware. Column encodings that shrink floating-point data, statistics that let a scan skip whole files, and vector types that avoid per-row object allocation all reduce the bytes that ever reach an accelerator. The Apache Parquet and Apache Arrow communities are actively working all three of those right now, and the improvements land through a library upgrade rather than a purchase order.

What This Means for Data Teams

Pull the four sections together and a few things follow for anyone running data infrastructure under AI workloads.

Price tasks rather than tokens. Every major release this week competed on efficiency rather than list price. A model at $10 per million that finishes in one pass beats a model at $0.75 per million that loops six times, and thinking tokens bill as output on models that deliberately think more. Instrument total tokens, tool calls and wall-clock time per completed task on your own workloads. Vendor benchmark tables cannot answer that question for you.

Watch the dated cliffs. Gemini 3.8 Flash pricing doubles on January 1, 2027. GPT-6 Astra doubles input pricing above 272,000 tokens per request. Promotional windows on several models end in November and December. Build the batch path and the caching path now, while the rates are favorable, rather than discovering them during a budget review.

Design for the gate. Capability-gated model tiers are now standard practice across three labs, and enrollment runs weeks to months. If your security or compliance roadmap depends on a restricted variant, start the access process before you need it.

Plan MCP migrations with dates attached. Dynamic Client Registration, Roots, Sampling, Logging and HTTP with SSE all have deprecation clocks running. The offramps are generous, which is exactly why the work gets deferred until it is urgent.

Budget memory as its own line item. The DRAM squeeze touches everything you run, not just accelerators, and the supply relief is years out. Quote early, quote often, and expect quarterly contract terms instead of annual ones.

Keep an eye on the format layer. Encoding and type work in Parquet and Arrow moves the cost of AI data more reliably than most hardware cycles, and it lands through a dependency bump rather than a procurement cycle.

What to Watch Next Week

OpenAI DevDay is September 29 in San Francisco, where Astra's wider release and its full evaluation suite are expected. Meta's open-weights Muse Spark release remains on the roadmap with no date, and the same promise for 1.2 weights went unfulfilled, so treat any date you hear as provisional. Independent evaluations of Astra should start appearing as API access widens beyond the initial organizations, and those numbers will matter more than the launch table. On the standards side, the next MCP specification cycle is working on server-initiated events and the Server Card discovery conventions, both of which change how agent platforms get built. And the memory market gets its next real read when Q3 contract pricing settles.

If you want to go deeper on the data and AI stack behind all of this, from lakehouse architecture to agentic AI workflows, I write books on it. You can browse the full catalog at books.alexmerced.com.

Alex Merced, Data Lakehouse and AI Evangelist

AI Weekly: Cheap Tokens, Tight Safeguards, and a Two Million GPU Order

Alex Merced — Wed, 02 Sep 2026 19:03:57 +0000

Week of August 26 to September 2, 2026

By Alex Merced, Data Lakehouse and AI Evangelist

Three labs shipped models this week and every one of them led with price. Anthropic cut cache reads 75%, Z.ai put a natively multimodal 320B model on the table at $0.15 per million input tokens, and Alibaba previewed its next architecture with a model that computes six billion parameters per token. Underneath the model news, AWS and NVIDIA committed to two million more GPUs, AMD shipped a version-10 software stack built around agents, and MCP published a roadmap that puts agent identity at the center of the next spec.

Models: Anthropic ships Fable 5.1 and Mythos 5.1

Anthropic released Claude Fable 5.1 and Claude Mythos 5.1 on September 1. The two are the same underlying model with different safeguard levels. Fable 5.1 is generally available on the Claude API, Claude.ai, Claude Code, and Claude Cowork, and it runs on AWS, Google Cloud, and Microsoft Azure. Developers call it with the identifier claude-fable-5-1.

Mythos 5.1 goes only to vetted participants in two trusted access programs. The Cyber Verification Program covers defensive security work. The Life Sciences Verification Program, built with the US government, enrolled its first participants and plans to widen access. Anthropic also moved Claude Security, its codebase vulnerability scanner, onto Mythos 5.1.

Token pricing stays at $10 per million input and $50 per million output. The change is in cache reads, which drop 75% to $0.25 per million. Anthropic measured four weeks of real August usage and reports roughly 25% lower cost on typical workloads and up to 45% on context-heavy agentic work. For anyone running long agent loops where cached context dominates the bill, that second number is the one that matters.

On benchmarks, all figures below are vendor-reported and run with production safeguards enabled. Fable 5.1 scored 52.6% on Terminal-Bench-Science 0.1 against 24.7% for Fable 5, 29.0% for Opus 5, and 22.4% for GPT-5.6 Sol. Anthropic notes a standard error of 3.5 to 4.5 points on that benchmark, so read the gap as large rather than exact. Terminal-Bench 4.0 came in at 55.8%, with Mythos 5.1 at 60.9%. The spread between the two reflects tasks where cyber safeguards intervened on Fable.

Other numbers from the same table: 31.4% on AutomationBench against 17.1% for Fable 5, 73.4% on CursorBench 3.2.0, 60.9% on Humanity's Last Exam without tools, 1853 on GDPval-AA v2, and 77.9% partial credit on the August task release of OSWorld 2.0. The pattern is consistent. Long-horizon agentic work moved a lot, and short-horizon reasoning moved a few points.

The safeguard changes deserve as much attention as the scores. Anthropic reports that its updated biology safeguards fire 85% less often on benign elementary biology and medical questions. Cyber safeguards block 60% fewer false positives, in part because Fable 5.1 is now permitted to identify software vulnerabilities. Exploit development, penetration testing, and binary vulnerability scanning still route to Opus models. If you build security tooling on Claude, the routing map changed this week and your evals should account for it.

Two other changes affect anyone building on the API. Enterprise Frontier Safeguards store customer data on customer-controlled cloud infrastructure and give the privacy properties of a zero data retention agreement while keeping misuse detection in place. Rollout starts this fall, and eligible customers get zero data retention on Fable 5.1 until then. Separately, Anthropic added anti-distillation measures: new API accounts created from launch day forward cannot manually edit Claude's prior context in a multi-turn conversation while preserving the transcript of its earlier thinking. Existing accounts are unaffected for now. A small number of custom integrations will need adjustments.

Anthropic also confirmed it is watermarking outputs of models released after August 2, 2026, under the EU AI Act's Code of Practice on Transparency of AI-Generated Content, which it signed in July alongside 190 other organizations. The watermark is a statistical signal, invisible without the detection API, and carries no information about the user or the conversation. A detection API is in private preview for regulators, researchers, media, and enterprises with their own compliance obligations.

The science results are the part of this release that points somewhere new. Given open-source protein design and folding tools, Mythos 5.1 designed binders whose affinities on three targets ran ten times higher than the best entries in Adaptyv Bio's design competitions. Its hit rate reached nearly 50% across twelve targets, against a typical 10% to 15% in the field. Fable 5.1 trained a network on 30-year-old NASA Magellan radar data to build an elevation map of a third of Venus at two to three kilometer resolution, up from 10 to 20, with heights up to 25% more accurate. Anthropic released the map under a Creative Commons license ahead of the NASA VERITAS and ESA EnVision missions. Mythos 5.1 also wrote custom GPU kernels that sped up seven open-source genomics and protein models by as much as 2.5 times on an H100, cutting estimated GPU cost on genome-wide analyses by 30% to 60%.

That last result is worth pausing on. The work took days instead of the weeks a performance engineering team normally spends, and Anthropic plans to open-source the optimizations. Kernel optimization is exactly the kind of expensive, specialized work with outsized payoff most academic labs cannot afford. A model that does it cheaply changes who gets to run large-scale experiments.

Z.ai puts a multimodal 320B model at Flash prices

Z.ai released GLM-5.3-Flash on August 26. It is a mixture-of-experts model with 320 billion total parameters and 18 billion active per token, a 1,048,576-token context window, and native image and video input. Weights ship on Hugging Face under an open license, and the model runs natively in FP8.

The architecture is where the cost story comes from. Z.ai combined sparse and linear attention to hold down long-context serving cost, and the model starts from a newly trained base rather than a post-training pass on GLM-5.2. Self-reported numbers put it at 63.4 on DeepSWE against 46.2 for GLM-5.2, and 48.8 on AutomationBench against 26.2. Those are the lab's own figures and remain unverified by third-party evaluators.

List pricing runs $0.15 per million input, $0.03 cached, and $0.50 per million output, with a 50% launch promotion that expires September 9 at 16:00 UTC. Budget against the list rate, not the promo. Note that this is a different model from the text-only GLM-5.3 flagship, which lists at $1.40 and $4.40 and whose weights have not shipped.

The model spent twelve days on OpenRouter as an anonymous entry called Ox Alpha before the announcement. Reporting from MarkTechPost says it was served on domestically produced Chinese AI chips during that period. If that holds up, it is a signal about inference supply independent of the model itself.

Alibaba previews the Qwen4 architecture

Alibaba's Qwen team open-sourced Qwen3.8-Flash-Next on the same day, framing it as an architecture preview of the coming Qwen4 generation rather than a flagship. The team used the same pattern before, shipping Qwen3-Next ahead of the Qwen3.5 series so the community had time to build tooling.

The shape is unusual. A 125B backbone pairs with a 51B N-gram embedding table and a 4B multi-token prediction head, and only 6 billion parameters activate per token. The 48-layer stack mixes 36 Gated DeltaNet linear-attention layers with 12 full-attention layers using Qwen Sparse Attention, trained with the Muon optimizer. Native context is 262,144 tokens, extensible toward a million. It ships under a community license with weights on Hugging Face and ModelScope.

Reported scores include 62.5 on SWE-bench Pro and 91.7 on GPQA Diamond. The-decoder reports the model lands just below the Qwen3.8-Max flagship at roughly one twelfth the price on both input and output. The N-gram table is offloadable to host RAM, which changes the local-inference math for anyone running on a workstation rather than a rack.

Tencent also open-sourced a Hy4 preview on August 28 with 770 billion parameters and a one-million-token context, per TechNode. Details beyond that are thin, so treat it as an early signal rather than a deployable option.

Three open-weight releases in four days, two of them explicitly optimized for cost per token, is the shape of this market now. The frontier labs compete on capability at the top and the open-weight labs compete on the price floor underneath them. Anthropic cutting cache reads 75% in the same week is not a coincidence.

Tooling: coding agents get session management and vendor skills

Claude Code shipped a set of changes aimed at teams rather than individuals. Enterprise plans can now turn on skill and plugin security scanning, which checks third-party skills and plugins for malicious content when someone uploads or edits them. Given that skills are just instructions plus code that an agent will execute, scanning them at upload is a control that should have existed from the start.

Cloud sessions now sync plugins from claude.ai, showing them as name@synced and never overriding a same-named plugin installed locally. The release notes also record a fix worth reading if you run on Bedrock: streaming behind proxies that strip the response Content-Type header silently doubled billed API calls by re-running every turn non-streaming. That is a billing bug that produces no error, which is the worst kind. The usage-limit message now also reports when session and weekly limits reset, not only the monthly spend limit.

Fable 5.1 defaults to High effort in Claude Code and Medium in Claude Cowork and on Claude.ai. Effort level drives both quality and cost, so anyone moving to the new model should verify which default applies in each surface before comparing bills.

OpenAI's Codex spent the week on session and task management. The release notes list a new interactive codex agents dashboard for searching, starting, opening, renaming, and stopping tasks, plus a codex queue command for sending messages into existing local or remote sessions. New /cd, /pwd, and /cwd commands manage the working directory inside TUI sessions. The codex doctor command now diagnoses endpoint protection, network and proxy failures, desktop app state, and update connectivity.

SDK users can pass exact CLI config overrides and select max or ultra reasoning effort. A later drop added @ mentions across Codex tasks, letting agents read, create, and message other tasks from the terminal. Both tools are converging on the same realization: once agents run for hours, the interesting product surface is not the chat box, it is the queue.

The most interesting tooling item came from a chip vendor. AMD's ROCm 10 release includes AMD Skills, which packages validated AMD hardware knowledge and workflows into a form that Claude Code, Cursor, and Codex consume directly. A hardware company shipping its documentation as agent skills rather than as a PDF is a meaningful shift in how vendor knowledge reaches developers. Expect more of it, and expect skill provenance to become a security question fast, which is precisely what Anthropic's scanning feature anticipates.

On the adoption side, Cognition said it moved Devin's Opus 5 traffic to Fable 5.1 on launch day, starting with code review, and credited the cache read pricing for making a Fable-class model economical for workloads it had kept on cheaper tiers. That is the practical effect of a pricing change: it reshuffles which model sits in which part of the pipeline.

Standards: MCP puts agent identity at the center

The Model Context Protocol maintainers published a new roadmap on August 22, setting direction for the next spec release after the large 2026-07-28 revision. Five priority areas now govern which proposals get expedited review.

Agent identity is the one to watch. MCP authorization today assumes a person clicking approve in a browser. That model breaks when the caller is a cloud workload with its own identity, acting for a user who is not present, or delegating narrower authority to a sub-agent. The roadmap commits to finalizing Demonstrating Proof of Possession and driving its adoption, and to defining an opinionated path for agent identity and delegation through Workload Identity Federation, the ID-JAG grant behind Enterprise-Managed Authorization, and standard token exchange. The maintainers also plan to keep engaging the IETF OAuth and WIMSE working groups.

This is the right problem to solve next. Long-lived API keys pasted into agent configs are how most production MCP deployments authenticate today, and that pattern does not survive contact with an auditor. Building on OAuth machinery that enterprises already run is a better answer than inventing agent-specific credentials.

The second item practitioners will feel is progressive discovery. Connecting to a server with a hundred tools means the model pays for that entire surface before the user asks anything, and tool selection degrades as the list grows. The roadmap starts an effort to let a server expose a small entry point and reveal more of its catalog as the conversation narrows. Anyone who has watched an agent pick the wrong tool from a large catalog knows the cost of the current design.

The other three areas cover agentic messaging primitives, including server-initiated events through webhooks and channels so clients stop polling, transport unification so local servers speak Streamable HTTP over stdio, and result-type improvements so a server developer knows which form of a tool result a client will actually put in front of the model. That last one sounds small and is not. Ambiguous result contracts are why the same MCP server behaves differently across two hosts.

On the browser side, OpenAI introduced Site tools, its implementation of the proposed WebMCP standard. A website exposes actions directly to an agent alongside the interface people use, and in the ChatGPT desktop app's built-in browser, ChatGPT Work and Codex discover and call those tools against the same live page and signed-in session. WebMCP is the piece the agent stack has been missing. MCP connects agents to servers, A2A connects agents to each other, and WebMCP gives the existing web a way to expose actions without anyone building a separate API.

Two more standards-adjacent items from this week. Anthropic's anti-distillation change is a de facto API contract change, since editing prior assistant context while preserving thinking transcripts stops working for new accounts and will apply to all accounts on future model releases. And the EU AI Act watermarking requirement now has a working implementation with a detection API, which sets a template other signatories will follow.

Infrastructure: two million GPUs and a memory squeeze

AWS and NVIDIA announced a major expansion of their collaboration on August 26. AWS plans to deploy two million additional Blackwell Ultra, Rubin, and Rubin Ultra GPUs across its global infrastructure in 2027 and 2028. That comes on top of the one million GPUs AWS committed to at GTC 2026, which demand has already outrun.

The deal covers more than GPU count. NVIDIA Vera CPUs come to AWS for agentic workloads that need heavy CPU compute next to accelerators. NVLink Fusion extends with custom NVIDIA high-bandwidth memory inside Trainium racks. The two companies will build AI factories for the US government, including 100,000 GPUs on secure AWS infrastructure. EC2 G7 instances add RTX PRO 4500 Blackwell Server Edition GPUs.

NVIDIA followed on August 27 by confirming Vera CPU shipments at scale, with AWS receiving its first Vera CPU server and Vera Rubin GPU in Seattle. Earlier deliveries went to Oracle Cloud Infrastructure and to Anthropic, OpenAI, and SpaceXAI. NVIDIA's CFO said the company expects Vera deployment across every major hyperscaler, neocloud, AI lab, and system OEM. Vendor-reported figures put Vera at up to 1.8 times faster per core on selected agentic workloads with twice the energy efficiency of traditional infrastructure, and those comparisons are not independently verified.

NVIDIA also moved Groq 3 LPX into full production, positioning it as a decode-phase accelerator for latency-sensitive agentic work, with vendor figures of 3,400 output tokens per second on 100,000-token long-context use cases. Splitting prefill and decode across different silicon is the direction inference hardware has been heading, and a production part built specifically for token generation makes that split concrete.

AMD shipped ROCm 10 on August 27, ten years after ROCm 1.0. The headline is ROCm.AI, which bundles AMD Skills, the new ROCm CLI, and Hyperloom, an agentic system that profiles inference workloads, finds bottlenecks, modifies code, and benchmarks the result. AMD's internal testing reports an average 3.3 times inference improvement and 2.4 times training improvement against ROCm 7, measured on eight Instinct MI355X GPUs running GLM-5, Kimi-K2.5, and DeepSeek-R1-0528. Read that as a tuned configuration against an untuned baseline, not a blanket speedup.

The rest of the release addresses fragmentation, which has been AMD's real problem. Windows and Linux now share the ROCm Core SDK, and the separate Windows HIP SDK is retired. The TheRock build pipeline is production ready. RCCL advances to NCCL 2.30.4 with GPU-initiated networking, and vLLM v0.2x is supported. An open stack that an agent can drive is a more credible challenge to CUDA than another round of raw performance claims.

Memory is where the cost pressure sits. Kioxia and SanDisk committed more than $31 billion in Japan through 2032 to expand flash production, including a new building at Kitakami. SK hynix broke ground on its HBM production base in Indiana on August 28. TrendForce projects cloud provider capital expenditure rising 98% year over year in 2026 and another 50% in 2027, with DRAM and NAND accounting for 47% of that spending in 2026 and 68% in 2027.

That last figure is the one to sit with. When memory takes two thirds of cloud capital spending, the binding constraint on AI capacity stops being GPU allocation and becomes DRAM and HBM supply. Fabs take years. Every efficiency gain that reduces bytes moved per token, from linear attention in GLM-5.3-Flash to cache read pricing at Anthropic, is a response to the same physical limit.

What this means for the data layer

Three threads from this week land directly on anyone running data infrastructure.

Cache economics now shape architecture. When cache reads cost a quarter of what fresh input costs, the winning pattern is a stable, reusable context prefix with the variable part at the end. That favors agents that hold a fixed schema catalog, a fixed set of tool definitions, and a fixed instruction block, then append the query. Teams that rebuild context from scratch on every turn are paying full freight for work the provider will discount by 75%.

Progressive tool discovery in the MCP roadmap matters more for data platforms than for most MCP servers, because a data catalog is exactly the case where the tool surface is enormous. A server that exposes every table as a tool poisons model attention. A server that exposes search and drill-down, then reveals the specific tables the conversation needs, works. Design for that now rather than waiting for the spec.

Agent identity is the governance question. Column-level and row-level restrictions enforced at a catalog only mean something if the catalog knows which agent is asking, on whose behalf, with what delegated authority. The Iceberg REST catalog community voted on finer grained read restrictions this same week. MCP is working the credential side of the same problem. Those two lines of work need to meet, and today they do not.

What to watch

The GLM-5.3-Flash promotional price expires September 9 at 16:00 UTC, and independent evaluators have not yet verified its self-reported DeepSWE and AutomationBench numbers. Z.ai still owes the community the GLM-5.3 flagship weights it promised. Alibaba's full Qwen4 family follows the Flash-Next architecture preview, with no date announced.

Anthropic said it plans to bring Fable 5.1's improvements to the rest of the Claude model family, so watch for Opus and Sonnet updates. Enterprise Frontier Safeguards begin phased rollout this fall. The anti-distillation context restriction applies to all accounts on future model releases, not just new ones, so integrations that rely on editing prior assistant turns have a limited runway.

On the standards side, the MCP working groups are taking SEPs in the five roadmap areas, with agent identity and progressive discovery the two most likely to change how you build. And keep an eye on memory pricing. If DRAM and NAND really reach 68% of cloud capital spending next year, inference cost curves will bend for reasons that have nothing to do with model architecture.

If you want to go deeper on agentic AI, lakehouse architecture, and the data infrastructure underneath both, I keep a full catalog of my books at books.alexmerced.com.

Apache Data Lakehouse Weekly: August 26 to September 2, 2026

Alex Merced — Wed, 02 Sep 2026 18:47:12 +0000

By Alex Merced, Data Lakehouse and AI Evangelist

Release season arrived across the lakehouse projects this week. PyIceberg 0.12 shipped, Arrow Rust 59.3.0 passed its vote, Parquet Java 1.18.1 collected its binding +1s, Polaris lined up 1.8.0, DataFusion started backport triage for 55.1.0, and the Iceberg Terraform provider recovered from a failed release candidate on the strength of one careful license review. Underneath the release traffic sat a harder set of questions about who owns what. Iceberg contributors debated moving the DataFusion integration to a different PMC, Parquet contributors worked through what a format version number actually promises, Iceberg's catalog crowd argued over whether label metadata belongs to the catalog or the user, and the Ossie podling spent the week deciding what its own specification is for. Six projects, one recurring theme: boundaries.

Apache Iceberg

The headline release is PyIceberg 0.12, announced by release manager Alex Stephen on September 1 after the 0.12.0rc2 vote closed with binding +1s from Daniel Weeks, Kevin Liu, Fokko Driesprong, and Sung Yun, plus non-binding verification from Jared Yu, Xuanwo, Neelesh Salian, Yuya Ebihara, and Jean-Baptiste Onofré. The verification detail in that thread is worth reading if you have never watched an ASF release vote up close. Yuya Ebihara checked GPG signatures on every wheel and the source tarball on Apple Silicon with Python 3.13.3. Neelesh Salian ran the same checks on Python 3.12. Jean-Baptiste Onofré flagged a license issue in the source distribution, and Kevin Liu opened a GitHub issue to track it rather than block the release. Fokko Driesprong named conflict resolution as his favorite feature in the new version, which tells you where the Python implementation is heading. PyIceberg started as a metadata reader. It now handles write paths with the kind of concurrency semantics that used to require the Java library.

The Iceberg Terraform provider took a longer road. Sung Yun cast a binding -1 on RC2 after finding dependency drift and LICENSE-binary gaps that appeared between RC1 and RC2, specifically around the iceberg-go dependency. Jean-Baptiste Onofré added a 0 with a full checklist of what he verified in the source distribution. Matt Topol, the release manager, opened a fix PR the same day and cut RC3 two days later. That candidate collected +1s from Xuanwo, Alex Stephen, Fokko Driesprong, and Sung Yun, who confirmed that LICENSE-binary now covers the modules missed in RC2. This is the release process working exactly as designed. A binding voter caught a real licensing problem, the release manager fixed it in under 48 hours, and the first release of a new Iceberg subproject goes out clean.

The most consequential governance discussion of the week is the proposal to move the iceberg-datafusion integration out of the apache/iceberg-rust repository. Shawn Chang summarized the state of play after the community sync: the crate has a reviewer bandwidth problem that is slowing development. Gabriel Musat laid out the two facts driving the decision, namely the review bottleneck and the tight coupling between the crate and DataFusion's own release cadence. Renjie Liu pushed back on testing concerns as a blocker, noting that most of the sqllogictests live in PRs that modify the integration anyway. Then Xuanwo made the argument that seems to be carrying the room: DataFusion is the largest dependency of iceberg-datafusion, and Comet is its largest downstream consumer. Andy Grove added a +1 from the Comet side, calling the integration very important to that project. Renjie Liu closed the week by writing out the Comet and DataFusion relationship for people who do not follow the Rust side daily.

Read that thread alongside the Java release work and you see two different Iceberg ecosystems operating at different speeds. Neelesh Salian cleaned up the 1.12.0 milestone and reported on September 2 that the milestone is down to a handful of open PRs. Danny Jones from AWS volunteered as release manager for iceberg-rust 0.11 with Shawn Chang backing him on process, and Renjie Liu, Kevin Liu, Kurtis Wright, and Alexander Bailey all signed on within days. The Rust implementation moves fast and reorganizes itself when structure gets in the way. The Java implementation moves on a milestone and carries a decade of compatibility obligations.

Flink version support turned into the week's sharpest engineering debate. Péter Váry opened the question for 1.12.0 with a proposal to adopt an "LTS plus the two latest releases" support matrix. Maximilian Michels backed it as the approach that keeps Iceberg aligned with the Flink versions Flink itself supports at release time. Manu Zhang raised the production concern that matters to operators: Iceberg has never jumped two Flink versions at once, and teams running older Flink builds need an upgrade path that does not strand them. Talat Uyarer proposed a middle position that preserves version overlap, then went ahead and opened a PR to unblock the release candidate, with an explicit note that the PR does not preempt the discussion. Péter Váry asked Maximilian Michels how hard it will be to fold Talat's suggestion into the existing PR. The compromise looks close. The underlying tension is permanent, because Flink and Iceberg release on different clocks and every support matrix decision picks a winner between currency and stability.

On the spec side, the vote on finer grained read restrictions in the REST catalog opened August 31 and drew a wall of support: a binding +1 from Yufei Gu, and non-binding votes from Nevin Zheng, Andrei Tserakhau, huaxin gao, Alex Stephen, Gianluca Graziadei, and Holden Karau, among others. Prashant Singh drove this proposal through weeks of dedicated syncs, and the two loose ends closed just before the vote. The overlapping column projections question on nested types resolved toward disallowing overlap on both the catalog and client side. Prashant also added a compatibility kit for read restrictions so implementations have something to test against rather than a prose spec and good intentions. Column-level and row-level restrictions enforced at the catalog rather than the engine changes the security story for multi-engine lakehouses, because the restriction travels with the table instead of living in whichever query engine happens to be reading.

The catalog thread that generated the most heat was table and column label metadata in the REST catalog. Andrei Tserakhau brought the proposal back to the list after the catalog community sync, framing the motivating case as catalog-to-catalog federation. Prashant Singh raised concerns about labels used for governance decisions. Ryan Blue drew the line hard, rejecting the idea of reusing table config for this purpose and warning against mixing user-controlled properties into catalog-controlled configuration. That objection is the crux. Config already exists, labels look like config, and the difference is who owns the value and what trusts it. By September 2 Andrei and Prashant had narrowed the disagreement to how labels get surfaced in logging and AI context, with Andrei conceding that his earlier framing was too broad. Watch this one. Label metadata is the seam where catalogs stop being table registries and start being governance systems.

Several v4 metadata questions moved in parallel. Ryan Blue proposed a scheme for tracking field IDs for non-materialized columns, assigning table field IDs to values that never get written into the table. Gianluca Graziadei suggested splitting the treatment between expressions that are intrinsically deterministic and those that are not. Péter Váry raised schema evolution, pointing out that expression result types change as the schema underneath them changes, and noted the proposal makes index definitions much simpler. Sergei Nikolaev opened a separate v4 question on Avro timestamp types, arguing for dropping Iceberg-specific conventions in favor of the Avro 1.12.0 spec. Fokko Driesprong replied with the history, since he added the fixed[16] UUID encoding and timestamp-nanos to Avro in the first place. Andrei Tserakhau also split out a collations question about file prunability when engines run different ICU versions, which is the kind of problem that only shows up once you take multi-engine correctness seriously.

Smaller items worth your attention. Russell Spitzer explained the resolution of the EagerInputFile discussion: Varun moved the eager read into the Parquet file reader itself, so Parquet now buffers any file of one megabyte or less in a single request. For metadata-heavy workloads against object storage, request count is the cost driver, and collapsing small reads into one round trip pays off immediately. Rahul Mahadev proposed a standard User-Agent format for REST catalog clients so a catalog can tell which client is talking to it in a parseable way. Hongyue Zhang closed the loop on position deletes with row data, with the community agreeing to remove that handling from maintenance actions in 1.12.0. Tomohiro Tanaka responded to feedback on the table_properties_log metadata table, addressing read cost and server impact. Hemanth Boyina proposed extending validate-from-snapshot-id to MERGE so optimistic concurrency validation starts from a known snapshot the same way it does for overwrites. William Hyun and Sung Yun kept file-level access delegation in the REST spec moving, with Sung splitting Dan Weeks's two scenarios into three by separating presigned URLs returned from scan planning from those returned elsewhere.

The Iceberg community also picked up a new integration point from outside its own ecosystem. Gianluca Graziadei announced storm-iceberg, an external Apache Storm module that ingests streaming tuples straight into an Iceberg table from inside a Storm topology. Talat Uyarer scheduled a File Type sync for September 2 to align on the first-class file type proposal. And Danica Fine reminded both the Iceberg and Polaris lists that Lakehouse Day EU 2026 lands October 10 in Glasgow, co-located with Community Over Code.

Apache Polaris

Polaris spent the week getting 1.8.0 ready and settling three design questions that all trace back to the same root: how much of a table's physical layout does the catalog get to decide?

Jean-Baptiste Onofré volunteered to drive the 1.8.0 release at the beginning of September. Robert Stupp backed the schedule and argued for keeping the release focused on fixes and improvements that are ready today rather than stretching to include in-flight features. Jean-Baptiste agreed, noting that a monthly cadence means nothing needs to be forced into any single release. Yufei Gu made the case for one exception, PR 5053, on the grounds that the work is nearly done and has been in flight for a while. Dmitri Bourlatchkov seconded, and Jean-Baptiste agreed to take another review pass. Monthly releases change the psychology of scope negotiation. When the next train leaves in four weeks, nobody fights to get on this one.

Eundo Lee's work to make the relational JDBC schema name configurable is the community-health story of the week. Eundo sent a polite reminder that PR #4945 had been waiting while maintainers focused on the release. Alexandre Dutra responded within hours, proposed merging by end of day unless anyone objected, and then did. Yufei Gu registered a +0 with a real design reservation, arguing that the schema name should be a Polaris-owned property that Polaris maps to the backend rather than a passthrough, and chose not to block on it. Eundo folded in EJ Wang's review suggestions before the merge landed. A first-time-ish contributor got a clear answer, a dissenting reviewer said what he thought without stopping the work, and the feature ships in 1.8.0.

Yufei Gu opened the semantics of custom namespace locations by pointing out that ALLOW_NAMESPACE_CUSTOM_LOCATION currently skips parent-location validation entirely, so a namespace lands anywhere rather than under its parent. Dmitri Bourlatchkov challenged the premise that strict nesting applies universally, arguing that namespaces serve an organizational purpose that does not have to mirror storage layout. Jean-Baptiste went further and said the naming problem hides a deeper one, since whether tables follow namespace nesting is already governed by a different property, ALLOW_UNSTRUCTURED_TABLE_LOCATION. Two flags controlling overlapping behavior with names that suggest different scopes is a config bug waiting to become a security bug, and this thread is the right place to fix it before 1.9.0.

Alexandre Dutra also worked through forwarding user-defined principal properties in PolarisPrincipal with Prithvi S, reviewing the design doc, proposing an alternative, and signing off once Prithvi incorporated the changes. The threat model stayed constant through the discussion, with authorizers remaining trusted components. Alexandre returned to multiple StorageConfigurationInfos per catalog after a summer slowdown, picking up Srinivas's writeup. EJ Wang posted a status update on the Tag spec, with the API contract PR ready for review and a second PR covering definition CRUD. And Sung Yun reported that ASF Infra created the terraform-provider-polaris repository, scaffolded with LICENSE, NOTICE, a stub README, and an .asf.yaml.

Note the pattern across two projects. Iceberg is releasing a Terraform provider this week. Polaris just got its repository created. Infrastructure-as-code support for catalogs and table formats is becoming table stakes, because nobody wants to click through a UI to provision the governance layer of a production lakehouse.

Apache Arrow

Arrow's week centered on removing something rather than adding it. Antoine Pitrou pinged the dormant discussion about deprecating Tensor and SparseTensor messages in the IPC protocol, asking two questions: does anyone object, and does this need a formal vote? Curt Hagenlocher answered that a vote seems right, and made the observation that settles the substance. The only complete implementation of these messages is the C++ one, which means the feature does not meet today's bar for a format change. Matt Topol agreed on both the vote and the deprecation. Antoine opened the vote on September 1 and it filled up the same day with binding +1s from Curt Hagenlocher, Rok Mihevc, Micah Kornfield, Matt Topol, and Sutou Kouhei, plus non-binding support from Rusty Conover and Jacob Quinn.

Removing tensor support from a columnar format in 2026 reads odd until you remember what Arrow is for. Tensors in Arrow IPC never got multi-language implementations, so no cross-language interoperability existed to protect. Machine learning workloads that need tensor transport have their own well-supported paths. Carrying an under-implemented message type in the spec taxes every new implementation with no payoff. The word "informal" in the vote title matters too, since this deprecates without breaking anything that currently works.

Andrew Lamb ran the Arrow Rust 59.3.0 RC2 vote after RC1 hit a problem, and announced the result on September 1 with six +1 votes, four of them binding. Verification came in across Intel Mac from Ed Seidl, M4 Mac from Jeffrey Vo and L. C. Hsieh, and x86-64 Fedora 44 from Adam Reeve and Kosta Tarasov. The crate is published. The platform spread in that vote is quietly important, because arrow-rs sits underneath DataFusion, Comet, iceberg-rust, and a long tail of Rust data tools that all inherit whatever it does or does not verify.

Two format discussions stayed open. Mandukhai Alimaa posted a status update on the arrow.big_decimal extension proposal, noting that the effort started as a canonical arbitrary precision and scale type and has taken on more shape since. Micah Kornfield responded that a long pause is unnecessary and framed his own comments as pushing for conscious choices rather than rushed ones. Kosta Tarasov followed up on the Variant extension spec being inconsistent with the Parquet shredding spec and asked anyone working on Variant to take a look. That inconsistency is the sort of thing that becomes very expensive later, when two implementations both claim Variant support and disagree about what shredded data means on disk.

Wes McKinney revisited the status of Arrow's conbench data and the conbench open source project, asking whether anyone wants to engage before he takes a new codebase in a different direction under a different name. Rok Mihevc replied with a draft schema design and connected Wes with someone facing similar problems. Benchmark infrastructure rarely gets attention until it stops working, and Arrow's performance claims rest on the data it collects. Ian Cook also ran the biweekly community meeting on August 26.

Apache Parquet

Parquet is renegotiating its own compatibility contract, and this week showed the whole spread of that work: two votes passed, a patch release voted through, and one long-running design thread that hit a governance snag.

Alkis Evlogimenos closed the vote to remove self-references from the FILE logical type with seven binding +1s and eight non-binding, no dissent. Russell Spitzer, Antoine Pitrou, Micah Kornfield, Daniel Weeks, Ryan Blue, Julien Le Dem, and Gang Wu carried the binding side, with Rok Mihevc, Burak Yavuz, Divjot Arora, and Prateek Gaur among the non-binding voters in the vote thread. Divjot Arora followed with a second vote to specify handling for unrecognized logical and physical type combinations, where a reader drops to the physical type instead of failing. Andrew Lamb and Antoine Pitrou both voted +1 binding, with Antoine calling it short and useful. That is exactly the right instinct. Graceful degradation rules cost almost nothing to specify and save every future reader implementation from guessing.

Fokko Driesprong drove the patch release. After regressions surfaced in 1.18.0, he proposed a follow-up and cut 1.18.1 RC1 on August 31. Gábor Szádovszky verified the tarball, built and tested from source, checked artifacts with Steve's Auditor tool, and validated against Dremio builds and tests. Gang Wu covered signatures, KEYS, SHA512, tag and commit, a Maven Java 11 build, and license checks. Russell Spitzer, Peter Toth, and Gidon Gershinsky added their votes, and Fokko ran the release against an Iceberg PR to confirm nothing broke downstream. Testing a Parquet RC against Iceberg before it ships is a good habit, since Iceberg is where most Parquet regressions get discovered in production.

The versioning proposal is the thread to read in full. Ryan Blue summarized the open questions after a Wednesday call and followed up September 1 with a resolution on magic bytes, confirming with the modular footer authors that encryption can be handled within the footer, which frees the magic bytes to signal format version. Divjot Arora agreed with the summary and pushed back on using magic bytes for encryption signaling. Russell Spitzer endorsed the PAR3, PAR4 progression as the safest version signal. Micah Kornfield redirected the discussion toward requirements before logistics, arguing the community should agree on what the version number promises before deciding where the bits live.

Then Antoine Pitrou raised a process objection that deserves attention. He said the thread has become difficult to follow because it references discussions that happened privately or at least off-list, and that it is drifting from the original scope. He is right to say it, and the fact that he said it on the dev list is the system working. Parquet's design work now spans a versioning thread, a modular footer proposal, a Google doc from Julien Le Dem, a weekly sync, and a dedicated footer sync that Jiayi Wang ran on September 1 for its fifth session. Every one of those venues produces decisions. Only the mailing list produces a record. Projects that let the record fall behind the decisions lose the contributors who cannot attend meetings, which over time means losing the contributors who do not work at the two or three companies with the most people on the call.

On the encoding side, Andrew Lamb kept the ALP blog post review moving with Kosta Tarasov, adding BYTE_STREAM_SPLIT with ZSTD to the comparison after feedback from Antoine Pitrou, Arnav, Russell, and Jigao. Floating point compression in Parquet has been an open opportunity for years, and having measured numbers published in the open changes what implementations choose by default. Julien Le Dem ran the regular Parquet sync on August 26.

Apache DataFusion

DataFusion's week was short on threads and long on consequences. Tim Saucer opened the 55.1.0 release discussion, asking contributors to comment on the tracking issue with anything they want backported. Andrew Lamb crowdsourced the September ASF board report with a draft doc and a tracking ticket, which is the quarterly ritual that keeps a large, fast-moving project legible to the foundation.

The substantive thread is Comet. Andy Grove proposed a dedicated weekly sync call on the grounds that contributor count and velocity have both climbed sharply. Kazuyuki Tanimura, Kumar Ujjawal, Parth Chandra, and Marko Milenković all signed on. Bhargava Vadlamani proposed Friday mornings at 10:30 Pacific and offered to coordinate the meeting. Andy suggested documenting the call in the Comet contributors guide, and Bhargava agreed to open the doc PR and send the invite. A subproject earning its own weekly call is a growth signal, and Comet has been earning it. A Spark accelerator built on DataFusion sits at the intersection of the two ecosystems where most enterprise Spark workloads are heading.

Which brings the iceberg-datafusion question back around. The same thread about moving the integration ran on both dev lists, and the DataFusion side of the argument is straightforward. Comet depends on iceberg-datafusion, Comet lives under DataFusion, and the reviewers with the deepest context on both sit in the DataFusion community. Renjie Liu spent September 2 writing out the Comet and DataFusion relationship for readers who do not track the Rust ecosystem daily. If this move happens, DataFusion picks up ownership of a piece of the Iceberg stack, and Iceberg gains a maintainer pool it does not have to grow itself.

Apache Ossie

The Ossie podling is doing something none of the other five projects have to do anymore. It is deciding what it is.

Julian Hyde, who designed Apache Calcite and wrote the measures-in-SQL work that Ossie contributors already cite, introduced himself on the list after lurking for a while. Khushboo Bhatia welcomed him and explained that Ossie has been deliberate about keeping authoring concerns separate. Jean-Baptiste Onofré framed the project's purpose as standardizing the semantic model. Julian then asked the question that reframed the whole thread: if Ossie converts a semantic model from tool A to tool B, who decides whether the translated model means the same thing as the original? He offered one answer, which is to make Ossie a database-style API so semantics are defined by execution rather than by document structure. Matthew Mullins from Coginiti welcomed the pot-stirring, liked the proposal, and asked the community to do better than JDBC, which he described as inconsistently implemented everywhere. Khushboo replied that Ossie is not an interchange specification, that dropping the word "interchange" from the name was intentional, and that the repository now carries an expression language spec. Matthew's response was fair and pointed: if it is not an interchange spec, the README needs to stop reading like one. Will Pugh landed the synthesis, noting that an interchange without a definition of what it means cannot guarantee you get the same thing on the other side.

That exchange is the whole problem with semantic layer standardization in one thread. A semantic model is a set of definitions, and definitions only mean something relative to an evaluator. Two tools can accept the same YAML and produce different numbers for the same metric, and both of them will insist they implemented the spec. Ossie's answer to this is a compliance suite, which is why the PRs on foundational semantics and the compliance suite matter more than any feature PR in the repo. Khushboo argued for narrow, well-defined semantics that work end to end across many vendors over more powerful semantics that nobody defines precisely or adopts widely. Chris Eubank recapped the metric language working group meeting for people who missed it, and separately proposed seeding a BI SQL corpus as a follow-up action item from that meeting. A corpus of real BI SQL is how you find out whether a spec survives contact with what tools actually generate.

Release mechanics got their own debate. Jean-Baptiste opened the first Ossie releases discussion, and Yufei Gu argued for a source-only distribution since the community has not settled how to release individual converters. Yong Zheng asked whether the OSI to Ossie rename PR should land first, and Yufei agreed it should. Julian Hyde made the case for releasing the whole repository as one unit for the first few releases, noting that ASF releases carry legal weight and that the convenience of patching a single processor is a secondary concern. Yufei also kept the Python converter consolidation thread going, arguing that shared components exist even if release and review overhead cut the other way.

Vendor participation is picking up. Damian Waldron, a product manager at ThoughtSpot, brought the ThoughtSpot converter question to the list, covering scope, licensing, and where the code should live. Jean-Baptiste gave the standard ASF answer: Apache License 2.0, and code coming from another product needs a software grant agreement and a license change. Damian split the specification piece into a separate thread proposing THOUGHTSPOT be added to the Dialect enum, which is the right instinct, because adding a vendor to a spec enum is a different decision than accepting a code contribution.

Feature design ran hot on GitHub-backed discussions too. A proposal for shared filters, shared dimensions, and metric references drew eleven messages, with Josh Klahr digging into filter scope problems and cross-model references. Josh separately proposed dataset-scoped metrics, letting metrics be declared on an individual dataset instead of only on the semantic model, and Khushboo backed it while noting the subtleties belong on the PR. A spatial dimension type discussion picked up interest from CARTO, and a contributor from Databricks pushed on making relationship cardinality explicit and on how model-level semantic filters flow through consumers. Ankit Tandon posted notes from the September 1 Ontology working group sync.

The podling also grew. Jean-Baptiste announced Yong Zheng as a committer, and Khushboo Bhatia announced Josh Klahr joining the PPMC. Khushboo and Yufei both signed off on the September incubator report that Jean-Baptiste drafted.

Cross-Project Themes

Ownership beats architecture. The iceberg-datafusion move, the Parquet versioning debate, the Iceberg labels argument, and Ossie's interchange identity crisis are all the same class of question wearing different clothes. Nobody is arguing about whether the code works. They are arguing about who decides what it means and who has to maintain it. Xuanwo's case for moving the DataFusion integration rested on dependency direction and reviewer location, not on technical merit. Ryan Blue's objection to reusing config for labels rested on who controls the value, not on schema design. Ossie's whole week reduced to whether a spec that defines structure can promise meaning. As these projects mature, the interesting decisions stop being architectural and start being about boundaries between communities.

The verification bar keeps rising, and it is catching real problems. Sung Yun's binding -1 on the Terraform provider found license drift between two release candidates. Jean-Baptiste flagged a license issue in PyIceberg that Kevin Liu turned into a tracked issue. Gábor Szádovszky ran the Parquet RC through a third-party auditor tool and against downstream builds. Fokko tested Parquet 1.18.1 against an Iceberg PR before voting. This is not ceremony. Four of these projects ship artifacts that every other one depends on, and a bad release propagates through the stack in days.

Spec work is converging on compliance testing. Prashant Singh shipped a compatibility kit alongside the Iceberg read restrictions spec. Ossie is building a compliance suite as the mechanism that makes its semantics real. Divjot Arora's Parquet vote specifies reader behavior for combinations nobody anticipated. Kosta Tarasov flagged the Variant and Parquet shredding inconsistency precisely because two specs describing the same bytes differently is a bug that only testing surfaces. The community learned from a decade of table format ambiguity that a spec without a test suite is a suggestion.

Rust is where the reorganization happens. iceberg-rust is cutting 0.11 with a new release manager, splitting out its DataFusion integration, and shipping fast enough that structural changes get proposed and resolved inside a single week. arrow-rs shipped 59.3.0 with verification across four platform combinations. Comet is spinning up its own weekly sync on contributor growth. The Java implementations of these same projects operate on milestones, LTS matrices, and multi-year compatibility promises. Both models are correct for their constituencies, and the seam between them is where most of the interesting engineering now lives.

Semantic metadata is climbing into the catalog. Iceberg's labels proposal, Polaris's Tag spec, and Ossie's entire existence point the same direction. Teams want meaning attached to data at a layer that survives whichever engine reads it. The disagreements in the Iceberg labels thread are the early version of a fight every catalog will have, because once metadata drives governance decisions, the question of who writes it stops being a convenience question and becomes a security one.

Looking Ahead

Watch for the iceberg-rust 0.11 release candidate now that Danny Jones has consensus, and for a formal proposal on where iceberg-datafusion lands. The Iceberg 1.12.0 milestone is down to a few PRs, so an RC is close, with the Flink support matrix as the last real blocker. Polaris 1.8.0 should cut in the next week or two with PR 5053 included. Parquet 1.18.1 has its binding votes and needs only a result thread. Arrow's Tensor deprecation vote closes with more than enough support. DataFusion 55.1.0 is collecting backports, and the first dedicated Comet sync happens Friday morning Pacific. On the Ossie side, the first source release and the compliance suite PRs are the two things to track, since one makes the project real to users and the other makes the spec real to implementers.

If you want to go deeper on any of the topics in this issue, I keep a full catalog of my books on Apache Iceberg, lakehouse architecture, and agentic AI at books.alexmerced.com.

Inside the Puffin File Format

Alex Merced — Tue, 01 Sep 2026 20:20:13 +0000

A query joins a 2-billion-row fact table to a 40,000-row dimension table. The optimizer has to decide which side to broadcast and which side to hash. It reads the manifests and finds row counts, min and max values, and null counts for every column in every file. What it does not find is how many distinct customer IDs exist in the fact table. Without that number it guesses, and a wrong guess means shuffling terabytes that a broadcast join avoids.

The same engine, a few minutes later, deletes 300 rows from a data file that holds 4 million. In format version 2 it writes a position delete file: a Parquet file listing the path of the data file and the position of each deleted row. Every subsequent read of that data file has to open the delete file, decode Parquet, build a set of positions, and filter. Do that across ten thousand data files and delete handling dominates query time.

Both problems have the same shape. Iceberg's manifests are the wrong place for the answer. Manifests are optimized for per-file scalar statistics that fit in a few bytes each. A distinct-value sketch is kilobytes. A delete bitmap is arbitrary size. Neither belongs inline in an Avro record that the planner reads for every file on every query.

Puffin is the file format Iceberg uses for information that does not fit in a manifest. It is a simple container: a magic number, a sequence of opaque blobs, and a JSON footer that describes what each blob is, what it was computed for, and where it sits in the file. Today the spec defines two blob types. One holds a Theta sketch for estimating distinct values. The other holds a deletion vector for row-level deletes in format version 3. This article takes the format apart byte by byte, explains both blob types from first principles, shows how table metadata and manifests reference Puffin content, and covers what goes wrong operationally. I work at Dremio, whose query engine consumes Puffin statistics, but nothing here is vendor-specific.

Why Manifests Were Not Enough

Understanding Puffin starts with understanding what manifests already do and where they stop.

An Iceberg manifest is an Avro file with one entry per data file or delete file. Each entry carries the file path, format, partition tuple, record count, file size, and a set of per-column metrics: value counts, null counts, NaN counts, and lower and upper bounds. The planner reads these entries for every query. Bounds let it skip files whose value ranges cannot match a predicate. Counts let it estimate scan size.

These metrics share three properties. They are small, a few bytes per column per file. They are cheap to compute during the write, because a writer already sees every value. And they are per file, which is exactly the granularity the planner needs for pruning.

Table-level statistics for a cost-based optimizer violate all three. The number of distinct values (NDV) in a column across the whole table is not a per-file quantity, and you cannot sum per-file NDVs because the same value appears in many files. Computing it accurately requires a pass over the whole table or a mergeable sketch. And the sketch itself, the data structure that lets you merge partial results, is thousands of bytes, not a handful.

Row-level deletes have a different mismatch. A delete for a single data file is a set of row positions. The natural encoding is a bitmap. A bitmap for a 4-million-row file with scattered deletes compresses to a few kilobytes. That is too large to store inline in a manifest entry, and the manifest has to be rewritten on every delete if the bitmap lives there, which defeats Iceberg's append-only metadata design.

The Iceberg community's answer, proposed around 2022 alongside the Trino integration work, was a dedicated sidecar format with three design goals. It had to be trivially parseable by any language, so a new engine adopts it without a large dependency. It had to support random access to individual blobs, so a reader that wants one statistic does not read the whole file. And it had to be extensible, so new statistic and index types get added without changing the container.

The result was named Puffin, and the magic bytes spell out the joke: PFA1 stands for Fratercula arctica, the Atlantic puffin, version 1.

File Layout Byte by Byte

A Puffin file is a flat sequence with no internal structure beyond what the footer describes:

Magic  Blob₁  Blob₂  ...  Blobₙ  Footer

The leading Magic is four bytes: 0x50 0x46 0x41 0x31, the ASCII characters P, F, A, 1. Every blob follows immediately, back to back, with no headers, length prefixes, or padding between them. A blob is whatever bytes the writer chose to put there. The container does not interpret them. That interpretation is entirely the footer's job.

The footer sits at the end of the file and has its own fixed structure:

Magic  FooterPayload  FooterPayloadSize  Flags  Magic

Reading it backward from the end of the file: the last four bytes are the magic again. Before that, four bytes of flags. Before that, a four-byte integer holding the size of the footer payload. Before that, the payload itself. And before the payload, the magic once more, marking where the footer begins.

All four-byte integers in Puffin are signed, two's complement, little-endian. The flags field is four bytes, but only one bit is defined today. Bit 0 of byte 0 indicates whether the footer payload is compressed. Every other bit is reserved and must be written as zero.

When the compression bit is set, the footer payload is a single LZ4 frame with content size present. When it is clear, the payload is raw bytes. In both cases the decompressed payload is UTF-8 JSON describing a single FileMetadata object.

The reason the layout ends with the magic and puts the size just before the flags is that it lets a reader locate the footer with two range reads and no scanning. Read the last 12 bytes of the file. Verify the trailing magic. Extract the flags and the payload size. Compute the payload's starting offset as file_size - 12 - payload_size, and do a second read of payload_size + 4 bytes to pull the leading magic plus the payload. Two requests against object storage, and the reader knows every blob's type, location, and length.

Iceberg's table metadata makes this even cheaper by recording file-footer-size-in-bytes for every statistics file. A reader that has the table metadata skips the first probe entirely and fetches the footer in one range read of exactly the right size.

Once the footer is decoded, fetching a blob is one more range read at the offset and length the footer specifies. A reader that needs the NDV sketch for one column reads three small ranges from a file that is otherwise never touched. That is the random-access goal delivered.

The Footer Payload: FileMetadata and BlobMetadata

The JSON payload is where the format gets its meaning. It has two levels.

FileMetadata is the root object. It has one required field, blobs, which is a list of BlobMetadata objects, and one optional field, properties, a flat map of string keys to string values for information about the file as a whole. The spec recommends that writers set a created-by property identifying the application and version, such as "Trino version 381". That property is diagnostic gold when a stats file behaves oddly and you need to know which engine produced it.

Each BlobMetadata object describes one blob:

Three of these fields deserve a closer look.

fields is a list because a blob can describe several columns at once. A multi-column sketch, for instance a distinct-count of the combination of customer_id and region, lists both field IDs. Order matters, because the spec states that the order is used when computing sketches. A single-column NDV sketch has a one-element list. Using field IDs rather than names means the blob survives column renames, which is the same reason manifests use field IDs for their metrics maps.

snapshot-id and sequence-number pin the blob to a point in table history. A Theta sketch computed against snapshot 4 describes the data as of snapshot 4. After twenty more commits, it describes the data poorly. The planner compares the blob's snapshot to the current snapshot and decides how much to trust it. For deletion vectors, this pinning does not apply, and the spec requires both to be set to -1 because a delete file is written before the snapshot that contains it exists.

compression-codec is limited to two values for a reason the spec states plainly: for maximal interoperability, other codecs are not supported. lz4 means a single LZ4 frame with content size present. zstd means a single Zstandard frame with content size present. Both are single-frame encodings, so a reader decompresses the blob with one call and needs no framing logic. A Puffin reader in any language needs an LZ4 library and a Zstandard library and nothing else.

Here is a footer payload for a statistics file with two NDV sketches:

{
  "blobs": [
    {
      "type": "apache-datasketches-theta-v1",
      "fields": [2],
      "snapshot-id": 7168742983117921046,
      "sequence-number": 14,
      "offset": 4,
      "length": 32912,
      "compression-codec": "zstd",
      "properties": { "ndv": "1249831" }
    },
    {
      "type": "apache-datasketches-theta-v1",
      "fields": [5],
      "snapshot-id": 7168742983117921046,
      "sequence-number": 14,
      "offset": 32916,
      "length": 96,
      "compression-codec": "zstd",
      "properties": { "ndv": "6" }
    }
  ],
  "properties": {
    "created-by": "Spark 4.1 / Iceberg 1.11.0"
  }
}

The first blob starts at offset 4, immediately after the magic. The second starts at 32916, which is 4 plus 32912, immediately after the first. The ndv property on each is the pre-computed estimate, so an engine that only wants the number reads the footer and stops. The second column has an NDV of 6 and its sketch is under 100 bytes, because a Theta sketch of six values holds six 8-byte hashes plus a header. The first column has over a million distinct values and its sketch is about 32 KB, which is the saturation size of a compact Theta sketch at the DataSketches default of 4,096 nominal entries. Sketch size does not grow with cardinality past that point, which is the entire reason the structure is useful.

Blob Type One: The Theta Sketch for Distinct Values

The apache-datasketches-theta-v1 blob type stores a compact Theta sketch from the Apache DataSketches library. To understand why Iceberg chose this structure over a simple count, it helps to see how the sketch works.

Counting exact distinct values requires remembering every value you have seen, which for a billion-row column means a billion-entry hash set. A Theta sketch instead keeps a small, fixed-size sample of hashed values and uses the sample to estimate the total. The idea, known as K-Minimum Values, goes like this. Hash every value to a uniformly distributed 64-bit number, which maps it to a point in the interval [0, 1). Keep only the smallest k hashes you have seen. If you have seen n distinct values spread uniformly across [0, 1), the k-th smallest one sits at roughly k / n. Call that position theta. Then the estimate for n is k / theta. Duplicates hash to the same point and never increase the sample, so the estimate counts distinct values by construction.

DataSketches' Theta family generalizes this. Rather than a fixed k, the sketch tracks a threshold theta, keeps every hash below theta, and lowers theta as the sketch fills. The Alpha variant that Iceberg specifies uses a more sophisticated update rule that trades a bit of accuracy at small sizes for lower memory and faster updates. With the library's default of 4,096 nominal entries, the relative standard error on the estimate is about 1.6 percent. Doubling the size roughly divides the error by 1.4.

The property that matters most for Iceberg is that Theta sketches are mergeable. Two sketches built over two different sets of files union into one sketch that estimates the distinct count of the combined set, with no loss of accuracy versus building one sketch over everything. A writer that computes a sketch per data file, or per partition, or per Spark task, merges them into one table-level sketch. Re-analyzing after appending new files means sketching only the new files and merging with the old sketch. Intersection and set difference are also supported, which lets an optimizer estimate the overlap between two columns' value sets for join cardinality.

The spec fixes the inputs so sketches from different engines merge correctly. The sketch is built with the default seed. Each distinct value is converted to bytes using Iceberg's single-value serialization, the same encoding used for partition values and bounds in manifests. An int becomes four little-endian bytes, a string becomes UTF-8, a decimal becomes its unscaled big-endian two's complement bytes, and so on. If Trino used one byte encoding and Spark used another, the same value hashes differently and a union double-counts it. The shared serialization rule is what makes cross-engine merging valid.

The stored form is the "compact" serialization, which is the sorted array of retained hashes plus a small header holding theta and a few flags. The compact form is read-only and space-optimal, which is what you want in a file that gets written once and read many times.

The blob metadata for a Theta sketch carries an ndv property with the estimate already computed, stored as a decimal string with no leading or trailing spaces. The spec says the property "may" be included, but in practice every engine that writes sketches includes it, and the dev list has discussed making it required. Trino and Presto read the ndv property directly as their source of truth rather than deserializing the sketch. Spark's compute_table_stats procedure writes both the sketch and the property. The property is the fast path. The sketch is for engines that need to merge or intersect.

Blob Type Two: The Deletion Vector

The deletion-vector-v1 blob type was added to the Puffin spec for Iceberg format version 3, and it is the storage format for deletion vectors, the v3 replacement for position delete files.

A deletion vector is a bitmap over the row positions of one data file. A set bit at position P means row P is deleted. Reading the data file with the vector applied means skipping every row whose position is set. The engine gets a bitmap it can test in constant time rather than a set of positions it has to build from a Parquet file.

The bitmap encoding is Roaring. Roaring bitmaps partition the 32-bit integer space into 65,536 chunks of 65,536 values each, and store each chunk in whichever of three containers is smallest for its density: a sorted array of 16-bit values for sparse chunks, a 8-kilobyte bitset for dense chunks, and a run-length list for chunks with long consecutive runs. A vector marking 300 scattered rows out of 4 million uses a handful of array containers and totals a few hundred bytes. A vector marking rows 1,000,000 through 2,999,999 as deleted uses run containers and totals a few dozen bytes. This adaptivity is why Roaring became the standard for this job in Delta Lake, Lucene, and now Iceberg.

Iceberg rows can have positions above 2^32, since a single data file can in principle hold more than 4 billion rows. The spec handles this by splitting a 64-bit position into a 32-bit key from the high four bytes and a 32-bit sub-position from the low four bytes. For each distinct key, one 32-bit Roaring bitmap holds the sub-positions. Testing a position means finding the bitmap for its key, then testing the sub-position. Files under 4 billion rows, which is all of them in practice, have exactly one key and one bitmap. The structure supports the full 64-bit range without paying for it.

The serialized blob has a fixed envelope around the bitmap:

Four bytes, big-endian: the combined length of the magic and the vector.
Four magic bytes: D1 D3 39 64.
The vector, in Roaring's portable 64-bit format.
Four bytes, big-endian: a CRC-32 checksum over the magic and the vector.

Inside the vector, the portable format is: an 8-byte little-endian count of 32-bit bitmaps, then for each bitmap in unsigned key order, a 4-byte little-endian key followed by the standard 32-bit Roaring serialization.

The endianness mix is deliberate and the spec explains it. The Roaring format itself is little-endian, as defined by the Roaring specification. The length and CRC envelope is big-endian for byte compatibility with the deletion vectors Delta Lake already stored. Delta and Iceberg deletion vectors are wire-identical inside the envelope, which is one of the concrete outcomes of the cross-format convergence work that also produced the variant type.

The blob metadata for a deletion vector has strict requirements. It must include a referenced-data-file property whose value equals the data file's location in the table metadata, so a reader can pair the vector with the file it applies to. It must include a cardinality property with the number of set bits, so planners can estimate live row counts without decoding the bitmap. It must omit compression-codec, because Roaring is already compact and the spec forbids compressing deletion vectors. And snapshot-id and sequence-number must both be -1, since the vector is written before the commit that includes it.

Many deletion vectors can live in one Puffin file. A single delete operation that touches 500 data files writes 500 vectors into one file, back to back, and the footer lists all 500 with their offsets and lengths. This is what keeps the file count under control. In v2, that same operation wrote up to 500 position delete files. In v3 it writes one Puffin file.

How Table Metadata and Manifests Point at Puffin

A Puffin file on its own is inert. It becomes part of the table when Iceberg metadata references it, and the two blob types are referenced in completely different ways.

Statistics files are referenced from table metadata. The table metadata JSON has an optional statistics list. Each entry is a struct with the snapshot ID the file belongs to, the statistics-path, file-size-in-bytes, file-footer-size-in-bytes, an optional key-metadata for encryption, and a blob-metadata list that mirrors a subset of the Puffin footer: each blob's type, snapshot ID, sequence number, field IDs, and properties. This duplication is intentional. An engine that reads the table metadata already knows every statistic available and its ndv estimate without opening the Puffin file at all. Only an engine that wants the sketch itself, for merging or intersection, goes to storage.

Statistics are informational. The spec is explicit that a reader can ignore them and that support is not required to read the table correctly. A table can hold many statistics files for different snapshots, and each is associated with exactly one snapshot ID. When a snapshot is expired, the statistics file tied to it is removed from the statistics list and its file becomes an orphan to be cleaned up by orphan-file removal.

There is a second, related list called partition-statistics. These files are not Puffin. They are Parquet, Avro, or ORC files with a fixed schema of per-partition row counts, file counts, and sizes, produced by the compute_partition_stats procedure. People conflate the two because both are "statistics" and both hang off the table metadata. Only column-level sketches use Puffin.

Deletion vectors are referenced from delete manifests. This is the more interesting integration, because deletion vectors are not informational. They are required for correctness. A reader that ignores them returns deleted rows.

A delete manifest entry for a deletion vector is a normal manifest entry with content set to position deletes, file_format set to puffin, and file_path pointing at the Puffin file. Three fields added in v3 do the rest. referenced_data_file holds the location of the one data file the vector applies to. content_offset holds the byte offset of the vector's blob inside the Puffin file. content_size_in_bytes holds the blob's length. The spec requires that these two values exactly match the offset and length in the Puffin footer for that blob.

The effect is that a reader never has to parse the Puffin footer to apply a deletion vector. The manifest entry already says: open this file, seek to this offset, read this many bytes, and you have a bitmap for that data file. One range read per vector, no footer decode. The Puffin footer still exists and is still valid, which keeps the file inspectable by generic tooling, but the hot path bypasses it.

Two rules from the spec govern the lifecycle. First, there can be at most one deletion vector per data file in a snapshot. A writer that adds deletes to a file that already has a vector must read the old vector, union in the new positions, write a new vector, and replace the manifest entry. This is different from v2 position deletes, where multiple delete files for one data file accumulated and every reader merged them. Second, when a data file is removed, the writer must remove its deletion vector from the delete manifests, but is not required to rewrite the Puffin file containing that vector. The vector's bytes stay in the file as dead space until the file has no live references and gets cleaned up.

The result is a very different file count profile from v2. A table with a million data files and frequent updates in v2 accumulates position delete files at roughly one per touched data file per commit. In v3 it accumulates one Puffin file per commit, holding as many vectors as that commit touched files. Ten thousand small update commits produce ten thousand Puffin files rather than millions of delete files.

Walkthrough: Reading a Puffin File From Scratch

Nothing demonstrates a format's simplicity like a reader that fits on one screen. The following Python reads a Puffin footer and lists its blobs, using only the standard library plus lz4 for the optional footer compression. It does not need Iceberg, PyIceberg, or any JVM.

import json
import struct

MAGIC = b"PFA1"

def read_puffin_footer(path):
    with open(path, "rb") as f:
        f.seek(0, 2)
        file_size = f.tell()

        # Trailer: FooterPayloadSize (4) + Flags (4) + Magic (4)
        f.seek(file_size - 12)
        trailer = f.read(12)
        payload_size, flags, magic = struct.unpack("<ii4s", trailer)
        assert magic == MAGIC, "bad trailing magic"

        # Payload plus the magic that precedes it
        f.seek(file_size - 12 - payload_size - 4)
        head_magic = f.read(4)
        assert head_magic == MAGIC, "bad footer-start magic"
        payload = f.read(payload_size)

    compressed = flags & 0x01
    if compressed:
        import lz4.frame
        payload = lz4.frame.decompress(payload)

    return json.loads(payload.decode("utf-8"))

def read_blob(path, blob):
    with open(path, "rb") as f:
        f.seek(blob["offset"])
        raw = f.read(blob["length"])
    codec = blob.get("compression-codec")
    if codec == "zstd":
        import zstandard
        return zstandard.ZstdDecompressor().decompress(raw)
    if codec == "lz4":
        import lz4.frame
        return lz4.frame.decompress(raw)
    return raw

meta = read_puffin_footer("stats.puffin")
print("created-by:", meta.get("properties", {}).get("created-by"))
for b in meta["blobs"]:
    print(b["type"], "fields", b["fields"],
          "snapshot", b["snapshot-id"],
          "ndv", b.get("properties", {}).get("ndv"))

Walking through it. The trailer is read as one 12-byte chunk and unpacked with struct using the < prefix for little-endian and i for signed 32-bit integers, matching the spec's integer rule. The payload's starting position is computed arithmetically from the file size and payload size, then the reader verifies the magic that precedes the payload before trusting it. The compression bit is bit 0 of the flags integer, tested with a bitwise AND. read_blob seeks to the offset from the footer, reads exactly length bytes, and decompresses based on the codec string. The only third-party dependencies are the two compression libraries, and only when a codec is actually used.

Decoding a Theta sketch blob past this point needs the DataSketches library for your language. Decoding a deletion vector needs a Roaring bitmap library and the envelope logic from the spec:

import struct
import zlib

DV_MAGIC = bytes.fromhex("D1D33964")

def decode_deletion_vector(blob_bytes):
    (length,) = struct.unpack(">i", blob_bytes[:4])
    magic = blob_bytes[4:8]
    assert magic == DV_MAGIC, "bad deletion vector magic"
    vector = blob_bytes[8:4 + length]
    (crc,) = struct.unpack(">I", blob_bytes[4 + length:8 + length])
    assert zlib.crc32(blob_bytes[4:4 + length]) == crc, "crc mismatch"

    # Portable 64-bit Roaring: count (8 LE), then key (4 LE) + 32-bit bitmap
    (n_bitmaps,) = struct.unpack("<q", vector[:8])
    return n_bitmaps, vector[8:]

The length and CRC use > for big-endian, the magic is checked, and the CRC is computed over the magic and vector together, exactly as the spec states. What comes back is the count of 32-bit bitmaps and the raw Roaring bytes, which the pyroaring package or any Roaring implementation deserializes. The point of showing this is not that you should write your own reader. It is that a complete, correct reader is under a hundred lines, which is what "trivially parseable" was supposed to mean.

To see how table metadata references a stats file, query the metadata JSON directly. In Spark, the statistics list is not exposed as a metadata table, but you can read the current metadata file location from the metadata_log_entries table and inspect it:

SELECT file FROM db.orders.metadata_log_entries
ORDER BY timestamp DESC LIMIT 1;

Opening that JSON and looking at the statistics array shows the statistics-path, file-footer-size-in-bytes, and the embedded blob-metadata with ndv properties. That is the fast path an optimizer uses.

Producing and Consuming Statistics Across Engines

Puffin statistics are not written automatically. Every engine that supports them requires an explicit analyze step, and the commands differ.

In Spark with the Iceberg extensions, the procedure is:

CALL polaris.system.compute_table_stats(
  table => 'sales.orders',
  columns => array('customer_id', 'product_id', 'order_status')
);

Without the columns argument it computes sketches for every column, which on a wide table is expensive and mostly wasted. Restrict it to join keys, filter columns, and group-by columns. An optional snapshot_id argument computes against an older snapshot. The procedure returns the path of the Puffin file it wrote and registers it in the table metadata in the same commit.

In Trino, the command is the standard ANALYZE sales.orders, optionally with a columns property to restrict scope. Trino was the first engine to write Theta sketches to Puffin, in 2022, and its optimizer reads the ndv property during planning. Presto reads them the same way.

Dremio's cost-based optimizer consumes NDV statistics when planning joins, and statistics collection is triggered through the platform's own commands rather than the Spark procedure. Amazon Athena and Redshift Spectrum read Puffin NDV statistics from tables analyzed by other engines. The pattern across the ecosystem is that reading is more widely supported than writing, and a single analyze job in Spark or Trino benefits every reader that shares the table.

What each engine does with the number varies. The common use is join ordering: a three-way join has six possible orders, and the intermediate result size between the best and worst can differ by 100x. NDV estimates on the join keys let the optimizer predict output cardinality for each order and pick the smallest. The second use is broadcast decisions: a table with low distinct-count keys and few rows is a broadcast candidate. The third is aggregation sizing: GROUP BY customer_id with an NDV of 1.2 million tells the engine to plan for a 1.2-million-entry hash table rather than guessing.

Deletion vectors, unlike statistics, are produced automatically by any v3-capable writer that performs a delete, update, or merge. Spark 3.5 and 4.x with Iceberg 1.8 and later write them by default on v3 tables. Flink's dynamic sink gained deletion vector support in Iceberg 1.11. Any engine that reads v3 tables must apply them, and every engine claiming v3 read support does. There is no analyze step and no opt-in beyond upgrading the table's format version.

Failure Modes: What Breaks and the Warning Signs

Puffin is simple, and most Puffin problems are not format problems. They are lifecycle problems: stale content, orphaned files, and mismatched expectations between engines.

Stale statistics that the optimizer trusts. A sketch is pinned to a snapshot. Nothing forces an engine to distrust it after the table has moved on. If you analyzed a table when it had 10 million rows and it now has 800 million, the ndv for customer_id reflects the old population, and the optimizer plans joins against a number that is off by an order of magnitude. The warning sign is a join that was fast and turned slow with no query change. Comparing the snapshot ID in the statistics entry to the current snapshot ID tells you immediately how far behind the stats are.

No statistics at all. Because writing is opt-in, most tables have never been analyzed. The optimizer falls back to row counts from manifests and heuristics for NDV. Query plans are frequently reasonable anyway, which hides the problem until a workload arrives where join order matters. Checking whether the statistics list in table metadata is empty is a thirty-second diagnostic that many teams never run.

Statistics computed by one engine that another engine ignores. Every engine reads the ndv property, but not every engine deserializes the sketch. If you rely on an engine to intersect sketches for join selectivity and it only reads the property, you get a cruder estimate than you expected. Engines also differ in how they weight stale stats. Knowing which of your engines does what is part of running a shared table.

Orphaned Puffin files after snapshot expiry. When expire_snapshots removes a snapshot, the statistics file registered to it is dropped from the metadata list. The file is not deleted. Deletion vectors follow the same pattern: when data files are rewritten by compaction, their vectors are dropped from manifests but the Puffin files stay on storage. Over months, a busy table collects thousands of unreferenced Puffin files. They cost storage and, on some object stores, slow down listing. Regular remove_orphan_files runs are the fix, and the same job cleans up orphaned data files, so most teams already have it scheduled.

Deletion vector accumulation. A v3 table that receives frequent small updates and is never compacted ends up with a Puffin file per commit and a deletion vector for a large fraction of its data files. Reads stay correct, and each vector is a single range read, so the per-file cost is low. But the aggregate still adds up: ten thousand data files each with a vector means ten thousand extra range requests per full scan. The signal is scan latency rising with the number of delete manifests. rewrite_data_files merges deletes into new data files and drops the vectors, and it should run on the same cadence as any other compaction.

A vector that does not match its manifest entry. The spec requires content_offset and content_size_in_bytes in the manifest to match the blob's offset and length in the Puffin footer exactly. A writer bug or a manually edited manifest that breaks this produces a reader that seeks to the wrong bytes. Good readers verify the deletion vector magic and CRC and fail loudly. A CRC mismatch error on read is the sign, and the fix is to rewrite the affected data files.

Mismatched serialization between sketch writers. If two engines build sketches with different value serializations and you union them, the same value counts twice. The spec fixes the serialization to prevent this, but a nonconforming writer breaks it silently. The symptom is a merged NDV that exceeds the sum of the parts' plausible ranges. In practice this has not been a common problem because the writer count is small, but it becomes one as more implementations appear.

Puffin files written with an unsupported codec. The spec allows only lz4 and zstd. A writer that uses another codec produces a file no conforming reader opens. This does not happen with mainstream engines, but a home-grown stats writer is a place to check.

Operational Guidance: Cadence, Scope, Cleanup, and Monitoring

A handful of practices keep Puffin content useful and keep the file count under control.

Analyze on a schedule tied to growth, not time. Refresh statistics when the table has grown or changed enough that the old sketch is misleading. A rule that works: re-run compute_table_stats when the row count has changed by more than 20 percent since the snapshot the current stats were computed against, or after any large backfill or rewrite. For slowly changing dimension tables, once a month is plenty. For a fact table that doubles weekly, tie the analyze job to the ingestion pipeline.

Restrict the column list. Sketch only the columns the optimizer uses: join keys, common filter columns, common group-by columns. A 200-column event table with sketches on every column produces a 6-megabyte statistics file and spends an hour of cluster time on columns no query joins on. Twenty well-chosen columns cover almost every plan.

Use incremental merging where the engine supports it. Since Theta sketches merge, an engine that sketches only new files since the last analyze and unions with the prior sketch does the job in a fraction of the time. Check whether your engine's analyze implementation is incremental. If it is not, and the table is large, schedule the full analyze during a low-traffic window.

Compact deletion vectors on the same cadence as data files. Treat a high ratio of delete manifests to data manifests as a compaction trigger. rewrite_data_files with the default settings rewrites files that have deletes attached and removes the vectors. Running rewrite_position_delete_files on v3 tables is less relevant since vectors are already one per file, but it still helps consolidate Puffin files that hold only a few live vectors each.

Run orphan-file removal monthly. Puffin files become orphans through both snapshot expiry and compaction. The standard remove_orphan_files procedure handles them along with everything else. Set the older_than threshold to comfortably exceed your longest-running job so an in-flight write's files are never swept.

Monitor three numbers. The age of the current statistics in commits or days. The count of delete manifests relative to data manifests. The count of Puffin files on storage relative to the count referenced in metadata. Each one drifting upward has a specific fix, and each is cheap to compute from the metadata tables.

Record created-by and check it. When a Puffin file behaves strangely, the created-by property in its footer tells you which engine and version wrote it. Encourage every writer in your stack to set it, and include it in any debugging checklist.

Encrypt if the table is encrypted. Iceberg's table encryption, which gained envelope encryption and key management integration in 1.11, extends to statistics files through the key-metadata field in the statistics entry. A Puffin file holding a sketch of customer IDs leaks value hashes, not values, but a deletion vector file discloses which rows changed. If the data files are encrypted, the Puffin files should be too.

Where the Ecosystem Is Heading

Puffin was designed to hold more than two blob types, and the pressure to add more is growing.

More statistics blob types. The obvious candidates are histograms for range selectivity, which let an optimizer estimate what fraction of rows fall between two values rather than assuming uniform distribution, and most-frequent-value lists for skew detection. Both are mergeable in sketch form and both are well understood from decades of database work. Proposals for these come up on the dev list with regularity.

Indexes, not just statistics. A blob type for a Bloom filter or a min-max index over a whole partition lets an engine prune at a finer grain than per-file manifests without touching Parquet footers. Spatial indexes for the v3 geometry and geography types are a natural fit: bounding boxes in manifests are coarse, and a cell-based index in Puffin gives the planner a second, finer cut. Vector-search indexes for embedding columns are further out but follow the same pattern.

Non-JVM readers and writers. PyIceberg, iceberg-rust, and iceberg-go all read deletion vectors as part of their v3 support, because reads are not correct without them. Writing statistics from these implementations is newer. As DuckDB, Polars, and the Rust-based engines become first-class Iceberg writers, expect them to produce Theta sketches with the DataSketches ports for their languages, and expect the shared serialization rule to matter more as the writer count grows.

A second Puffin version. The spec currently defines a single version and reserves every flag bit but one. A version bump becomes likely when a blob type needs a container-level feature, such as per-blob encryption keys or a blob-level checksum for statistics (deletion vectors already have one). The design leaves room for this without breaking the two-range-read footer discovery.

Format version 4. The v4 spec restructures manifests and moves column statistics into typed structs. It does not change Puffin. Deletion vectors and statistics files continue to be referenced the same way, which is a sign the container has held up.

Conclusion

Puffin solves two problems that Iceberg's manifests were never designed for: table-wide statistics that need mergeable sketches, and row-level deletes that need bitmaps. It solves them with a container so simple that a complete reader is a hundred lines in any language. A magic number, a run of opaque blobs, a JSON footer with offsets and lengths, and a trailer that locates the footer in two range reads.

The two blob types show the range of what fits in that container. A Theta sketch is a probabilistic structure that estimates distinct counts within a couple of percent, merges across engines because the spec fixes the value serialization, and ships its estimate in the footer so most readers never decode it. A deletion vector is a Roaring bitmap wrapped in a Delta-compatible envelope, referenced directly from delete manifests by offset and length so the hot read path skips the footer entirely.

The operational lessons are about lifecycle rather than format. Statistics are opt-in and go stale, so analyze on a growth-driven cadence and scope it to columns the optimizer uses. Deletion vectors accumulate and get orphaned, so compact and clean up on the same schedule as data files. Do those two things and Puffin is invisible infrastructure that makes joins faster and deletes cheap. Skip them and you have a table with statistics from six months ago and ten thousand small delete files that nobody sweeps.

Keep Going

If this piece was useful, I have written a lot more on the Iceberg metadata layer and how engines use it to plan and execute queries. Apache Iceberg: The Definitive Guide from O'Reilly covers manifests, snapshots, row-level deletes, and the statistics that feed query planning, which is the context every section of this article sits inside. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet

Alex Merced — Tue, 01 Sep 2026 19:27:36 +0000

A logistics team stores 40 million delivery stops in an Apache Iceberg table. Every row has a latitude and a longitude. The analyst wants every stop inside a polygon that outlines one metro area. The query engine scans every data file in the table, because nothing in the table metadata tells it which files contain points inside that polygon. Forty million rows get read to return two hundred thousand.

That was the normal state of spatial data on the lakehouse for most of a decade. Coordinates lived in two double columns or in an opaque binary column. The table format did not know the column was spatial. The file format did not know either. Every optimization that Iceberg applies to timestamps, integers, and strings, from min/max pruning to partition transforms, simply did not apply.

Iceberg format version 3 changes this by adding two native primitive types: geometry and geography. Apache Parquet 2.11 added matching logical types at the file level. Together they give spatial data the same standing as any other column: a declared type, a coordinate reference system that travels with the schema, and per-file bounding-box statistics that let an engine skip files before reading a single shape.

This article explains the mechanism. It covers what the two types mean, how coordinate reference systems and edge interpolation are encoded, how bounding boxes are stored and used for pruning, how the Iceberg types relate to Parquet and to the older GeoParquet convention, and what breaks when you deploy this in production. I work at Dremio, which ships Iceberg v3 support, but the material here is spec-level and applies to any engine.

How Spatial Data Lived in Tables Before v3

Before format version 3, an Iceberg table had no vocabulary for a shape. Teams picked from a short list of workarounds, and every option lost something.

The simplest approach stored longitude and latitude as two double columns. This works for points and nothing else. A polygon, a route, or a service boundary cannot fit in two numbers. Min/max statistics on the two columns do give you crude bounding-box pruning for point data, which is why many teams stuck with this pattern for years.

The more general approach stored shapes as Well-Known Binary (WKB) in a binary column, or Well-Known Text (WKT) in a string column. WKB is the Open Geospatial Consortium (OGC) standard byte encoding for points, lines, polygons, and their multi-part variants. Every spatial library reads it. The problem is that the Iceberg schema saw only binary. The manifest recorded byte-wise min and max bounds for the column, which are meaningless for pruning. No engine skipped a file based on those bounds. Every spatial predicate became a full scan followed by row-by-row geometry parsing.

The coordinate reference system (CRS) was the other casualty. A CRS defines how a pair of numbers maps to a location on Earth. Longitude 30, latitude 10 means one place under WGS84 and a completely different place under a projected national grid. With a plain binary column, the CRS lived in a wiki page, a column comment, or someone's memory. Two teams writing to the same table with different assumptions produced silent corruption that no validation caught.

Engines with spatial support, such as Apache Sedona, built their own conventions on top of Iceberg to fill the gap. Sedona's Havasu extension added CRS metadata, bounding-box statistics, and format annotations through a fork of Iceberg. This worked for Sedona users but did not travel. A Sedona-written table opened in another engine went back to being bytes.

The v3 spec work pulled these ideas into the standard. The design was driven largely by the Wherobots team, who had run the Havasu approach in production since 2022 and contributed the design upstream to both Parquet and Iceberg. The Parquet logical type proposal collected over 400 review comments. The Iceberg type spec collected 240 more. That review volume is a sign of how many decisions hide inside "just add a geometry type."

Geometry Versus Geography: Two Types, Two Models of the Earth

Iceberg v3 defines two spatial types rather than one because there are two different ways to compute with coordinates, and mixing them produces wrong answers.

The geometry type treats coordinates as points on a flat plane. Distance is Euclidean. A line between two points is straight in the coordinate space. This is the right model for data in a projected CRS such as a state plane or UTM zone, where the projection has already flattened a region of the Earth onto a plane. It is also the right model for non-geographic data such as floor plans, chip layouts, or any coordinate system where "the Earth is round" is not a relevant fact.

The geography type treats coordinates as positions on the surface of an ellipsoid or sphere. A line between two points follows a geodesic, the shortest path over the curved surface, rather than a straight line in longitude and latitude. Distance is computed along that surface. This is the right model for global data stored in longitude and latitude, where a "straight" line across a thousand kilometers in planar math bends noticeably away from the true shortest path.

The difference shows up in ordinary queries. Take two airports 8,000 kilometers apart. Planar distance on raw longitude and latitude gives a number in degrees that means nothing. Geodesic distance gives kilometers. Take a polygon that covers Alaska. Under planar math its western edge crosses the antimeridian at longitude 180 and the polygon appears to wrap around the entire planet. Under geographic math the polygon is a small region on a sphere and behaves correctly.

The spec encodes this distinction in the type definitions. geometry(C) is parameterized by a CRS C. geography(C, A) is parameterized by a CRS C and an edge-interpolation algorithm A. Both default the CRS to OGC:CRS84, which means longitude and latitude on the WGS84 datum with longitude first. Geography defaults the algorithm to spherical.

The choice between them is not cosmetic. An engine reading a geometry column runs Cartesian computations regardless of what CRS string is attached. The spec states this directly: for geometry, the CRS does not affect geometric calculations. The CRS is carried as metadata so downstream tools can reproject or display correctly, but the storage layer computes on a plane. If your longitude-latitude data needs correct global distances and containment, geography is the type that asks for that.

Coordinate Reference Systems and Edge Interpolation in the Schema

The CRS parameter is a string, and the spec is deliberate about what that string can and cannot contain.

The recommended form is <context>:<identifier>. Examples from the spec are OGC:CRS84, EPSG:4326, IGNF:ATI, and SRID:0. The EPSG registry (originally the European Petroleum Survey Group) is the most widely used catalog of CRS definitions, and EPSG:4326 is the code for WGS84 with latitude-first axis order. OGC:CRS84 is the same datum with longitude-first order, which matches the WKB convention of X then Y. The default is OGC:CRS84 for exactly that reason: WKB always stores X (longitude or easting) before Y (latitude or northing), so the default CRS declares the same order.

For a custom CRS that does not have a registry code, the spec allows a reference of the form projjson:<property-name>. PROJJSON is the JSON encoding of a CRS definition from the PROJ library. The definition itself goes in a table property under that name, and the type string only points to it. The spec forbids inlining PROJJSON directly into the type string and forbids implementations from parsing the type string as PROJJSON. The reason is size. A full PROJJSON definition runs to kilobytes, and the schema is embedded in every metadata file and every manifest list. Inlining it bloats metadata reads across the whole table.

For geography, the CRS has an added constraint: it must be geographic, with longitudes in [-180, 180] and latitudes in [-90, 90]. A projected CRS on a geography column is invalid.

The edge-interpolation algorithm A on geography selects how the engine computes the curve between two vertices. The spec lists five values:

spherical: edges are geodesics on a perfect sphere. Cheapest to compute, accurate to within about 0.3 percent for most distances. The default.
vincenty: Vincenty's iterative formulae on the ellipsoid. Accurate to millimeters, fails to converge for nearly antipodal points.
thomas: Paul Thomas's 1970 spheroidal geodesic method.
andoyer: Thomas's 1965 navigation model, a lower-cost ellipsoidal approximation.
karney: Charles Karney's 2013 algorithm as implemented in GeographicLib. Converges everywhere and is accurate to nanometers.

Most teams never change this from spherical. The parameter exists so that two engines reading the same table agree on what "the edge between these two points" means. If a writer computed containment using Karney geodesics and a reader used spherical ones, a point sitting a few meters from a polygon boundary flips between inside and outside depending on who asks. Storing the algorithm in the type removes that ambiguity.

In the schema JSON, the types serialize as strings. A geometry column in a default CRS is written as "geometry". With a custom CRS it becomes "geometry(srid:4326)". A geography column with both parameters looks like "geography(srid:4326, spherical)". Any engine that already parses Iceberg type strings extends its parser to handle the parenthesized parameters.

What the Type Changes in Metadata, Files, and Partitioning

Adding a type to a table format touches more than the schema. Several rules in the v3 spec exist only because these two types exist.

Default values are restricted. Iceberg v3 introduced initial-default and write-default so a column added later can be populated for old rows without rewriting files. For geometry and geography, along with variant and unknown, the spec requires that both defaults be null. A non-null default for a shape column is invalid. This avoids embedding WKB byte strings inside the schema JSON, and it sidesteps the question of what a "default polygon" even means.

Partition transforms are limited. The identity transform is defined for every primitive type except geometry and geography. The bucket transform's list of valid source types does not include them either. You cannot partition directly on a shape column. The reasons are practical. Identity partitioning on a polygon produces one partition per distinct polygon, which is useless. Bucketing by hash of the WKB bytes scatters spatially adjacent shapes across buckets at random, which defeats the point of spatial locality. Spatial partitioning is done today through derived columns, covered later in this article.

Physical storage is WKB everywhere. In Avro, both types map to bytes in WKB. In Parquet, both map to binary, annotated with the GEOMETRY or GEOGRAPHY logical type where the writer supports it. In ORC, both map to binary with an iceberg.binary-type attribute set to GEOMETRY or GEOGRAPHY, because ORC has no native spatial logical type. Single-value serialization for partition values and bounds uses WKB. JSON serialization, used in places like default values and some REST catalog payloads, uses WKT so the value is human-readable.

The Parquet logical type is what makes cross-engine reads work. This point deserves emphasis. If a writer produces a Parquet file with a plain binary column and no logical type annotation, a reader that opens that file without the Iceberg schema sees bytes. The PyIceberg implementation notes this explicitly: binary columns cannot be distinguished from geometry without the Iceberg schema metadata. When the writer applies the Parquet GEOMETRY logical type, the file itself declares the column as spatial, and any Parquet reader that understands Parquet 2.11 recognizes it. That is the difference between spatial data that works in one engine and spatial data that works everywhere.

The Parquet logical type also carries the CRS. Parquet's GEOMETRY and GEOGRAPHY types have their own CRS field and, for geography, their own edge algorithm field. Iceberg writers set these to match the Iceberg type parameters. A file written for a geography(OGC:CRS84, karney) column carries that same CRS and algorithm in its Parquet footer. Readers that trust the Parquet footer and readers that trust the Iceberg schema arrive at the same answer.

Bounding Boxes: How Files Get Skipped

The most valuable thing the v3 types add is a per-file bounding box that the query planner reads from the manifest. This is the mechanism that turns a 40-million-row scan into a handful of files.

For every primitive column, Iceberg manifests store lower_bounds and upper_bounds. For an integer column these are the smallest and largest values in the file. For a geometry or geography column, the spec defines the bounds as two points. The lower bound is a point whose X, Y, and optional Z and M coordinates are each the minimum of that coordinate across every shape in the file. The upper bound is the point of maximums. Together they define the axis-aligned bounding box that contains every object in the file.

Z is elevation and M is a fourth measure such as a milepost or timestamp. Both are optional in WKB. The spec handles missing dimensions carefully. Null or NaN coordinate values are skipped during bound computation. If a dimension has only null or NaN values across the whole file, that dimension is omitted from the box. If either X or Y is missing entirely, no bounding box is produced at all, because a box without both planar axes cannot prune anything.

In v3, the two bound points are serialized as raw binary: an x:y:z:m concatenation of 8-byte little-endian IEEE 754 doubles. X and Y are mandatory. The encoding shrinks to x:y when Z and M are absent, x:y:z when only M is absent, and x:y:NaN:m when only Z is absent. The NaN placeholder keeps the byte offsets unambiguous.

In v4, the bounds move into typed structs called geo_lower and geo_upper inside the new content_stats structure. Each struct has required x and y doubles and optional z and m doubles. The struct field IDs are assigned by fixed offsets within the column's stats ID range, so a geometry column with field ID 4 gets its lower-bound X at stats ID 10,810 and its upper-bound X at 10,814. The information is the same as v3. The difference is that engines read typed fields instead of parsing a variable-length byte array.

The geography type has one special rule for bounding boxes that catches people out. For geography columns, the X value of the lower bound is allowed to be greater than the X value of the upper bound. This encodes a box that crosses the antimeridian at longitude 180. A file containing shapes around Fiji, which straddles that line, gets a lower X of 178 and an upper X of negative 179. Under normal min/max logic that box is empty. Under the geography rule, an object matches if its X satisfies x >= xmin OR x <= xmax. The spec ties this to geographic vocabulary: xmin is westernmost, xmax is easternmost, ymin southernmost, ymax northernmost. Bounds are further restricted to the canonical ranges of [-180, 180] and [-90, 90].

For geometry, no wraparound applies. The X of the lower bound is always less than or equal to the X of the upper bound, because planar coordinates do not wrap.

When a query arrives with a spatial predicate such as ST_Intersects(geom, <polygon>), the planner computes the bounding box of the query polygon and compares it to each file's stored box. If the boxes do not overlap, the file cannot contain a match and is skipped without being opened. If they do overlap, the file is read and the precise predicate is evaluated row by row. This is the same inclusive-bound logic Iceberg uses for every other type, extended to two dimensions.

The pruning is only as good as the boxes are tight. A file whose shapes are scattered across a continent has a box that overlaps nearly every query. A file whose shapes cluster in one city has a small box that most queries miss. Data layout determines whether the statistics do anything, which is why the operational section of this article spends time on sorting.

GeoParquet, Native Parquet Types, and Iceberg: Three Layers That Now Line Up

Anyone who has worked with spatial data on object storage has encountered GeoParquet, and the relationship between GeoParquet and the new Iceberg types confuses people. The short version: they solved the same problem at different layers and at different times, and they now converge.

GeoParquet 1.0, standardized in 2022 by the OGC community, defined a convention for spatial data in ordinary Parquet files. Geometry columns were stored as BYTE_ARRAY containing WKB. A JSON document under a geo key in the file's key-value metadata declared which columns were spatial, what CRS they used, what geometry types they contained, and an overall bounding box. GeoParquet 1.1 added a covering option: an extra struct column with xmin, ymin, xmax, and ymax per row, so that Parquet's own per-row-group statistics on those four doubles gave engines a way to skip row groups.

This worked and got wide adoption. Its weakness was structural. The geometry column was still a plain binary column. An engine had to opt in to reading the sidecar JSON, and engines built for general analytics rarely did. Table formats had the same problem: Iceberg needed a first-class Parquet type to build interoperable table-level semantics, and sidecar metadata cannot provide that.

Parquet 2.11, released in March 2025, added GEOMETRY and GEOGRAPHY as logical types in the format specification itself. They annotate a BYTE_ARRAY in WKB, carry a CRS and (for geography) an edge algorithm, and produce native column statistics that include a bounding box per column chunk. The Parquet community refers to this direction as GeoParquet 2.0, and the GeoParquet 2.0 specification is written on top of the native types. GeoParquet 2.0 requires geometry columns to use the native logical types, requires them to sit at the root of the schema rather than nested inside structs or lists, and keeps the geo metadata key for optional extras the core Parquet spec does not cover.

Iceberg v3 sits above both. The Iceberg schema declares the column type, CRS, and algorithm. The Parquet files carry the matching logical type and per-row-group statistics. The Iceberg manifests carry per-file bounding boxes computed from those files. Three layers, one set of semantics.

The practical consequence is that a GeoParquet 1.x data lake and an Iceberg v3 spatial table are not competitors. GeoParquet files are an input format. You read them with any GeoParquet-aware tool, write the rows into an Iceberg v3 table with a geometry or geography column, and the Iceberg writer produces native-typed Parquet on the way out. The Sedona documentation makes this argument plainly: Iceberg with native geo types gives you what GeoParquet gave you, plus transactions, schema evolution, and row-level updates.

How It Fits Together in Practice

A spatial Iceberg table in production has three moving parts: the writer that produces native-typed files, the catalog and manifests that carry the statistics, and the engines that evaluate spatial predicates. Each part has a different maturity level as of late 2026, and knowing where each stands saves you from debugging problems that are really version gaps.

Writers. The Java reference implementation shipped the type system, the bounding-box types, and spatial predicates across releases 1.10 and 1.11. The Parquet read and write path that stamps the GEOMETRY logical type onto files went through a long review. PyIceberg added GeometryType and GeographyType in early 2026, stores values as WKB, and gains full GeoArrow extension-type support with CRS and edge metadata when installed with the geoarrow extra. Without that extra, PyIceberg writes plain binary columns and relies on the Iceberg schema for type information.

Catalogs. Any catalog that stores v3 table metadata handles the new types, because the catalog stores JSON and the types are strings. Apache Polaris, the REST catalog implementation that graduated to an Apache top-level project on February 18, 2026, validates schemas against format version but does not interpret spatial semantics. The same is true of Nessie, Unity Catalog, AWS Glue (which shipped v3 support in November 2025), and every other REST catalog. Catalogs are not where spatial support lives.

Engines. Snowflake was the first major engine to ship v3 geometry and geography on Iceberg tables in 2026. Apache Sedona reads and writes Iceberg spatial columns and has the deepest spatial function library, including CRS-aware transforms and support for CRS forms beyond integer SRIDs. Dremio has GA support for format version 3 in its cloud platform. Spark, Flink, and Trino connector support for spatial types is rolling out release by release, and the honest guidance is to check the specific connector version before moving a production spatial workload. An engine that supports v3 tables in general does not necessarily evaluate spatial predicates or push them down to bounding boxes.

The data flow that works today looks like this. Source shapes arrive as GeoJSON, shapefiles, GeoParquet, or WKT strings. A Sedona or GeoPandas process parses them into geometries in a known CRS. The writer casts them to the Iceberg column type and writes Parquet files with the native logical type and per-row-group bounding boxes. The Iceberg commit records per-file bounding boxes in the manifest. A downstream engine plans a spatial query by comparing the query polygon's box against manifest boxes, opens only overlapping files, and evaluates the exact predicate on the rows.

The one architectural decision that matters more than engine choice is data layout. Files must be spatially coherent for the bounding boxes to prune anything. A table where every file spans the whole world has statistics that are technically correct and practically useless. Layout is covered in detail under operational guidance.

Walkthrough: Defining, Writing, and Querying a Spatial Table

This section builds a table of delivery stops and service zones and runs a containment query against it. The schema comes first, because seeing the JSON makes the type parameters concrete.

A v3 table metadata file with two spatial columns carries a schema like this:

{
  "type": "struct",
  "schema-id": 0,
  "fields": [
    { "id": 1, "name": "stop_id", "required": true, "type": "long" },
    { "id": 2, "name": "delivered_at", "required": true, "type": "timestamptz" },
    { "id": 3, "name": "location", "required": false, "type": "geography" },
    { "id": 4, "name": "zone_id", "required": false, "type": "string" },
    { "id": 5, "name": "zone_footprint", "required": false,
      "type": "geometry(EPSG:3857)" }
  ]
}

The location column is geography with no parameters, so it defaults to OGC:CRS84 and spherical edges. Points are longitude-latitude on WGS84 and any distance math is geodesic. The zone_footprint column is geometry in EPSG:3857, the Web Mercator projection used by most map tiles. Zone boundaries drawn in a mapping tool arrive in that projection, and planar math on them is correct within a metro area. Two columns, two types, two CRSs, and the schema records all of it so no downstream reader has to guess.

Creating the table from Python uses PyIceberg's type classes. This requires a PyIceberg release with v3 spatial support and the geoarrow extra installed:

from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import (
    NestedField, LongType, TimestamptzType, StringType,
    GeographyType, GeometryType,
)

catalog = load_catalog("polaris")

schema = Schema(
    NestedField(1, "stop_id", LongType(), required=True),
    NestedField(2, "delivered_at", TimestamptzType(), required=True),
    NestedField(3, "location", GeographyType(), required=False),
    NestedField(4, "zone_id", StringType(), required=False),
    NestedField(5, "zone_footprint",
                GeometryType(crs="EPSG:3857"), required=False),
)

table = catalog.create_table(
    "logistics.delivery_stops",
    schema=schema,
    properties={"format-version": "3"},
)

The format-version property is the part people forget. Spatial types are rejected on v1 and v2 tables. PyIceberg raises a validation error through its format-version compatibility check rather than silently writing a binary column.

Writing rows from a GeoPandas frame goes through Arrow. With the geoarrow extra installed, PyIceberg recognizes GeoArrow extension arrays and maps them to the Iceberg types, preserving CRS metadata:

import geopandas as gpd
import pyarrow as pa

stops = gpd.read_parquet("s3://raw/stops/2026-08.parquet")
stops = stops.set_crs("OGC:CRS84", allow_override=True)

arrow_table = pa.Table.from_pandas(
    stops[["stop_id", "delivered_at", "location", "zone_id"]]
)
table.append(arrow_table)

Each append commits a snapshot whose manifest entries carry bounding boxes for the location column. You can verify this from the metadata tables. In Spark with the Iceberg extensions loaded:

SELECT file_path,
       record_count,
       lower_bounds[3] AS location_lower,
       upper_bounds[3] AS location_upper
FROM logistics.delivery_stops.files
LIMIT 5;

The map key 3 is the field ID of location. In a v3 table the values are the binary x:y encodings described earlier. Engines with spatial support decode them for display, and in v4 tables the same query reads typed geo_lower and geo_upper structs directly.

Querying is where engine support matters. In Apache Sedona on Spark, a containment query against one zone reads like this:

SELECT s.stop_id, s.delivered_at
FROM logistics.delivery_stops s
WHERE ST_Intersects(
  s.location,
  ST_Transform(
    ST_GeomFromWKT('POLYGON((-81.6 28.3, -81.2 28.3, -81.2 28.7, -81.6 28.7, -81.6 28.3))'),
    'EPSG:4326', 'OGC:CRS84'
  )
);

ST_GeomFromWKT parses the polygon. ST_Transform reprojects it to match the column's CRS. ST_Intersects is the spatial predicate. An engine with v3 pushdown computes the polygon's bounding box, compares it to each file's manifest box, and skips files whose boxes fall outside the rectangle from longitude -81.6 to -81.2 and latitude 28.3 to 28.7. Files that pass the box check are opened, and Sedona evaluates the exact intersection on each row.

The reprojection step is not optional. If the polygon is in EPSG:4326 (latitude-first) and the column is in OGC:CRS84 (longitude-first), the coordinates are the same numbers in swapped order. Skip the transform and the query returns rows from a polygon near the equator in the Indian Ocean. This class of bug is the single most common spatial error, and it happens silently.

Failure Modes: What Breaks and How You Notice

Spatial tables fail in ways that ordinary tables do not, and most of the failures produce wrong answers rather than errors. Knowing the patterns in advance is the difference between catching them in staging and catching them in a customer report.

Mixed CRS within one column. The type declares one CRS. Nothing at the storage layer verifies that every WKB value was actually produced in that CRS, because WKB does not carry a CRS. A pipeline that ingests one source in WGS84 and another in a national grid, and writes both to the same geometry(OGC:CRS84) column, produces a table where half the shapes are in the wrong place by thousands of kilometers. The bounding boxes for those files span absurd ranges, which is your first clue. A sanity check that every file's box falls inside the plausible extent of your data catches this on the first commit.

Geometry where geography was needed. A team stores global longitude-latitude points in a geometry column because it was the first type they saw. Distance queries return degrees. Buffer operations produce ellipses that stretch as latitude increases. Nothing errors. The fix is a new geography column and a backfill, not a type change, because the two types have different computational semantics and Iceberg does not support promoting between them.

Bounding boxes that never prune. If files are written in ingestion order rather than spatial order, each file contains points from wherever deliveries happened that hour, which is everywhere. Every file's box covers the service area, every query overlaps every box, and the planner reads everything. The table looks correct and the statistics look populated. The only symptom is that spatial queries are no faster than they were on v2. Checking the files metadata table and looking at how many boxes overlap a small test polygon tells you within minutes whether layout is working.

Antimeridian polygons in geometry columns. A polygon that crosses longitude 180 stored in a geometry column gets a planar bounding box from -180 to 180. It matches every query. Worse, planar intersection logic treats the polygon as spanning the world rather than a small region across the dateline. geography handles this correctly with the wraparound bound rule. Data that touches the Pacific belongs in geography.

Engines that read the table but not the type. An engine with v3 support but no spatial support opens the table, sees the type string, and either fails to parse it or maps it to binary. Some engines return WKB bytes for the column and evaluate no spatial predicates. Others refuse the table entirely. Every engine in the path needs to be checked individually, and a shared table that must serve an engine without spatial support needs either a parallel binary column or a wait until that engine catches up.

Very large shapes in a file of small ones. One country-sized polygon in a file of city blocks expands that file's box to the whole country. Every query anywhere in that country now opens that file. Boundary datasets with mixed scale deserve their own table or at least their own partition so their boxes do not pollute point data.

Z and M dimensions that are inconsistently present. If some rows carry elevation and others do not, the bounding box for Z is computed only from rows that have it, per the spec's NaN-skipping rule. That is correct but surprising: a Z-range filter will not exclude rows with no Z. Decide up front whether a column carries Z and M, and make it consistent.

Writer produces binary without the Parquet logical type. An older writer, or PyIceberg without the geoarrow extra, writes valid Iceberg data with the Iceberg schema type set correctly but with plain binary Parquet columns underneath. Iceberg-aware readers work fine. A direct Parquet reader, or a tool reading the files through a GeoParquet path, sees bytes with no CRS. Inspecting a Parquet footer with parquet-tools or PyArrow and checking for the GEOMETRY logical type confirms which situation you are in.

Operational Guidance: Layout, Partitioning, Migration, and Monitoring

Getting the types right is the first day. Keeping the table fast is every day after. The practices below are the ones that matter most.

Sort spatially before writing. Since bounding-box pruning depends on spatial coherence within files, the write path has to cluster nearby shapes together. The standard technique is to compute a space-filling curve index for each row and sort on it. A geohash string, an H3 cell index, or a Hilbert curve value all work. Compute it as an ordinary column, sort the write by it, and files naturally contain neighbors. Iceberg's RewriteDataFiles action with a sort order on that column does the same job for existing data during compaction.

Partition on a derived cell, not on the shape. Since identity and bucket transforms are not allowed on spatial types, partitioning uses a derived column. A coarse H3 resolution (resolution 3 gives cells around 12,000 square kilometers) or a short geohash prefix works as a partition column. Choose the resolution so that a typical query touches a small number of partitions and each partition holds a healthy number of files. Partition on the cell column with the identity transform, and sort within partitions on a finer cell for file-level coherence.

Keep polygons and points in separate tables. Point tables prune beautifully because each point is a single coordinate. Polygon tables prune less well because polygons have area. Mixing them in one table gives you the worst of both. Two tables joined at query time is almost always the faster design.

Migrating from a binary column. If you have a v2 table with WKB in a binary column, the path is: upgrade the table to format version 3 (a metadata-only change that does not touch data files), add a new geometry or geography column, run an update or a full rewrite that casts the binary values into the new column, verify counts and bounding boxes, then drop the old column. Do not try to change the existing column's type. Promotion from binary to a spatial type is not a supported type promotion, and no engine will do it in place. Also remember that once the table is on v3, engines that only support v2 can no longer read it, so the upgrade gates on every reader being ready.

Confirm statistics after the first write. Query the files metadata table and look at the bounds for the spatial column's field ID. If they are null, the writer did not compute them, and no pruning is happening. If they are present, spot-check a few against known data ranges.

Monitor files scanned per spatial query. The single best health metric is the ratio of files opened to files in the table for a representative small-area query. On a well-laid-out point table that ratio should be in the low single-digit percent. When it drifts upward after weeks of ingestion, the table needs a sort-order compaction.

Compaction has to preserve spatial sort. A compaction job that merges small files without a sort order destroys the spatial coherence the ingestion path created. Always pass the cell column as the sort key when rewriting, and consider making it the table's default sort order so every engine's compaction respects it.

Pick the geography algorithm once. The default spherical is fine for nearly all analytics. Switch to karney only if you need sub-meter agreement with a surveying system, and be aware that not every engine implements every algorithm. Changing the algorithm later means a new column, because it is part of the type.

Where the Ecosystem Is Heading

Spatial support in Iceberg is at the point where the spec is settled and the implementations are catching up. Several developments are worth watching.

Engine coverage will widen. The pattern with every v3 feature has been that the reference Java implementation lands first, then Spark and Flink connectors, then Trino, then the commercial engines. Spatial types follow the same curve. Expect the Spark and Trino connectors to reach full read, write, and pushdown parity over the next several releases, and expect the Rust and Go implementations that back DuckDB, ClickHouse, and the growing family of non-JVM readers to add spatial types as their v3 support matures.

Spatial partition transforms are under discussion. The community has talked through native transforms based on space-filling curves, such as a Hilbert or Z-order transform that takes a spatial column as its source. A native transform lets Iceberg partition on a shape column directly, with the engine computing the cell rather than the pipeline. This is not in the spec today, and derived columns remain the answer, but it is the obvious next step and the design work is visible on the dev list.

Spatial indexes in Puffin. Bounding boxes are the coarsest possible index. Finer structures such as R-trees or cell-based inverted indexes for a whole table or partition are a natural fit for the Puffin file format, which already stores deletion vectors and distinct-value sketches as blobs. A spatial index blob type lets an engine prune at row-group or row level before opening files.

GeoArrow closes the in-memory gap. GeoArrow is the Apache Arrow extension type specification for spatial data. With PyIceberg, Sedona, DuckDB, and GeoPandas all speaking GeoArrow, spatial data moves between tools without WKB serialization round trips. Iceberg's Parquet logical types and GeoArrow's extension types share the same CRS and edge vocabulary, so the mapping is direct.

v4 typed statistics simplify readers. The move from binary-encoded bounds in v3 to typed geo_lower and geo_upper structs in v4 removes a parsing step and makes bounding boxes visible to any tool that reads manifests, including tools with no spatial library at all. Expect metadata inspection tooling to display spatial bounds natively once v4 tables are common.

Agents and spatial data. Language-model agents that query lakehouse tables through the Model Context Protocol (MCP) work best when the schema tells them what a column means. A column typed geography with a CRS is self-describing in a way that a binary column named geom_wkb is not. Native types make spatial data usable by tooling that never had a GIS specialist in the loop.

Conclusion

For most of Iceberg's life, spatial data was a second-class citizen: bytes in a binary column, a coordinate system documented somewhere else, and no way for the planner to skip a file. Format version 3 fixes this at the root. geometry gives you planar shapes with a declared CRS. geography gives you geodesic shapes with a declared CRS and a declared edge algorithm. Both carry per-file bounding boxes in the manifest, both map to native Parquet 2.11 logical types, and both line up with the GeoParquet 2.0 direction so the file-level and table-level ecosystems finally agree.

The mechanism is simple once you see it: type in the schema, WKB in the file, bounding box in the manifest, logical type in the Parquet footer. The discipline is in the details. Choose the right type for your computational model. Reproject before you compare. Sort spatially before you write. Partition on a derived cell. Check the boxes after the first commit. Do those things and spatial queries on Iceberg prune like any other query. Skip them and you have a v3 table that scans like a v2 table.

Keep Going

If this piece was useful, I have written a lot more on the Iceberg table format and the metadata mechanics that make it work. Apache Iceberg: The Definitive Guide from O'Reilly covers the spec, the metadata layer, and how engines plan queries against manifests, which is the foundation everything in this article builds on. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Open Standards for Agentic Harnesses

Alex Merced — Mon, 31 Aug 2026 15:12:19 +0000

Every team that gets serious about AI agents hits the same wall, usually around month three. The agent works. It reviews code the way you want, or it triages tickets, or it maintains your data pipelines. Then someone asks a simple question: can we run this somewhere else? Can we move it to the tool the platform team standardized on? Can we share it with the team in another office that uses a different product?

The answer, in most shops, is no. The agent is not a thing you own. It is a configuration scattered across one vendor's product: a system prompt in one screen, tool grants in another, accumulated context in a proprietary store, approval rules in a settings page nobody remembers configuring. The model behind the agent is swappable. The harness around it is not, and the harness is where everything you built actually lives.

This article is about the standards effort to fix that. I am going to walk through six specifications that, together, make the pieces of an agentic system portable: the Model Context Protocol (MCP), Agent Skills, Agent2Agent (A2A), the Open Agent Profile (OAP), the Agentic Graph Specification (AGS), and the Agent Approval Interchange Specification (AAIS). Full disclosure up front: I authored the last three of those, and I work at Dremio, which ships an MCP Server as part of its platform. I will keep the analysis honest anyway, including where each standard is young, unproven, or the wrong tool.

Why Harnesses Became the New Lock-in Point

For most of the last decade, the lock-in conversation in data and AI centered on two layers. First it was storage and table formats, which is the fight Apache Iceberg largely settled by making tables an open specification any engine can read. Then it was models, which the market settled through sheer competition. Today you can route a request to a frontier model from any of a half dozen providers, or run an open-weight model on your own hardware, and switch between them in an afternoon.

The harness is the layer that quietly inherited the lock-in. A harness is the runtime around a model: the software that holds the conversation loop, executes tool calls, enforces permissions, manages context, and turns a model's text output into actual work. Claude Code is a harness. OpenAI's Codex CLI is a harness. Cursor's agent mode, Goose, OpenCode, and the internal orchestrators enterprises build on frameworks are all harnesses. The model does the thinking. The harness does everything else.

Everything else turns out to be everything that matters for ownership. Consider what accumulates inside a harness after six months of real use. Agent definitions, meaning the roles, instructions, and personas your team refined through hundreds of corrections. Tool connections, each one configured, authenticated, and scoped. Procedural knowledge, the documented workflows the agent follows for releases, migrations, and reviews. Work plans, the decompositions of big jobs into steps. Approval rules, the record of what requires a human and what does not. And learned state, the facts an agent picked up about your systems that make it useful on day 180 in a way it was not on day one.

None of that has anything to do with which model you use. All of it, absent standards, lives in one product's shape. Switching harnesses means reconstructing it from memory, which is expensive enough that most teams never do it. That is lock-in in its purest form: not a contract, just a moat made of your own accumulated work.

The pattern rhymes with what happened in data infrastructure, and I say that as someone who has spent years teaching that history. Before open table formats, your tables were trapped inside whichever warehouse wrote them. The fix was not a better warehouse. The fix was specifications: Parquet for files, Iceberg for tables, Polaris for catalogs. Each one turned a proprietary internal structure into a document any conforming system reads. The agentic stack is now going through the same transition, one artifact type at a time.

Six Standards, Six Questions

The useful way to hold these six specifications in your head is not as competitors. Each answers a different question about an agentic system, and a complete system needs an answer to all six.

Three of these have institutional weight behind them. MCP and A2A both live at the Linux Foundation now, and Agent Skills is stewarded through the Agentic AI Foundation with adoption across directly competing vendors. The other three are young, and I wrote them, so weigh my enthusiasm accordingly. What I will argue is that the questions they answer are real regardless of whether these particular documents win. If OAP, AGS, and AAIS all get replaced by better specifications next year, the gaps they name will still need filling: agent identity, work shape, and approval interchange have no portable home in the three established standards.

One more framing point before the details. A pile of open components does not automatically produce an open system. The test that matters is whether each artifact type can move: can you take your tool connections, your skills, your agent definitions, your plans, and your approval flows to a different runtime without rewriting them? Every section below is really an answer to that question for one artifact type.

MCP: What an Agent Can Reach

The Model Context Protocol is the oldest of the six and the closest thing the agentic stack has to settled infrastructure. Anthropic released it in November 2024 as an open protocol for connecting AI applications to tools and data. In December 2025 it was contributed to the Agentic AI Foundation under the Linux Foundation, which put it under neutral governance alongside other agent-era building blocks.

The mechanism is straightforward. An MCP server exposes three kinds of things: tools an agent can invoke, resources it can read, and prompts it can use as templates. A client, meaning the harness, connects to servers over stdio for local processes or HTTP for remote ones, speaks JSON-RPC, discovers what each server offers, and makes those capabilities available to the model. The protocol standardizes discovery, invocation, and results. It deliberately does not standardize what the tools do.

The reason MCP matters for harness portability is the shape of the integration problem it dissolves. Before a tool protocol, every harness needed its own connector for every system: N harnesses times M systems means N times M integrations, each one written by whichever vendor got around to it. With MCP, a system exposes one server and every conforming harness can use it. The database vendor writes one server. The ticketing system writes one server. Your internal platform team writes one server for your proprietary services. When you switch harnesses, the connections come with you, because the connections were never the harness's property.

This is where my employer shows up as a worked example, so let me flag it and move on. Dremio ships an MCP Server that lets agents query governed data through the platform's semantic layer, which means an agent in any MCP-capable harness can run SQL against approved datasets with the same access controls a human analyst gets. I am not going to argue that is the right architecture for you. The point that generalizes is that the data platform exposes capability once, through a protocol it does not control, and every harness benefits equally. That is what an open standard buys both sides of the connection.

MCP's limits are worth naming because people ask it to do jobs it was never designed for. It says nothing about which tools an agent should be allowed to use, only how to call them. Authorization lives in the harness. It says nothing about what an agent is, how work decomposes, or how a human approves a dangerous action. It is a reach protocol. Treating it as the whole standards story, which a lot of 2025-era architecture diagrams did, leaves the other five questions unanswered.

The operational caution with MCP is the security surface. Every server you connect is code that feeds content into your agent's context, and content is exactly the channel prompt injection travels through. A malicious or compromised server can return tool results crafted to steer the model. The mitigations are the boring ones: treat servers like dependencies, pin and review them, run them with the least access they need, and keep dangerous capabilities behind approval gates. That last mitigation is a preview of why AAIS exists, and we will get there.

Agent Skills: What an Agent Knows How to Do

Agent Skills is the standard with the most surprising adoption story, and the structure of the spec explains why. A skill is a folder. Inside the folder is a file named SKILL.md with YAML frontmatter carrying two required fields, a name and a description, followed by a Markdown body of instructions. The folder can also carry scripts, reference documents, and templates the instructions point to. That is the whole format.

The runtime behavior is progressive disclosure. The harness loads only each skill's name and description at startup, which costs a few dozen tokens per skill. When a task matches a description, the harness loads the full body, and the agent reads any bundled files only as needed. The design lets an agent carry a large library of procedures without paying the context cost of all of them on every request.

Anthropic shipped skills as a Claude feature in October 2025 and published the format as an open specification at agentskills.io on December 18, 2025. What happened next is the part worth studying. Microsoft added support in VS Code within days. OpenAI adopted it in ChatGPT and the Codex CLI. By mid 2026 the official showcase lists roughly 40 products reading the same format, including Gemini CLI, GitHub Copilot, Cursor, JetBrains Junie, Goose, OpenCode, and offerings from Databricks and Snowflake. Directly competing vendors adopted a competitor's format in weeks, which almost never happens, and it happened because the spec is small enough to implement in an afternoon and the value of a shared skills library is obvious to everyone's customers.

For the portability argument, skills solve the procedural knowledge problem. The release checklist, the incident triage protocol, the way your team writes migration scripts: before skills, that knowledge lived in tool-specific configuration files, a .cursorrules here, a CLAUDE.md there, none of it portable. A skill written to the spec moves between every conforming product unchanged. I use this daily in my own content work. The skills that produce my newsletters and articles are folders in version control, and nothing about them belongs to any one harness.

Two honest cautions. First, quality varies enormously in the public skill ecosystem. Community directories now index skills by the hundreds of thousands, and a February 2026 security audit that scanned 3,984 public skills found 36 percent carried at least one security flaw, including prompt injection payloads. A skill is instructions your agent will follow and sometimes scripts it will execute. Review community skills the way you review an open-source dependency, because that is exactly what they are. Second, a skill is not a capability grant. It tells the agent how to do something, not whether it is permitted to. If your permission model lives inside skill text, you do not have a permission model. You have a suggestion.

Agent2Agent: How Agents Talk to Each Other

MCP connects an agent to tools. A2A connects an agent to other agents, and the distinction is easy to state: a tool is a passive capability you invoke, while a peer agent is an actor with its own reasoning, its own tools, and its own opinion about how to accomplish a task. Delegating to a peer is a different problem from calling a function, and A2A is the protocol built for it.

Google announced A2A in April 2025 and donated the specification, SDKs, and tooling to the Linux Foundation that June, where an independent project now governs it with backing from AWS, Cisco, Microsoft, Salesforce, SAP, ServiceNow, and others. By its first anniversary the project reported more than 150 supporting organizations and integrations across the major cloud agent platforms, with SDKs in Python, JavaScript, Java, Go, and .NET.

The mechanics center on two ideas. The first is discovery through Agent Cards. An A2A server publishes a JSON document at a well-known path describing what the agent can do, what skills it advertises, which transports it speaks, and what security it requires. A client agent reads the card and knows whether this peer can handle the task at hand. The second idea is the task lifecycle. A2A models delegated work as a task object that moves through explicit states: submitted, working, input required, auth required, and terminal states for completed, failed, canceled, and rejected. Long-running tasks stream status over server-sent events or push notifications, and the lifecycle survives disconnects.

The task lifecycle is the design decision that separates A2A from a fancy REST wrapper. Agent-to-agent delegation is slow, stateful, and frequently interactive. The peer agent works for minutes or hours, sometimes needs more input, sometimes needs the delegating side to authenticate, and sometimes fails halfway. Modeling all of that as first-class protocol state means both sides agree on where a piece of work stands without inventing a convention per integration.

Where does A2A fit next to the other five? It is the horizontal protocol in a stack of mostly vertical ones. MCP runs between an agent and its tools. Skills, profiles, and graphs are documents a single harness consumes. A2A runs between organizations, or between departments, wherever the two sides of a delegation do not share a runtime. That also defines its limits. Inside a single harness, spinning up A2A between your own subagents adds protocol overhead where a function call did fine. The fair criticism of A2A's first year was exactly that: enthusiastic architectures used it where simpler mechanisms served, and the protocol earned some skepticism it did not deserve on the merits. Use it at trust boundaries. Skip it inside them.

Open Agent Profile: Who the Agent Is, and What It Has Learned

Now we reach the three specifications I authored, starting with the one that addresses the gap I felt most personally. Here is the problem in one paragraph. You spend months refining an agent: a code reviewer that knows your conventions, a data engineer that has learned your table layouts, a researcher that cites the way you want. That definition and everything it learned lives in one product, in that product's shape, and often only for the length of a session. The agent, as an artifact you own, does not exist.

The Open Agent Profile makes it exist by persisting the agent as a file. A profile is a YAML or JSON document with three top-level parts, and the boundary between them carries the whole design. Metadata holds the name, description, and a revision number. Spec holds the contract: role instructions, the model selection, the tool policy, permissions, and lifecycle settings. This is the part a human writes and approves. State holds what sessions learned: a summary, discrete facts with confidence and provenance, and open threads with status. This is the part sessions write. A harness reads the file, runs a fresh session, and writes an updated revision back when the session ends. Nothing stays resident. The file is the agent.

A portable file describing what an agent is permitted to do is a security problem before it is a convenience, and the spec's answer is three rules that hold under every configuration.

First, a profile narrows and never widens. A harness grants the intersection of what the profile requests and what its own policy already allows. There is no field or trust marker that reverses this, which means accepting a profile from a stranger is safe. The worst case is an agent with fewer capabilities than you already permit. Without this rule, portable agent files become an escalation mechanism: run a file from somewhere and receive whatever authority it claims.

Second, an agent cannot rewrite its own contract. Sessions emit a structured delta at the end, and delta operations only touch the state section. A change to tools, permissions, model, or instructions goes into a proposals block with a written rationale and waits for a human. This holds even under fully automatic writeback. A boundary that configuration can relax is not a boundary, just a default.

Third, learned state is untrusted content. Text an agent wrote about itself gets injected into future sessions as information, never as authority. A state entry claiming shell access no longer needs approval changes nothing. This rule closes the nastiest failure in persistent agents: without it, one successful prompt injection becomes permanent, because the attacker convinces the agent once and the agent writes the instruction into its own memory. Treating state as data keeps a one-time injection one-time.

The proposals mechanism deserves a paragraph because it solves the problem that kills least-privilege in practice. Narrow permissions fail socially, not technically: legitimate work gets blocked, friction builds, and someone widens the grant to stop the complaints. A proposal turns that pressure into evidence. When a session hits a wall, it records the specific change it needs and a rationale explaining what it was unable to do. A reviewer reads a request for shell access attached to an explanation that the agent was unable to verify a flaky test claim without running the suite, and makes an actual engineering decision. The mechanism produces the artifact a reviewer needs, at the moment the need is fresh.

The spec sits at version 1.0 with support libraries at 1.0.5 in Python, TypeScript, Go, Rust, and Java, all Apache licensed and tested against a shared conformance corpus that includes negative fixtures a correct implementation must reject. Profiles get canonical digests, so the exact content that was approved is verifiable regardless of encoding or field order. Three conformance levels let a harness be honest about partial support, from read-only instantiation up through full state persistence and composition, and an implementation is required to publish what it does not implement. Silent degradation is the failure that kills trust in portable formats: someone reviews a profile, runs it elsewhere, and gets a different agent than the one they read. I implemented OAP across my own harnesses, Loro and MagAgent, and in the Merced AI broker, so the spec has running code behind it, and I will be plain that adoption beyond that is early. The mitigating factor for you is that the artifact is declarative text describing your agents. If a different profile standard wins, translating files is a small job next to reconstructing agent definitions from a product UI.

Agentic Graph Specification: The Shape of the Work

Every serious harness already decomposes big jobs into steps. It does so internally, in its own shape, and the plan evaporates when the session ends. AGS makes the decomposition a document, and four familiar frustrations fall out of that one change.

You cannot review a plan you never see, so a wrong decomposition is discovered after the tokens are spent. You cannot move a plan trapped in one harness's memory, so the planning work is discarded at the session boundary. Without declared acceptance criteria, done is whatever the model says, and self-reported completion accumulates silent failures. And without a declared capability demand per step, every step gets the same model, which sends trivial work to expensive models and hard decisions to cheap ones.

An Agentic Graph is a directed acyclic graph where each node is a bounded agentic loop, one unit of work an agent runs end to end, and each edge is a control-flow dependency. The specification is implementation neutral, written in YAML or JSON with the two encodings equivalent, at version 1.0 under Apache 2.0 with libraries at 1.0.4.

The node is where the format earns its opinionated reputation. Each node declares a brief written to stand alone, so an agent that has seen nothing else can act on it. Typed inputs and outputs, so the harness checks that a node produced something of the right shape instead of trusting a claim. Success conditions, machine-checkable where possible and always human-readable, evaluated by the harness rather than asserted by the model. A normalized capability tier instead of a model name, so the graph stays valid when models are deprecated and portable to harnesses configured with different providers. Required tools, permissions, and budgets, declared per node. And failure handling, chosen from retry with feedback, fallback to an alternative approach, escalation to a stronger tier or different node, and human checkpoint.

Two structural elements lift this above a task list. Decision nodes branch on an outcome, ready or needs work, which lets a graph express remediation without becoming a cycle: the fix-it path rejoins downstream rather than looping back. Gates hold for an explicit human decision, and placing a gate immediately before the first irreversible action or the first expensive fan-out is the single highest-value structural choice in any graph.

The success-conditions rule carries the most weight, so let me defend it directly. A model asked whether it finished will usually say yes, not from dishonesty but because grading your own work against a criterion you also interpreted is unreliable. Systems built on self-reported completion rot quietly: a half-working step is reported done and the next step builds on it. Moving evaluation into the harness turns completion into a check. A condition stating that the test suite passes gets run. A condition that is only human-readable at least tells a reviewer what to look at, and an unchecked criterion still beats an unstated one.

A validated graph is also useful before anything runs. Planning tools derive execution order and parallelism, flag unreachable nodes, compute worst-case cost bounds when every retry path fires, summarize how much of the work demands an expensive tier, report which features this environment does not support, and produce a stable digest that ties a review to exact content. Knowing the worst-case bound before spending it is the difference between a budget and a hope.

The honest boundary: graphs cost structure, and structure applied everywhere makes an idea useless. Release processes, migrations, incident response, and multi-stage builds have real shape worth reviewing. Exploratory work does not. A question with unknown shape cannot be decomposed in advance, and forcing it into nodes produces a document that is wrong by step two. Explicit structure removes the room an agent has to improvise, which is precisely the point in high-consequence work and precisely the loss everywhere else.

AAIS: How a Human Says Yes

The last of the six covers the smallest surface and, in production, one of the most consequential. Every harness eventually needs to pause and ask a person: the agent wants to run this command, send this email, drop this table. Approve or deny?

Today that handoff is almost always a terminal prompt blocking on standard input, which fails in every direction that matters at scale. The person is not at the terminal, they are on their phone. The process restarts and the pending question is gone. The approval UI is welded to one harness, so an organization running three harnesses builds three approval experiences. And the record of what was approved, if it exists at all, is a line in a log.

The Agent Approval Interchange Specification makes the approval itself a portable, durable protocol. AAIS 1.0 is a transport-neutral contract for one handoff: a runtime needs permission for an action, and a person decides from whatever trusted interface they are actually using, a CLI, a web page, a desktop app, or an automated policy service. It covers chats, subagents, background jobs, and graph nodes without defining any of those runtimes. Messages travel over whatever you have: MCP, HTTP with server-sent events, WebSocket, or stdio.

The design holds one line firmly: the harness stays the authority. A client presents the exact requested action and returns a selected decision. It cannot grant itself capability. Before acting, the harness revalidates the decision against current policy, the action digest, expiry, and the choices it originally offered. Four properties make the loop safe. Decisions bind to a canonical digest of the exact action reviewed, computed under RFC 8785 canonicalization, so what was approved is what runs, byte for byte in meaning. Choices are bounded, so a client only selects among scopes the harness offered. The lifecycle fails closed: expired, stale, conflicting, malformed, and replayed decisions are rejected. And requests carry provenance while retries stay idempotent, so the audit trail records who asked, for what, and what was decided.

Durability is the operational feature people feel first. A pending approval is application state, not a blocked process. Ordered events and snapshots let a browser or desktop client reconnect and recover outstanding decisions, including ones raised hours ago by a long-running graph node. The approval you did not answer at your desk is waiting on your phone.

AAIS ships as a 1.0 protocol with 0.1.0 support libraries in Python, TypeScript, Go, Rust, and Java, published to the standard registries and verified against shared fixtures so a message created in one language validates in another. It deliberately excludes chat, model reasoning, tools, and authentication, and it carries concise activity, risk, choices, decisions, and receipts rather than private chain-of-thought. Same disclosure as before: I wrote it, it is young, and the questions it answers stop being optional the moment agents act on systems that matter.

How the Six Compose

The composition story is where the stack stops being a list of acronyms and becomes an architecture, so let me trace one delegation end to end.

A profile defines your data engineer agent: its instructions, its permitted tools, its ceiling of authority, and everything past sessions taught it. A graph defines this week's migration: twelve nodes, typed handoffs, per-node budgets, a gate before the schema change. The harness loads both and grants each node the intersection of what the profile allows and what the node declares it needs, which yields per-step authority narrower than either document alone. Skills supply the procedures nodes follow, the migration checklist and the validation routine, loaded on demand. MCP supplies reach, connecting the agent to the warehouse, the catalog, and the ticketing system through servers those platforms publish. When node seven hits the gate, the harness emits an AAIS request, you approve the exact schema change from your phone an hour later, and the harness revalidates the decision before executing. When one node's brief calls for a legal review your organization delegates to another department's agent, the harness discovers that peer through its A2A card and hands off a task with a real lifecycle instead of a fire-and-forget API call.

Notice what the harness became in that story: an engine. Every artifact it consumed, the profile, the graph, the skills, the tool connections, the approval flow, and the delegation protocol, is a document or contract that outlives it. Swap the engine and the work moves. That is the whole thesis, and it is the same thesis open table formats proved in data: when the durable artifacts are specifications rather than internals, the runtime becomes a choice you revisit instead of a decision you married.

A Worked Example on Disk

Abstractions earn trust when you see the files, so here is a trimmed but real-syntax pair: an OAP profile and an AGS graph fragment that references it.

# reviewer.oap.yaml
oap_version: "1.0"
metadata:
  name: code-reviewer
  description: Reviews pull requests against team conventions
  revision: 14
spec:
  role: |
    You review pull requests for correctness, style, and risk.
    Flag anything touching auth or billing for human review.
  model:
    provider: anthropic
    id: claude-opus-5
    tier: frontier          # portable fallback when the id is unavailable
  tools:
    mode: allowlist
    allow: [git.read, files.read, tests.run]
  lifecycle:
    writeback: propose      # state deltas apply, contract changes wait
state:
  summary: Reviews Go and SQL. Team prefers table-driven tests.
  facts:
    - text: Migrations live in /db/migrations, numbered.
      confidence: high
      source: session-2026-08-12
      pinned: true
  threads:
    - title: Flaky auth test on CI
      status: open
proposals:
  - change: add tool tests.run_integration
    rationale: Unable to verify flaky-test claims from unit suite alone.
    status: pending

Read the file the way a reviewer does. The spec block is the contract: an allowlist of three read-mostly tools plus test execution, a named model with a portable tier fallback, and writeback set to propose. The state block is what fourteen revisions of sessions accumulated, each fact carrying confidence and provenance so stale entries can be pruned, with one fact pinned to survive summarization. The proposals block shows the mechanism working: the agent hit a wall, documented it, and the request waits for a human. Nothing in state or proposals changed the contract.

# release-check.ags.yaml (fragment)
ags_version: "1.0"
nodes:
  - id: review
    brief: >
      Review the diff in inputs.diff against team conventions.
      Produce findings as structured JSON.
    agent_profile: code-reviewer      # binds the OAP profile above
    inputs:  { diff: {type: fileset} }
    outputs: { findings: {type: json} }
    intelligence: { tier: standard }
    success:
      - check: outputs.findings validates against findings.schema.json
    on_failure:
      retry: { max: 2, feed_failure: true }
      then: escalate
  - id: gate-merge
    kind: gate
    brief: Human approves merge based on review findings.
edges:
  - from: review
    to: gate-merge

The review node runs at a standard tier because review does not need frontier capability, its success condition is a schema validation the harness executes, and its failure handling retries twice with the failure fed back before escalating. The gate holds for a person, and in a harness that speaks AAIS, that gate arrives on whatever device the approver is carrying. The two files together express who works, on what, with which authority, and where a human stands in the path, and neither file names the harness that will run them.

Failure Modes and What Breaks

Standards do not remove failure. They move it somewhere visible, and knowing where to look is most of the operational skill.

Silent partial support. The failure that destroys trust in portable formats is a runtime that accepts a document and quietly ignores half of it. A harness that reads a profile at Level 1 does not persist state, which changes what the profile is for. A runtime that ignores a tool denylist turns a control into a description. Check the conformance statement of anything you depend on, and prefer implementations that publish their gaps over ones that look complete.

Injection through every content channel. MCP tool results, skill bodies, and profile state are all text that reaches the model, and all three have carried real attacks. The 36 percent flaw rate in that audit of public skills is the number to keep in mind when someone proposes installing community skills wholesale. The defenses stack: review skills like dependencies, pin MCP servers, treat profile state as untrusted by rule, and keep irreversible actions behind AAIS-style gates so injected intent still meets a human.

Stale documents. Profiles accumulate facts that stop being true. Graphs reference tools that got renamed. A confident agent running on stale declarations is worse than an ignorant one, because it acts. Prune profile state using the confidence and provenance fields, and validate graphs in continuous integration like any other artifact.

Over-decomposition. Twenty graph nodes where four serve produces coordination overhead and context loss at every boundary. A node is a unit of work an agent completes, not a single action. The matching mistake with skills is the mega-skill, a body so long the progressive-disclosure economics invert. Small, sharp, and few beats large and many in both formats.

Standards where they do not belong. A2A between your own subagents, graphs wrapped around exploratory questions, profiles stuffed with domain knowledge that belongs in a knowledge store: each is a real pattern I have seen proposed, and each adds ceremony without adding portability. The test is always the artifact: if nothing durable needs to move across a boundary, you do not need the interchange format at that boundary.

Budget surprises. Failure handling multiplies cost. A node with three retries, a fallback tier, and an escalation path is cheap on the happy path and expensive in the worst case. Plan against the worst-case bound the graph tooling computes, and let an alarming bound prompt the better question: does this node fail because the brief is unclear?

Where This Is Heading

Reading the direction of travel is easier if you accept one premise: the agentic stack is recapitulating the data stack's history at roughly five times the speed. Formats standardize first, then catalogs and governance, then the engines commoditize. Skills standardized in weeks. MCP took about a year to become assumed infrastructure. A2A found its footing at trust boundaries after a year of being tried everywhere.

The unresolved layer is exactly the one OAP, AGS, and AAIS aim at: identity, work shape, and authority. Whether those particular documents win is the least interesting question. Watch instead for three signals. First, whether the major harness vendors expose import and export for agent definitions at all, because a vendor that will not let an agent leave has told you its answer on portability. Second, whether the institutional homes, the Agentic AI Foundation and the A2A project, expand scope to cover identity and approvals, which is the natural place for consolidation. Third, whether enterprises start requiring reviewable, digest-identified plans and approval receipts for agent actions in regulated workflows, because compliance demand is what turned data governance from a slideware topic into a purchase requirement, and the same forcing function is already visible for agents.

My own bet is on the pattern, not any single spec: durable artifacts as open documents, harnesses as replaceable engines, humans holding explicit gates. Every layer of infrastructure I have worked on eventually arrived at that shape. The ones that arrived early spared their users years of reconstruction work.

Conclusion

Six specifications, six questions. MCP answers what an agent can reach, and it is settled enough to build on without hesitation. Agent Skills answers what an agent knows how to do, and its cross-vendor adoption made procedural knowledge the first truly portable agentic artifact. A2A answers how agents cooperate across trust boundaries, with a task lifecycle built for slow, stateful, interruptible delegation. OAP answers who the agent is and what it has learned, with narrowing, contract protection, and untrusted state as its safety spine. AGS answers what shape the work takes, turning plans into reviewable, priceable, movable documents with harness-checked completion. AAIS answers how a human authorizes the moment that matters, durably, from any trusted surface.

Adopt them in the order your risk dictates. Tool connections and skills first, because the standards are mature and the wins are immediate. Then write one profile for your most capable agent, because writing down its authority surfaces at least one grant nobody defends. Then graph one process where a wrong plan is expensive. Gate the irreversible steps. At each stage, the test stays the same: when you imagine switching harnesses next year, what moves with you, and what do you rebuild? Every artifact in the second pile is a decision you have not finished making.

Keep Going

If this piece was useful, I have written a lot more on agentic architecture and the data foundations beneath it. Hands-On Agentic Engineering covers building multi-agent systems in practice, from harnesses and tool protocols to governance. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

The Great Despecialization: Why AI Changes the Shape of Jobs Instead of Deleting Them

Alex Merced — Thu, 27 Aug 2026 16:18:09 +0000

I run a handful of personal websites. A book catalog, a blog, a few project sites. Not long ago, keeping those running the way I wanted meant one of two things. Either I did everything myself, badly and slowly, or I assembled a small team: a web developer for the layout and build, a copy editor for the writing, a graphic designer for the covers and banners. Three specialists, three sets of handoffs, and a lot of waiting on other people for a site that earns nothing.

Today I do all three jobs myself with AI assisting on each one. The layout gets scaffolded by a model, the copy gets a first editing pass from a model, the graphics get generated and adjusted with a model. None of that works unless I know enough about web development, editing, and design to describe what I want and to recognize when the output is wrong. The specialists did not get replaced by software. Their procedural work got absorbed into a wider version of my job, and what the job now requires of me is judgment across three areas instead of skill in one.

That is the real subject of this article. The conversation about artificial intelligence and work keeps asking one question: how many jobs will AI eliminate? I think that is the wrong question, and the wrong question is producing wrong answers. My argument is that we are entering a period I call the great despecialization. For roughly two centuries, productivity gains came from splitting work into narrower roles. AI reverses the incentive. When one person with the right tools can carry a piece of work across boundaries that used to require handoffs, the economics favor breadth over depth. Jobs do not vanish. They widen. And if my websites ever grow to the point where I need a team again, that team will not be specialists. It will be generalists, each owning additional end-to-end workflows toward the same goal.

I work at Dremio and spend a lot of time around data engineers, so data teams show up in the examples below. The pattern applies well beyond them.

The Substitution Story Gets the Unit of Analysis Wrong

Most AI job predictions start from a list of occupations and score each one for how automatable it looks. The method feels rigorous. It produces big scary numbers. It also measures the wrong thing.

A job is not a single activity. A job is a bundle of tasks that some organization decided to hand to one person. A data engineer writes ingestion code, debugs failed runs, sits in requirements meetings, negotiates with a source system owner, documents schemas, answers Slack questions from analysts, and estimates timelines. AI is very good at two or three of those tasks and mediocre at the rest. Scoring the job as a whole hides that variation.

The newer research has started to catch this. The 2026 PwC Global AI Jobs Barometer looked at more than one billion job advertisements across 27 countries and found a two-track market. Roles where AI automates routine tasks so that human judgment gets more emphasis are growing faster than roles that AI has made easy enough for non-experts to perform. Read that carefully. The roles growing fastest are the ones where AI removed the routine parts and left a human holding a wider set of responsibilities.

The layoff data tells a similar story. Of the roughly 1.2 million US layoffs announced in 2025, only about 4.5 percent explicitly cited AI according to Challenger, Gray and Christmas. S&P Global's purchasing managers survey put the global net employment effect of AI adoption at negative 5 points over the past year, a modest number, with process efficiency and productivity cited as the goal far more often than headcount reduction. These are real effects, but they are not the wholesale deletion the substitution story predicts.

The place where displacement is clearest is the entry level. Stanford's Digital Economy Lab measured about a 13 percent relative employment decline for 22 to 25 year olds in the most AI-exposed occupations. I will come back to that number, because it is the strongest evidence against my thesis and it deserves a direct answer. For now the point is simpler. When you measure tasks instead of occupations, the picture is not "jobs disappear." The picture is "jobs get rebundled."

What Specialization Was Actually For

To understand why AI rebundles work, you have to understand why we unbundled it in the first place.

Adam Smith opened The Wealth of Nations with a pin factory. One worker doing every step of pin-making produced maybe twenty pins a day. Ten workers, each doing one step, produced forty-eight thousand. The gain came from three sources. Each worker got better at a narrow task through repetition. Nobody lost time switching between tools and tasks. And narrow tasks were easier to turn into machines.

Every knowledge-work org chart from the last fifty years is a pin factory with laptops. We split "get data to the people who need it" into source system owner, ingestion engineer, warehouse engineer, analytics engineer, BI developer, data analyst, and data steward. Each role exists because the skill it requires takes years to build, because switching between those skills is expensive, and because narrow roles are easier to hire for and measure.

Specialization has a cost that the pin factory story leaves out. Ronald Coase won a Nobel Prize partly for pointing out that coordination is not free. Every boundary between two specialists is a handoff. Every handoff needs a ticket, a meeting, a shared definition of done, a translation between two vocabularies. The waiting I described between a developer, an editor, and a designer is pure coordination cost. No pins get made while three people wait on each other.

Organizations tolerate coordination cost because the alternative was worse. One person cannot hold enough expertise to do all seven of those data jobs well. The human brain and the human calendar have limits. So we accepted the handoffs, hired project managers to grease them, and built tooling like Jira to track them. The whole apparatus of the modern knowledge-work company is a machine for managing the cost of specialization.

That is the key insight for what comes next. Specialization was never the goal. It was a workaround for the fact that expertise is expensive to acquire and slow to switch between. Change those two constraints and the workaround stops paying for itself.

Why AI Attacks the Reason for Specialization, Not the Specialist

Here is what a large language model actually does when it helps a working professional. It lowers the cost of acquiring just enough expertise to do a task, and it lowers the cost of switching between tasks. Those are exactly the two constraints that specialization existed to route around.

Consider the switching cost first. A data engineer who needs to write a Terraform module for a new bucket used to face a choice. Spend two hours relearning HCL syntax and the provider's quirks, or file a ticket for the platform team and wait three days. In 2026 that engineer describes the bucket, gets a working module in under a minute, reads it, adjusts the lifecycle policy, and moves on. The switching cost dropped from hours to minutes. The ticket, and the handoff it represents, no longer makes sense.

Now consider the acquisition cost. Expertise has two layers. There is the layer of knowing how to do a thing, which is mostly recall of syntax, procedures, and conventions. And there is the layer of knowing what to do and why, which is judgment built from seeing things go wrong. AI has commoditized the first layer almost completely. It has barely touched the second. A model will write you a correct window function. It will not tell you that your business partner's definition of "active customer" has changed twice this year and the dashboard is quietly wrong.

This split matters because it tells you which half of every specialist role gets absorbed. The recall half. The procedural half. The half that took the longest to learn and contributed the least judgment. What remains is the judgment half, and judgment is portable across domains in a way that syntax never was.

A person with good judgment about data quality, plus an AI that handles the procedural work of five adjacent roles, can now cover ground that used to need five people. Not because the person got five times smarter, but because the five roles were mostly procedural work stacked on top of a thin layer of judgment each. Collapse the procedural work and the judgment layers stack up into one job.

That is the mechanism. AI is not competing with the specialist for the specialist's job. AI is dissolving the boundaries that made the specialist's job a separate job at all.

The Great Despecialization, Defined

Let me state the thesis precisely so it can be argued with.

The great despecialization is the shift in the economics of knowledge work from rewarding depth in one task to rewarding breadth across many tasks, driven by AI reducing the cost of switching between tasks and the cost of acquiring procedural competence in a new one. It changes the shape of jobs before it changes the count of jobs.

Three predictions follow from that definition, and each one is testable.

First, job descriptions get wider. The average posting asks for more distinct skill areas than it did five years ago, and the premium for "AI fluency" shows up as a demand for people who can apply AI across those areas rather than in one. Stanford's AI Index and Lightcast data already show AI skills appearing in about 2.5 percent of all US job postings, up 55 percent year over year, and mentions of agentic AI skills grew more than 280 percent in a single year. Those mentions are not asking for machine learning researchers. They are asking for accountants and marketers and engineers who can direct AI tools.

Second, team sizes shrink while team scope grows. The eleven-person data team becomes a four-person data team that owns more of the value chain, not less. Headcount per unit of output drops. Total output rises. Whether total employment drops depends on whether demand for output grows, which is the same question every previous productivity wave faced.

Third, the skills premium moves from "can you do X" to "can you tell whether X was done correctly." Review, verification, and judgment become the scarce inputs. This flips the traditional career ladder, where you spent years doing the thing before you were trusted to review the thing. I will get to why that flip is painful.

If you want a historical analogy, do not reach for the Luddites. Reach for the spreadsheet. VisiCalc and then Lotus 1-2-3 did not eliminate accountants. They eliminated the bookkeeping clerks who did arithmetic, and they turned every manager into a person who does financial modeling as one task among many. The number of people doing financial analysis went up. The number of people whose whole job was financial arithmetic went to zero. The job of "manager" got wider. That is despecialization, and it happened forty years ago.

The bank teller is a second example worth keeping in mind. Automated teller machines arrived in the 1970s and everyone expected teller employment to collapse. Instead the number of tellers in the United States rose for three decades, because cheaper branches meant more branches, and the teller's job shifted from counting cash to selling accounts and handling exceptions. The procedural core of the role was automated away and the role got wider. It took decades for teller headcount to finally decline, and when it did, the cause was online banking removing the branch itself rather than the machine inside it. Despecialization came first. Elimination, where it happened at all, came a generation later through a different mechanism.

What It Looks Like Inside a Data Team

Abstract arguments about labor economics are easy to nod along with and hard to act on. So let me walk through a hypothetical data team, since that is the kind of team I talk to most often.

A typical mid-sized data organization in 2022 looked something like this. Two platform engineers ran the Kubernetes clusters and the object storage. Three data engineers wrote Spark or Airflow pipelines. Two analytics engineers built dbt models. One database administrator (DBA) tuned the warehouse. Three analysts wrote SQL and built dashboards. One data steward maintained the catalog and lineage. That is twelve people, six distinct specialties, and at least five handoff boundaries between raw data and a chart a VP looks at.

Now trace a single request through that org. Marketing wants churn by acquisition channel. The analyst files a ticket because the channel field is not in the model. The analytics engineer discovers the field is not in the warehouse either. The data engineer finds the source system exposes it but the ingestion job drops it. The DBA warns the new column will blow up a partition scheme. Three weeks and four tickets later, marketing gets a chart. Every person involved did their job correctly. The system produced a three-week latency out of correct individual behavior.

Here is the same request in a despecialized team of five, each running AI agents against the platform. The analyst, who is now something closer to a "data generalist," opens an agent session connected to the catalog through an MCP (Model Context Protocol) server. MCP is an open standard that lets an AI agent discover and call tools, so the agent can inspect table metadata, run queries, and read lineage without a human copying things between windows. The agent confirms the field exists in the source, drafts the ingestion change, proposes a dbt model update, runs the query against a branch, and flags the partition concern. The generalist reviews each step, rejects the partition change in favor of a different approach, and merges. Two days, one person, zero tickets.

This is the workflow that tools like Dremio's MCP Server against an Open Catalog powered by Apache Polaris are built for, and other stacks support the same pattern. The vendor matters less than the shape: a catalog with rich metadata, an agent that can read it, and a human whose job is to direct and verify rather than to execute each step by hand.

Notice what did not happen. Nobody got fired in that story. The twelve-person team did not become a five-person team through layoffs. It became a five-person team because the next three people who left were not backfilled, and the work absorbed into wider roles. That is how despecialization actually arrives in most organizations: through attrition and scope creep, not pink slips.

Notice also what the five remaining people need to know. Each of them touches ingestion, modeling, query tuning, and governance in a single week. None of them are the deepest expert in any of those. All of them need enough judgment in each to catch an agent's mistakes. The DBA's knowledge did not disappear. It got spread thin across five people and one model.

The table below shows the shift in what each role spends time on. The percentages are illustrative of the pattern, not a survey result.

Activity	2022 specialist team	2026 despecialized team
Writing code, SQL, and config by hand	45%	15%
Waiting on or coordinating handoffs	25%	5%
Reviewing and verifying work (own or AI's)	10%	35%
Talking to business stakeholders	10%	25%
Learning adjacent skills	5%	15%
Meetings about who owns what	5%	5%

The bottom row is a joke, but only partly. Ownership fights do not go away. They change from "whose ticket is this" to "who is accountable when the agent gets it wrong."

It Is Not Only Data Teams

Data teams are the example I reach for because of where I work, but the same collapse is happening wherever a value stream got sliced into specialist roles.

My own websites are the smallest possible case. Developer, editor, designer: three roles that existed because each skill took years to build. AI compressed the procedural half of all three into tools I direct, and the judgment half of all three into one person. If the sites ever needed a second person, that person is not a specialist designer. That person is another generalist who owns a new end-to-end workflow, say a newsletter or a course pipeline, from draft to publish, and who can step into mine when needed.

Take a marketing organization. In 2022 a campaign passed through a strategist, a copywriter, a designer, a web developer who built the landing page, an email specialist who set up the sequence, and an analyst who reported on it. Six roles, five handoffs, a two-week cycle for a single campaign. In 2026 a "growth marketer" drafts copy with a model, generates and adjusts layout with a design tool, ships the landing page from a template an agent modifies, configures the email flow, and reads the results out of an agent connected to the analytics warehouse. The strategist and the analyst are often the same person. The cycle is two days. The designer still exists, but as one senior person reviewing output across a dozen campaigns rather than producing one at a time.

Take a small software company. The old shape had frontend engineers, backend engineers, a DevOps engineer, a QA engineer, and a technical writer. The new shape has "product engineers" who own a feature from database migration to documentation, with agents writing the tests and the docs and a senior engineer reviewing the architecture. The QA role did not vanish because testing stopped mattering. It vanished because testing became a task every engineer directs an agent to do, and the judgment about what to test moved into the engineer's head.

Take finance. A financial planning and analysis (FP&A) team used to have people who built models, people who pulled data, people who made decks, and people who presented. The person who presents now builds the model with an agent, pulls the data through a connector, and generates the deck. The three procedural roles compressed into one judgment role.

The pattern is identical in each case. Find the value stream. Count the handoffs. Each handoff existed because switching skills was expensive. Remove that expense and the handoffs collapse into the person closest to the outcome. That person's job gets wider, the people whose whole role was a handoff get absorbed or not backfilled, and the total number of people producing the outcome drops while the outcome's cycle time drops faster.

What changes across industries is how thick the judgment layer is at each step. Marketing has a thin one at the procedural level and a thick one at the strategic level, so it despecializes fast. Finance has regulatory sign-off at the end, so the last step stays narrow. Software has a deep specialist layer in infrastructure and security that resists. The direction is the same everywhere. The speed and the stopping point differ.

A Task Inventory You Can Run on Your Own Role

The most useful exercise I know for thinking about this is a task inventory. List everything you do in a typical month. For each task, estimate two things: how much of it is procedural (recall, syntax, following a known sequence) versus judgment (deciding what should happen and whether it did), and how much of it exists only because of a handoff to or from another specialist.

Below is a small Python script that does the arithmetic. It takes a list of tasks with rough weights and produces two numbers: how much of your current job is exposed to procedural automation, and how much is coordination overhead that disappears if the boundary around you dissolves.

from dataclasses import dataclass

@dataclass
class Task:
    name: str
    hours_per_month: float
    procedural_share: float   # 0.0 to 1.0, fraction that is recall/syntax
    handoff_driven: bool      # exists mainly because of a role boundary

tasks = [
    Task("Write ingestion jobs",            30, 0.70, False),
    Task("Debug failed pipeline runs",      20, 0.40, False),
    Task("Answer analyst schema questions", 15, 0.30, True),
    Task("Write tickets for platform team", 10, 0.80, True),
    Task("Requirements meetings",           12, 0.10, False),
    Task("Document schemas in catalog",     8,  0.60, True),
    Task("Estimate timelines",              5,  0.20, False),
]

total = sum(t.hours_per_month for t in tasks)
procedural = sum(t.hours_per_month * t.procedural_share for t in tasks)
handoff = sum(t.hours_per_month for t in tasks if t.handoff_driven)
judgment = total - procedural

print(f"Total hours:              {total:.0f}")
print(f"Procedural (AI-absorbable): {procedural:.0f}  ({procedural/total:.0%})")
print(f"Judgment (stays human):     {judgment:.0f}  ({judgment/total:.0%})")
print(f"Handoff overhead:           {handoff:.0f}  ({handoff/total:.0%})")
print(f"Hours freed for wider scope: {procedural + handoff * 0.5:.0f}")

Run it with the sample numbers and you get 100 hours total, 49 hours procedural, 51 hours judgment, and 33 hours of handoff-driven work. The last line estimates hours freed for wider scope by assuming AI absorbs the procedural work and half of the handoff overhead evaporates once you can do the adjacent task yourself.

Walk through what each part means. The procedural_share field is the honest question: when I do this task, how much of the time am I remembering how versus deciding what? Writing ingestion jobs is mostly how. Requirements meetings are almost entirely what. The handoff_driven flag asks whether the task exists because someone else owns the next step. Writing tickets for the platform team is pure handoff. If you owned the platform change, the ticket disappears.

The output is not a prediction of your job's survival. It is a map of which hours are about to become available and which hours are the reason your employer still needs a human. The engineer in the sample has roughly half their month in judgment work. That half is the seed of the wider role. The other half is what gets refilled with adjacent tasks.

Try running it against your own month. If procedural comes out above 70 percent, the honest read is that your current role is mostly a bundle of recall tasks and the bundle is going to be repackaged. If judgment comes out above 60 percent, you are already doing generalist work and the shift is going to feel like getting more tools rather than losing ground.

What Breaks: The Failure Modes of the Generalist Shift

I am making an optimistic case, so I owe you the parts that go wrong. Despecialization has real failure modes, and some of them are already visible.

The apprenticeship ladder collapses first

This is the strongest objection and the one I take most seriously. That Stanford figure, a 13 percent relative employment decline for 22 to 25 year olds in AI-exposed occupations, is the sound of the bottom rung breaking. The traditional path into expertise ran through years of procedural work. You wrote the boring SQL for three years, and while writing it you absorbed the judgment that let you review someone else's SQL in year four. AI takes the boring SQL. So where does the judgment come from?

There is no clean answer yet. The National Association of Colleges and Employers reported in spring 2026 that just over a quarter of employers say AI has reduced the need for tasks entry-level workers performed, while more than half are in active discussions about it. The generalist role is a great destination and a terrible starting point. A 23-year-old asked to direct agents across ingestion, modeling, and governance has never seen any of those go wrong and cannot tell a plausible agent output from a correct one.

Organizations that want a pipeline of future generalists have to build apprenticeship deliberately, since the work no longer provides it for free. That means pairing juniors with seniors on review work, not just execution work. It means giving juniors ownership of small end-to-end slices instead of narrow tasks. It costs money in the short term and most companies are not doing it.

The jagged frontier eats the unwary generalist

Ethan Mollick at Wharton coined the phrase "jagged frontier" for the fact that AI capability is uneven in ways that do not match human intuition. A model that writes flawless Python fails at a date calculation a child gets right. A generalist working across five domains is, by definition, not deep enough in any one of them to always know where the frontier sits.

The failure looks like this. The generalist asks the agent to add a column to an Iceberg table and update downstream models. The agent does it and reports success. What the agent did not know, and the generalist did not know to check, was that the table used a partition transform on a column that a downstream engine reads in a version-specific way. The change was syntactically correct and operationally wrong. A specialist DBA catches it on sight. A generalist finds out in production.

The mitigation is not "become a specialist in everything." It is building verification habits: test in a branch before merging to main, use catalogs and formats that make changes reversible, and treat every agent output as a pull request from a confident junior rather than a finished product. Apache Iceberg's snapshot model is a real asset here, because a bad table change is a rollback rather than a restore-from-backup.

Depth erodes when nobody is paid to maintain it

If every team despecializes, who keeps the deep knowledge alive? Somebody has to understand Parquet encoding at the byte level, or query planner internals, or the edge cases of a specific regulatory regime. Generalists consume that knowledge through AI tools. They do not produce it.

I think the honest answer is that deep specialists do not go away. They get rarer and more concentrated. They cluster in the companies that build the tools, in open source projects, and in a smaller number of very senior roles at large organizations. The specialist-to-generalist ratio in the average company drops from something like one-in-two to one-in-ten. That is a real shift in what a specialist career looks like, and it means fewer specialist jobs at typical companies even if it means more specialist jobs in aggregate at the platform layer.

Accountability does not despecialize

When an agent-directed generalist approves a change that corrupts three months of financial data, whose fault is it? The answer today is the generalist's, and that is a heavier load than the old specialist carried, because the specialist only owned one step. Wider scope means wider blast radius. Organizations that widen roles without widening the review process, the rollback tooling, and the psychological safety to say "I am not sure about this one" are setting up their generalists to fail loudly.

Coordination cost does not vanish, it moves

The pipeline meeting with eleven people goes away. In its place comes a new coordination problem: five generalists each running agents against the same catalog. Two of them change the same model in the same afternoon. The agent-to-agent conflicts are a new class of problem with immature tooling. Catalogs with branching, like the Iceberg REST catalog implementations that support it, help. So do conventions borrowed from software engineering: feature branches, required reviews, protected main. Most data teams have not adopted those conventions yet. They are about to be forced to.

Where Specialists Still Win

I do not want to overstate the case. There are places where depth beats breadth and AI does not change that.

Licensed and legally accountable roles keep their shape longest. An auditor signs an opinion. A physician signs a chart. A structural engineer stamps a drawing. The signature carries legal weight that a generalist directing an agent cannot substitute for, and regulators are not going to change that quickly. These roles will use AI heavily and stay narrow.

Roles where the frontier is the job stay specialized too. If your work is pushing the boundary of what is known in a field, whether that is query optimizer research or protein folding, AI is a tool for a specialist, not a replacement for one. The model knows what has been written. The specialist knows what has not been written yet.

Physical work is the obvious third case. Despecialization is a knowledge-work phenomenon. The electrician and the surgeon are not being asked to also do the plumbing and the anesthesia because a chatbot got good at reading manuals.

The fourth case is subtler: roles where the cost of being wrong is catastrophic and detection is slow. Security engineering is a good example. A generalist who is 90 percent as good as a specialist across five domains is a wonderful thing in most contexts. In security, the 10 percent gap is the breach. Some functions will resist despecialization purely because the organization cannot afford the tail risk.

The pattern across all four is the same. Specialization survives where the judgment layer is thick, the accountability is personal, or the error cost is extreme. It dissolves where the procedural layer was thick and the error cost is a rollback.

How to Prepare, as a Person and as a Manager

Prediction is cheap. What should you actually do?

If you are an individual contributor, stop optimizing for depth in your current role and start optimizing for judgment across adjacent ones. Concretely, that means spending time on the tasks upstream and downstream of you. If you are an analytics engineer, learn enough about ingestion to review an agent's ingestion change and enough about BI to review the dashboard that consumes your model. You are not trying to become the best at either. You are trying to be able to say "that looks wrong" with reasons.

Build a verification practice. Write down what "correct" looks like before you ask an agent to do something. Test against a branch. Keep a personal list of the mistakes agents have made in your domain, because that list is the beginning of the judgment that used to take years of procedural work to acquire. The generalists who thrive are the ones with the best error catalogs, not the best prompts.

Learn the open standards rather than the vendor interfaces. Iceberg, Parquet, Arrow, MCP, and SQL itself are the shared vocabulary that lets one person move across tools. Vendor-specific expertise was a fine specialist asset. It is a weak generalist asset, because the whole point is to move across systems without relearning each one.

If you manage a team, resist the temptation to treat despecialization as a headcount exercise. The gains come from removing handoffs, and removing handoffs requires rethinking scope, not just cutting the fourth engineer. Redraw roles around end-to-end ownership of a value stream. Give a person the churn dashboard, source to chart, with agents to do the procedural work and a review process to catch their mistakes. Then measure cycle time, not utilization.

Invest in the apprenticeship problem before it invests in you. In three years you will need senior generalists and there is no longer a natural pipeline producing them. Pair juniors on review. Rotate them through the full stack in months rather than years. Accept that they will be slower and make more mistakes than an agent, because the mistakes are the curriculum.

Fix your platform for multi-agent concurrency now. A catalog that supports branching, a table format with snapshot rollback, and a review workflow that treats agent changes like pull requests are table stakes for a team of generalists. Without them you get five people stepping on each other and blaming the tools.

Warning signs you can watch for

You do not have to wait for a reorg to see despecialization arriving in your organization. The leading indicators show up months earlier.

Ticket volume between teams drops while output stays flat or rises. That means people are doing adjacent work themselves instead of asking for it. Backfill requests stall in the budget process, not because the budget is tight but because the hiring manager cannot articulate what the narrow role does that the existing team is not already covering. Job postings from your own company start listing four or five skill areas where they used to list one. Senior people spend more of their calendar on review and less on execution, and they say so in one-on-ones. And the loudest complaints shift from "I am waiting on another team" to "I approved something I did not fully understand."

That last complaint is the one to act on immediately. It is the sound of scope widening faster than judgment, and it is fixable with review pairing and better rollback tooling. Ignore it and the next signal is an incident.

Finally, be honest with your team about what is happening. The people on it can see that the tickets are drying up and the scope is widening. Naming the shift, and describing what the wider role looks like and how they get there, does more for retention than any amount of reassurance that "AI will not replace you." They know it will not replace them. They want to know what it is turning them into.

Where This Is Heading

The World Economic Forum's Future of Jobs Report 2025 projected 170 million new jobs and 92 million displaced by 2030, a net gain of 78 million and about 22 percent structural churn. I hold that projection loosely, because every such projection has been wrong in the specifics. I hold the churn number more tightly, because churn is what despecialization looks like from the outside. Roles get deleted and recreated with wider definitions. The person often stays. The job title changes.

Three things I expect to see by the end of the decade.

Job titles stop describing tasks and start describing domains. "Analytics engineer" and "data engineer" merge into something like "data owner for marketing" or "revenue data lead." The title tells you what business outcome the person owns, not which layer of the stack they touch, because they touch all of them.

Agent orchestration becomes a general professional skill, like email or spreadsheets, rather than a job. The 280 percent growth in agentic AI skill mentions in postings is the leading edge of this. Within a few years it stops being listed because it is assumed, the way "proficient in Microsoft Office" quietly disappeared from postings once everyone was.

The productivity gains show up as smaller companies doing bigger things rather than big companies doing the same things with fewer people. S&P Global's data already shows small firms forecasting net positive employment effects from AI while large firms trend negative. Small firms use AI to expand what a small team can cover. Large firms use it to remove handoffs they no longer need. Both are despecialization. They just feel different from inside.

The bear case for my thesis is that the judgment layer turns out to be thinner than I think, and agents get good enough at judgment that the generalist directing them becomes unnecessary too. I do not dismiss that. I think the timeline is longer than the loud voices suggest, because judgment in a real organization is inseparable from context, relationships, and accountability that models do not hold. But if I am wrong about that, I am wrong about the endpoint, not the shape of the next decade. Even in the bear case, the path runs through despecialization first.

Conclusion

The question "how many jobs will AI eliminate" assumes that jobs are fixed containers and AI either fills them or empties them. Jobs are not fixed. They are bundles of tasks that organizations assembled under a specific set of constraints, and the biggest of those constraints was that expertise was expensive to acquire and slow to switch between. AI relaxes both constraints at once.

The result is not empty containers. It is fewer, wider ones. Work that used to need a chain of specialists connected by tickets now fits inside one person directing agents across the chain. That person needs less recall and more judgment. They need to know what wrong looks like in five domains rather than what right looks like in one.

That is a harder job in some ways and a better one in others. It is harder because the blast radius is wider and the apprenticeship path that used to produce judgment is broken. It is better because the coordination overhead that ate a quarter of every specialist's week is gone, and because the work is closer to the outcome.

The three-person team I once needed for a personal website is already gone, replaced by one person with wider judgment and better tools. The eleven-person pipeline team is going the same way, replaced by a smaller group of generalists who each own a slice end to end. Our job, as individuals and as the people who run teams, is to make sure that person exists, knows how to check the agent's work, and is not a 23-year-old who has never seen a pipeline fail.

Keep Going

If this piece was useful, I have written a lot more on how AI reshapes work and the economics behind it. My book on AI and labor economics goes much deeper into the task-versus-job framing and what it means for careers and policy, and you can find it at a.co/d/06SeOKw8. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.