DEV Community: Alex Merced

Apache Data Lakehouse Weekly: October 28 to November 4, 2026

Alex Merced — Thu, 05 Nov 2026 02:32:21 +0000

This was a week of votes. Apache Iceberg passed or nearly passed three separate v4 spec changes, Apache Parquet opened and then reset a vote on a new vector type, and Apache Arrow, Iceberg Rust, DataFusion Comet and the Iceberg Terraform provider all shipped releases. The quieter story sits underneath those votes. Across six dev lists, maintainers kept running into the same two problems: specs that leave room for two correct readings, and a flood of AI-assisted pull requests that outpaces the people who review them.

The six lists produced 463 messages in the eight days ending November 4. Iceberg led with 117 messages across 37 threads. Apache Ossie, still incubating, came second with 125 messages, a sign of how much design work a young semantic standard needs before its first release. Polaris followed with 95, Parquet with 52, Arrow with 47 and DataFusion with 27.

Apache Iceberg

The v4 table format took three large steps forward in a single week. Ryan Blue closed the vote on stats-only fields for v4 with 15 +1 votes and no objections, as recorded in the result thread. The change adds a stats-fields list that assigns table field IDs to things that are not real columns, such as partition fields or an expression like to_lower_case. Engines can then store statistics for those derived values in the content_stats struct. In plain terms, Iceberg can now record min and max values for the expressions people actually filter on, not only for the raw columns they store.

The design discussion behind that vote, in the stats-only fields thread, carried a useful detail. Andrei Tserakhau raised collations, and Ryan proposed a new type with standardized metadata for those rather than an expression. His reasoning was simple: a collation always produces a string, so binding an expression adds work with no payoff. He also left the handling of unbindable expressions to implementations, arguing that a strict "must ignore" rule hides real errors.

Daniel Weeks then opened a vote on default value expressions for v4. The proposal expands write-default so a column default can be an expression from Iceberg's expressions spec, not only a literal. The thread collected binding +1 votes from Steven Wu, Russell Spitzer, Ryan Blue, Gang Wu, Eduard Tudenhöfner, Fokko Driesprong, Amogh Jahagirdar and Yufei Gu within two days. Péter Váry and Gianluca Graziadei left small review comments on the spec PR. For teams that evolve schemas often, this matters: a new column can get a computed default without a backfill job.

Steven Wu's vote to add a _last_updated_time row lineage column had the most texture. The proposal records the wall-clock timestamp of the snapshot that last changed a row, next to the existing _last_updated_sequence_number. It adds an optional snapshot_timestamp field in v4 manifests, and writers leave it null so readers inherit the snapshot's timestamp-ms. Upgrading to v4 needs no manifest rewrites. Daniel Weeks voted +1 but asked to change the type from timestamptz to timestamp, since the inherited value carries no time zone. Ryan Blue then wrote that the spec text still needs more editing before it is ready. Expect a revised PR before this one closes.

Two more v4 threads show where the format is heading. Kevin Liu asked directly whether v4 should stop writing ORC and Avro data files. Existing files stay readable. His case rests on two points. Iceberg's own format test kit already lists features those formats lack, such as variant shredding, and the draft vector type does not support ORC. He also noted that recent ORC and Avro format changes Iceberg needed came from Iceberg contributors themselves, with Gang Wu adding geo types to ORC and Fokko adding nanosecond timestamps and fixed-size UUIDs to Avro. The thread poses four questions, including what bar a format must clear to become writable again.

Pachu Shrinivas revived the deprecation of equality deletes in v4 with a practical worry. Equality deletes do not conflict with concurrent compaction, because compaction can preserve the data sequence number from planning time. Deletion vectors (DVs) are scoped to one data file. So when compaction rewrites that file, either the streaming writer or the compaction job fails. Streaming shops will want an answer to this before v4 lands.

That same theme showed up from an unexpected direction. Michalis Papadimitriou from the TornadoVM team shared a proof of concept for GPU-accelerated merge-on-read scans and table maintenance. TornadoVM compiles plain Java methods into GPU kernels. The library sits on Iceberg 1.12 public APIs, so Iceberg still plans and commits while the GPU handles per-row work. The team reports 3.9x to 4.8x faster merge-on-read scans at 5 billion rows on two A10 GPUs versus 128 CPU threads. It also reports equality-delete to DV conversion on TPC-DS SF100 in 0.77 seconds versus 81.9 seconds for Spark compaction. These are the team's own numbers. Their pitch ties directly to the v4 decision: if new equality deletes are forbidden, converting existing ones becomes required work. They asked whether the code belongs out of tree, like Comet, or in a contrib module.

Steven Wu also consolidated the Monday sync into a single "V4 Metadata" meeting. It now covers the adaptive metadata tree, column files, content stats, relative paths, constraints and row timestamps. The Tuesday column-files sync goes away. The Thursday file and vector types sync stays for a few more sessions, which Philipp Fischbeck and Russell Spitzer used this week in the file and vector types thread to agree that a vector should be a primitive type in Iceberg. Russell's reasoning: consumers treat a vector as one whole value, not a list of cells to edit.

Alex Stephen added an AI angle on top of that work with a proposal for AI functions for FILE and VECTOR types. The idea is a vendor-neutral generative function and an embedding function in the Spark and Flink connectors, so users can work with multimodal Iceberg data without picking one AI vendor's SQL dialect. Alex Cruise posted a separate proposal for pluggable Puffin blob types with a compaction-aware lifecycle. His goal is "progressive indexing": adding new index types to large, old files without rewriting the data. He framed it as complementary to the v4 secondary index proposal, and he published a gentler backgrounder alongside the formal text.

Index work also moved in a dedicated sync. Huaxin Gao summarized how databases, catalogs and REST APIs model indexes. Every catalog API she checked scopes an index to its table. SQL databases split on the question. Daniel Weeks gave the proposal a +1, and Péter Váry said he prefers nesting indexes under the table path in the REST spec.

REST catalog and auth

The REST catalog spec got a useful clarification this week, and the trigger came from Polaris. Anand Kumar Sankaran asked what the metrics reporting endpoint should return when the server cannot store a report. Eduard Tudenhöfner replied that metrics reporting is best-effort, the Java RESTMetricsReporter never surfaces errors, and clients are not expected to retry. Yufei Gu wanted that written into the spec, asking where the line sits if a failed write can return 204. Daniel Weeks drew the line clearly. A server should never return a success code for a failed request, but whether a client retries is a client choice. The spec stays as is. The practical guidance: a metrics endpoint behaves like any REST endpoint and can return a 4xx or 5xx when it fails.

Alex Dutra brought news from Community over Code in Glasgow. The group agreed to downscope the OAuth2 AuthManager v2 proposal. The reboot supports one grant (client_credentials), two client authentication methods, simplified token renewal and limited legacy v1 support. It now fits in a single PR. Christian Thiel, who backed the approach in Glasgow, put it on the catalog sync agenda.

Youngrae Kim closed the vote to add none to the X-Iceberg-Access-Delegation header values with seven +1 votes. A client can now tell the server it wants no credential vending at all. That small change unblocks a Polaris discussion covered below. Yong Zheng also proposed replacing example with examples in the REST OpenAPI document to match OpenAPI 3.1.

Spec gaps found by readers

Sylwester Lachiewicz found a real interoperability bug in the spec's text on name mapping for files that carry only some field IDs. He checked four implementations and found four behaviors. Java resolves by ID only if any field has an ID. PyIceberg uses IDs only if every field, including list elements and map keys, has one. Iceberg Rust lets the first top-level column decide. Iceberg Go resolves per field. This is not theoretical. Delta tables with column mapping, written without IcebergCompatV2, store IDs on top-level columns but not on map keys and values. Read the same table through PyIceberg and Java and you get different results. He wants the community to pick one reading and write it down.

Maintenance code drew attention too. Sebastian Daberdaku asked for eyes on a fix that makes remove_orphan_files use the catalog's Hadoop configuration. When a catalog reaches storage with different credentials than the cluster, the listing step fails with AccessDenied or sees no files. Daniel Weeks suggested reading the configuration from the FileIO instead, and Russell Spitzer backed avoiding new API surface. Sebastian showed why that falls short: S3FileIO, GCSFileIO and ADLSFileIO do not implement HadoopConfigurable. Vova Kolmakov raised a sharper risk in the file_list_view location thread. A raw startsWith check on s3://bucket/table also matches s3://bucket/table-backup, so files in a sibling directory can be treated as orphans and deleted. The fix treats the location as a directory. Munendra S N asked for review of a GCS fix that separates real delete failures from missing objects, since the current code silently swallows permission errors.

Aaron Niskode-Dossett resurfaced a stale PR that pushes aggregates down for GROUP BY on identity partition columns in Spark. Today Iceberg answers COUNT, MIN and MAX from metadata only when there is no GROUP BY. Grouping by an identity partition column is the exact case where the metadata already holds the answer.

Releases and community

Iceberg Rust 0.11.0 shipped after Danny Jones ran an RC3 vote that passed with three binding and seven non-binding votes, per the result. The release adds client-side encryption on the write path and incremental v3 support, including row lineage, nanosecond timestamps, delete vectors and Variant in metadata. The delay came from licensing. Danny closed the loop on the pyiceberg-core attribution issue: the compiled Python wheels now carry every bundled dependency's NOTICE and LICENSE text. Kevin Liu tested the RC against PyIceberg's integration suite and confirmed every crate passes a cargo publish dry run.

Matt Topol announced the Iceberg Terraform Provider v0.2.0 after a vote with four binding +1s. It adds an iceberg_tables data source, SigV4 request signing and better OAuth2 and schema evolution support, and it ships to both the Terraform and OpenTofu registries. On the Python side, Orest Tokovenko proposed dropping Python 3.10 from PyIceberg. Python 3.10 reached end of life on October 1, and PyArrow 26 no longer ships 3.10 wheels. Fokko Driesprong agreed, and the PR is up. Neelesh Salian scheduled an Iceberg Go sync for November 17 to cover v4 adoption and the remaining breaking changes before a 1.0 tag.

The most human thread of the week came from a dashboard. Russell Spitzer shared Iceberg community health stats, and several people, including Kevin Liu and Ryan Blue, noted his first email never reached them, part of an ongoing ASF mail delivery problem. Jared Yu offered a theory for a weak "first to second contribution" number. He sees new contributors arriving with AI tools, expecting fast progress, then stalling because a small group of maintainers cannot review the volume. His phrase for it was a "dopamine crash." That observation connects to almost every other list this week.

Two meetups are on the calendar: a virtual Iceberg meetup on November 13 about building an open lakehouse with Spark, Iceberg and the Polaris REST catalog, and a Tokyo meetup on December 7.

Apache Polaris

Polaris spent the week finishing 1.9.0 and arguing, productively, about where policy belongs. Jean-Baptiste Onofré posted the November board report draft, and it gives a clear snapshot of the project eight months after graduation. Polaris has 33 committers and 19 PMC members. Ayush Saxena became a committer on October 12. From August through October, 646 pull requests were opened and 529 merged. Excluding dependency bots, those numbers drop to 391 and 294. The open PR backlog grew from 102 to 137, and 76 people opened PRs this quarter, up from 57. The report says it plainly: the bottleneck moved from writing changes to reviewing them.

The report also lists what shipped in 1.7.0 and 1.8.0. Highlights include event listeners that publish Polaris events to Kafka and OpenTelemetry, GCS principal attribution so the Polaris principal shows up in Google Cloud audit logs, and dedicated privileges for semantic models. Two CVEs were published this quarter, one low and one important, and both were fixed in a release before disclosure. No release shipped in October. 1.9.0 is being finalized now and includes a first beta of Polaris Directories.

Who decides about credentials

The longest Polaris thread asked who should control credential vending. Dmitri Bourlatchkov pushed typed storage config options to skip credential vending in Azure and GCS, implemented by Vignesh in PR 5884. S3 already had this option. His use case is workload identity: engines already hold ambient credentials, so vended credentials add overhead and a second, unneeded way to reach storage. Yufei Gu countered that clients can simply not send the X-Iceberg-Access-Delegation header, as Spark and Trino do by default. Dmitri replied that a server cannot assume every client behaves. Yufei then argued that vending is a property of the catalog, namespace or table, not the storage config, since one storage config will soon be shared across many entities. Dmitri disagreed and asked for more voices. This is a real design fork, and it lines up with the new Iceberg none delegation value.

That shared storage config is its own thread. Srinivas Rishindra posted a spec-only PR for multiple storage configurations per catalog. Storage configurations become named, immutable entities under a catalog. Admins can map a namespace or table name to a configuration before the object exists. The PR replaces two earlier attempts.

The /v1/config endpoint got closer scrutiny too. The soft-roll authorization thread for GET /v1/config started as a simple merge. Yufei and JB pointed out the deeper issue: Polaris returns the entire catalog properties map as client defaults, mixing client settings like rest-page-size with server settings like default-base-location. Gating all of it on one privilege means a client with narrow grants loses settings it needs. Dmitri proposed a compromise. Vignesh dropped the second operation, kept GET_CATALOG_CONFIG behind a flag that defaults to false, and documented that the default will not flip until a follow-up defines a client-facing config. Dmitri wants it in 1.9.0 and flagged it in the release thread.

Yufei raised a sharp edge in the ALLOW_UNSTRUCTURED_TABLE_LOCATION thread. If an admin turns the flag off, a renamed table whose data still sits under the old namespace starts failing with 403 errors later. Alex Dutra and Russell Spitzer both argued against blocking the change, because it blocks a move from a less secure setup to a more secure one. Russell's framing was clean: an admin who wants a hole closed wants it closed now, and failing closed is fine. The group settled on documentation in 1.9.0 and a read-only admin report after it.

Metrics, discovery and federation

The metrics endpoint debate that reached the Iceberg list started here. Anand's PR to drop reports instead of returning 500 on a database failure logs the failure and increments a polaris.metrics.reports.dropped counter. Dmitri argued retries belong on the server, since metric reports cannot conflict with other writes. Anand agreed and offered to build server-side buffering as a follow-up. Yufei asked to settle the semantics with Iceberg first, which Anand did. Given Daniel Weeks's answer on the Iceberg list, returning 204 for a failed write now looks like the wrong call.

Dennis Huo proposed catalog discovery for data-plane clients. Today an engine has to learn the catalog name out of band, even though Iceberg's /config endpoint is multi-catalog. Polaris's existing listCatalogs is a management API that needs a root privilege. The new listAccessibleCatalogs endpoint returns only catalogs the caller can call getConfig on. JB and Dmitri approved the PR. This also fills a gap for the in-progress Polaris Shares API.

Federation and auth produced several smaller threads. Laurent Goujon proposed chain-aware authorization for views using Iceberg's optional referenced-by parameter, which Polaris accepts and then drops today. Passing the chain of views lets an authorizer support definer views. Arnab Karmakar asked for a custom STS endpoint for SigV4 federated catalog connections, matching what storage vending already supports. Austen Tomek posted a draft for Cloudflare R2 with scoped credential vending, and Tommy Bollhofer offered to test resumed remote signing work on FlashBlade storage with Trino, Spark, PyIceberg and DuckDB clients.

Qiegang Long found a correctness bug worth reading. In the conditional entity writes thread, he showed that a multi-table commit can assert a table is unchanged without including it in the atomic write. Two concurrent commits can both succeed, each changing one table while asserting the other. That is classic write skew. He opened the SPI change as a draft and brought it to the list first, as CONTRIBUTING asks for extension-point changes.

Ossie inside Polaris

Arnab also drafted semantic model JSON Schema validation, validating stored semantic models against a bundled Apache Ossie schema and capping document size at 1 MiB. JB asked him to wait for the Ossie REST API proposal so Polaris does not lock in a validator interface that the standard then changes. Arnab agreed and linked his draft on the Ossie PR. That small exchange is the clearest sign yet that Polaris plans to be a home for Ossie semantic models.

How the project works

Yufei opened a timely thread on PR quality and sharing the review load. He asked authors who use AI to act as the first reviewer of what the AI writes. Dmitri agreed and suggested sharpening AGENTS.md. JB pointed out that CONTRIBUTING.md already has a section on AI-assisted contributions and asked reviewers to link it rather than invent new rules. Youngrae Kim shared two practices from the contributor side: write the failing reproduction test first, and review PRs in areas you just touched. Yufei separately proposed preferring Java records for simple value types, noting that AI agents now write the boilerplate that once justified the Immutables library. The group agreed for new code only.

Alex Dutra also revived the CHANGELOG merge conflict thread and implemented changelog.d fragments using Dmitri's naming scheme. Robert Stupp wrote a reply worth quoting in spirit. When contributors report recurring friction, saying others hit it less often does not resolve it. A useful reply names a drawback, proposes an alternative, or agrees to try.

Apache Arrow

Arrow shipped a patch release, got new funding, and started a conversation about AI contributions that mirrors Polaris and Iceberg. Raúl Cumplido announced Apache Arrow 26.0.1 after an RC1 vote that drew detailed verification reports across macOS, Debian, Fedora and Ubuntu. The release resolves 19 GitHub issues. The headline fix is security. Javier Guerrero asked whether the Brotli CVE fix lands in 26.0.1, since Arrow statically linked Brotli 1.0.9, which is affected by CVE-2026-111497. JB confirmed during verification that the bundled Brotli is now 1.2.1. If you ship PyArrow wheels inside a product, upgrade.

The funding news came from Alenka Frim. Germany's Sovereign Tech Agency contracted research and development work for PyArrow. Alenka, Nic Crane, Rok Mihevc and Tadeja Kadunc handled the application, with Open Source Collective as fiscal host. The contract funds an average of 1.9 full-time engineers through the rest of 2026 and all of 2027. It targets work that stalled because no corporate sponsor wanted to pay for it, starting with PyArrow type annotations. Antoine Pitrou, Weston Pace, Neal Richardson, Matt Topol and others replied with congratulations. Public money for core data infrastructure is still rare, and this is a good model.

Contributions, human and otherwise

Andy Grove gave the Java community a heads-up that DataFusion Comet plans to send PRs upstream for workarounds it carried against arrow-java. He was direct about the method: a human in the loop, AI writing the code, and real issues behind each PR. David Li thanked him and asked for patience, since contributor capacity on arrow-java is low. JB started reviewing. Raúl and Nic Crane, meanwhile, discussed the CPython AI usage guidelines. Python uses one label that asks a PR author to explain the change in their own words, and a second label that closes the PR when that expectation is not met. Nic liked the "give them a chance, then close" approach and the automation that labels allow.

JB also posted a heads-up for Arrow Java 20.0.0 release prep, and Fred Park asked for review of a hostnameOverride property for the Flight SQL JDBC driver. The property lets users keep TLS certificate checks on when they connect through SSH tunnels, private endpoints or raw IP addresses. Today the only workaround turns certificate verification off.

Format and tooling

Mandukhai Alimaa bumped the canonical extension type for FILE and said a vote comes soon if nobody objects. This is Arrow's half of a cross-project effort. Parquet already merged a FILE logical type, Iceberg is designing FILE and VECTOR types, and Arrow needs a canonical in-memory form so engines can pass these values around without custom glue.

Wes McKinney shared an update on Arrow's conbench data. His team has run a Go port, now called BenchDB, in production for a couple of months, with historical data offloaded to Parquet files managed by an embedded DuckDB. He offered help porting Arrow's benchmark runner and said his company is willing to sponsor cloud costs. Arrow Rust 60.1.0 passed its vote and is on crates.io, and ADBC 25 closed its RC2 vote.

Jeffrey Vo raised the CI issue every Apache Rust and Java project now faces. In the pull_request_target migration thread, he warned that GitHub will block the trigger by default on December 2, with brownouts on November 9 and November 16. A scan found arrow, arrow-java, arrow-site and arrow-erlang still use it. Raúl opened a tracking issue the same day.

Antoine proposed an informal Arrow gathering at Compute! Paris on the morning of Thursday, November 26. JB and Raúl plan to attend. Ian Cook posted the November 4 community meeting at 17:00 UTC.

Apache Parquet

Parquet had the most dramatic vote of the week. Rok Mihevc called a vote to add the FINITE_VECTOR logical type as specified in parquet-format PR 624. The type annotates the same structure used for LIST, so readers that do not know it still see a list. Every non-null vector has exactly num_elements elements, and every element is finite: no NaN, no null, no infinity. Support came fast from Julien Le Dem, Ryan Blue, Daniel Weeks, Gang Wu, Russell Spitzer and others.

Then Antoine Pitrou voted -1 (binding). His objection was not about vectors. It was about the sentence "Writers must enforce these rules." Parquet does not require writers to validate STRING data as UTF-8, and Parquet C++ does not do so. If FINITE_VECTOR demands writer-side validation, then the natural expectation is that STRING and JSON do too. That is a much larger change in Parquet's philosophy. Antoine said writers can validate, but it should stay an implementation choice. Rok asked him to propose wording on the PR. He did. Rok updated the PR with feedback from Antoine and Andrew Lamb and moved the vote to the new commit, asking earlier voters to look again. The vote stays open for at least 72 hours.

That vote grew out of the focused-call follow-up on numeric vector physical representation. Will Edwards worried that the annotation will "orphan" huge volumes of existing embedding data stored as plain lists of floats. Daniel Weeks told the group to stop arguing about performance, since speed is not the goal of the type, and confirmed that both Iceberg and Delta want it with these semantics. Ryan Blue expects the Iceberg spec to require writers to produce the annotation. Jan Finis drafted an alternative in PR 633 with two composable building blocks: an optional elementCount on lists and an optional isFinite property on float types. Rok replied that FINITE_VECTOR is narrow by design and does not block a general fixed-size list later. Julien made the meta point that settled the tone. Consensus on new types stalls easily, and narrowing scope is how projects keep moving.

Getting the spec right

Two threads this week dealt with how Parquet changes its own rules. Julien posted a taxonomy of format changes that defines backward compatibility, forward compatibility, correct reads and "correct failed reads," where a reader errors out instead of returning wrong data. It also defines partial compatibility, where a reader can read the subset of columns it understands. Julien wants agreement on definitions before the implementation discussion resumes.

The FILE type then gave that taxonomy an immediate test. Stefan Grafberger found an ambiguity when a FILE value sets both inline and uri. If the object at the URI changes after the write, one reader returns the inline bytes and another returns the new object, and both follow the spec. He proposed "inline wins," with locator fields kept as provenance. Divjot Arora asked whether changing read rules for a released type counts as a compatible change. Julien answered directly. When the spec under-defines behavior, the project has to pick the correct reading, declare the others bugs, ship fixes, and keep compatibility with the clarified spec from then on.

Jiayi Wang posted the revised Parquet Modular Footer proposal. The new version stops treating row groups as physical boundaries. A column chunk is now just a contiguous set of pages for one column, and different columns can have different numbers of chunks. Stats are defined over row ranges, not row groups. The confusing EncodedArray concept is gone, replaced by Parquet pages as modules. Julien called it "fairly transformational" and asked everyone to review, since the project wants to avoid changing the footer again for a while.

A release, with lessons

Parquet Java 1.18.2 is close. Fokko Driesprong proposed the patch release to carry Jackson 2.22.4 and Thrift and Snappy backports, and Russell noted the Jackson bump lets Iceberg remove a Trivy CVE exception. Manu Zhang volunteered as release manager and used the new automated release scripts. RC0 failed when Gang Wu found two problems. The signing key was missing from the KEYS file, and the source zip contained an entire Thrift build tree with 41 compiled object files and an 80 MB executable. Manu fixed the script and cut RC1. His verification even ran Iceberg's CI against the RC, with the Jackson CVE ignores removed. Russell then spotted an incorrect SCM tag in the POM and pushed a fix to the release automation. New release tooling always finds its bugs on the first real run, and this team found them in public and fixed them fast.

Apache DataFusion

DataFusion's list was short but pointed. Andy Grove ran the vote for DataFusion Comet 1.2.0, the Spark accelerator that runs Spark plans on DataFusion. Manu Zhang voted -1 (non-binding) after finding that the JAR file is missing lz4 and zstd licenses, and he opened a fix. Martin Grigorov argued it should not block the release. Manu cited ASF release policy, which requires LICENSE and NOTICE files to cover everything in a package, including binary JARs. Andy approved the fix for the next release and moved forward. The vote passed with nine +1 votes, five binding. The licensing question echoes the Iceberg Rust delay a week earlier: compiled artifacts carry obligations that source releases do not.

Andrew Lamb opened two threads that set the tone for the next release cycle. He proposed slowing feature velocity to work down the bug backlog and asked for feedback on a GitHub issue. A few days later he announced that the 56.0.0 release is starting, with Xudong cutting branch-56 and an RC expected in days. He also called the vote for sqlparser-rs 0.64.0, which passed verification with 1,917 tests and no failures in one report.

CI cost shaped two more threads. Andy announced that Comet fork PRs now run CI on the contributor's fork, not on shared ASF runners, matching what DataFusion and arrow-rs already do. Andrew said the change made a large difference in the runner time the ASF gets charged for. Jeffrey Vo then posted a call to migrate away from pull_request_target before GitHub's December 2 block, and Kumar Ujjawal found a PR already in progress. Andy also opened a discussion on the future of the datafusion-spark crate after the weekly sync.

Apache Ossie

Apache Ossie, the incubating open semantic interchange standard, was the busiest list after Iceberg. The project is preparing its first Apache release, and the week's threads read like a spec hardening under real data.

Yufei Gu opened a thread on stability expectations for the first release, which will be either 0.2.0-incubating or 0.3.0-incubating. His reading: 0.x is beta, minor releases can break things, patch releases cannot, and 1.0 brings backward compatibility within 1.x. He argued for shipping now, since converters need a version to build against instead of a moving main branch. JB agreed and said 1.0 can follow quickly once something is stable and verified. Kyoung Min Kim added a practical warning. The Python models hard-code the version string 0.2.0.dev0 in a Literal type, so a patch release breaks five converters unless that changes too. JB then posted the release resources prep with three PRs that stage release scripts, bump the version to 0.3.0-incubating and align converter dependencies. Ossie will use the ASF's ATR release tooling.

What a table name means

The deepest technical thread asked what Dataset.source should guarantee. It ran 17 messages and pulled in real data from three producers. JB started from a gap one step before connections. Catalogs such as Apache Polaris allow dots in table names, yet the Ossie schema treats source as a plain string with no delimiter or quoting rule. John DaCosta proposed a list form, identifier: [prod, sales, eu.customers], with dotted strings kept as shorthand. Damian Waldron ran the proposal against 164 datasets from 31 real ThoughtSpot models. All of them stayed valid shorthand, so the change costs nothing on real data. Chad CH Kim ran 53 sources from his generator and got the same result.

Richard SG Kim then reframed the problem. The repo's own example uses DATABASE.SCHEMA.RUNWAYS, where the first two parts are placeholders. A tool reading that file looks for a database literally called DATABASE. He proposed a qualification field that separates two cases. A relative reference means the consumer must supply the missing parts. An absolute reference with too few parts is a broken document. Damian changed his mind on the list, and JB asked for three conformance behaviors written into the spec: import a relative reference the consumer can bind, reject one it cannot bind while naming the missing parts, and treat a short absolute reference as invalid. JB's key line: a converter should fail loudly, not write an empty value and exit 0.

Query semantics

Chris Eubank proposed the Layer 3 wide-table query interface. A Layer 3 query names dimensions, metrics and an optional filter. The engine picks the joins and lowers the query to Layer 2. Measure evaluation reuses Layer 2's AGG() function, so the spec has one source of truth for measures. Aleksandr Kosachev confirmed the worked examples on ClickHouse and gave it a +1. A GitHub discussion on how semantic engines should interpret relationship definitions explained the split. At Layer 2, relationships are descriptive metadata. At Layer 3, they drive join selection. Julian Hyde added that Layer 3 is limited on purpose: it lets you pick dimensions, metrics, filters and sort order, but offers no joins or unions.

The AGG() work itself moved in the dataset-scoped metrics thread. Justin Talbot split his PR in two after JB's review. The first reserves AGG() in the expression language. The second lets dataset metrics reference source columns, and it now requires the source. qualifier, so adding a field named quantity later cannot silently change what SUM(quantity) means. Julian Hyde pushed on terminology: the project needs to pick "metric" or "measure" and use it consistently. The October 28 working group notes track both workstreams.

David Schwarzenbacher proposed fixing two OSSIE_SQL_2026 rules by lazy consensus on November 9. ROUND breaks ties half away from zero, and / means true division, so 5 / 2 is 2.5. Engines differ on both today, which means the same portable metric returns different numbers on different engines. Johannes Zemlin opened the ANTLR4 reference grammar PR with valid and invalid conformance fixtures, keeping the written spec normative.

Converters, validators and the ecosystem

Validation got a careful survey. Kyoung Min Kim mapped what the 23 converters check today and where a vendor hook belongs. JB proposed two limits: a vendor checker only reads its own payload and dialects, and vendor findings never change core results. Andrew Nepogoda asked for stable, documented error codes and severities instead of free-form strings. Andrew also proposed publishing the apache-ossie Python package to PyPI, after main-branch changes broke converters he did not own. JB asked him to wait for the release prerequisites, citing legal checks on published artifacts. His related thread on shared models and model-based validation moves ontology types into the shared package.

New contributors kept arriving. Yurii Melnychuk asked how to contribute Coginiti's Rust converter, and Yufei confirmed apache/ossie-converters is the home, with thin wrappers fine. Chen Ning from Dameng Database introduced himself with a consumer-side view: a text-to-SQL system is only as good as the semantic model it is grounded on. The GASP SaaS metrics standard announced it publishes its 306 metrics as Ossie files, with 164 staged datasets. Mike Carlo asked for a vote on making custom_extensions data an object instead of a string, and Purvaja Narayanaswamy proposed an Action concept for the ontology spec to describe business operations next to terms and metrics. Two GitHub discussions explored graph-based ontology integration with an RDF mapping and whether Ossie is an interchange format or a source of truth. JB's answer to the second: both, in layers. Mario De Felipe gave the SAP view, where the ERP system owns structure and a small human-owned layer sits on top.

Cross-Project Themes

The biggest theme this week was ambiguity in specs, and how each project chose to resolve it. Iceberg found four readers giving four answers for partial field IDs. Parquet found two readers returning different bytes for the same FILE value. Ossie found that its own example file contains placeholder table names a tool will treat as real. Polaris and Iceberg disagreed for a day about whether a failed metrics write can return success. In every case, the fix followed the same pattern Julien Le Dem described on the Parquet list. Pick one correct reading, write it into the spec, and declare the other behaviors bugs. The Ossie thread went one step further by asking for conformance behaviors, so a tool that cannot do the right thing fails loudly instead of guessing. That principle belongs in every open format.

The second theme was review capacity in the age of AI-written code. Polaris's board report put numbers on it: 137 open PRs and a backlog that grew 34% in a quarter. Iceberg's health dashboard surfaced the same pattern as a drop in second contributions. Arrow discussed borrowing CPython's labels for AI-generated PRs, and Andy Grove opened his Comet contributions with a note that a human stayed in the loop. DataFusion is pausing features to fix bugs. None of these projects banned AI tools. They are converging on one rule: the person who opens the PR owns it end to end and must explain it in their own words.

The third theme was vectors and files moving across the whole stack at once. Parquet is voting on FINITE_VECTOR and clarifying FILE. Arrow is about to vote on a canonical FILE extension type. Iceberg treats vectors as primitives, proposed vendor-neutral AI functions over FILE and VECTOR columns, and is weighing whether v4 should write only Parquet. Read together, these threads describe an open lakehouse that stores embeddings and unstructured references as first-class typed data, with the same type meaning the same thing from file to memory to table. That only works because the same people sit on several of these lists. Daniel Weeks, Ryan Blue and Russell Spitzer voted on Parquet's vector type while shaping Iceberg's.

Release hygiene was the fourth thread. Iceberg Rust waited weeks to fix NOTICE files in compiled wheels. Comet shipped with a known JAR licensing gap and a fix queued. Parquet's new release automation packed a Thrift build tree into a source zip on its first run. Arrow 26.0.1 existed mostly to pick up a Brotli CVE fix. And Arrow, DataFusion and Comet are all racing GitHub's December 2 block on pull_request_target. Binary artifacts and CI pipelines now take as much care as the code itself.

What to Watch Next Week

The FINITE_VECTOR vote resets its 72-hour clock on the updated commit, so watch whether Antoine Pitrou's wording change holds the earlier +1 votes. Iceberg's default value expressions vote is ready to close, and the _last_updated_time vote needs Daniel Weeks's type change and Ryan Blue's editing pass. Polaris 1.9.0 is close, and the credential vending debate needs more voices before PR 5884 merges on November 6. Ossie's lazy consensus on ROUND and division closes November 9, and its first release candidate depends on the version string fix. DataFusion 56.0.0 should produce an RC, and GitHub's first pull_request_target brownout lands November 9, so expect red CI on any repo that has not migrated.

Keep Learning

If you want to go further on Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet and the open lakehouse, I have written a lot more on these topics. You can find all of my books, including Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide, along with my self-published guides on lakehouses and agentic AI, at books.alexmerced.com.

AI Weekly: Gemini 4 Carbon, Open Mistral, AMD's Surge

Alex Merced — Thu, 05 Nov 2026 02:20:51 +0000

Week of October 28 to November 4, 2026

Two frontier releases set the tone this week, and they pointed in opposite directions. Google announced its strongest model yet and gave it to almost nobody, while Mistral handed the world a trillion-parameter model to download. Around them, the tooling layer got faster and more careful, the agent standards gained another production-grade SDK, and AMD posted a quarter that says the AI buildout is far from finished.

This issue follows the usual order. Models come first because they set what is possible. Tooling comes next because it decides who can use those models. Standards follow because they decide whether the pieces connect. Infrastructure closes the main sections because it sets the cost of everything above it.

Models: Gemini 4 Carbon Leads, but Access Is Narrow

Gemini 4 Carbon

Google announced Gemini 4 Carbon on October 28, 2026. Carbon builds on Gemini 4 Argon and targets security and coding work that runs for days, according to Cybersecurity News. Google's own October recap calls Carbon its strongest model yet for long-horizon engineering and cybersecurity defense. The timeline matters here. Google introduced Argon on September 30, opened it broadly on October 13, and shipped Carbon 15 days after that expansion, as VentureBeat laid out.

Access is the catch. Carbon starts with selected cyber defenders in Google's Fairwind program, not a public launch. As of October 30, Google's public model catalog had no Carbon entry and no announced API date, per EvoLink's review. The only Gemini 4 model in the API list is still gemini-4-argon, and there is no Carbon model ID yet, according to SmartScope. Google also says Carbon sits inside the U.S. government's voluntary pre-release testing process.

The benchmark numbers are strong, and all of them are vendor-reported. Google reports 88.2% on SEC-bench Pro for vulnerability discovery, 54.8% on ExploitGym for exploitation and 71.0% on CWE-bench for patching. On Terminal-Bench 4.0, Google lists Carbon at 65.2% against Argon's 57.1%. Carbon's GraphWalks score rises from 84.2% to 96.2%, and its ExploitGym score rises from Argon's 42.0%, per EvoLink. Google also says high-severity misalignment alerts fell by almost 80% in internal coding environments, and unsupported claims dropped by roughly half compared with Argon.

Independent signals line up with Google's claims so far. Carbon scores 59 to Argon's 50 on the Artificial Analysis Intelligence Index v5.0, according to Fello AI's tracker. Pricing at launch is reported at $2 per million input tokens and $10 per million output tokens, the same introductory rates as Argon, per TeamoRouter. Argon's $2 and $10 rate is set to run until November 30. Google has not restated a context window for Carbon. Third-party listings show 1 million tokens, which matches Argon's published spec.

So what does Carbon change for practitioners this week? Very little, directly. You cannot call it. What it changes is the planning horizon. Carbon arrived inside the Gemini 4 generation number rather than as Gemini 4.1 or Gemini 5. OrcaRouter's analysis read that as a sign Google will ship variants, not versions. If that holds, teams building on Argon today can expect a stronger sibling under the same API family, not a migration to a new one.

Carbon also confirms a pattern that now covers three frontier labs. OpenAI's GPT-6 Astra rolled out first to its Daybreak cybersecurity program in September. Anthropic's Mythos-class models reached cyber defenders through Project Glasswing before wider release. Google now ships Carbon to Fairwind partners first. The frontier labs agree on one thing: the most capable cyber model goes to defenders before it goes to everyone.

How to Read This Week's Cyber Benchmarks

Carbon's launch leans on security benchmarks, so it helps to know what each one measures. SEC-bench Pro tests whether a model can find real vulnerabilities in real code. ExploitGym, a benchmark built by UC Berkeley researchers with several frontier labs, tests whether a model can turn a known weakness into a working exploit. CWE-bench tests whether a model can write a correct patch for a weakness class from the Common Weakness Enumeration list. Terminal-Bench 4.0 measures long, multi-step work in a command-line environment, which is closer to how coding agents run in practice.

Two cautions apply to every number in this section. First, these are lab-run results on lab-chosen settings. Reasoning effort, tool access and time budgets change scores by many points, and labs rarely publish all of those settings at launch. Second, exploitation scores cut both ways. A higher ExploitGym score means a better red-team assistant and a more dangerous model in the wrong hands. That is exactly why Google, OpenAI and Anthropic all gate their strongest cyber models.

For a data platform team, the useful signal is the patching and secure-code results, not the exploit score. Google says Carbon more than doubled Argon's success rate on an internal secure-code generation test, per Cybersecurity News. If that holds when Carbon reaches the API, the main benefit for most teams is safer generated code in pipelines, connectors and SQL, where a single injection flaw exposes a whole warehouse.

Mistral Large 4 Goes Open Weight

The other major release went the opposite way. On October 29, Mistral released the full downloadable weights for Mistral Large 4, nicknamed "Le Chonk," three weeks after a preview. SiliconANGLE covered the launch, which Mistral posted on its news page. The weights live on Hugging Face as mistralai/Mistral-Large-4-1T-A52B, with an NVFP4 build for Blackwell GPUs. The API model ID is mistral-large-4, per Developers Digest.

The model has roughly one trillion total parameters. Reports on active parameters differ. One summary of Mistral's materials cites 49 billion active parameters and training on 3,800 NVIDIA GPUs in Europe, per KuCoin's news digest. The repository name says 52 billion. Check the model card before you size your hardware.

The license deserves a careful read. Large 4 ships under the Mistral Ecosystem Open-Weight License, a modified MIT license. You can use, modify, fine-tune and deploy it commercially. Cloud providers that resell it as a hosted service need a separate commercial license from Mistral, per SiliconANGLE and Trending Topics. That is a shift from the Apache 2.0 license Mistral used for Large 3. In practice, it means fewer third-party hosts and more self-hosting.

On independent rankings, the released Large 4 scores 26 points on Artificial Analysis's redesigned open-model index, one more than the preview. That puts it eighth overall and first among models from outside China, per Trending Topics. DeepSeek V4.1 Flash leads the open index at 32. Meta's Muse Spark Open sits at 25. Moonshot's Kimi K3.1 Pro scores 46, but its weights are not yet public.

Mistral's own numbers, all vendor-reported, focus on coding agents. Mistral cites about 41% on Terminal-Bench 4.0, roughly level with GLM-5.3's reported 40%, per Online Tech Tips. Mistral also claims Large 4 uses 47% fewer tokens per task. Its reported resilience to prompt-injection attacks rose from 93.3% in the preview to 97.1%, according to ForkLog.

Here is why Large 4 matters even though it does not top any chart. It is a frontier-class model that a European government or bank can run inside its own walls, under a license that permits commercial use. Data teams that cannot send sensitive tables to a hosted API now have another option at this capability level. The 47% token reduction claim, if it holds up in independent tests, also matters more than a few benchmark points. Agents burn tokens, and fewer tokens per task means lower cost per answer.

Self-hosting a trillion-parameter model is not a laptop project. Even with an NVFP4 build, you need a multi-GPU node with hundreds of gigabytes of accelerator memory before you serve the first request. The realistic path for most teams starts with Mistral's hosted API to test quality, then a decision about whether the data sensitivity justifies owning the hardware. The license shapes that decision too. Because third-party resellers need their own deal with Mistral, the usual menu of cheap inference providers is shorter for Large 4 than for Apache-licensed models.

Choosing a Model This Week

The practical picture for teams picking models right now is simple. If you need the strongest Google model you can actually call, that is still Gemini 4 Argon at gemini-4-argon, with the introductory $2 and $10 pricing through November 30. Carbon is a planning signal, not an option. If you need open weights at the frontier tier and can run your own hardware, Mistral Large 4 is the strongest choice from outside China on the Artificial Analysis open index. If your security team needs the strongest cyber capability, every major lab now runs a vetting program, and you apply rather than sign up.

Build for swapping. The gap between Argon and Carbon was 15 days. The gap between Large 4's preview and its weights was three weeks. Teams that hard-code one model ID into prompts, evaluations and cost models pay a migration tax every month. Keep model choice in configuration, keep an evaluation set that runs against any model, and track cost per completed task rather than cost per token.

The Week's Model Ledger

Two notes on the ledger. Both rows describe what each lab said or what reporters found, and every benchmark above is vendor-reported unless it cites Artificial Analysis. And no other major lab shipped a new frontier model in this window. OpenAI, Anthropic, Meta, xAI and DeepSeek all spent the week on products, pricing and infrastructure instead.

Tooling: Faster Agents, Harder Boundaries

Claude Message Threads Cut Latency for Long Sessions

Anthropic added message threads to the Messages API in public beta on November 2, per Anthropic's announcement. The feature solves a problem every agent builder knows. A coding agent sends hundreds of requests in one session, and each request resends every earlier message and tool result. Once the conversation reaches hundreds of thousands of tokens, the upload alone adds time before the model starts to answer.

With message threads, the first request sends the full conversation and starts a thread. Each later request sends only the new messages plus the ID of the previous response. Anthropic holds the thread in memory between turns, encrypted and not written to disk. A thread expires five minutes after its last request, and the client then resends the full conversation.

The numbers are concrete. With message threads on by default for most Claude Code users, requests between 500,000 and 1 million tokens saw median time to first token drop by about 35%. Token usage does not change, since Claude still reads the whole conversation each turn. Standard token rates and rate limits apply. The feature is live on the native Claude Platform, Claude Platform on AWS, Claude in Microsoft Foundry and Claude Models on Google Cloud (Anthropic managed).

For teams building analytics agents, this is a quiet but real gain. An agent that pulls query results, schema descriptions and sample rows into context grows fast. Every tool result becomes upload overhead on every turn. Threads shift that overhead from per-turn to per-session, which matters most for exactly the long, context-heavy workflows data teams run.

Anthropic also shipped governance features for enterprise buyers on October 28. Claude Enterprise now offers smart reports, in-app surveys and pooled group budgets, per Anthropic. Pooled budgets give a group one shared monthly amount, with individual limits kept as a guardrail. Admins get email alerts at 50%, 75%, 95% and 100% of a budget. This is the billing side of agentic work. Agents spend tokens unevenly, and per-seat caps punish the heaviest, most productive users first.

Microsoft Opens the Copilot Agent Harness

Microsoft opened the GitHub Copilot agent harness to outside developers through Microsoft Foundry, according to The New Stack's October 30 daily. The harness entered public preview through Foundry Agent Service on Wednesday, October 28. It includes built-in sandboxing, reusable skills, MCP integrations and support for subagents. Microsoft runs the agent and manages its execution environment. Developers choose a supported model and configure instructions and tools.

This continues a trend that every major lab and platform now follows. The harness, meaning the loop that manages context, tools, permissions and sandboxes around a model, is becoming a product of its own. Anthropic sells Claude Managed Agents. OpenAI ships Codex and ChatGPT Work. Microsoft now rents out the harness behind Copilot. The practical question for builders is portability. The New Stack's own follow-up headline flagged it: the harness makes agents easy until you want to switch. Skills and MCP tools carry across harnesses. Harness-specific orchestration features do not.

Embedded Browsers Become an Attack Surface

Security researchers published findings on October 29 about flaws in the embedded browsers of Cursor, GitHub Copilot, Codex, VS Code and Claude Desktop, per Cybersecurity News. The researchers named the attack class EMBRACE, short for an embedded browser trusted like a real one. The flaws enabled code execution, file theft and address-bar spoofing on developer machines.

The core problem is simple. Coding assistants added browsers to preview projects, read pages and track tabs. Those browsers often got extra access to the host app. In one example, Cursor exposed a window.cursorBrowser.send() bridge that let any loaded page push content into the agent's instruction box without a user gesture. That is prompt injection with a direct pipe. Forever Security says Cursor patched its flaw on October 1. GitHub fixed the spoofing issue in Copilot 1.1.29 on October 13.

The lesson for teams is about trust boundaries, not one vendor. Anything an agent reads is untrusted input. An embedded browser that can write into the agent's instructions removes the boundary entirely. Check that your tools are on patched versions, and treat any feature that lets web content talk to the agent as a security review item.

AI-Written Code Meets Maintainer Capacity

The coding tools are working, and open source maintainers are feeling it. The Apache Polaris November board report, drafted on the dev list by Jean-Baptiste Onofré, put numbers on the shift. From August through October, 76 people opened pull requests, up from 57 the quarter before. The open PR backlog grew from 102 to 137. The report states plainly that the main challenge moved from producing changes to reviewing them.

Other projects describe the same pattern. On the Iceberg list, a community health dashboard showed a weak rate of second contributions, and one contributor blamed a flood of AI-generated code landing on a small group of maintainers. Arrow committers discussed borrowing CPython's labels that ask a PR author to explain a change in their own words, then close the PR if they cannot. Polaris maintainers agreed that anyone using AI must review the generated code end to end before submitting it.

None of these projects banned AI tools, and several maintainers use them daily. The rule they are converging on is ownership. The person who opens the PR answers for every line. For companies that point coding agents at open source dependencies, that rule has a cost: budget human review time for every upstream contribution, or expect it to sit in a queue.

GitHub Sets a Hard Date for pull_request_target

GitHub announced on November 2 that, starting December 2, 2026, GitHub Actions will block pull_request_target events by default in all public repositories, unless an Actions event policy explicitly allows them, per the GitHub changelog. Enforcement was originally scheduled for November 2. Two brownouts come first. During each window, blocked workflows create failed runs with an error annotation and do not execute.

This trigger runs a workflow in the context of the base repository, with access to its secrets, even when a fork opens the pull request. Combined with a checkout of the untrusted PR code, it gives attackers a path to repository secrets. GitHub already hardened actions/checkout v7 in June to block the most common unsafe pattern, as InfoWorld reported.

The open source data stack is already scrambling. On the Apache Arrow dev list, Jeffrey Vo warned that arrow, arrow-java, arrow-site and arrow-erlang still use the trigger, with brownouts on November 9 and November 16. He posted a matching call to action on the DataFusion list. If your team maintains a public repo with automation that labels or comments on PRs, audit it this week.

Model Retirements to Track

The October change most likely to break your setup is a model retirement, not a feature. GitHub Copilot drops Claude Opus 5, Claude Opus 4.8 and GPT-5.6 Terra on November 16, an announcement from October 15 tracked by Devshot. Saved model settings in shared configs and team instructions point at specific models. Update them before the cutoff, or expect fallback behavior that nobody tested.

A Short Tooling Checklist

This week's tooling news adds up to a handful of concrete actions for teams that run coding and data agents:

Turn on message threads in any long-running Claude agent you operate, and measure time to first token before and after on your largest sessions.
Patch Cursor, Copilot and any other assistant with an embedded browser, and review which features let page content reach the agent's instructions.
Search your public repositories for pull_request_target before November 9, and move labeling and commenting jobs to safer triggers.
Replace hard-coded model names in shared Copilot settings and team instruction files before November 16.
If you evaluate managed harnesses such as Foundry Agent Service or Claude Managed Agents, keep your skills and MCP tools in a form you can move to another harness.

Standards: MCP Adds a Tier 1 SDK, A2A Spreads

The Ruby SDK for MCP Reaches Tier 1

The Model Context Protocol (MCP) defines how AI applications connect to tools and data sources. On October 30, the official Ruby SDK for MCP became a Tier 1 SDK, per the MCP blog. Ruby joins TypeScript, Python, C#, Go and Rust at the top tier. Tier 1 means full spec feature support, a 100% conformance pass rate, support for new spec versions by their release date, issue triage within two business days and critical bug fixes within seven days.

Starting with version 1.2.0 of the mcp gem, the Ruby SDK fully supports the 2026-07-28 MCP spec. That revision made MCP stateless at the protocol layer. In Ruby terms, that means four concrete changes:

The stateless lifecycle (SEP-2575) runs alongside the classic session handshake, including server/discover and the subscriptions/listen notification stream.
Multi round-trip requests (SEP-2322) work on both server and client, with a shim that lets new-style handlers serve older clients.
Cache hints (SEP-2549) are emitted on the modern path, and the client sends standard and custom HTTP request headers (SEP-2243).
Roots, Sampling and Logging are marked deprecated per SEP-2577, and clients warn when they declare those capabilities on a 2026-07-28 connection.

The climb was fast. The SDK was assessed at Tier 3 in March 2026, reached Tier 2 with its 1.0 release in July, and was confirmed at Tier 1 in September. Server conformance went from 25 of 30 scenarios to every scored scenario. Documented features went from 22 of 48 to 46. The team also committed to support the next MCP spec, targeted for late March 2027, by its release date.

It is worth spelling out what "stateless" buys you, because it changes how MCP servers get deployed. Under the older spec, a client and server ran an initialize handshake and then held a session. Every request in that session had to reach the same server instance, which forced sticky load balancing and made horizontal scaling awkward. Under 2026-07-28, protocol version, client identity and capabilities travel with each request. Any server instance behind a plain load balancer can answer any call. For a data team exposing a catalog, a semantic layer or a query engine over MCP, that means the MCP server scales like any other stateless HTTP service.

The Ruby SDK serves both eras side by side, so current clients keep working while 2026-07-28 clients use the stateless path. That dual-mode design is the right pattern for any server you run in production right now. Agents in the field will span several MCP versions for months.

Why should data teams care about a Ruby SDK? A large share of internal business software runs on Rails. Billing systems, admin panels and internal tools hold exactly the operational data that agents need. A Tier 1 Ruby SDK means those teams can expose MCP servers with the same guarantees as Python or TypeScript shops, without a translation service in between.

The SDK roadmap also covers the tasks extension and the newer Skills extension. The TypeScript SDK added Skills over MCP (SEP-2640) in version 2.4.0 on October 23. A server can publish a folder of skills, and a client can list, read and verify them, with size and digest checks on every file. Skills are reusable instructions for how to do a task. Serving them over MCP means an organization can publish its procedures once and let any compliant agent load them.

A2A Keeps Showing Up in Products

The Agent2Agent (A2A) protocol lets agents from different vendors discover each other, delegate tasks and exchange results. It moved to the Agentic AI Foundation in August, joining MCP under the same roof, per Axios. This week brought two product signals.

On November 4, Adobe put its marketing AI agent inside ChatGPT, Claude and Copilot. The agent tracks campaign results, forecasts audiences and helps with planning, and Adobe built it on both MCP and A2A, per The Next Web. On October 30, Cognigy shipped experimental A2A support in its releases 2026.19 and 2026.20, per Cognigy's product update. Its A2A server endpoint exposes a Cognigy agent to other platforms, supports both A2A 1.0 and 0.3, and streams over server-sent events. A separate node lets a Cognigy agent call a remote A2A agent as a single tool.

The pattern is consistent. Vendors use MCP to reach tools and data, and A2A to reach other agents. The Cognigy detail about supporting both 1.0 and 0.3 is the most practical point here. Version skew is now a real cost. Anyone building an A2A endpoint should plan to serve at least two protocol versions for a while.

Open Data Standards Are Becoming AI Standards

The most important standards work for data-driven AI this week happened on Apache mailing lists, not in AI foundations. Three threads stand out.

Apache Parquet opened a vote to add a FINITE_VECTOR logical type. The type marks a list as a fixed-length vector of finite numbers, with no NaN, null or infinity values. Readers that do not know the type still see a plain list. Antoine Pitrou cast a binding -1 over one sentence that required writers to enforce the rules, and Rok Mihevc revised the PR and reset the vote. Iceberg and Delta both want the type. Once it passes, an embedding column in Parquet carries a typed contract instead of a guess.

Apache Iceberg is building on that base. Alex Stephen proposed vendor-neutral AI functions for FILE and VECTOR types in the Spark and Flink connectors, covering both generation and embeddings. Kevin Liu asked whether Iceberg v4 should stop writing ORC and Avro data files, in part because the draft vector type does not support ORC. The open table format is choosing Parquet as the home for AI-ready data.

Apache Ossie, the incubating open semantic interchange standard, spent the week on a question every text-to-SQL agent runs into. In the Dataset.source thread, contributors tested proposals against 164 real BI datasets and agreed on a rule: a converter that cannot resolve a table reference must fail loudly and name what it needs, not write an empty value and exit cleanly. A new contributor building text-to-SQL at Dameng Database put the stakes plainly: a text-to-SQL system is only as good as the semantic model it is grounded on. Ossie is also fixing ROUND tie-breaking and true division by lazy consensus on November 9, so the same portable metric returns the same number on every engine. Agents that compute business metrics need that guarantee more than humans do, because nobody checks the agent's arithmetic by eye.

Catalog standards moved in the same direction. On the Polaris list, Dennis Huo proposed catalog discovery for data-plane clients, a new listAccessibleCatalogs endpoint that returns only the catalogs a caller is allowed to use. Today an engine must learn the catalog name out of band. For an agent, that gap is worse than an annoyance, because the agent has no human to tell it where to look. A discovery endpoint scoped to the caller's own permissions lets an agent find the data it is allowed to see and nothing else.

The Iceberg REST spec also got a clarification that every agent tool builder will recognize. In a thread on the metrics reporting endpoint, Daniel Weeks made the rule plain: a server should never return a success code for a failed request, and retry policy belongs to the client. MCP tool servers face the same choice whenever a side effect fails after the main work succeeds. The answer is the same. Report what happened, and let the caller decide.

Infrastructure: AMD's Record Quarter and the Memory Squeeze

AMD's Data Center Business Nearly Doubles

AMD reported third-quarter results on November 3, and the numbers say the AI buildout is accelerating, not plateauing. Total revenue reached about $13.4 billion, up 45%. Data center revenue hit a record $8.5 billion, up 97% year over year, and made up 64% of total sales, per the Investing.com transcript. A year earlier, data center was nearly 47% of AMD's revenue, per Opening Bell Daily.

The guidance was the bigger story. AMD expects fourth-quarter revenue of $17 billion, plus or minus 3%, well above the $16.3 billion analysts expected, per Bloomberg. On the call, AMD said it expects data center revenue to grow 50% sequentially in Q4. It also said that Q4 will be the quarter its data center AI business passes its server CPU business for the first time.

The product behind the surge is the Instinct MI450 series, which launched this quarter. The MI450 clusters 72 chips into a single rack-scale system with up to 432 GB of HBM4 memory per GPU, per Startup Fortune. Oracle Cloud Infrastructure committed to 50,000 MI450 GPUs starting in 2026. That is the largest single public cloud commitment to AMD accelerators to date.

For practitioners, the AMD story is about supply and price. A second vendor shipping rack-scale systems at volume puts pressure on GPU rental rates and on the per-token prices labs charge. It also raises the stakes for software portability. Code that assumes one vendor's kernels and libraries now leaves money on the table. Open formats help here too. A query engine reading Arrow buffers and Parquet files does not care which accelerator sits under it.

Memory Is the Real Bottleneck

The supply chain reports this week point to memory, not compute, as the gating factor. Samsung reportedly concluded pricing negotiations for its 2027 high-bandwidth memory (HBM) supply and targets volume production of HBM4E in the first half of 2027, per a November 4 industry brief. The same brief cites a reported multi-year agreement with AMD covering HBM4 and standard DRAM.

Base dies are the newer pressure point. An HBM stack sits on a logic base die, and next-generation designs customize that die per customer. Reports say Samsung's Taylor, Texas Fab 2 plans to dedicate most of its planned HBM base-die capacity to AMD, with equipment move-in in 2029 and volume production in 2030. Separate reports say NVIDIA has asked for its custom HBM base dies to be built at TSMC, per a November 3 industry brief. Treat both as reports, not confirmed contracts. The direction is clear either way. Memory capacity is being locked up years in advance, and the customization of base dies ties each memory supplier more tightly to each chip designer.

Analysts also flagged HBM supply as the main near-term risk to AMD's own GPU shipments, per TradingKey. When memory is scarce, waste at the model and data layers stops being a rounding error. A model that uses 47% fewer tokens per task, as Mistral claims for Large 4, needs less KV cache per request. A context-handling feature like message threads does not cut tokens, but it cuts transfer and wait time. Every layer of the stack is now being judged on how much memory it wastes.

Power is the other constraint. The same November 4 brief reports that NVIDIA is working with utilities to explore throttling data center power draw during grid peaks. Treat that as a report until NVIDIA describes it publicly. The idea is sound either way. A training cluster that can shed load for an hour when the grid strains is easier to permit and cheaper to connect than one that demands a fixed peak.

What the Infrastructure News Means for Buyers

Three practical points come out of this week's numbers. First, accelerator choice is widening. AMD shipping rack-scale MI450 systems at volume, with a major public cloud commitment behind them, means more capacity on the market in 2027 and more pricing pressure on hosted inference. Second, memory shortages will shape what gets built. Models and serving stacks that cut KV cache use, and data formats that let engines read less, buy real capacity when HBM is booked years ahead. Third, portability now has a dollar value. Workloads tied to one vendor's kernels cannot follow the best price, while engines built on open in-memory formats like Apache Arrow move between accelerators with far less friction.

Formats and Accelerators Meet in the Lakehouse

Two Apache threads this week show where accelerated hardware meets open data formats. On the Iceberg list, the TornadoVM team shared a proof of concept for GPU-accelerated merge-on-read scans and table maintenance built on Iceberg 1.12 public APIs. TornadoVM compiles plain Java methods into GPU kernels. The team reports 3.9x to 4.8x faster merge-on-read scans at 5 billion rows on two A10 GPUs versus 128 CPU threads. It also reports converting equality deletes to deletion vectors 106x cheaper than Spark compaction on TPC-DS SF100. Those are the project's own numbers. The interesting design choice is that Iceberg still plans and commits every change. The GPU only does the per-row work in between.

On the Parquet list, Jiayi Wang posted a revised proposal for the Parquet Modular Footer. It stops treating row groups as physical boundaries and defines statistics over row ranges instead. Julien Le Dem called it "fairly transformational." A footer that describes columns independently makes it easier for readers to fetch only the metadata they need, which matters for wide AI feature tables with thousands of columns.

The Python Data Stack Gets Funded and Patched

One more infrastructure item sits under nearly every AI data pipeline. Germany's Sovereign Tech Agency contracted research and development work for PyArrow, the Python library that moves columnar data between Pandas, Polars, DuckDB, Spark and most model-training data loaders. The contract funds an average of 1.9 full-time engineers through the rest of 2026 and all of 2027, aimed at work no corporate sponsor wanted to pay for. The first target is PyArrow type annotations, which help both human developers and coding agents write correct code against the library.

The same week, Apache Arrow released version 26.0.1. The patch bundles Brotli 1.2.1, replacing a statically linked Brotli 1.0.9 affected by CVE-2026-111497, as a user asked on the list. If you ship PyArrow inside a container or a product, rebuild on 26.0.1. Arrow is also preparing a vote on a canonical FILE extension type, the in-memory partner to Parquet's FILE logical type, so engines can pass references to images, audio and documents without custom glue.

Practitioner Takeaways

Plan for gated frontier models. The strongest model from each major lab now reaches vetted security partners first. Build your roadmap on what you can call today, and treat announcements as previews.
Price open weights by total cost. Mistral Large 4 is free to download, but serving it needs serious hardware, and its license limits who can host it for you.
Cut wasted context. Message threads lower latency without lowering tokens. Pair them with tighter tool results, so agents carry less text from turn to turn.
Audit trust boundaries. Embedded browsers, CI triggers and agent harnesses all moved this week because untrusted input reached privileged code. Look for the same pattern in your own agent tools.
Standardize the data layer. Typed vectors in Parquet, AI functions in Iceberg and portable metrics in Ossie give agents data they can use without guessing. Push your vendors toward these open standards.

The Data Layer Tie-In

Every section above lands on the same question: does the agent understand the data it touches? NVIDIA's Agent Knowledge Foundation, announced at GTC Berlin on October 21, gives one answer. It bundles retrieval models, a new Auto Ontology library and models for structured data, per The New Stack. Auto Ontology is an Apache 2.0 framework that builds a graph-based semantic layer over databases so agents generate SQL with approved definitions and lineage. It exposes that context through an MCP server, a CLI and a REST API.

The open lakehouse community is working on the same problem from the standards side. Apache Ossie defines a portable semantic model. Apache Polaris contributors are preparing to store and validate Ossie models, and they paused that work so it aligns with Ossie's REST API before anything locks in. Iceberg and Parquet are giving vectors and files first-class types. MCP is the common door agents use to reach all of it.

Put those together and the stack for trustworthy agentic analytics comes into focus. Data sits in open table formats on object storage. A catalog governs access. A semantic layer, described in an open standard, tells the agent what each metric means and how to compute it. MCP connects the agent to that layer. Each piece is open, and each can be replaced without rewriting the others. That is the outcome worth pushing for, because the alternative is a semantic layer locked inside one vendor's agent.

What to Watch Next Week

GitHub's first pull_request_target brownout hits on November 9, so expect failed CI runs on public repos that have not migrated. Ossie's lazy consensus on ROUND and division closes the same day. Copilot's model retirements and the second brownout both land on November 16. Watch Google's Gemini API changelog for the first gemini-4-carbon entry, since that, not the announcement, is when Carbon becomes something you can build on. Watch for independent benchmark replication of Mistral Large 4's coding and token-savings claims. And watch the Parquet FINITE_VECTOR vote, which resets its 72-hour clock on the revised commit. Argon's introductory $2 and $10 pricing also ends November 30, so budget for whatever comes next.

Keep Learning

The tools change every week, but the foundations under them move slower. If you want to go deeper on agentic AI, AI-assisted development, MCP, and the open lakehouse that gives agents trustworthy data, you can find all of my books at books.alexmerced.com.

Apache Data Lakehouse Weekly: October 21-28, 2026

Alex Merced — Wed, 28 Oct 2026 14:17:35 +0000

Last week the projects decided what new things the lakehouse should hold: files, vectors and semantic models. This week they worked on whether anyone can trust what those things say about themselves. Iceberg passed votes on tight stats, table constraints and a new REST header value, and opened another on stats-only fields. Polaris worked out who is allowed to skip credential vending, how storage configuration should live in a catalog, and how a transaction should span tables. Parquet kept arguing about what a vector promises and what a null count actually counts. Ossie, the incubating semantic interchange project, started preparing its first release.

Underneath the spec work sat a harder problem that showed up on almost every list. Maintainers on Iceberg, Arrow and Polaris all said, in different words, that AI-generated pull requests and shared CI capacity are now growing faster than the number of people who can review and run them. That theme gets its own section near the end, because it may shape these projects more over the next year than any single spec change.

Apache Iceberg

The v4 spec learns to make promises

The biggest outcome of the week was Anoop Johnson's vote to replace per-column tight_bounds with a file-level tight_stats flag. It passed on October 27 with 9 binding +1s from Amogh Jahagirdar, Daniel Weeks, Eduard Tudenhöfner, Fokko Driesprong, Gang Wu, Russell Spitzer, Ryan Blue, Steven Wu and Szehon Ho. Andrei Tserakhau, Anurag Mantripragada, Hongyue Zhang and Huaxin Gao added non-binding +1s, and nobody voted against it.

The change sounds small but it matters a great deal for query planning. In the earlier design, each column carried its own flag saying whether its lower and upper bounds were exact. The new design puts one tight_stats flag on the data file, and that flag covers null counts, NaN counts and value counts as well as bounds. If the flag is missing, readers must assume the stats are not tight. This is the vote that came out of Anoop's discussion last week about extending "tight" beyond min and max. With it in place, an engine can answer count(*) WHERE col IS NULL from metadata alone and know the answer is exact.

Two details in the spec are worth calling out for anyone who runs table maintenance. Deletion vectors no longer widen stats automatically, so applying a delete does not quietly downgrade a file's metadata. And engines may re-tighten stats without rewriting data files. That opens the door to a cheap maintenance job that recomputes stats on files whose rows were partly deleted, instead of a full compaction. The spec PR is apache/iceberg#18585.

Table constraints arrive, and primary keys replace identifier fields

Huaxin Gao's vote to add table constraints to the v4 spec also passed on October 27, with 8 +1s, 5 of them binding by Huaxin's count. Ryan Blue, Russell Spitzer, Steven Wu and Eduard Tudenhöfner marked their votes binding, and Szehon Ho also voted +1. Andrei Tserakhau, Anoop Johnson and Junwang Zhao voted non-binding.

The spec adds three constraint kinds to table metadata: check, unique and primary-key. Each one carries an enforced flag, so a table can declare a constraint that the writer promises to honor without the catalog trying to police every commit. More precisely, the flag controls whether writers must verify the rows they add or modify. An unenforced constraint is still useful. It tells the optimizer and downstream tools what the data is supposed to look like, which is how many warehouses have long treated declared keys. Each snapshot also records a status for every constraint: validated, valid, invalid or unvalidated. That last part is the clever bit. It means a reader can tell whether a constraint was actually checked for the data it is about to read, rather than trusting a declaration that may have gone stale three commits ago.

In v4 the primary key replaces identifier field IDs, the older mechanism used mostly for upserts and CDC. Upgraded tables get an unenforced primary key built from their identifier fields, so nothing breaks on migration. During review, Ryan raised a question about type promotion for columns referenced by a check constraint. Huaxin's reply says his comments were addressed, so check PR #17822 for the final rule before relying on it. For practitioners, this is the change that will matter most in a year's time. Optimizers in many engines already use declared keys for join elimination and better cardinality estimates, and Iceberg tables will finally have a standard place to declare them.

Stats-only fields head to a vote

Ryan Blue opened a discussion on stats-only fields on October 22 and moved it to a formal vote on October 27. Earlier drafts called the idea non-materialized columns. A new stats-fields list assigns table field IDs to things that are not columns, such as the output of a partition transform or an expression like to_lower_case(name). Their stats can then live in content_stats and drive pruning without ever appearing in the schema.

Ryan explained the name change in his opening mail. "Stats-only fields" says what the feature does, while "non-materialized columns" described what it is not. He also considered unifying the idea with generated columns, such as default-value expressions, and decided against it for the same reason the group kept expressions inside check constraints. Splitting them would mean adding an expression and then a column that depends on it, and that indirection buys nothing. The spec PR is apache/iceberg#18405.

Péter Váry initially wanted to tie the feature to derived or generated columns, which he revived last week. Ryan argued that keeping it out of the schema is simpler and avoids committing the spec to generated-column semantics too early. Péter came around and gave his +1. In the vote thread, binding +1s from Daniel Weeks, Amogh Jahagirdar and Steven Wu arrived quickly. Anoop Johnson and Maximilian Michels added non-binding support. The vote runs 72 hours and should close around October 30.

The proposal already has a follow-up. Andrei Tserakhau started a thread on collation bounds, which would let an engine keep separate min and max values for a case-insensitive or locale-aware ordering of a string column. Ryan prefers a new type with standardized collation metadata over overloading stats-only fields. Either way, the pattern is clear: v4 is turning stats from a best-effort hint into a typed, declared contract.

The File type creates its first follow-up work

Last week the community voted a File type into v4, letting a column hold a reference to an external object such as an image or PDF. This week the follow-ups began. Sung Yun proposed a batch remote signing endpoint for the REST spec. His reasoning is simple. With remote signing, a client asks the catalog to sign each object request. When a table holds one Parquet file per few million rows, that is manageable. When each row can point at its own object, the number of signatures grows with rows instead of files. The design is at s.apache.org/irc-batch-signing and the PR is #18254.

A related REST change passed outright. youngrae kim's vote to allow "none" as an X-Iceberg-Access-Delegation value passed with 7 +1s, including binding votes from Yufei Gu, Sung Yun and Eduard Tudenhöfner. A client can now tell the server it does not want any delegated credentials, for example because it already has its own storage access. This closes a long-running awkward case where a server could not satisfy a delegation request and had no clean way to say so. As the Polaris section shows, it landed just in time for Polaris to use it.

Auth, CDC and correctness

Alex Dutra brought back an idea from a Community Over Code Glasgow session on REST client auth. He proposed a new BearerAuthManager that reads a token from a local file and simply sends it. The work on a second-generation OAuth2Manager stalled on scope creep, so the plan is to shrink that effort and separate who produces a token from who uses it. A sidecar or platform agent refreshes the file, and the Iceberg client stays out of the identity business. The Glasgow group agreed on two tracks. The first is a downscoped OAuth2Manager v2 with minimal functionality, done right and standards-compliant. The second is the BearerAuthManager, which is opinionated, lightweight and has no dependencies. Users who need in-process authentication would keep OAuth2Manager, and everyone else could switch to the file-based approach. Alex is starting with the bearer manager because it is simpler, and he promised a separate thread for the downscoped OAuth2 work. It addresses an issue that has been open for about a year (#13550).

On streaming ingestion, koodin 9 proposed Kafka Connect CDC writes without equality deletes. The sink coordinator would resolve deletes against the base snapshot into deletion vectors and commit the data and DVs together in one RowDelta. Since v4 already forbids writing new equality deletes, the sink needs some answer here. The background matters. An earlier delta writer for the sink (#14797) was never merged, mainly because equality deletes committed by a streaming sink slow down every reader until compaction runs. Flink takes a different route, converting equality deletes into deletion vectors after commit with a maintenance task that keeps a primary key index in Flink state. The new proposal does the lookup before commit with a stateless scan that prunes candidate files by partition and deleted keys. It reads only the key columns and row positions, and hides the lookup behind an interface so a real index can replace it later. For concurrency, the RowDelta validates from the base snapshot, uses the deleted keys as the conflict filter, and checks that the files the new DVs point to still exist, which catches a concurrent compaction. Andrei Tserakhau asked whether the sink should do CDC at all, since append-only ingestion plus a Spark MERGE already works. Péter Váry asked for performance testing first and warned that a slow resolve step could starve concurrent commits.

Two correctness threads deserve attention from anyone maintaining an Iceberg implementation. Marius Grama reported a silent correctness bug, first seen in Trino, where stats on non-key columns in equality delete files wrongly prune delete manifests. The fix in PR #18338 ignores those stats, and Péter's view is that equality deletes are being phased out anyway. Andrei also found that NaN and Infinity do not round-trip in JSON single-value serialization. Java writes the strings "NaN" and "Infinity" but cannot read them back. iceberg-rust, PyIceberg and iceberg-cpp write null and lose the value. iceberg-go refuses to write them at all. Raghvendra Yadav backed standardizing on the strings Java already writes. It is a small bug, but five implementations disagreeing on one value is exactly the kind of drift a multi-language spec has to catch early.

Yuya Ebihara's HLL sketch proposal for Puffin, covered last week, picked up +1s from Denys Kuzmenko, Junwang Zhao, Raghvendra Yadav and Jian Chen. It would add apache-datasketches-hll-v1 next to Theta for distinct-value estimates. The REST OpenAPI thread from Yong Zheng also drew replies from Alex Dutra, Péter Váry and Yufei Gu about making the spec valid OpenAPI 3.1.

Releases, subprojects and security

The only release vote on the list was the Iceberg Terraform Provider. Matt Topol's 0.2.0 RC0 got a full verification and +1 from Xuanwo, then a -1 from Alex Stephen because LICENSE and NOTICE files were out of date. Matt cut RC1 on October 26. Version 0.2.0 matters because it is the release that lets the provider be listed in the Terraform Registry, which makes managing Iceberg catalogs as infrastructure code much easier.

The Rust sync notes, posted by Danny Jones, say the group agreed to ship 0.11 from the current branch "ASAP" without recutting. The DataFusion Iceberg integration has moved out of the repo and will see its first release under DataFusion governance. Matt opened a v4 tracking issue, iceberg-rust#3513, and the group discussed a File Format API like the one in Java, with a Vortex proposal doc linked. The group also agreed to start a roughly three-month community roadmap modeled on DataFusion's, partly to give contributors a clear signal about which PRs are wanted. Kevin Liu uploaded the recording to YouTube.

For Python users, Orest Tokovenko proposed dropping Python 3.10 from PyIceberg. Python 3.10 reached end of life on October 1, and PyArrow 26 ships no 3.10 wheels. The proposal sets requires-python >=3.11 and raises the PyArrow floor from 18 to 26 in one PR. If you pin old PyArrow versions in production, now is the time to plan an upgrade.

Russell Spitzer also posted a Confluent connector vulnerability update. The vulnerable 1.9.2 Kafka Connect zip on Confluent Hub was uploaded by Confluent, not built by the project, so the project will not patch that artifact. PR #15212 adds an official runtime zip instead. The only remaining scanner findings are Jackson CVEs inside Parquet's bundled copy, which ties neatly into the parquet-java 1.18.2 release discussed below.

Community health

The thread most worth reading in full is Russell Spitzer's Iceberg Community Health Stats. Russell built a dashboard at russellspitzer.com/iceberg-community-health and flagged a drop in contributors who come back for a second PR after their first, especially in the Java repo. He took the idea from Patrick McFadin's Cassandra project health dashboard, which he saw at Community Over Code. Russell said he has no prescriptions yet, only worry about the decline in "First to Second" contributors across the ecosystem. He was also open about how he built it, describing the site as "pretty viciously vibe-coded." The data and code are on GitHub, and he invited people to dig in. Jared Yu said AI-generated PRs are burying a small group of maintainers, which slows reviews for everyone, including genuine first-timers. Several people said Russell's original mail landed in spam, and Kevin Liu is chasing ASF Infra about delivery. Hongyue Zhang asked to put the dashboard on the community sync agenda. Yufei Gu and Ryan Blue also joined the discussion.

On the event side, Talat Uyarer finalized the Bay Area meetup agenda for November 2 at Google in Sunnyvale. Anurag Mantripragada announced that the Iceberg and Spark sync goes monthly at 9 AM Pacific, starting November 11.

Apache Polaris

1.9.0: scope settles, the date holds for now

Polaris 1.9.0 is still in the scoping thread JB Onofré opened on October 4. This week Yufei Gu asked to include PR #5263 because it fixes a potential table corruption issue. Dmitri Bourlatchkov added #5856 to the milestone. JB agreed to both and still wants Directories in the release as a beta feature. He wrote that he wants to put the release to a vote before Halloween, even after losing power and internet in a storm midweek. As of October 28 there was no release candidate.

If Directories ship as beta, 1.9.0 will be the first Polaris release where the catalog governs raw object-store prefixes next to tables. Last week's review raised real concerns, including Ankur Kotwal's scanner bug and Tornike Gurgenidze's argument that the inventory table should be the source of truth. So the beta label will matter. Treat it as a preview to test, not a feature to put production pipelines on.

Storage access gets more flexible

The busiest design area was how Polaris talks to object storage. Dmitri Bourlatchkov proposed typed storage config options to skip credential vending on Azure and GCS. They are AzureStorageConfigInfo.skipSasTokenGeneration and GcpStorageConfigInfo.skipCredentialDownscoping, matching the existing AWS stsUnavailable option. Ayush Saxena weighed in, vignesh a implemented it in PR #5884, and Dmitri linked it directly to the Iceberg "none" vote. Together they give a clean path for engines that already hold their own storage credentials, such as workloads running inside a trusted network.

At the opposite end, Tommy Bollhofer described a real deployment that needs more than credential vending can offer. His team runs Polaris against an on-prem FlashBlade whose STS supports only AssumeRoleWithWebIdentity. As a result they hand static storage keys to every engine, which defeats the point of a governed catalog. The keys go to every service in every environment: Polaris itself, Trino, Spark for table maintenance, and PyIceberg and DuckDB clients. Each new environment adds rotation work and widens the blast radius of a leaked key. Tommy also noted his client versions are mixed. Trino runs on Iceberg 1.12, Spark on the 1.10 runtime, and PyIceberg and DuckDB use their own Iceberg implementations, so client-side support for remote signing will vary. His longest jobs are Spark maintenance runs that take minutes, so he thinks remote signing could ship without token refresh support at first. His team offered to test early builds against FlashBlade and report which engines work. Alex Dutra judged that enough evidence to restart remote signing and is preparing an experimental PR on top of the earlier work. Combined with Sung Yun's batch signing proposal in Iceberg, remote signing looks set to become a first-class option rather than a niche one.

Yufei Gu also changed direction on multiple storage configurations per catalog. After Srinivas Rishindra's input, the group agreed to skip a transitional phase and make storage configuration its own catalog-scoped entity with REST endpoints. The open question is how a table binds to one. Yufei prefers a table property set at create time over a "pending binding" model. Separately, Yufei proposed blocking a risky config change: flipping ALLOW_UNSTRUCTURED_TABLE_LOCATION from true to false can strand renamed tables behind 403 errors.

Transactions, sharing and discovery

Qiegang Long's multi-table transaction work moved from proof of concept to design. He opened umbrella issue #5833 and posted the design doc. Engines would get BEGIN ... COMMIT across statements and tables. Writes are staged on Iceberg table branches and published with the existing multi-table commit API. The proof of concept runs Spark and Trino at READ COMMITTED and REPEATABLE READ isolation. Ankur Kotwal argued for a server-side publish step, citing the gaps he hit while adding commit_transactions to PyIceberg. His examples were concrete. PyIceberg can fast-forward a branch but cannot cherry-pick, so a Python client could not re-apply appends onto a main branch that has moved. He also found that a delete matching no rows staged nothing, so the transaction went out without its "main has not moved" check. Gaps like that are easier to close once on the server than in every client. On snapshot expiry, Ankur suggested rejecting snapshot removal while a registered read point still covers it, while keeping a 410 response for unregistered handles so routine expire jobs do not start failing. An Iceberg cross-post is coming, since any real answer has to work for every REST catalog, not just Polaris.

The long-running Open Sharing design also moved. Dmitri Bourlatchkov pushed for a Shares API that does not assume any particular authentication scheme. Dennis Huo split the work into two API specs, one for shares and membership and one for consumers, listings and credentials. Madhan Neethiraj, who works on Apache Ranger, asked how authorization would work. Dennis answered that every layer becomes pluggable: what is shared, who can read it, how it is authorized and how consumers authenticate.

Dennis also proposed a catalog discovery endpoint, GET /api/catalog/polaris/v1/catalogs. Today a client with only a catalog role needs to know the catalog name before it can configure itself. Polaris already has a listCatalogs call, but it is a management API. It needs a root-level privilege and returns the full catalog model, including storage configuration. The new endpoint is for data-plane readers. At first it would return only catalog names, the value a client passes as the warehouse to the REST /config endpoint. Dennis also tied it to the in-progress Shares API, where a consumer needs some way to discover what has been shared with it. JB supported the idea, suggested the name listAccessibleCatalogs, and set a simple rule: a catalog is listed if and only if getConfig would succeed for that caller. He also asked for paging from day one. This is a small feature that would make life easier for tools and agents that connect to Polaris without being told where to look.

Labels, semantic models and other threads

The IRC Labels design settled on option B, in which Iceberg REST labels are a read-only view over Polaris tags. Dmitri clarified that importing labels from federated catalogs happens only when sub-catalog RBAC for federated catalogs is enabled, and Andrei Tserakhau agreed.

Arnab Karmakar asked for review of PR #5205, which validates semantic models against a bundled Apache Ossie JSON Schema, version 0.2.0.dev0, with a 1 MiB size cap. It is an early sign of Ossie definitions being consumed inside a catalog, which is the whole point of a semantic interchange standard.

Not every thread was about features. vignesh a proposed dropping metrics reports instead of returning HTTP 500 when the reporter's database write fails. Yufei objected that the Iceberg spec never says metrics are best effort, and that some deployments feed metrics into billing or auditing. He wants the question raised with Iceberg first. Harshita Joshi asked for review of a fix for staged creates, used by DuckDB, which were dropping a catalog's default table properties. Prithvi S explored opt-in table soft-delete, and Dmitri moved to soft-roll authorization on GET /v1/config.

The AI review bot thread from last week continued. Robert Stupp asked which tool runs the reviews and who pays. Yufei answered that the tool is his own project, can use more than one model provider, and is funded by his employer, and that ASF Infra sponsorship is being explored. Every comment will be labeled as AI-generated. vignesh a worried that the bot misses decisions made on the list and in syncs, and that newer contributors will treat its comments as required.

Apache Arrow

A busy release week

ADBC Libraries 25 passed its vote with 6 binding +1s from Bryce Mecum, Curt Hagenlocher, Gang Wu, Matt Topol, Raúl Cumplido and Sutou Kouhei, plus a non-binding +1 from Xuanwo. David Li announced it on October 26. The headline is the C# core reaching 1.0.0, with C# drivers at 0.25.0. The C, C++, GLib, Go, Python and Ruby packages are at 1.13.0, and Java, R and Rust at 0.25.0. Bryce worked with Dewey to get three R packages, adbcdrivermanager, adbcpostgresql and adbcsqlite, accepted on CRAN. npm publishing still needs rework.

On the Rust side, Jeffrey Vo ran votes that delivered patch releases on three older lines: 57.3.2, 58.4.1 and 59.3.2, each passing with 3 binding +1s. That completes the multi-branch backport plan Andrew Lamb announced last week. Jeffrey also ran a vote on 60.1.0, a minor release on the newest line, which passed on October 28 with binding votes from Jeffrey, Andrew Lamb and L. C. Hsieh. Xuanwo found that the release audit script fails on a missing license header in .config/nextest.toml, which Jeffrey will fix next release. Matt Butrovich tested arrow-pyarrow on Python 3.15 using PyO3's forward compatibility flag.

Arrow Java 20.0.0, which JB flagged as imminent last week, saw no vote. No list message explains the delay. On GitHub, a release-candidate CI job for macOS on ARM has been failing since a bundled protobuf upgrade, and a fix PR is open. That is the likely cause, but it is an inference, not a statement from the project.

Funding for PyArrow

Alenka Frim announced that the Sovereign Tech Fund is investing in PyArrow. Germany's Sovereign Tech Agency will fund about 1.9 full-time roles for the rest of 2026 and all of 2027. The applicants were Alenka Frim, Nic Crane, Rok Mihevc and Tadeja Kadunc. The money covers PyArrow type annotations, the long-standing bug backlog, documentation, extension type support and tooling to help maintainers handle agentic PRs. Alenka said the contract funds work that has stalled and is unlikely to attract regular corporate sponsors. Alenka, Rok and Tadeja have already started on type annotations, building on work Rok has invested in for a long time. Nic is working through bug reports that have sat open for years. In 2027 Alenka plans to research building the full documentation without PyArrow installed, then improve the Python User Guide, API examples and Cookbook. Tadeja will focus on extension types and related compute kernels. Open Source Collective is fiscal host, on behalf of Arctos Alliance. Curt Hagenlocher, Antoine Pitrou and Raúl Cumplido replied with congratulations. For a library that sits under pandas and much of the Python data stack, paid maintenance time is good news for everyone downstream.

AI contributions and CI pressure

The AI contributions thread picked up again. Raúl Cumplido opened a PR to write down reviewer guidance. Nic Crane posted an impact report that found about a third of recent PRs skip AI disclosure. His seven recommendations include moving the PR cap into .asf.yaml before November, mandatory disclosure, and a bot that converts non-disclosing PRs to drafts. He also suggested giving active non-committer reviewers the collaborator role so they can help triage. Raúl pointed to CPython's new AI triage labels as a model.

Nic's report is worth reading in full because its diagnosis applies well beyond Arrow. He named three problems. Maintainers spend time reviewing PRs that are later abandoned, or that need more rounds of feedback because they were mostly AI-generated. Maintainers also spend time enforcing AI rules, while non-committer reviewers have little power to flag problem PRs. And the volume itself keeps growing. His other recommendations include adopting a rule against AI-written review replies and auto-drafting PRs that sit idle for 14 days with conflicts, failing CI or requested changes. He also suggested gating PR opening on triage state, as pandas does, and tracking monthly how many new-contributor PRs get no human response. He singled out growing the merge group from people already triaging as the only measure that adds capacity rather than cutting inflow. Nic noted, with some irony, that he used AI to help analyze the data.

Jarek Potiuk opened the most heated thread of the week on making Arrow's GitHub Actions CI lighter. Using an Apache Magpie workflow-optimization skill, he estimated Arrow used 110,000 to 125,000 runner minutes the previous week, with Linux packaging about 40 percent of that. He suggested 30 to 40 percent savings, partly by running extra workflows nightly instead of per PR. He also noted that pull_request_target is being disabled for ASF repos in November. That change affects three Arrow workflows, including the PR limit bot. Jarek suggested replacing the bot with the .asf.yaml creation_cap setting, which enforces the same limit of three open PRs per contributor with no workflow at all. He also spotted that nightly failure reports had been silent since October 21. The report job installs the latest Python 3, now 3.15, and a dependency has no 3.15 wheel yet, so the job fails before sending email. Pinning Python 3.13 fixes it. Benson Muite suggested Fedora's Packit for RPM builds. Raúl disputed one of Jarek's findings, saying nightly reports do work. Andrew Lamb reported 3 to 7 hour CI queue waits for arrow-rs and DataFusion contributors, and his numbers showed Airflow using far more shared runner time than the Arrow repos. The exchange got sharp before Jarek apologized on October 27. The underlying issue is real: the ASF's shared runner pool is now a bottleneck that contributors feel directly.

Flight SQL and community

Tornike Gurgenidze's Flight SQL proposal to advertise a server's Substrait dialect continued. Curt Hagenlocher asked whether clients could negotiate a dialect version. Tornike replied that GetSqlInfo cannot take parameters, so negotiation would need a new RPC. He argued the existing minimum and maximum Substrait version codes are enough for now. Ian Cook posted logistics for the October 21 community meeting.

One housekeeping note: mail-archive.com is missing several Arrow messages from this week, including the 60.1.0 vote opener and some of Andrew Lamb's replies. JB reported on the Ossie list that some ASF list mail was not being delivered and that Infra was working on it, which likely explains the gaps.

Apache Parquet

parquet-java 1.18.2 becomes the release to watch

Manu Zhang proposed a parquet-java 1.18.2 patch release on October 26. The 1.18.x branch already carries several fixes. They stop writers from flushing incomplete files after a failure and fix a dictionary lookup collision between dotted field names and nested paths. They also correct Bloom filter column path handling, release row-group buffers properly and fix protobuf handling of empty and recursive messages. The release also bumps Jackson to 2.22.3 to clear CVEs. Steve Loughran asked to include master's Thrift and Snappy upgrades, Fokko Driesprong backed the release, and Manu opened backport PRs. Divjot Arora also replied in support.

This matters beyond Parquet. The Jackson CVEs inside Parquet's bundled copy are the last scanner findings blocking a clean Iceberg Kafka Connect runtime, as Russell Spitzer noted on the Iceberg list. Meanwhile the 1.19.0 release candidates have sat unreleased since September, so 1.18.2 is the realistic near-term upgrade for Java users.

The vector debate goes another round

The numeric vector thread, started by Rok Mihevc in September, kept going. Jan Finis proposed an alternative: put finiteness on float elements and an optional fixed size on LIST, rather than adding a new FINITE_VECTOR type. He argued that design stays forward compatible, while a new union member would make older engines reject the column. Ryan Blue held to FINITE_VECTOR. He floated allowing both LIST and FINITE_VECTOR annotations on the same column, and argued that types carry guarantees, as UTF-8 does for strings. Finiteness lets readers skip per-element checks and catches bad data at write time.

Curt Hagenlocher pointed out that SQL Server, MySQL and SAP ASE already reject non-finite floats. Will Edwards suggested putting VECTOR on hold until there is a broader proposal for ML tensors and sparse or quantized data. He warned that writers that do not know VECTOR will rewrite files as plain lists, and repeated that Iceberg may be the better home for finiteness. Fokko Driesprong said the group is "running in circles," while noting people are willing to do the work and that Parquet should be self-contained. Rok's format PR #624 is still open. For anyone storing embeddings in Parquet today, nothing changes yet, but the decision will shape how Iceberg's own vector type is defined.

Precise definitions and safer readers

Andrew Bell asked a deceptively simple question about the difference between num_values and num_rows in DataPageHeaderV2. Adam Reeve explained that num_values counts leaf-level values including nulls, num_rows counts logical rows, and V2 pages never split a row. Antoine Pitrou went further. He noted that the null-count fields are ambiguous at nesting levels deeper than one and proposed precise definitions, in which num_values really counts levels. This could become a spec clarification, which would help every implementer who has had to reverse-engineer the answer from parquet-java.

Antoine also continued his fuzzing thread. Steve Loughran asked which code areas fuzzing hits most, and Antoine named three bug classes: unvalidated metadata such as negative lengths or invalid enums, memory allocations sized from metadata instead of actual file size, and unchecked integer overflow in arithmetic on metadata values. It is a useful checklist for anyone writing or auditing a Parquet reader. His Community Over Code Glasgow talk is linked in the thread. Julien Le Dem posted the October 21 sync logistics.

Apache Ossie

First release prep and a new committer

The incubating semantic interchange project took two steps toward maturity. Josh Klahr announced Yufei Gu as a new committer, with congratulations from Russell Spitzer, Dmitri Bourlatchkov, Chris Eubank, Ayush Saxena and others. Yufei's Polaris background is useful here, since Polaris is already validating semantic models against Ossie's schema. After the converters moved to their own repository, JB Onofré started the release-prep PRs for the first core release: version properties, source release, distribution and Maven publication. No vote has been called yet.

The spec takes shape

Chris Eubank put the Wide Table Query Interface under lazy consensus through November 4. A Layer 3 query names dimensions, metrics and a filter, and the engine lowers it to Layer 2 using AGG(). Justin Talbot proposed reserving AGG() for evaluating measures and letting dataset-scoped metrics reference source columns directly. Together these define how a BI tool or AI agent asks for "revenue by region last quarter" without writing SQL against physical tables.

John DaCosta asked what Dataset.source should guarantee. Chris said the group agrees on a source discriminator, table, query or file, with connection details kept in vendor extensions. He framed the core goal as sharing semantics across tools while querying data where it already lives. A real ERP export then started a debate over which time field slices a metric, after a 14 versus 15 mismatch traced back to that choice. Julian Hyde argued the time hint belongs on the dataset, not the metric, except for semi-additive measures like balances.

The expression portability work continued too. Richard SG Kim migrated 176 expressions to the portable form and found it 22 to 27 percent shorter. Chris told contributors that the ClickHouse and Trino dialect PRs only need a rebase, since adding a dialect is no longer a core spec change. Johannes Zemlin shared an initial ANTLR4 grammar for Ossie expressions, and Ayush Saxena fixed NOT-operator precedence. Ossie has no mail-archive.com mirror, so these links go to lists.apache.org.

Cross-Project Themes

Metadata you can trust

Look at the week's spec decisions side by side and they share one goal. Iceberg's tight_stats flag says the counts are exact. Table constraints say whether a key was actually validated for this snapshot. Stats-only fields give expression stats a declared identity. Parquet's FINITE_VECTOR debate is about whether a type should guarantee its contents. Antoine's null-count definitions would make a page header mean one thing in every reader. In each case, the project is turning metadata that is "probably right" into metadata that is guaranteed.

That shift matters most for automated consumers. A human analyst can sanity-check a suspicious number. A query planner skipping files, or an AI agent answering a question from metadata, cannot. The more of the lakehouse that carries guaranteed facts, the more work engines and agents can skip safely, and the cheaper and faster the whole stack gets. Ossie's spec work fits the same pattern one layer up, where semantic definitions become portable contracts rather than tool-specific settings.

Storage access is being rethought from both ends

Iceberg's "none" delegation value and Polaris's skip-vending options cover clients that already have storage access. Remote signing in Polaris and batch signing in Iceberg cover the opposite case, where clients should never hold storage credentials at all. The File type is forcing the question, because a column of object references multiplies the number of storage requests. Expect the REST spec and Polaris to keep moving together here.

Maintainers under pressure

The most important theme was not a spec. Russell Spitzer's dashboard shows fewer returning contributors in Iceberg Java. The Iceberg Rust group wants a roadmap partly to manage AI-generated PRs. Nic Crane found a third of Arrow PRs skip AI disclosure. Polaris is trialing an AI reviewer while contributors worry about its blind spots. Arrow's CI thread shows the shared ASF runner pool is a real bottleneck, and the Sovereign Tech Fund grant explicitly pays for tooling to handle agentic PRs.

Contributions are getting cheaper to produce, while review time and CI minutes are not. These projects are building the open infrastructure that AI agents will read from, and they are also among the first communities to feel what agents do to open source maintenance. Expect disclosure rules, PR caps and funded maintainer time to spread across ASF data projects quickly. If you contribute to any of them, disclose AI assistance and keep PRs small. It is the easiest way to keep your work out of the triage pile.

Looking Ahead

Watch for the stats-only fields vote result around October 30 and a Polaris 1.9.0 vote by October 31, including whether Directories ship as beta. A parquet-java 1.18.2 vote, the iceberg-rust 0.11 release, binding votes on the Iceberg Terraform Provider 0.2.0 RC1 and Arrow Java 20.0.0 are all possible in the next two weeks. Ossie's Wide Table lazy consensus closes November 4, and its first core release vote could follow. The Iceberg Bay Area meetup runs November 2 in Sunnyvale, and the monthly Iceberg and Spark sync starts November 11.

Resources & Further Learning

Get Started with Dremio

Try Dremio Free: Build your lakehouse on Iceberg with a free trial
Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow: Learn how Dremio brings the open lakehouse stack together

Free Downloads

Apache Iceberg: The Definitive Guide: O'Reilly book, free download
Apache Polaris: The Definitive Guide: O'Reilly book, free download

Books by Alex Merced

AI Weekly: GPT-6.1 Terra, Agent Skills and Anthropic's S-1

Alex Merced — Wed, 28 Oct 2026 14:01:55 +0000

Week of October 21-28, 2026

OpenAI shipped a cheaper mid-tier model built for computer use. Anthropic handed Agent Skills to a neutral foundation and then filed to go public. Its S-1 put real numbers on what frontier AI costs to run.

Coding agents also kept moving approval decisions from people to classifiers. Here is what changed, what it costs and what you need to do about it.

Models: GPT-6.1 Terra Cuts the Mid-Tier Price in Half

This was a mid-tier week, not a flagship week. OpenAI released GPT-6.1 Terra, Moonshot shipped Kimi K3.1 Pro, and MiniMax posted open weights. Anthropic released no new public model. It did describe an unreleased research model that built working exploit chains, and it launched an agent app on top of Opus 5.5.

OpenAI GPT-6.1 Terra (October 26)

OpenAI released GPT-6.1 Terra on October 26. The API model ID is gpt-6.1-terra, and it is the only snapshot so far. OpenAI pitches it for visual reasoning, computer use and clear writing. It sits between GPT-6 Luna below and GPT-6.1 Sol and GPT-6 Astra above.

The model card lists these specs:

Context window: 1,050,000 tokens, with a max input of 922,000 tokens
Max output: 128,000 tokens
Knowledge cutoff: June 1, 2026
Input: text and image. Output: text only
Reasoning effort: none, low, medium (default), high, xhigh and max
Endpoints: Chat Completions, Responses and Batch. Tool calling needs the Responses API

Pricing is the headline. Standard rates are $1 per million input tokens and $5 per million output tokens. Cached input costs $0.05 per million, and cache writes with 30-minute retention cost $1.25 per million. OpenAI says that is 50% below GPT-5.6 Terra and one-tenth the per-token price of GPT-6 Astra.

Read the fine print on long prompts. Prompts over 272,000 input tokens cost 2x on input and cache rates and 1.5x on output. Batch and Flex run 50% below Standard. Regional processing adds 10%, and fast mode costs 2x Standard.

The benchmark numbers all come from OpenAI, so treat them as vendor-reported. Decrypt reports that Terra solved 50.3% of Terminal-Bench 4.0 tasks at max effort, up from 21.5% for GPT-5.6 Terra. On an internal test of prompts built to trick models, the factual error rate at low effort fell from 53.6% to 16.7%. At max effort the error rate dropped to 4.3%. OpenAI also says Terra matches GPT-6 Astra on Agents' Last Exam and finishes tasks more than 3x faster than Astra and Sol on average. OpenAI posted no numeric score for that Agents' Last Exam claim.

Terra is live in ChatGPT Work and Codex for Plus, Pro, Business, Enterprise and Edu users, per the Codex changelog. It is not available in Chat, and it does not support UAE inference residency. GitHub added Terra to Copilot the same day.

One practical detail came from the developer forum, not OpenAI. A user reports that reasoning.effort: "none" on Terra accepts temperature and top_p, which GPT-6.1 Sol blocked. That is unconfirmed, so test it before you rely on it.

What does Terra change? It makes computer-use agents cheaper to run at volume. A model that reads screens and fills forms at $1 in and $5 out lets you put an agent on back-office work that did not pencil out at Sol prices. OpenAI itself still points heavy research and complex coding to Sol or Astra.

Moonshot Kimi K3.1 Pro (October 23)

Moonshot released Kimi K3.1 Pro on October 23. The quickstart guide lists two model IDs: kimi-k3.1-pro and kimi-k3.1-pro-highspeed. Moonshot says the highspeed variant runs 2-3x faster. Context is 1M tokens. Output defaults to 131,072 tokens and can go as high as 1,048,576.

Reasoning is always on. You pick low, high or max through a top-level reasoning_effort field, and max is the default. You cannot turn thinking off. The model takes image and video input and supports tool calling and strict JSON Schema output.

Moonshot's guide says pricing is flat, with no tiering by context length. Its pricing page did not render the K3 table when we checked. Simon Willison reported $2 per million input tokens and $10 per million output tokens. Confirm those numbers in the Kimi console before you budget against them.

The benchmark story is messy. On launch week, Moonshot's own quickstart still showed placeholder values ("xx.x%") for Terminal-Bench 4.0 and Terminal-Bench-Science. The one hard number in the guide is a finance hallucination test. Moonshot says its claim-level hallucination rate fell from 23.35% on K3 to 6.85% on K3.1 Pro. That figure is vendor-reported, and Moonshot calls the test a stress test, not a general error rate.

Secondary coverage cites a Moonshot figure of 65.7% on Terminal-Bench 4.0, against 66.4% for Claude Opus 5.5. We did not find that number on a Moonshot page, so treat it as unverified. Press reports also put K3.1 Pro at 55 on the Artificial Analysis Intelligence Index.

Moonshot says full weights will ship by November 4. It has not named a license. If you plan to self-host, wait for the license file before you commit.

MiniMax M3.1 Flash Weights Go Public (October 23)

Last week we noted that MiniMax promised weights for M3.1 Flash. They landed. The repo MiniMaxAI/MiniMax-M3.1-Flash is public and ungated on Hugging Face. Coverage puts the model at 428 billion total parameters with 23 billion active per token, a 1M context and text, image and video input. Hugging Face lists about 220 billion stored parameters in the safetensors files, so check the model card for how MiniMax counts.

The license is not Apache or MIT. Hugging Face tags it as "other." Coverage describes a community license that requires written permission from MiniMax for products above $20 million in yearly revenue. One write-up also says commercial products must display "Built with MiniMax M3.1-Flash." Read the license file in the repo before you ship anything on it. MiniMax recommends SGLang, vLLM and TokenSpeed for serving.

Anthropic: Duet, Google Cloud and an Unreleased Cyber Model

Anthropic shipped no new public model this week. It did three things that matter for model users.

First, Anthropic introduced Claude Duet on October 21. Duet is a proactive agent app built on Claude Opus 5.5. We cover it in the Tooling section below.

Second, Claude Models on Google Cloud (Anthropic managed) went live on October 22. Anthropic runs the Claude API, and you reach it through your Google Cloud project. Google Cloud IAM controls access, and usage lands on your Google Cloud bill and draws down existing spend commitments. The release notes confirm fast mode for Opus 5.5 is available there too. The older Vertex AI offering is now called Claude Models on Google Cloud (Google managed). Pick that one if you need Google as the data processor or strict regional controls.

Third, Anthropic published a research post on October 27 about an unreleased internal research model. Anthropic says the model found and chained flaws in Chrome, Safari, Android, Windows and Linux into proof-of-concept exploits. Google assigned 62 Chrome CVEs, all rated critical or high, including 13 sandbox escapes. On Linux, Anthropic says the model took 48 minutes to leak a kernel pointer and about five more minutes to reach shell execution, through CVE-2026-98432.

These claims come from Anthropic's own account. The Hacker News notes that it is hard to tell how much of each chain the model built alone. All flaws were reported and patched. Anthropic did not say whether it will release the model. It did point to its Cyber Verification Program, which gives vetted security teams access to stronger cyber capabilities.

The lesson for everyone else is simple. Patch cycles need to get shorter. If a model can turn four Chrome bugs into a sandbox escape in a lab, attackers will try the same thing.

Google: Genie 4 and Two Gemini API Changes

Google DeepMind introduced Genie 4 on October 27. It is a real-time world model that generates audio, ambient sound and dialogue along with the scene. You steer it with natural language and keyboard controls. Google uses Gemini 3.8 Flash as a guiding agent to keep scenes coherent. Google gave no resolution, frame rate or session length. It says details "can still shift or drift over time." Genie 4 is a research preview for trusted testers, with Google AI Ultra and Gemini API early access to follow.

Two quieter changes in the Gemini API changelog matter more for data teams. On October 26, Google turned on request storage by default for new paid-tier projects. Calls to generateContent, streamGenerateContent and batchGenerateContent now default to store=true. Existing projects keep their current setting. The changelog does not say how long Google keeps stored requests.

If you send customer data or regulated data through new Gemini projects, set store=false per request or turn off logging in the Logs and Datasets settings. Do it before your first production call, not after.

On October 27, Google released the Gemini API CLI (gemini-api) in beta. It is a stateless, zero-dependency command-line client for coding agents, scripts and CI pipelines. It works with the Interactions API and can run Gemini managed agents.

OpenAI Platform: Decisions API and Service Tier Controls

The OpenAI developer changelog had two smaller entries. On October 21, OpenAI added settings to restrict which service tiers each project or a whole organization can use. That helps you stop a test project from burning money on a priority tier.

On October 22, the Decisions API reached general availability on gpt-6-luna. Each request now handles up to 400 questions, up from 200 in beta. Shared input can be cached across requests for prompt-cache discounts, and you can pass structured data with input_json.

Quiet Labs

We found no new model from Meta, xAI, Mistral, DeepSeek or Qwen dated in this window. DeepSeek's news page has nothing after its October 17 Flash update. xAI's release notes show only the Grok Voice 3.0 items we covered last week. Mistral's Large 4 weights deadline of October 31 is still ahead.

The community filled some of the gap. ISTA-DASLab posted GGUF quantizations of DeepSeek-V4.1-Flash-1017, including a 2-bit build. Press reports say it runs on a single 12GB GPU. We did not test that claim.

What This Means for Practitioners

The price floor keeps dropping in the middle of the market. A frontier lab now sells a 1M-context computer-use model at $1 in and $5 out. Open weights keep arriving, but with custom licenses that cap revenue or require sign-off. And defaults keep shifting under you, like Gemini's new storage setting. Read changelogs every week, not every quarter.

Tooling: Approval Classifiers Go Mainstream

The tooling theme this week was who says yes. GitHub made LLM-judged approvals generally available. Anthropic launched an agent that asks before it pays or signs. Claude Code and Codex both tightened permission defaults. The common pattern is a classifier that approves safe actions and stops for a human on the risky ones.

GitHub Copilot Assisted Permissions Reach GA (October 26)

GitHub made assisted permissions generally available in Copilot CLI, the Copilot app and VS Code. When Copilot wants to run a command, touch a file path or open a URL, an LLM judges the request. Requests it rates "clearly safe" run on their own. Everything else waits for you.

Copilot now has three approval modes:

Manual: always ask
Assisted: the classifier approves safe requests
Allow All: approve everything

You turn it on with /permissions in the CLI, the Tool permissions picker in the app or the permissions picker in VS Code. GitHub did not publish the classifier's criteria.

The admin controls are the important part. Enterprise settings can block Allow All while keeping Assisted. Admins can also disable Assisted, set the default mode for new sessions and fall back to Manual when the default is not allowed. Platform teams finally get a middle path between "ask every time" and "approve everything."

Copilot Server-to-Server Auth and Usage Metrics

On October 27, Copilot added server-to-server authentication in public preview. Agents built with the Copilot SDK can now call Copilot for an organization without a signed-in user. You grant a GitHub App the read-only Copilot Requests permission, install it on the org and pass its short-lived installation token to the SDK.

Billing follows the app. Usage is billed to the organization that owns the installation, and that org's model policies, budgets and spending limits apply. Owners can spot app usage in AI usage CSV reports through the app_client_id field. This is how you run a Copilot-powered bot in CI without tying it to one employee's seat.

GitHub also widened its usage data. On October 22, the usage metrics API started reporting custom skills, agents and MCP servers. On October 23, it added usage from tools like OpenCode. A global default for Copilot features also reached GA on October 22.

GPT-6.1 Terra arrived in Copilot on October 26 for Pro, Pro+, Max, Business and Enterprise plans. GitHub says it bills "at provider list pricing under usage-based billing." New models turn on by default unless an admin disables the global default or the model itself. If your org pins models, check your policy before the rollout reaches you.

GitHub Universe runs October 28-29. Expect more Copilot news next week.

Claude Code 2.1.307 Through 2.1.312

Claude Code shipped six releases between October 21 and October 27, according to npm. The changelog leaned toward permissions, sandboxing and cloud sessions.

On approvals and safety, these changes stand out:

Bypass mode and rm: In the Desktop app and VS Code, the dangerous rm and rmdir prompt in bypass permissions mode now denies the command at once with a rewrite hint. Unattended sessions keep going instead of hanging.
PowerShell fix: Some Set-Location and cd commands with extra arguments were auto-approved. They now prompt.
Windows hook fix: PreToolUse and PermissionRequest hooks let some tool calls through when Git Bash was missing. That is fixed.
ultrareview trust: claude ultrareview now requires workspace trust. Scripts must pass --trust-workspace.
Sandbox fallback: Commands excluded from the sandbox now ask before writing to protected paths when the sandbox is on but cannot run.

Telemetry went back and forth. Version 2.1.307 added a claude_code.safety_event OpenTelemetry event for every tool call that a rule, auto mode, hook or sandbox stopped. Version 2.1.312 stopped sending that event six days later. If you built a dashboard on it, check it now.

Version 2.1.307 also added /claude-api api-version, which moves your Claude API code to API version 2026-10-21. That pairs with the breaking API release we covered last week.

On models and cost, ultrathink in the terminal now runs its turn at xhigh effort on Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 through Anthropic's API. Usage messages now say "5-hour limit" instead of "session limit." On Bedrock, Vertex, Foundry and Mantle, Claude Code now loads model settings from a signed file on downloads.claude.ai. You can turn that off with CLAUDE_CODE_MODEL_CATALOG=off.

For enterprises, 2.1.312 added oidc.user_id_claim to the Claude apps gateway. You can now identify people by a claim other than sub, such as Microsoft Entra ID's oid. Version 2.1.310 added cloud sessions and Remote Control permissions to Enterprise custom roles.

Codex CLI 0.164.0 and 0.165.0

OpenAI shipped Codex CLI 0.164.0 on October 23. Credentials got the most attention:

Enterprise MCP connections can now sign in with eligible Codex account grants or a dedicated sign-in, including OAuth clients that need secrets.
Codex masks credentials during supported remote Bash and Zsh snapshot sessions, including credentials from custom providers.
On Windows, HTTPS and WebSocket connections now answer proxy challenges with the user's Negotiate or NTLM credentials.
Queued prompts now wait for pending permission choices before they continue.

The release also added /presets, ten numbered slots for saved prompts and slash commands. You can now set model, effort, speed and permission profile inline with /model and /permissions. The Amazon Bedrock setup wizard is on by default.

Codex CLI 0.165.0 followed on October 27. It added GPT-6.1 Terra to the model catalog and a built-in Google Cloud Vertex AI provider, set with model_provider = "gcp". Reasoning effort changes mid-conversation are now on by default for supported models. Memory v2 is the default for fresh profiles and can import memories from other agents.

Two fixes matter for approvals. Fresh TUI sessions now honor the server's default approval reviewer. And Codex closed a marketplace URL validation gap that let Git connect to a host other than the one checked against allow patterns.

Claude Duet: An Agent That Asks Before It Pays

Anthropic introduced Claude Duet on October 21 as a research preview. Duet runs on Opus 5.5 in its own cloud virtual machine with a browser, so your laptop can be closed. It sends a morning brief from your connected apps and checks them every few hours. It messages you only when something needs a decision.

The approval model is the interesting part. Duet reads and drafts on its own. Before it sends a message, makes a payment, signs a document or deletes anything, it shows you the action and waits. Anthropic says an auto mode classifier handles those approval checks. Two actions are off-limits even with approval: trading securities and moving money between accounts.

Anthropic also published a safety and privacy note on October 23. A dedicated classifier blocks sharing of card numbers and government IDs. Duet never sees passwords or sign-in tokens directly.

Duet is included in Pro and Max plans and shares their usage limits. Background checks count toward those limits. Access starts with a US waitlist for Pro and Max users on iOS, Android and claude.ai. Team and Enterprise plans are not included yet.

Claude Security and Claude Enterprise

On October 21, Claude Security started scanning continuously in public beta for Claude Enterprise. It runs a baseline scan, then rescans changes on a schedule set by risk. Findings close when the fix lands and reopen if the issue comes back. Anthropic says thousands of organizations have found over a million vulnerabilities with it since the April beta.

The new pieces are an API, personal access tokens, webhooks and an MCP connector for Claude.ai and Claude Code. Role-based access controls decide who can view, triage and create projects. Anthropic is giving organizations $25,000 in free usage credits per month for two months, for usage through December 31, 2026.

On October 23, chat and Cowork merged for Claude Enterprise. A conversation starts as chat, and tasks that need code or files move to a cloud environment. The admin controls cover cost directly:

Model entitlements by role
Spend limits at the organization, group and member level
Pooled monthly budgets for groups
A new setting that keeps the experience on Sonnet and Haiku models to save money

Network egress is off by default for Enterprise cloud sessions, and connector credentials stay outside the sandbox. Organizations on zero data retention, HIPAA-ready setups or customer-managed keys stay on their current setup for now.

On October 23, Anthropic also opened a Claude Tag API in beta for Enterprise orgs. It manages Claude Tag settings on Slack channels through Admin API keys with read:claude_tag or write:claude_tag scopes.

Quiet Week Elsewhere

Cursor's changelog had no entries dated in this window. Its last post was October 20. We found nothing dated October 21-28 from Google Antigravity, Jules, Devin or Kiro.

Standards: Agent Skills Moves to the Agentic AI Foundation

Standards had the busiest week of the quarter. AGNTCon + MCPCon North America ran October 22-23 in San Jose. Anthropic donated Agent Skills, OpenAI donated Open Responses, and MCP set a new release calendar. The talks focused less on new features and more on security and identity.

Agent Skills Joins AAIF With Three Other Projects (October 22)

Anthropic donated Agent Skills to the Agentic AI Foundation on October 22. Agent Skills is the open format where a skill is a folder with a SKILL.md file plus any scripts or reference files it needs. Anthropic handed over the standard, its name and logo, the GitHub organization and the agentskills.io domain.

The adoption numbers are large. Anthropic says more than 70 agent platforms have integrated Agent Skills since the standard was published in December 2025. It names ChatGPT, Cursor, GitHub Copilot and VS Code. Anthropic also reports 1 million published skills and nearly 280 million installs.

Governance now looks like MCP's. Engineers from Anthropic, Google, OpenAI and others will maintain the standard as individuals. David Soria Parra and Mahesh Murag are the lead maintainers. The proposed roadmap adds versioned releases, a formal proposal process and security guidance for people who write and install skills. No dates came with that roadmap. You do not need to join AAIF to use or contribute to the spec.

The AAIF announcement named four new hosted projects:

That brings AAIF to 10 hosted projects, up from three at its December 2025 launch. The other six are A2A, Agent Router, agentgateway, AGENTS.md, goose and MCP. The announcement did not state licenses for the four new projects.

Open Responses deserves a closer look. It turns the shape of OpenAI's Responses API into a shared schema with common streaming events. If other providers adopt it, one client library can talk to many model APIs. That is the same move MCP made for tools.

Why does this matter? Skills, tools, model interfaces, sandboxes and training environments now sit under one neutral roof. A skill you write for one agent has a real path to run in others. And no single vendor controls how the format changes.

MCP Locks In a Twice-Yearly Release Calendar (October 27)

The MCP project merged SEP-3398: Release Cycle Updates on October 27. Core Maintainers accepted it on October 14. It sets a fixed schedule for the Model Context Protocol spec.

Here is how it works. Spec releases land on the last Tuesday of March and September. Each release gets an eight-month cycle, and cycles overlap by two months. The SEP text sets these milestones before each release:

The near-term impact is a date change. The SEP moves the December 2026 release target to late March 2027. If you planned around a December spec, plan for March instead. Proposals not finished by soft freeze move to the next cycle, and the release date does not slip.

Each SEP now needs a sponsor, either a Working Group or the Core Maintainer group. A primary and a secondary release manager run each cycle, and the secondary becomes primary next time. For anyone shipping an MCP client or server, this is good news. You get SDK betas six weeks before each release and two dates a year to plan around.

MCPCon Talks: Tasks, Auth and 4.17 Billion Downloads

InfoQ's recap of AGNTCon + MCPCon captured the stage claims. InfoQ notes that several figures came without published data, so treat them as speaker-reported.

Tasks in Claude Code: David Soria Parra announced MCP tasks support in Claude Code. Tool calls can now run for minutes, hours or days.
Enterprise-managed authorization: Den Delimarsky said it is live in Claude and marked stable on the auth roadmap. Proof-of-possession tokens and workload identity federation sit in draft. The roadmap ends with agents that sign in as themselves instead of borrowing a user token.
Downloads: Delimarsky put cumulative MCP SDK downloads at 4.17 billion.
Conformance: About one-third of the 116 conformance scenarios now test security.
Roadmap: InfoQ lists agent messaging, skills over MCP, identity and an HTTP-native transport among five priority areas.

Two talks dealt with tool safety. Thomas Sarlandie of OpenAI said OpenAI rejects submitted MCP servers unless every tool carries annotations. Annotations let ChatGPT fast-track read-only calls and confirm destructive ones. He also shared data from the ChatGPT plugin directory: about three in four tool schemas are under 1 kB, and 56% of plugins use server instructions.

Sarah Drasner of Chrome presented WebMCP, a proposed web standard now in origin trial. It lets websites expose tools that agents call directly instead of scraping the page. Chrome's benchmarks showed actions 2.5x to 7.5x faster. WebMCP adds read-only and untrusted-content hints to help agents resist prompt injection.

If you run MCP servers, the action item is clear. Add tool annotations now. At least one major client already requires them, and the rest will follow.

MCP TypeScript SDK 2.4.0 Ships the Skills Extension (October 23)

Last week we reported that only four providers serve skills over MCP the way the Skills Extension spec describes. The official SDK just made that easier. @modelcontextprotocol/server 2.4.0 shipped on October 23 with the Skills extension (SEP-2640) under new ext/skills subpaths.

The server side gets a SkillsExtension that serves skills/list, skills/get and a skill:// resource template. A Node helper, loadSkills(folder), loads skills from a directory. The client side gets a SkillsClient that verifies size, digest and SKILL.md frontmatter on every read. The extension works on both the 2025-11-25 and 2026-07-28 protocol versions. The attach method is marked experimental, and the rest of the Skills API is stable.

Two breaking-ish changes ride along. The SDK now rejects x-mcp-header on number-typed tool parameters, matching the 2026-07-28 spec. Use an integer type instead. And a newer bundled validator rejects some $id and $ref values with invalid URI schemes, such as my_tool:output. Check your tool schemas before you upgrade.

The MCP registry also shipped versions 1.8.4, 1.8.5 and 1.8.6 between October 23 and October 27. Version 1.8.6 now requires at least three characters for nonempty search terms. Shorter searches return HTTP 422.

A2A and Agent Identity

A2A had no release this week. The latest tag on the A2A repo is still v1.0.1 from May. Agent identity moved forward through vendor features, not a new standard. Copilot's server-to-server tokens, Claude's enterprise-managed MCP authorization and Codex's enterprise MCP sign-in all landed in the same week. Each one gives an agent its own scoped credential instead of a person's password or long-lived token.

Infrastructure: Anthropic's S-1 Puts a Price on Frontier Compute

The infrastructure story came from an IPO filing. Anthropic's S-1 gave the clearest public view yet of what a frontier lab commits to spend on compute. SK hynix then reported a quarter that shows where much of that money ends up.

Anthropic Files Its S-1 (October 26)

Anthropic announced it filed a Form S-1 with the SEC on October 26. It plans to list Class A common stock on the Nasdaq Global Select Market under the ticker "ANT." Share count and price range are not set yet. Goldman Sachs and Morgan Stanley are the named sources for the preliminary prospectus.

The S-1 itself carries the numbers. We pulled these directly from the filing:

Revenue: $16.3 billion for the six months ended June 30, 2026, up from $1.2 billion a year earlier
Q2 2026: revenue of $11.6 billion, up from $787 million, with operating income of $1.0 billion and an 8% operating margin
Operating loss: $693 million for the first half of 2026, down from $2.1 billion
Prior years: revenue of $41 million in 2023, $386 million in 2024 and $4.6 billion in 2025

Now the compute side. As of June 30, 2026, Anthropic reports about $265.6 billion in expected fixed or minimum guaranteed compute commitments, with terms from one to 20 years. On top of that, it has agreements with no fixed dollar figure, where payments depend on capacity delivered and pricing.

Press coverage has quoted totals from $417 billion to over $600 billion. Those higher figures add estimates and leases on top of the contractual minimums. The $265.6 billion number is the one in the filing's commitment disclosure.

Four Chip Families, Five Clouds

The S-1 describes a "multi-chip compute architecture across AWS Trainium, Google TPUs, NVIDIA GPUs, and AMD GPUs." It names the partners behind each one.

AMD: In July 2026, Anthropic signed a hardware supply framework with AMD for Helios rack-scale systems with Instinct MI450 series GPUs. It commits to an initial tranche in calendar 2027. If it buys the full tranche, Anthropic expects spending to exceed $20.0 billion. The deal allows a second tranche of about the same size. AMD agreed to buy up to $5.0 billion of Anthropic's Class N stock if conditions are met.
NVIDIA: Access to Grace Blackwell and Vera Rubin systems comes through NVIDIA and through Microsoft, SpaceXAI, Nscale, Lambda and other clouds.
SpaceXAI: Between May and October 2026, Anthropic signed and amended agreements with SpaceX.AI LLC, formerly X.AI LLC, for NVIDIA GPU compute.
Google and Amazon: Both have note purchase agreements of up to $20.0 billion each. They supply TPU and Trainium capacity.

Why does this matter to you? Model pricing depends on these contracts. A lab that spreads work across four chip families can shift workloads to the cheapest capacity. That is part of why mid-tier and fast-mode prices keep falling. It also shows the lock-in risk runs both ways. Labs commit for up to 20 years, so they need steady token demand to pay for it.

SK hynix Q3: Sold Out Through 2028, Short Through 2030 (October 28)

SK hynix reported third-quarter results on October 28 in Seoul. Investing.com's summary of the earnings slides gives these figures:

Revenue: 104.62 trillion won, up 32% from Q2 and 328% from a year earlier. It is the first quarter above 100 trillion won.
Operating profit: 83.33 trillion won, an 80% operating margin
Net profit: 39.78 trillion won, down 58% from Q2 on a 32.63 trillion won non-operating loss

Investing.com notes its summary was AI-assisted, so confirm figures against SK hynix's own filings.

The supply comments matter more than the profit. SK hynix has long-term agreements with 14 key customers, including major cloud providers, on standard five-year terms. Those deals cover more than 50% of total capacity for both 2027 and 2028. Management expects memory demand to outrun supply through 2030.

HBM4 is taking over. SK hynix expects HBM4 to pass HBM3E in both shipments and revenue in the fourth quarter. That tracks with NVIDIA's Vera Rubin ramp. The company also expects 2027 capex to rise significantly as it brings up its first Yongin fab.

Put this next to the Anthropic filing. Labs sign 20-year compute deals, and memory makers sign five-year supply deals. The AI hardware market is moving from spot buying to long contracts. If you plan to buy GPUs in 2027, the memory inside them is already spoken for.

NVIDIA and Oracle: Vera Rubin GA on OCI (October 27)

NVIDIA and Oracle announced on October 27 that Vera Rubin NVL72 systems and NVIDIA Vera CPUs are generally available on Oracle Cloud Infrastructure. The two companies plan to grow OCI Superclusters with Vera Rubin to multi-gigawatt scale. They gave no exact gigawatt figure.

Each NVL72 rack links 72 GPUs over NVLink. OCI uses Spectrum-X Ethernet with Spectrum-6 switches and BlueField-4 DPUs. A planned HGX Rubin NVL8 server pairs eight Rubin GPUs with two 64-core Intel Xeon 6 CPUs, 4 TB of DDR5 and 6.4 Tbps of RDMA networking. No date was given for NVL8.

The release adds one useful data point. OpenAI trained and served GPT-6 Astra on an OCI Supercluster in Abilene with more than 100,000 Blackwell GPUs. OpenAI also plans to run Vera CPUs in production on OCI for agent sandbox workloads.

That last detail is a trend. Agents run code, browse and call tools, and much of that work runs on CPUs, not GPUs. NVIDIA wants that CPU business too.

NVIDIA and Meta: $3 Billion and Vera Rubin (October 26)

NVIDIA and Meta expanded their collaboration on the Muse model family on October 26. Meta will deploy more NVIDIA infrastructure, including Vera Rubin, Vera CPUs, Spectrum-X Ethernet and BlueField-4. NVIDIA will invest $3 billion in Meta's equity offering, subject to closing conditions.

The two companies will also look at NVIDIA's OpenShell Reference Architecture for safe agent deployment, including the Sentry monitor we covered last week. The release gave no GPU counts, gigawatt figures or timeline.

What We Did Not See

We found no dated news in the window on Google TPUs, AWS Trainium or Microsoft Maia. Reports that Oracle will host OpenAI's Jalapeño chip and Google TPUs at Oracle AI World came from a single secondary source, and Oracle has not confirmed them in a release we found. We left them out. Intel and Samsung had not posted third-quarter results by our cutoff.

The Data Layer Takeaway

Three threads from this week point at the data layer.

First, defaults now carry governance weight. Gemini's new store=true default, Copilot's assisted permissions and Claude Code's sandbox changes all decide what happens to your data before you make a choice. Data teams need to read changelogs the way security teams read CVE feeds.

Second, agent work is getting longer. MCP tasks in Claude Code let a tool call run for hours or days. Duet checks your accounts every few hours and keeps working while you sleep. Long-running agents need data sources with stable identities, clear access rules and an audit trail. A governed lakehouse catalog gives an agent that boundary.

Third, skills are becoming the unit of reuse. With Agent Skills under AAIF and a Skills extension in the official MCP SDK, a skill that teaches an agent how to query your tables can travel between Claude, ChatGPT, Copilot and Cursor. Teams that write good skills for their semantic layer will get more out of every agent they adopt.

What to Watch Next Week

October 28-29: GitHub Universe
October 31: Mistral Large 4 weights deadline, and DeepSeek Search and Fetch billing starts for harness users
November 4: Kimi K3.1 Pro open weights, with a license still to be named
November 16: Copilot retires Claude Opus 4.8, Opus 5 and GPT-5.6 Terra
November 30: Claude Sonnet 4.5 retires
Ongoing: Anthropic's S-1 amendments, which should add share count and price range

Resources to Go Further

AI changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free - Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data - Dremio's agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community - Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development - Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis - A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

AI Weekly: Open Weights Surge and a Breaking Claude API

Alex Merced — Wed, 21 Oct 2026 19:51:46 +0000

Week of October 14-21, 2026

The big frontier launches landed just before this window. Claude Haiku 5.5, GPT-6 in ChatGPT, Meta Muse Wonder and Gemini 4 Argon all shipped between October 7 and October 13. This week was about what came after: a wave of open-weight models, a breaking Claude API version, a round of model retirements and an NVIDIA keynote that put supply chains ahead of chips.

Here is what changed, what it costs and what you need to do about it.

Models: Meta Muse Spark Open Leads an Open-Weight Week

No new closed frontier flagship shipped between October 14 and October 21. The open-weight side made up for it. Meta, NVIDIA, DeepSeek, MiniMax, Mistral and Tencent all released models or weights in the same seven days. For teams that self-host or fine-tune, this was the busiest week of the quarter.

Meta Muse Spark Open (October 15)

Meta released Muse Spark Open on October 15. The model ID on Hugging Face is meta-models/Muse-Spark-Open-816B-A42B. It is a mixture-of-experts model with 816 billion total parameters and 42 billion active per token. Context reaches 1 million tokens. Meta did not publish a max output figure.

The model is natively multimodal. Press coverage describes text, image, video and audio input with text output. Meta ships weights in BF16, and coverage also reports an NVFP4 build. Day-one engine support covers SGLang, vLLM, TensorRT-LLM, Transformers, MLX, llama.cpp and Unsloth.

Distribution is wide. Weights are on Hugging Face, Amazon SageMaker AI, Google Cloud Model Garden and Microsoft Foundry. Hosted APIs are live on Baseten, Fireworks, Groq, Modal, Nebius, OpenRouter and Together. Meta also demoed the model on two linked M5 Ultra Mac Studios. That is a useful signal for anyone sizing on-prem hardware.

Two caveats matter. First, Meta published no per-token price, so cost depends on the host you pick. Second, the license is the "Muse Model License," not Apache 2.0 or MIT. Meta describes it as carrying "limited commercial terms." One third-party reading says API resellers above $20 million in gross revenue need a separate deal with Meta. That reading is unconfirmed, so check with your legal team before building a business on it.

Meta calls Muse Spark Open "the most capable US open-weight model to date." Its blog gives no benchmark scores to back that claim. Artificial Analysis, an independent evaluator, scored it 39 on its Intelligence Index, according to press reports. That puts it behind MiMo-V2.6-Pro at 46 on the same index.

The strategic story is bigger than the score. Meta's previous flagship, Muse Spark, was closed. Muse Spark Open marks a partial return to the open-weight playbook that made Llama popular. For lakehouse teams running agents next to governed data, a 1M-context open model with broad engine support is a real option for private deployments.

NVIDIA Nemotron 3.5 Super VL and Bolt (October 15)

NVIDIA expanded the Nemotron family on the same day. The headline model is Nemotron 3.5 Super VL, a 120B-A12B vision-language mixture-of-experts model built for agents. NVIDIA trained it on about 26 trillion tokens, then added about 3.8 million post-training samples and a reinforcement learning stage.

NVIDIA says Super VL comes close to its largest reasoning model at about one-fifth the parameters. It also claims up to 5x the throughput of the next-best open model in its class, citing an Artificial Analysis chart. Treat that 5x as vendor-reported. HPCwire reports a score of 26.5 on the Artificial Analysis Intelligence Index, plus a context of up to 1M tokens and the OpenMDW-1.1 license. NVIDIA's own blog states neither the context nor the license, so confirm both on the model card before you deploy.

The more interesting release for many readers is Nemotron 3.5 Bolt 74B-A9B. NVIDIA trained it with native 2-bit quantization so the full model fits in 24GB of memory. That targets RTX desktops, DGX Spark and the new RTX Spark laptops. It runs on llama.cpp, Ollama, LM Studio, Unsloth and vLLM.

NVIDIA claims Bolt ranks first among open models in its class on PinchBench and second on BrowseComp. Both rankings are vendor-reported. Press reports put Bolt at 21.4 on the Artificial Analysis Intelligence Index. A smaller Bolt E4B that fits in 4GB is due October 23.

Super VL is available on Hugging Face, OpenRouter, ModelScope and NVIDIA NIM. If you need an agent model that reads screenshots and documents on your own GPUs, it belongs on your test list.

DeepSeek-V4.1-Flash-1017 and Search & Fetch APIs (October 17)

DeepSeek shipped a refreshed V4.1 Flash on October 17. The model ID is deepseek-ai/DeepSeek-V4.1-Flash-1017. Hugging Face metadata lists 763 billion total parameters. One secondary source said 552 billion, so trust the Hugging Face figure. Context is 1M tokens, and the weights carry an MIT license.

The gains over the September build are large on agent and coding tasks. These numbers come from DeepSeek's model card, so they are vendor-reported:

DeepSeek's own table lists Claude Opus 5 at 74.0 on DeepSWE v1.1. That puts the new Flash snapshot ahead on that one test, by DeepSeek's count. Wait for independent runs before you treat that as settled.

Pricing is murky. DeepSeek's announcement shows prices only in an image. A secondary source says pricing is unchanged at $0.15 input and $0.60 output per million tokens off-peak, and $0.30 and $1.20 at peak. We did not find those figures in a text source, so check the DeepSeek console.

The second launch matters as much as the model. DeepSeek opened its Search and Fetch APIs to all users at $0.75 per 1,000 calls per endpoint. Calls are free through server-side tools in DeepSeek Harness, Claude Code and Codex until October 31 at 16:00 UTC. Model labs now sell the tools around the model, not just the tokens.

MiniMax M3.1 Flash (October 19)

MiniMax released M3.1 Flash on October 19 under the model ID MiniMax-M3.1-Flash. It had circulated as "Space Bunny" since late September. Pricing is $0.15 per million input tokens, $0.60 per million output tokens and $0.015 per million cached input tokens.

MiniMax says the model costs 70% less on agentic tasks than its predecessor. It also claims about 3x the speed, with 200+ tokens per second at peak and 170+ at the median. MiniMax did not state a context window. One gateway lists 1M tokens.

The vendor-reported scores lean hard toward agents:

DeepSWE v1.1: 73.7%
OSWorld V2: 61.7
Agents' Last Exam: 29.6
BankerToolBench: 76.2
CyberGym: 82%

The model is live on the MiniMax API, the M Plan and MiniMax Code. MiniMax says weights "will be released later." Reports point to October 23, but that date is unconfirmed.

Note the pricing pattern. MiniMax M3.1 Flash and the reported DeepSeek off-peak rate both sit at $0.15 in and $0.60 out. Fast agent-grade models are converging on a price floor. That changes the math on running agents over large datasets.

Mistral Timestral, Tencent Hy-Memory and Grok Voice 3.0

Three smaller releases deserve a note.

Mistral released Timestral on October 15. It is a family of time-series forecasting models in 5M, 28M, 123M and 209M sizes under a modified MIT license. Context is 32,768 steps with a 1,024-step forecast horizon. Mistral reports the 209M model ranks first on GIFT-Eval, with a MASE of 0.658 and CRPS of 0.446, plus first place on TIME and fev-bench. Those rankings are vendor-reported. The model runs on CPU, and Mistral says a 209M forecast took 140 milliseconds on 8 threads.

Timestral is worth a look for data teams. Forecasting on lakehouse tables usually means exporting data to a separate ML stack. A 209M model that runs on CPU in milliseconds can sit much closer to the query engine.

Tencent posted Hy-Memory-30B-A3B to Hugging Face on October 15. It is a memory-writer model with about 3 billion active parameters under Apache 2.0. Its job is to decide what an agent should remember between sessions. Tencent reports a cross-harness average of 55.49, against 54.62 for Claude Opus 5. That comparison is vendor-reported.

xAI shipped Grok Voice 3.0 as grok-voice-3.0 at $0.08 per minute. If you call the grok-voice-latest alias, you are now on 3.0, and version 2.0 is deprecated. Press reports put it at 86.9 on the Artificial Analysis speech-to-speech index, up from 81.3. Latency rose from 0.70 seconds to 0.81 seconds. Pin your version if latency matters to your app.

Claude API Version 2026-10-21 Breaks Old Calls (October 21)

The most important model-platform change this week was not a model. Anthropic shipped API version 2026-10-21 along with new major versions of every official SDK. The new SDKs send the new version header by default. If you upgrade your SDK without reading the migration guide, some calls will fail.

Here are the new SDK versions:

Python 2.0.0
TypeScript 1.0.0
C# 13.0.0
Go 2.0.0
Java 3.0.0
PHP 1.0.0
Ruby 2.0.0
ant CLI 2.0.0

The SDKs and CLI also move from the MIT license to Apache 2.0.

Three request patterns now return a 400 error under the new version. First, sending temperature, top_p or top_k fails, even at default values. Second, a document block placed directly in an assistant message fails. Third, a user text block with non-empty citations fails. Workloads that need sampling parameters can stay on the 2023-06-01 version for now.

The SDK changes go further. Text Completions is gone from the TypeScript, Go, Java and Ruby SDKs and from the CLI. API keys now travel as Authorization: Bearer, so check any proxy that inspects the old header. Each client now holds one credential, resolved in a fixed order: ANTHROPIC_API_KEY, then ANTHROPIC_AUTH_TOKEN, then the profile, then workload identity federation. The ant CLI drops its --api-key and --auth-token flags. The Admin API drops old field names and replaces workspace_id on API keys with scope.

The practical advice is simple. Pin your SDK version in CI today. Then upgrade on a branch, run your test suite and fix the 400s before you ship. The full release notes list every change.

Other Anthropic Updates

Anthropic made several smaller changes in the same week.

October 14: OpenTelemetry export entered beta, and Managed Agents gained scheduled deployments on Claude Platform on AWS.
October 15: Opus 5.5 fast mode left research preview. It no longer needs a beta header, gets default rate limits and is available on Microsoft Foundry.
October 15: The Models API now reports whether each model supports thinking: {type: "between_tools"}, which turns off up-front thinking and lets the model think only between tool calls.
Retirement: Claude Sonnet 4.5 retires on November 30. The replacement is claude-sonnet-5-5.

On the policy side, Anthropic published an update on independent evaluators on October 15. METR now has employee-like access, with six staff onboarded and the right to publish. SecureBio is reviewing the chemical and biological chapter of Anthropic's August Risk Report. Outside review with publication rights is a meaningful step for frontier lab transparency.

Retirements Across Codex and Copilot

Model retirements hit two major coding platforms.

OpenAI retired GPT-5.5 in Codex on October 14 for users signed in with ChatGPT. API-key sessions are not affected. The replacement is gpt-6.1-sol.

GitHub deprecated six Copilot models on October 19:

GitHub also announced a second round for November 16. Claude Opus 4.8 and Opus 5 move to Opus 5.5. GPT-5.6 Terra moves to GPT-6.1 Sol or GPT-6 Luna. If your team pins models in Copilot policies, update those policies before mid-November.

What This Means for Practitioners

Three patterns stand out. Open-weight models now ship with 1M context, broad engine support and hosted APIs on day one. Fast agent models are settling at $0.15 in and $0.60 out per million tokens. And platform churn is real: in one week you got a breaking API version, eight model retirements and an alias that silently moved to a new voice model. Pin versions, track deprecation pages and budget time each month for upgrades.

Tooling: Approval Modes, Credentials and Multi-Model Coding

Coding agents had a security-heavy week. The common theme was control. Vendors tightened who approves an agent's actions, how credentials flow and how long tokens live.

Claude Code 2.1.299 Through 2.1.306

Claude Code shipped eight releases between October 14 and October 21, according to npm. The changelog shows the bulk of the work went into approval modes and permissions.

On approval modes, a new /auto-mode-setup command walks you through auto mode, which now works on more providers. Headless claude -p runs and SDK sessions now start in default mode on third-party providers. That is a safer starting point for CI pipelines. A new CLAUDE_CODE_AUTO_MODE_MODEL variable sets which model classifies actions in auto mode. And a check can now carry decider: "person", which means only a human can approve it.

Observability improved too. OpenTelemetry tool_decision events now include classifier_verdict and permission_mode. Security teams can now see why an agent action was allowed, not just that it happened.

Several permission rules got stricter:

MCP allow rules now match only the server they name.
Hooks in Cowork cloud sessions can no longer change the permission mode.
A new sandbox.allowNestedUserNamespaces setting exists, and the changelog warns that it can bypass sandbox file protections. Leave it off unless you know why you need it.
allowedMcpServers no longer filters some managed servers. Admins should use deniedMcpServers to block servers.

Credential fixes landed for forceLoginOrgUUID and MDM login settings, plus a login bug in cloud -p runs.

Billing and usage got clearer warnings. Idle compaction now runs for conversations over 200K tokens that sit idle for about 54 minutes. New /usage warnings appear, and Pro and Max users see a warning before resuming an idle conversation that eats a noticeable share of the 5-hour limit. Prompt suggestions are skipped after replies billed to credits.

One fix will save some people real pain. The --worktree flag no longer silently deletes a worktree and its branch at exit. New /advisor and /claude-api managed-agents-secure commands round out the releases.

Codex CLI 0.163.0 (October 15)

OpenAI shipped Codex CLI 0.163.0 on October 15. The feature list is practical:

A searchable /config screen
Steering Codex while it is mid-response
A leader key in /keymap
A /daemon command that restarts the background server
Parallel read-only tool calls

A feature called "Daybreak" is now on by default for eligible accounts. The release notes do not explain what it does.

The security fixes matter more. Resumed threads now restore their permissions and network approvals, so a resumed session no longer starts with stale or missing grants. The network proxy now checks permission before the DNS lookup and blocks credential-alias bypasses. New sandbox integrity checks run on Linux with bubblewrap, macOS with Seatbelt and Windows with MXC. Codex also stopped bundling its patched zsh. On the model side, Codex now advertises Ultrafast mode for GPT-6.1 Sol on Amazon Bedrock.

GitHub Copilot HydraFusion Max (October 14)

GitHub updated HydraFusion and added a HydraFusion Max option in research preview. HydraFusion combines several models into one answer. The model pool is Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna.

Drafts run at medium reasoning effort and reviews run at high effort. A new "Cascade" judging step asks for a repair only when the judge finds a problem. Critique now uses several critic models in parallel. HydraFusion works in VS Code, the Copilot app and Copilot CLI on the Pro, Pro+, Business and Enterprise plans.

One gap stands out. GitHub's post does not explain how HydraFusion Max bills premium requests. Several models per answer means several model calls per answer. Check your usage dashboard before you turn it on for a whole org.

GitHub shipped more on October 15. Agentic autofix for code scanning is now generally available. AI Scan for pull requests also reached GA, and Copilot in Slack and Teams gained multi-repo support. On October 19, GitHub released a model policy REST API at the enterprise, team, agent-app and org levels. Platform teams can now manage model access as code instead of clicking through settings.

GitHub Universe runs October 28-29. Expect more Copilot news there.

GitHub CLI 2.103.0: Expiring Tokens (October 21)

GitHub CLI 2.103.0 added opt-in OAuth tokens that expire and refresh on their own. GitHub says a leaked token "becomes unusable within hours." That is a big deal now that coding agents call gh on your behalf all day.

You opt in with gh auth login --expiring or gh config set expiring_credentials enabled. Scripts that cache the output of gh auth token will break, so update those first. The feature needs GitHub Enterprise Server 3.23 or later. GitHub plans to make it the default for new sign-ins in a future release.

GitHub also put a package firewall into public preview on October 20.

Cursor Canvases and Environments API (October 16)

Cursor's October 16 update lets cloud agents build canvases that you can share by link. An Environments API entered beta. Environments now support up to 300 repositories, and large ones load 5x faster, by Cursor's count. Cursor made no pricing changes.

Claude Platform: Per-Customer Safety Controls

Anthropic shipped two features for teams that build agents for their own customers. On October 20, user profiles arrived at /v1/user_profiles. They scope safety enforcement to one end customer instead of your whole organization. One bad actor in your app no longer puts your whole API account at risk.

The same day, organization webhooks entered beta. They fire when a user profile is restricted or banned. Inference hooks reached general availability. On October 14, Managed Agents web_search started following org-level domain restrictions.

These are the kinds of controls that make agents safe to put in front of paying customers. If you run a multi-tenant agent product, user profiles belong in your next sprint.

Quiet Week Elsewhere

We found nothing dated in this window from Google Antigravity, Gemini CLI, Devin Desktop or Kiro. Expect that to change as GitHub Universe and AGNTCon approach.

Standards: MCP Reorganizes Its Working Groups

Standards had the thinnest week of the four sections. No new MCP spec version shipped. A2A had no release, and the Agentic AI Foundation published no press releases in the window. The news came from governance, adoption data and identity.

MCP Creates a Tasks and Events Working Group (October 20)

The Model Context Protocol (MCP) is the open standard that lets AI apps connect to tools and data sources through one interface. On October 20, the project merged SEP PR #3440. It folds the task work and the event and trigger work into one new Tasks and Events Working Group.

The same change turns the Agents Working Group into an Interest Group. Interest Groups discuss ideas but have no decision rights. The message is clear. MCP's core team wants long-running tasks and event-driven triggers designed together, not in two separate rooms.

For context, the current spec is still version 2026-07-28. That release removed the initialize handshake and the Mcp-Session-Id header. It also deprecated Roots, Sampling and Logging. The latest SDKs are Python 2.3.0, released October 2, and TypeScript 2.3.1, released October 5. The MCP registry sits at version 1.8.1 and remains in preview.

Why does a tasks-and-events group matter to data teams? Most data work is not a single request and response. A pipeline run, a table compaction or a long query is a task that finishes later. Agents need a standard way to start that work, check on it and get told when it is done. This working group is where that standard will come from.

Skills Over MCP: Adoption Is Still Thin (October 21)

API Evangelist published an adoption study on October 21. The author probed 9,170 MCP endpoints. Only four providers serve skills the way the Skills Extension spec describes. Those four are Hugging Face, Glean, Renoolab and BestPrice. The Skills Extension, SEP-2640, reached Final status on September 13.

The protocol version data is even more telling:

Only 44 responding endpoints run the current spec. More than 500 still run a version from November 2024. That is a big gap between what the spec says and what servers in the wild support. If you build an MCP client, plan to support older protocol versions for a long time. If you run an MCP server, the July 2026 changes are worth the upgrade.

Agent Identity and Credentials

Agent identity did not get a new standard this week, but several vendors shipped identity features. Read together, they show where the industry is heading.

Anthropic user profiles scope safety enforcement to one end customer.
GitHub CLI expiring tokens shrink the damage window when an agent's token leaks.
The Claude SDK credential changes fix one credential per client and set a clear lookup order.

NVIDIA added a hardware angle on October 21. Its OpenShell Reference Architecture is an open-source secure runtime for agents. A BlueField-4 component called "Sentry" watches agents out-of-band and enforces policy in silicon. Partners include Anthropic, OpenAI, Google, Meta, Okta, 1Password and CrowdStrike. The announcement does not mention MCP and does not name a license, so the open-source claim needs a closer look once the code appears.

The standards track continues in parallel. The IETF draft draft-ietf-wimse-aims-00 on AI agent authentication was published September 15. Expect it to come up at the events ahead.

Personal Agent Protocol Still in Draft

The Personal Agent Protocol, nicknamed "Poppy," remains a draft. Its sample discovery file shows protocol_version "0.2" under Apache 2.0. Agents discover each other through /.well-known/poppy.json, sign in with OAuth and keep JWT sessions. Nothing changed in the window. One single-source report says the project missed its October targets.

AGNTCon + MCPCon This Week

AGNTCon and MCPCon North America run October 22-23 in San Jose. With a new MCP working group, a final Skills spec and thin adoption data on the table, expect protocol announcements there. We will cover them next week.

Infrastructure: NVIDIA Says Capacity Is the Constraint

The infrastructure story this week was supply, not speed. NVIDIA used GTC Berlin to talk about substrates, interposers and test capacity. Two of the biggest chip suppliers reported quarters that show demand still outrunning supply.

NVIDIA GTC Berlin: The Supply Chain Keynote (October 21)

Jensen Huang summed up the keynote in four words: "Capacity is the constraint." NVIDIA then laid out a supply chain expansion across packaging, testing and optics.

Substrates: package substrate capacity rises 2.5x over three years. IBIDEN gets a multibillion-dollar advance to triple capacity at its Ono plant, ramping in Q1 2028.
Interposers: Amkor targets more than 300% more interposer wafer capacity by the end of 2028.
Testing: test capacity doubles from 2026 to 2027. KYEC gets a $1.1 billion advance for US test sites, with volume production starting April 2027.
Optics: SPIL's co-packaged optics capacity rises 15x by mid-2027.

NVIDIA is now paying suppliers in advance to build plants. That tells you the bottleneck has moved past GPU design and wafer supply. It now sits in packaging, testing and optics. GPU supply will stay tight through at least 2027, and teams planning large AI clusters should plan for long lead times.

DSX POD: AI Factories in a Container

NVIDIA also introduced the DSX POD, a containerized Vera Rubin AI factory design. NVIDIA says deployment at a prepared site drops from six months to as little as 14 days. Deliveries start in early 2027. NVIDIA gave no GPU count or megawatt figure per POD.

A second announcement covered power. All of these figures are vendor-reported:

MaxLPS cuts power by up to 30% for the same output. At Nscale, it delivered 49% more throughput inside a 264 kW budget.
DSX Flex cuts power by up to 40% during simulated grid events. Data centers can now shed load when the grid is under stress.
Vera Rubin runs on 45°C coolant. Warmer coolant means less chilling and lower cooling bills.
DSX OS is open source.

Power is now the hard limit for many AI sites. Grid-aware load shedding helps data centers win interconnect approvals faster. Watch for utilities to start asking for it.

On October 16, RTX Spark PCs went on sale. Paired with Nemotron Bolt's 24GB footprint, NVIDIA is pushing hard on local agent inference.

TSMC Q3: Advanced Nodes Hit 79% (October 15)

TSMC reported third-quarter results on October 15. Revenue reached US$46.72 billion, up 41.2% year over year. Net income hit NT$763.06 billion, up 68.7%. Gross margin came in at 66.4%.

The node mix shows how fast AI chips move to new processes:

2nm: 8% of wafer revenue
3nm: 33%
Advanced nodes overall (7nm and below): 79%

TSMC guided fourth-quarter revenue to $52.3-54.3 billion. Secondary coverage puts high-performance computing at 68% of revenue. TSMC's own release does not confirm that figure or any capex update.

ASML Q3: Customers Ordering for 2029 (October 14)

ASML reported third-quarter results on October 14. Sales hit €11.66 billion with a 57.5% gross margin and €4.96 billion in net income. ASML guided Q4 to €14.2-15.2 billion and full-year 2026 to €44-45 billion.

The detail that matters is the order horizon. Some customers want to place orders for 2029 and 2030. ASML says its supply for 2027 and 2028 will not fully meet demand. ASML did not disclose a bookings figure. When the company that makes the lithography machines is sold out two years ahead, every chip downstream stays scarce.

Anthropic and Nscale: A Second Deal in Talks (Unconfirmed)

Bloomberg HT, citing Bloomberg, reported on October 19 that Anthropic is in talks with Nscale for a second deal of roughly $45 billion. We did not see the original Bloomberg story, so this rests on secondary write-ups. The reported deal covers about 460 MW at Nscale's Monarch campus in West Virginia, running Vera Rubin chips. Anthropic also gets a reported right of first refusal on a third building.

The talks are not final. Both companies declined to comment. Treat this as a report, not a done deal.

What We Did Not See

We found nothing dated in the window on HBM memory, Google TPUs, AWS Trainium or new gigawatt-scale power deals. The Open Compute Project Summit announcements landed on October 12, just before this window. Those included Meta adopting NVIDIA Spectrum-6, AMD's Helion rack and Microsoft's Maia 300 chip.

The Data Layer Takeaway

Three threads from this week point at the data layer.

First, agents are getting cheaper to run and easier to host. At $0.15 per million input tokens, an agent can read a lot of table metadata before the bill matters. Open models with 1M context make private deployment practical. The limit is no longer the model. It is whether the agent can find governed, well-described data.

Second, credentials and approvals got stricter everywhere. Expiring tokens, per-customer safety profiles and human-only approval checks all point the same way. Agents will act on your data, and every action needs an owner and an audit trail. A lakehouse with a governed catalog and fine-grained access control gives agents that boundary.

Third, MCP is reorganizing around tasks and events. Data work is long-running by nature. The protocol that lets an agent start a job, track it and hear when it finishes will shape how agents interact with query engines and catalogs.

What to Watch Next Week

October 22-23: AGNTCon + MCPCon North America in San Jose
October 23: MiniMax M3.1 Flash weights (reported) and Nemotron 3.5 Bolt E4B
October 28-29: GitHub Universe
October 31: DeepSeek Search and Fetch billing starts for harness users, and the deadline for Mistral Large 4 weights
November 16: Copilot retires Opus 4.8, Opus 5 and GPT-5.6 Terra
November 30: Claude Sonnet 4.5 retires, and Gemini 4 Argon introductory pricing ends

Resources to Go Further

AI changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free - Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data - Dremio's agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community - Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development - Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis - A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

Apache Data Lakehouse Weekly: October 14-21, 2026

Alex Merced — Wed, 21 Oct 2026 19:03:32 +0000

Many contributors were still flying home from Community Over Code in Glasgow, and the lists picked up where the hallway talks ended. Iceberg wrote unstructured files into its v4 spec. Polaris argued about who owns those files once they sit behind a catalog. Parquet spent the week deciding what a vector is and what a format version means. The common thread was AI data: embeddings, files and semantic models all moved toward being first-class citizens of the open lakehouse.

Apache Iceberg

The biggest outcome was the passing vote to add a File type to the v4 spec. Talat Uyarer opened the vote for the FILE type working group. It closed with 8 binding +1s, from Steven Wu, Péter Váry, Daniel Weeks, Gang Wu, Sung Yun, Amogh Jahagirdar, Yufei Gu and Russell Spitzer, plus 19 non-binding +1s. The new type stores a range of bytes either inline or as a reference to an external object, with fixed sub-fields for URI, offset, size, content type and checksum. It mirrors Parquet's FILE logical type, so images, PDFs and audio can sit in a table column with real metadata and stats. Kurtis Wright voted +0. He argued that maintenance jobs need a signal for when a referenced object is safe to delete, or tables sharing a media bucket will step on each other. Daehong Jeon and Orest Tokovenko have already volunteered to implement the type in iceberg-rust and PyIceberg.

That vote followed the result of the baseline v4 AMT spec vote, which passed with 9 binding and 9 non-binding +1s. With the Adaptive Metadata Tree in the main spec, the follow-ups started at once. Anoop Johnson proposed extending "tight" stats beyond min and max to null, NaN and value counts, so an engine could answer count(*) WHERE col IS NULL from metadata alone. Kevin Liu asked whether v4 should be Parquet-only for data files as well as manifests. He pointed to V3 feature gaps in ORC and Avro and thin support outside Java. Péter Váry and Junwang Zhao pushed back hard on narrowing the spec, and Junwang named Vortex, Lance and Nimble as formats worth keeping the door open for. Kevin will move it to a dedicated thread.

AI-oriented types kept coming. In the File and Vector sync thread, Péter Váry proposed int8 and float16 primitives for vector indexes. Philipp Fischbeck wanted them only as vector element types, and Russell Spitzer suggested voting the Vector type in first. Péter also revived derived columns, and Szehon Ho suggested landing them through the secondary index spec. On the stats side, Yuya Ebihara proposed an HLL sketch blob type for Puffin. Theta sketches cost Trino 10-20% on inserts, and without a standard HLL type every engine invents its own. Yong Zheng also flagged that the REST spec is not valid OpenAPI 3.1, which he found while upgrading Polaris.

Apache Polaris

JB Onofré put Polaris Directories up for review. A Directory is a table-like entity over an object-store prefix, with its contents tracked in a regular Iceberg inventory table. Ankur Kotwal found a bug where the scanner can delete a file a commit may have already landed, and he wants it fixed before 1.9.0. Tornike Gurgenidze argued the inventory should be the source of truth, like Iceberg manifests, so Polaris could offer snapshot listings and time travel over files. Yufei Gu suggested phasing the feature rather than rushing it into the release JB has targeted for the end of October.

Qiegang Long shared a proof of concept for multi-table, multi-statement transactions, with Spark and Trino running BEGIN ... COMMIT across tables. JB welcomed it but set a process: design doc first, then small PRs, plus a persistence SPI so it does not depend on PostgreSQL. JB also asked how Polaris should support custom entities without patching core enums, since semantic models, tags, lineage and directories each touched about ten classes.

Process got attention too. Yufei Gu proposed an opt-in AI code review bot. JB backed it with labelling rules drawn from ASF guidance. Robert Stupp asked who pays, what data it keeps and why a second bot is needed next to Copilot. vignesh a reported that the bot missed context agreed on the list. Separately, the IRC Labels design settled on labels as a read-only projection of Polaris tags.

Apache Arrow

Raúl Cumplido announced Arrow 26.0.0, with 337 resolved issues. Arrow Rust 59.3.1 passed its vote with four binding +1s. Andrew Lamb then announced patch releases on four arrow-rs branches at once, a rare step for a project that normally ships forward. JB gave a heads up on Arrow Java 20.0.0, with a vote expected soon.

On the format side, Rusty Conover posted draft PRs across five languages so dictionary-encoded extension types survive IPC. The trigger was DuckDB failing to export a dictionary-encoded UUID column, and he plans a format vote in about a week. Tornike Gurgenidze proposed letting Flight SQL servers advertise their Substrait dialect, which ADBC would inherit. The PMC also welcomed Richard Baah as a committer.

Apache Parquet

The vector debate dominated. Jan Finis argued a finite vector type adds rules without new capability. Daniel Weeks and Gang Wu countered that logical types carry meaning, just as STRING promises valid UTF-8. Ryan Blue proposed two types: a finite vector for embeddings and a fixed-length list that allows NaN. Russell Spitzer backed the two-type path.

Alkis Evlogimenos laid out a concrete versioning proposal after auditing eight readers. Unknown encodings fail safely, but unknown page types and codecs can silently drop or corrupt rows in some readers. So encodings like ALP could ship in PAR1 files, while new page types or codecs would require a new compatibility level. Andrew Lamb and Divjot Arora were supportive. Daniel Weeks also rejected redefining distinct_count as an estimate, and Gunnar Morling contributed FILE type fixtures that parquet-java and arrow-rs both read correctly.

Apache Ossie

The incubating semantic interchange project was the busiest list of the week. Julian Hyde asked whether the project is truly open, calling Slack and working groups a walled garden. Chris Eubank agreed the point was fair and made it explicit that decisions happen on dev@. A run of SQL dialect PRs led Chris to propose making dialect a free-text field. The converters moved to their own repo, and JB is now preparing the first core release.

Cross-Project Themes

Unstructured data arrived at three layers at once. Parquet has a FILE logical type, Iceberg voted in a matching File column, and Polaris is designing Directories over raw prefixes. All three hit the same open question: who decides when a referenced object can be deleted, and which layer holds the truth. Kurtis Wright's +0 in Iceberg and Tornike's inventory argument in Polaris are the same concern from two angles. Expect a shared answer to emerge rather than three separate ones.

The vector debate also crosses projects. Parquet is defining the physical type while Iceberg debates element types like int8 and float16, and Will Edwards suggested Iceberg might be the better home for the finiteness contract. Meanwhile, two projects openly wrestled with AI in their own process, from Polaris's review bot to contributors disclosing LLM-assisted reports on Polaris and Parquet.

Looking Ahead

Watch for a Polaris 1.9.0 release candidate and whether Directories make the cut. Arrow Java 20.0.0 and ADBC 25 votes should land, along with Rusty Conover's dictionary extension format vote. In Iceberg, look for Kevin Liu's format thread and a Vector type vote. In Parquet, watch whether the two-type vector proposal and the versioning design move toward a vote.

Resources & Further Learning

Get Started with Dremio

Try Dremio Free: Build your lakehouse on Iceberg with a free trial
Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow: Learn how Dremio brings the open lakehouse stack together

Free Downloads

Apache Iceberg: The Definitive Guide: O'Reilly book, free download
Apache Polaris: The Definitive Guide: O'Reilly book, free download

Books by Alex Merced

Apache Data Lakehouse Weekly: October 6-14, 2026

Alex Merced — Wed, 14 Oct 2026 17:08:21 +0000

Half the lakehouse community spent this week in Glasgow at Community Over Code, and the mailing lists show it. Syncs got cancelled, board reports went out late, and several threads open with a nod to a hallway conversation from Lakehouse Day. The work did not slow down. Apache Iceberg ran three overlapping v4 spec votes at once, Apache Parquet put a concrete versioning proposal on the table, Apache DataFusion Comet asked to become its own top-level project, and Apache Ossie (incubating) argued about where an MCP server belongs.

The thread that ties the week together is precision. Each community is replacing a loose rule with an exact one. Iceberg is pinning down snapshot timestamps and statistics guarantees. Parquet is fixing the meaning of a statistic that never had a firm definition. Polaris is redrawing the line between catalog configuration and server storage access. Ossie is deciding which parts of a semantic model must be portable before anyone can query one. These are the unglamorous rules that let engines, catalogs, and AI agents trust the same data without asking a human first.

This issue covers Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, Apache DataFusion, and Apache Ossie (incubating). All thread links point to the official archives at lists.apache.org.

Apache Iceberg

Three v4 votes in one week

The Iceberg v4 table spec moved further this week than in any week this fall. Three separate spec votes ran in parallel, and none of them drew a single -1.

The first to close was Steven Wu's proposal to require monotonically increasing snapshot timestamps in v4. The rule is simple. Every new snapshot on a branch in a v4 table must carry a timestamp-ms strictly greater than its parent's. Root snapshots stay unconstrained, and v1 through v3 tables keep their current behavior. Steven explained that out-of-order timestamps break time travel today, because a query for "the table as of 3:00 PM" picks the wrong snapshot when clocks drift between writers. The rule also opens the door to a row timestamp metadata column inherited from the snapshot timestamp, which Steven flagged as a separate follow-up change. The vote passed with 9 binding +1s and 11 non-binding +1s. Maninder Parmar added a note worth tracking: he wants the community to consider nanosecond precision (timestamp-ns) later, so high-throughput tables do not hit collisions at the millisecond level. Xuanwo restated the scope in his +1, confirming that pre-upgrade snapshots stay exempt.

The second vote is the big one. Amogh Jahagirdar opened a vote to merge the baseline v4 AMT spec PR into the main repo. AMT is the adaptive metadata tree, the new metadata structure at the center of v4 that changes how Iceberg organizes manifests and commits. Amogh was careful about what the vote means. It is a vote to land a baseline in the repository so other v4 work stops fighting rebases, and it is not a vote to adopt the design as final. Encryption, column file update rules, and dropping the partition tuple all remain open for later iteration. The thread filled with +1s from Steven Wu, Daniel Weeks, Russell Spitzer, Eduard Tudenhöfner, Fokko Driesprong, Gang Wu, Szehon Ho, and Péter Váry, plus a long list of non-binding supporters. Gábor Kaszab volunteered to cover the encryption piece in an immediate follow-up. Contributor 전대홍 said he looks forward to building AMT in iceberg-rust, which tells you the Rust implementation intends to track v4 closely rather than trail it.

The third vote opened on October 13. Talat Uyarer, writing for the FILE type working group, called a vote to add the File type to the Iceberg v4 spec. This is a new data type for a range of bytes, stored either inline or as a reference to an external file. Its value semantics follow the FILE logical type in the Parquet spec. A file field carries six fixed, implicit sub-fields: uri, offset, size, content_type, checksum, and inline. They never appear in the schema and cannot be added, removed, reordered, or promoted. Their field IDs are offsets one through six from the file field's own ID, so adding one file column moves last-column-id forward by seven.

The restrictions matter as much as the feature. Equality, ordering, and hashing are undefined for file values. A file column cannot be an identifier field or a source for partition or sort transforms. It must default to null. A struct with the same sub-fields is not interchangeable with a file and cannot be promoted into one. And a uri makes no promise that the referenced object still exists. Relative URIs resolve against the table location, which keeps tables portable when you move them. Each sub-field gets its own column statistics, and writers should produce stats for uri, content_type, and inline. The format mappings are clean: the FILE logical type in Parquet, a record in Avro, and a struct tagged iceberg.struct-type=FILE in ORC. By the end of the window the thread had binding +1s from Russell Spitzer, Steven Wu, Daniel Weeks, Gang Wu, and Sung Yun, with no objections. The 72-hour window closes this week.

Why does a file type matter? Because the lakehouse is absorbing unstructured data. Images, PDFs, audio clips, and model artifacts already sit next to tables in object storage. Today a table points at them with a plain string column and a convention. A typed file column gives engines the offset, size, content type, and checksum they need to read, verify, and govern those bytes with the same metadata machinery Iceberg uses for everything else. For teams building AI pipelines on Iceberg, this is the bridge between "a table of embeddings" and "a table of the documents those embeddings came from."

Statistics and vectors

Two design threads moved toward consensus without a vote. Anoop Johnson's Iceberg v4 stats tightness discussion settled on a single file-level flag. Hongyue Zhang laid out why bounds stop being tight. Deletion vectors added after a file is written remove rows, so the original min and max bounds no longer describe the live data. Variable-length columns such as strings and binary also carry truncated bounds by design. Steven Wu and Szehon Ho backed the file-level approach, citing Russell Spitzer's analysis that most use cases repair every bound at once and do not need per-column tracking. Anoop summarized the upgrade path. Files inside existing v3 manifests carry no flag, so readers treat their stats as wide. New v4 files carry true when no deletion vector is attached and false when one is. This matters for query planning. An engine that knows bounds are tight can answer some aggregate queries from metadata alone and prune more files with confidence.

On the vector side, Yan Yan extended the Thursday biweekly sync to cover both the File and Vector types and posted a draft spec PR built on the Parquet community's emerging vector work. The meeting notes set a narrow first scope. Iceberg will start with a one-dimensional numeric vector of fixed length with non-null, finite coordinates, where +0 and -0 both stay valid. The initial element types are int, long, float, and double. Narrower types such as int8 are deferred as possible future Iceberg primitives, though the Parquet representation can still carry them. Distance functions and sparse vectors are out of scope for now and will return with the index design. Formats without native vector support will use a list representation with writer-side enforcement, and the group will explore an Avro vector logical type with the Avro community. ORC support remains open.

REST catalog contract

Two REST catalog threads closed out with concrete changes. Rahul Mahadev posted the result of the User-Agent vote: the recommendation for a standard User-Agent format for REST clients passed with 19 +1 votes, 10 of them binding, and no objections. The reopened vote had collected late +1s from Gang Wu, Zheng Hu, Ryan Blue, Fokko Driesprong, John Zhuge, and Huaxin Gao. Zheng Hu flagged CVE scanning failures on the PR, so watch for that before merge. A shared User-Agent format sounds minor, but it gives catalog operators a reliable way to see which engines and library versions hit their servers, which helps with deprecation planning and incident response.

The second REST thread resolved a gap in credential vending. Youngrae Kim had asked what a server should do when the X-Iceberg-Access-Delegation header cannot be satisfied. Daniel Weeks suggested adding a none value, meaning the client does not need delegated access from the catalog. Yufei Gu and Dmitri Bourlatchkov agreed. Dan and Dmitri then argued against adding a sentence that explicitly allows a server to fail the request, since a server can always reject a request anyway. Youngrae opened spec PR #18480, which adds none to the enum with one defining sentence and leaves the rest of the header description alone. Clients that bring their own storage credentials now have a clean way to say so.

Smaller threads with real impact

Sandeep Gottimukkala proposed a spec clarification that readers must accept promotable types in any data file. Type promotion lets a field evolve, for example from DECIMAL(8, 0) to DECIMAL(25, 0), without rewriting old files. The clarification states that a reader must handle the narrower physical type wherever it appears. Russell Spitzer supported it but asked to drop language about whether the field was evolved, since that detail is not part of the invariant.

Revanth Ch confirmed the migration plan for the new expression JSON forms. With Ryan Blue's guidance that servers always omit residual filters, the new expression syntax only flows from client to server in plan requests, so no client capability signal is needed. Revanth will open a PR so the reference implementation stops sending a residual on every scan task, then continue building conformance fixtures.

Hongyue Zhang and a contributor writing as dzeri96 discussed making appendUpdate in BaseTransaction public. The use case is a custom index stored in a Puffin statistics file. Today that team injects custom pending updates through reflection to control conflict detection. The request is a small API change, but it points at a bigger need: index builders want first-class hooks into Iceberg's commit protocol. The Iceberg index sync that normally covers this was cancelled for October 12 because so many people were traveling. Huaxin Gao pointed everyone to the index spec PR in the meantime, and the next sync is October 26.

EJ Song posted a status update on row-granular concurrency for v3 deletion vectors. The first step merges concurrently added deletion vectors for pure delete commits. Huaxin Gao and Uroš Bojanić have reviewed it, CI is green, and it needs one more committer to sign off on the retry semantics in MergingSnapshotProducer. When this lands, two jobs that delete different rows from the same file will stop conflicting with each other.

On the integration front, Robert Kruszewski opened a PR for upgrading arrow-java from 15.0.2 to 19.0.0, which positions Iceberg for arrow-java 20 and its expected Decimal32 and Decimal64 support. Vladislav Sidorovich asked for another review on moving Delta Lake migration to Delta Kernel after addressing all comments. K Krishna raised two PyIceberg Azure fixes that had waited two weeks. One fixes a bug where FsspecFileIO wrote an inferred storage account back into shared properties, so tables spread across two storage accounts got the wrong filesystem. Fokko Driesprong picked them up.

Alex Stephen announced a new batch of public datasets behind a public Iceberg REST catalog: 72 tables with over 640 billion rows. If you want to test an engine against real Iceberg tables at scale without loading data yourself, this is a free starting point.

Community events filled out the calendar. Scott Haines announced the October virtual meetup for October 23 at 9:00 AM Pacific, featuring Liam Earley on improving scan latency with manifest work in iceberg-rust. Kevin Liu boosted the Seattle Iceberg meetup on October 21.

Apache Polaris

Where server storage access comes from

The most consequential Polaris discussion this week was about boundaries. Dmitri Bourlatchkov opened a thread on removing default table properties from server-side FileIO configuration, and Vignesh posted a parallel thread on dropping catalog table-default.* properties from server FileIO construction. Both grew out of review on PR #5328, the table metadata cache work.

Here is the problem. Polaris builds server-side FileIO objects from a mix of StorageAccessConfig and catalog-level default table properties. Vignesh noted that an earlier fix already stopped using caller-supplied metadata properties for server FileIO, tied to CVE-2026-97395. Commit, refresh, and register paths still pass catalog table-default.* properties into the FileIO factory. Everyone agreed the target state is a clean one: server-side FileIO should come from StorageAccessConfig alone.

The debate was about custom settings. Yufei Gu pointed out that deployments need to customize FileIO in ways nobody can predict. Dmitri first proposed letting the StorageAccessConfigProvider pull relevant catalog properties in. Jean-Baptiste Onofré preferred a different shape: a properties bag on PolarisStorageConfigurationInfo for custom FileIO settings, with MinIO and S3Mock endpoints and path-style access living on the AWS storage configuration where those fields already exist. Vignesh backed JB's shape and offered to write the cleanup PR, including moving test fixtures off table-default.s3.*. Dmitri agreed with the property bag but raised an upgrade concern. Environments that rely on table defaults today will break if Polaris stops reading them overnight, so he proposed a transition path. Expect the final PR to include a deprecation window.

This connects to Yufei's question about metadata immutability across storage endpoint changes. If two S3-compatible stores hold different files at the same s3:// path, can a storage endpoint change make an existing table's metadata URI return different content? Yufei argued table metadata should not change because a FileIO client was configured differently. Dmitri reframed it as a cache correctness question: Polaris does not control what external stores contain, so it cannot assume identical content across endpoints for caching. JB split the difference. Iceberg treats metadata files as immutable once written, so the URI acts like a content identifier in spirit. But an s3:// URI is not a content hash, so Polaris cannot enforce that. Robert Stupp summarized the two assumptions the system relies on and noted that neither follows from the URI and storage configuration staying the same. The practical outcome will shape whether the storage endpoint becomes part of the metadata cache key.

Dmitri also flagged expanding the config entries in StorageCredential responses for community awareness, since it touches credential vending. PR #5731 from Adam lets Polaris return the full client-visible storage access configuration next to credentials. JB supported it, noting that Iceberg defines StorageCredential.config as a prefix-scoped map that S3 and GCS FileIO overlay on their base properties. He asked for one rule to be made explicit before merge: if the credentials entry is meant to be the complete access config, it must hold in every case, including remote signing.

On the Polaris side of the access-delegation question, the matching Polaris thread confirmed that Polaris keeps its fail-fast behavior, returning HTTP 400 when delegation cannot be satisfied. Youngrae Kim noted the vended-credentials docs already explain both 400 responses and tell clients to drop the header and configure their own storage credentials. Once the Iceberg spec adds none, Youngrae plans to add support for it in Polaris. JB also noted that Iceberg 1.12 adds a remote signing configuration, so the community will need to revisit combined requests for vended credentials and remote signing.

Labels, branches, and atomic commits

Andrei Tserakhau opened a design discussion on IRC labels in Polaris now that the labels field has merged into the Iceberg REST spec and Polaris tag support is close. He proposed treating labels as a read-only projection over Polaris tags rather than a separate entity. Tags already carry definitions, allowed values, inheritance, and reverse lookup, and both models key on Iceberg field IDs. A second labels entity creates two sources of truth. Sung Yun agreed for the producer side and asked whether federated catalogs need to store foreign labels at all or can forward them straight to clients. Andrei answered that forwarding works for engines, but Polaris users need foreign labels represented as read-only tags to see them through Polaris discovery and reverse-lookup APIs. JB, picking up a conversation from Glasgow, backed the projection approach and asked about permissions and performance. Andrei proposed reusing tag visibility rules, so loadTable only returns labels the caller can already see through the Tag API, and caching effective tags per table so the hot path does not resolve inheritance on every call. He also suggested gating the feature behind a flag at first.

Ankur Kotwal started a discussion on catalog-level branches for atomic multi-table promotion, a feature users have requested in issue #3469. Iceberg already supports branches on a single table. What users want is a way to branch an entire catalog, write to many tables in isolation, and promote all of them to main in one atomic step. That is the lakehouse version of a database transaction across tables. After Dmitri questioned the need for a new entity type, Ankur dug into the code and found that commitTransaction is already atomic on both persistence families. He opened a test and doc PR (#5740) showing a recipe built on existing pieces, then shared a design doc for a native option where a branch becomes a second named pointer next to heads/main, backed by a per-catalog registry.

That work feeds directly into the long-running thread on consistent multi-object changes in Polaris persistence. EJ Wang's updated proof of concept centralizes business logic in a shared manager instead of reimplementing it across backends. JB praised the separation of concerns and noted it fixes a maintenance burden that dates back to the move from OpenJPA to JDBC and NoSQL. Ankur ran the multi-table promotion test from #5740 against EJ's branch on H2, plus two race tests, which is exactly the kind of cross-thread validation that makes a design trustworthy. Dmitri said he needs a couple of weeks to review the PoC properly.

Yufei also endorsed a proposal on avoiding validation of historical metadata references on every table commit. When write.metadata.path changes, Polaris does not need to revalidate every historical metadata file. Leaving old files in place preserves immutability for past snapshots, and moving them belongs in a separate migration process.

Operations and extensibility

Tobias Riemenschneider reported that NoSQL persistence retry timeouts are not configurable and run shorter than the recommended MongoDB timeouts. Running Polaris 1.8.0 on MongoDB, his team sees table commits fail after 15 seconds with zero retries when MongoDB stalls behind a lagging secondary. The retry loop checks its total budget only after an attempt ends, and nothing bounds a single attempt. Robert Stupp confirmed the diagnosis: RetryConfig was always meant to be configurable, but SmallRye Config ignores it because PersistenceParams.retryConfig() is a Java default method. Tobias filed issue #5777 and PR #5779 the same day. JB noted that JDBC already exposes max retries, max duration, and initial delay, so the NoSQL side will now match.

JB opened a broader thread on Polaris server extensions and custom entities, prompted by users he met in Glasgow. Authorizers and event listeners are easy to extend by shipping a module with a Jandex index. Custom entities are not. Users currently have to edit Polaris core and add values to the entities enum. JB opened PR #5776 to document the current state and asked the community how to make custom entities a supported extension point.

Vignesh landed the prerequisite for a shared CloudEvents envelope for Polaris events. PR #5717 adds a shared mapper next to the events SPI, with CloudEvents context at the top level, Kafka-parity fields in the data payload, and lowercase type names such as org.apache.polaris.after_create_table, following Dmitri's suggestion. The webhook listener in PR #5084 will rebase on it. Dmitri also pushed the REST endpoints for table metrics and events (PR #4115) into a final review round now that metrics grants merged separately.

Yong Zheng proposed a delete-dataset simulation for the benchmark tool. Re-running benchmarks against persistent backends gives skewed results because old metadata piles up. Yufei asked whether resetting the persistence layer directly is faster. JB preferred keeping the API-based delete as the default, since it works against any backend and keeps the benchmark a plain client of a running server. Dmitri, Yufei, and JB all signed off on the approach in polaris-tools PR #286. A smaller cleanup thread on role cleanup tasks concluded that the async cleanup flag on role deletion was a leftover from the initial code import. Ayush Saxena traced it to the first commit, and JB, Dmitri, and Vignesh agreed to remove it since roles have no storage to purge and grant records already get cleaned up synchronously.

Process and people

Alex Dutra's thread on reducing CHANGELOG.md merge conflicts drew 19 messages. Ayush Saxena brought numbers: 75 of 339 commits on main over 60 days touched CHANGELOG.md, roughly one in four and a half. Alex proposed per-PR changelog fragment files. JB questioned whether a hand-maintained changelog is worth the effort at all and suggested assembling release notes from labeled PRs. Robert Stupp suggested conventional commit messages. Yufei and Alex both raised AI-assisted summarization at release time as an option, with Alex listing the tradeoffs around missing context. Dmitri clarified that the changelog was never meant to have an entry for every PR. The group settled on a first step: Yufei's PR #5733 tightens the guidance so only release-note-worthy changes get entries. If conflicts persist, the changelog.d fragment approach is next.

JB announced that Ayush Saxena is a new Polaris committer, and congratulations poured in from across the community. Ayush's fingerprints are all over this week's threads, from changelog statistics to git archaeology on role cleanup. JB also cancelled the October 15 community meeting.

Apache Arrow

Arrow 26.0.0 ships

Raúl Cumplido posted the result of the Arrow 26.0.0 release vote. It carried with six binding +1s from Rossi Sun, Sutou Kouhei, Gang Wu, Raúl Cumplido, Adam Reeve, and David Li, plus non-binding +1s from Xuanwo, Tadeja Kadunc, and Jean-Baptiste Onofré. The verification thread reads like a tour of the platforms Arrow now has to support. Gang Wu verified on an Apple M5 Pro with Clang 21. Raúl ran source, reproducible-source, integration, binary, wheel, and APT checks, including a patch to validate a Python 3.15 release candidate. Adam Reeve tested on Fedora 44 with Python 3.14. Tadeja verified on macOS with Ruby 4.0 and found a couple of GLib and Ruby script fixes she plans to send upstream. A major Arrow release touches C++, Python, R, Ruby, GLib, and more, so the breadth of hands-on verification is the real story here.

The Rust implementation moved in parallel. Jeffrey Vo opened a vote on Arrow Rust 59.3.1 RC1, and L. C. Hsieh, Raúl Cumplido, and Andrew Lamb all cast binding +1s after verifying on M4 and M3 Macs and Debian 14. Arrow Rust is the foundation under DataFusion, iceberg-rust, and a growing set of Rust query engines, so its patch releases ripple outward fast.

ADBC grows up

Ian Cook brought back a new ADBC logo proposal after more design work. Broad support led to PR #4870 in arrow-adbc, which adds the logo, a new Visual Identity page, and an updated trademark notice. Rusty Conover gave detailed feedback: the project URL on the hex badge is hard to read at small sizes, and the text spacing did not sit in harmony with the chevron mark. Ian posted a revised version that addresses both points, and Dewey Dunnington approved. Barring objections, the logo passes by lazy consensus after 72 hours.

A logo matters because ADBC is becoming a brand people build on. That showed up in a second thread: Awanish Gupta from IBM asked to contribute a native C++ ADBC driver for IBM Db2. David Li explained that the adbc-drivers GitHub organization sits outside the Apache Arrow project, offered to set up access directly, and pointed to the driver authoring docs. Asked about scope, Awanish confirmed the driver targets Db2 for Linux, UNIX, and Windows only, not Db2 for z/OS. Every new ADBC driver widens the set of databases that can hand Arrow data to a lakehouse engine without row-by-row conversion.

Format work and community

The format track kept moving. Florian split the spec text for fixed and variable closedness range canonical extension types into its own PR (#53239) so the vote covers only the format change, and he asked to keep the vote open since no votes had arrived yet. David Li signed off on the latest revision of the JSON representation of Arrow schemas, a small but useful step for tools that want to exchange schemas without the Flatbuffers dependency.

Ian Cook reviewed Pearu Peterson's long-waiting CSV date and time parsing PR and noted that Pearu's emails, like many others recently, had landed in spam folders. Andrei Tserakhau reported the same thing on the Iceberg list this week. If you posted to an Apache list recently and heard nothing, check whether your message reached people's inboxes before assuming silence means disinterest.

Antoine Pitrou filed the Arrow board report, Ian Cook ran the October 7 community meeting, and Andrew Lamb announced on behalf of the PMC that Richard Baah is a new Arrow committer.

Apache Parquet

What does "distinct" even mean?

The sharpest Parquet debate this week was over a single word. Adrian Garcia Badaracco opened format PR #628 to classify distinct_count metadata as inexact after a sync call reached consensus. Micah Kornfield asked for the reasoning, since Andrew Lamb had pushed back. Russell Spitzer explained it. Exactness of the number of distinct values is not standardized for many types. Strings with collations treat "foo" and "Foo" as equal or not depending on the system. Floating point has its own traps: NaN against NaN, and -0.0 against 0.0. Most use cases only need approximate statistics anyway, so the group chose to correct the language rather than define exact semantics for every type. Daniel Weeks added that older readers and writers already disagree about logical and physical equivalence, so assuming current values are exact was never safe. Andrew still sees changing "count of distinct values" to "estimated count" as a real change in meaning, not a clarification. Dan agreed the proposed wording is not ideal and offered alternatives on the PR.

Russell also left a reminder that deserves repeating: when you cite a decision from a sync call, post notes to the list. Decisions that live only in a video call are invisible to the rest of the community.

The vector type fight

The numeric vector type thread turned into the most substantive design argument on any list this week. A focused call had produced a proposal for a FINITE_VECTOR logical type on fixed-length lists. Ryan Blue laid out what a logical type provides. It describes data for users and readers. It signals that repetition and definition levels reflect a fixed number of elements, which implementations can use to decode faster. It guarantees no null elements. And the finite variant guarantees no NaN or infinity values, so every vector has a finite length and works for distance calculations. Steven Wu agreed that FINITE_VECTOR brings value and argued against forcing every use case into a generic fixed-length list. Alkis Evlogimenos suggested naming it VECTOR to match other engines and type systems, leaving room for a non-finite variant later. Gang Wu supported separating FINITE_VECTOR (or VECTOR) from FIXED_LENGTH_LIST and moving the vector type first.

Then Jan Finis made the case against. He argued that the finite restriction adds little value on its own and that the fixed-length part should be a general feature rather than a one-off. Existing types already handle uncompressed vectors well, he said. The real win is quantization, where storage size and processing speed improve by orders of magnitude. Will Edwards made a similar point from the engine side. He wants every engine to offer vector functions on lists of floats without a new logical type, and he sees more value in a type that describes how to unpack quantized codes, scales, and codebooks, which is where machine learning is heading.

This is a genuine disagreement about what a file format owes the AI wave. One camp wants a simple, well-defined type now so engines and table formats can align, and Iceberg's vector work is already building on it. The other wants Parquet to skip the label and go straight to the compression problem. Both sides are right about something. A narrow VECTOR type gives Iceberg a stable target this year. Quantized storage is where the cost savings live. Watch for a proposal that ships the simple type first with an explicit path to quantized encodings.

A concrete plan for versioning

Alkis Evlogimenos restarted the stalled versioning discussion with a concrete proposal. His two goals are to ship features when they are ready rather than waiting for the next bundle, and to guarantee that a reader either returns correctly interpreted results or fails, never a false read. He sorts every format change into two kinds. "Add-everywhere" changes are valid at every compatibility level, including in today's files. They are safe when any reader that does not understand them either fails where it needs them or is unaffected. A new encoding is the classic example: a reader that does not know ALP fails on the ALP column and reads every other column. "Compatibility level" changes are for things existing readers misread instead of failing on, such as metadata redefinitions, removals, or a new footer. A reader that does not support the level fails before reading anything.

The most useful part of the proposal is the field survey. Alkis checked eight readers against their current source: Arrow C++, parquet-java, arrow-rs, DuckDB, Velox, arrow-go, cuDF, and fastparquet. Unknown encodings are mostly safe, though arrow-rs rejects the whole footer and parquet-java does the same when the encoding shows up in encoding stats. Unknown logical types are safe because every reader falls back to the physical type. Unknown page types are not safe: Arrow C++, DuckDB, and Velox skip them, so a new page type carrying rows gets dropped silently. Unknown compression codecs are not safe either: Arrow C++ treats them as uncompressed and decodes compressed bytes. Andrew Lamb praised the proposal and noted a third category, forward-compatible additions that older readers simply ignore, such as new statistics. This survey alone is a valuable document for anyone writing a Parquet reader.

Encodings and new types

Prateek Gaur reported that the C++ implementation of ALP encoding for floating-point data has merged, with follow-up items coming in a later PR. ALP compresses floating-point columns far better than the general-purpose codecs, which matters for sensor data, financial data, and embeddings. Asked who is working on new encodings, Andrew Lamb listed two active efforts: Prateek on PFOR (patched frame of reference) and Aarnav on FSST and OnPair, symbol-table-based string encodings.

Micah Kornfield pushed back on part of the WIRE logical type proposal for serialized wire-format messages. He does not want to move ahead with the projection component until an open source engine commits to using it, citing how hard variant was to get right in engines. He wants documenting serialized values kept separate from projection. Thomas Kissinger agreed to continue the extensible decimal floating-point type discussion in Costas Zarifis's design doc. Julien Le Dem submitted the October board report after a week of collecting status, and Russell noted that the timing collided with Community Over Code travel. Julien also announced the October 7 Parquet sync.

Apache DataFusion

Comet asks to leave home

Andy Grove posted a proposal to establish Apache Comet as a top-level project. His argument is that Comet differs from other DataFusion subprojects. The others extend DataFusion. Comet is an accelerator for Apache Spark that uses DataFusion as its execution engine, and its community has grown largely distinct from DataFusion's. The numbers back him up: 24 releases since July 2024, including 1.0.0 in August and 1.1.0 on October 6, and 146 people with merged PRs, 81 of them in the last 12 months.

The proposal is conservative by design. The name "Apache Comet" already cleared the podling name search. The initial PMC is drawn from existing DataFusion PMC members (Andy Grove, Andrew Lamb, Liang-Chi Hsieh, Matt Butrovich, and Oleks V.) with nobody promoted as part of the move, and Andy offered to chair. Initial committers are the existing DataFusion committers who contribute to Comet. The repository moves from apache/datafusion-comet to apache/comet, with docs at comet.apache.org. Joining Comet changes nobody's role in DataFusion. Confirmations came quickly from Bhargava Vadlamani, Manu Zhang, L. C. Hsieh, Matt Butrovich, Parth Chandra, Martin Grigorov, and Chao Sun, with Andrew Lamb adding his support. Andy plans a formal vote after about a week. He corrected the timeline in a follow-up: the next DataFusion board meeting is December 16, not November 18.

Comet also shipped this week. Andy announced that Comet 1.1.0 passed its vote with nine +1s, six binding, and published the jars to Maven. He also proposed an update to the Comet contributor guide on AI in PR #6734. Automated reviews can help, the guidance says, but every PR approval must come from a human, with or without AI assistance. That is a clear, enforceable line, and other Apache projects are likely to borrow it.

Releases across the family

The DataFusion family shipped on several fronts. Oleks V. announced that DataFusion 55.2.0 passed with five binding +1s and is published to crates.io. Andy reported that Ballista 55.0.0 passed with six +1s, four binding. Tim Saucer opened the DataFusion Python bindings 55.0.0 vote, which collected binding +1s from Marko Milenković, Andrew Lamb, L. C. Hsieh, and Andy Grove. Xuanwo's careful verification caught a missing NOTICE file in the source archive, a fix requested for the next release. Tim also closed out the Python bindings 54.1.0 vote with seven +1s, five binding.

Andy also proposed removing the Spark compatibility feature from Ballista. The experimental, off-by-default feature covered only a handful of expressions. After six years on Spark-compatible engines, Andy wrote, he knows Spark compatibility is a project that never finishes because Spark keeps changing, and other projects already fill that niche. Read alongside the Comet proposal, the direction is clear. Comet owns Spark acceleration. Ballista focuses on being a distributed DataFusion engine. Each project gets a sharper identity.

Apache Ossie (incubating)

Where does an MCP server belong?

Apache Ossie is the open semantic interchange specification, the standard for moving metric, dimension, and relationship definitions between BI tools, semantic layers, catalogs, and AI agents. This week its busiest thread asked a question every semantic layer team is facing: should Ossie ship an MCP server, and where should it live?

The thread started from GitHub discussion #510 and PR #529, which proposes an execute_query contract. Chris Eubank asked the key question: how tightly is execute_query tied to Ossie semantics? Is it meant to steer clients toward portable Ossie queries, or to standardize running any vendor's query language? As drafted, the model identity was optional and the result schema loose, so it looked closer to the second. Markus Cozowicz argued for pragmatism. Vendors will take time to build conversions, and each vendor has features the standard lacks, so allowing all registered Ossie dialects is a reasonable start. He also made an interesting point: if a query execution contract lands in the Ossie standard and then in LLM training data, nobody has to teach each agent how to run queries against a semantic model.

Jean-Baptiste Onofré drew the line clearly. Ossie is an interchange spec for semantic models. A running MCP server brings execution, connections, auth, and a fast-moving protocol surface into the project. He prefers that MCP servers live in engines and catalogs, with Ossie defining the contract and possibly a REST API, but not the server implementation. Chris agreed that the dividing line is whether a query resolves against an Ossie model, and said there is interest on his side in building query capabilities over the Layer 2 and Layer 3 query interfaces once they land in the core spec. Jakub Moravec asked whether defining skills in Ossie and leaving tools to vendors is enough. Ashfaqbs responded that skills are advisory text while tools are executable contracts, and that validation belongs in Ossie while conversion and query belong to vendors. Jakub agreed with that split.

The most valuable contributions came from people running code. Kyoung Min Kim ran an experiment on a real database of 33 datasets and 45 relationships, generated an Ossie model, and asked three models to return Layer 3 query specs that a compiler turned into SQL. Aleksandr Kosachev offered his ClickHouse executor as a test bed. It now accepts the aggregation query shape from PR #246 as a JSON object and returns #246 error codes such as E_NAME_NOT_FOUND, E_NO_PATH, and E_AMBIGUOUS_PATH inside the #529 error envelope. His first report found the executor raised 10 of the 28 #246 codes as specified. A later update showed it now follows paths across several many-to-one relationships, rejects a second path as ambiguous, and merges measures across datasets, with answers matching expected rows line by line. JB took the lesson that a small shared query shape plus error codes works, and that the next real test is a second engine running the same cases. Where two engines disagree, PR #246 is underspecified, and those are the spots to fix. Markus confirmed PR #529 is now narrowed to the declarative case.

Markus also captured a practical agent problem in the same thread. Filter values are where agents fail. A user asks about "Coca & Cola" when the data says "Coca-Cola Company." He laid out three strategies: put the vocabulary in the ontology when context allows, expose a dedicated tool for looking up column values, or return suggested values reactively when a query comes back empty. Each has tradeoffs in tool count and accuracy.

Pinning down the query layer

Several threads worked on the query layer that the MCP debate depends on. Chris Eubank opened a thread on model expressions versus query semantics, pointing out that the expression language spec scopes itself to defining a model and says the SQL interface comes in a separate document. Today, expressions define a model. They do not create a query interface. Justin Talbot pushed a revision of the Relational Query Interface spec (PR #354). The biggest change renames the measure evaluation function from MEASURE() to AGG() as the single normative spelling, with other spellings allowed as non-normative synonyms. Chris agreed with the changes and suggested deferring a SQL-native syntax for declaring measures to a future extension. Justin also opened a vote on the specification layers overview (PR #452), an informational document describing Ossie's four layers: Model Definition, Relational Query Interface, Wide Table Query Interface, and Ontology.

Ananth Tirumanur reworked his proposal to add an ICEBERG expression dialect to the core spec (PR #491). Reviewers worried that a new vendor dialect without a reference guide invites parser divergence. Ananth's fix: the dialect defines no new syntax. It is the predicate string accepted by PyIceberg's expression parser, which serves as the reference reader. Chris praised anchoring a dialect to an open source parser as a precedent worth encouraging. JB asked where the dialect applies, since the schema only uses dialect expressions for fields and metrics, and a predicate-only dialect fits boolean fields and nothing else. Kyoung Min Kim ran the PR's predicate table through PyIceberg 0.6.1 through current main. Thirty of 35 forms behave the same in every version, and every unsupported form is rejected everywhere. The five that differ include BETWEEN and double-quoted identifiers, which parse from 0.10.0. That data settles what minimum version the dialect needs to pin. For lakehouse teams, an ICEBERG dialect means a semantic model can carry a filter that an Iceberg-native engine pushes straight down to a table scan.

The expression language itself grew. The vote to add the FILTER (WHERE ...) aggregate modifier collected support from JB, Klahr Joshua, and Ayush Saxena after discussion with Julian Hyde and Chris. Chris noted that the || concatenation operator now renders correctly in the spec. Aleksandr opened a discussion on NULLS FIRST and NULLS LAST in window ORDER BY (PR #577). Chris volunteered to rewrite spec wording for dataset-scoped metrics, and Kyoung Min will narrow the validator so qualified references become errors. JB asked that the rule be stated in the spec text and backed by a conformance test.

Measuring whether Ossie works

Chris Eubank raised a question every interchange standard eventually has to answer: how do we measure semantic interoperability? He ran the simplest possible model, one sales dataset and one SUM(sales.amount) metric, through Ossie to vendor and back to Ossie across 14 converters. Eight of 14 preserved the ANSI_SQL expression. Only one of 14 preserved OSSIE_SQL_2026. The rest emitted a vendor dialect, dropped the metric, or lacked a vendor-to-Ossie path. Chris was clear that this measures preservation of the portable dialect, not semantic equivalence. He proposed a shared interoperability test suite in the repository with reference models, measurement utilities, and published results for every Ossie-hosted converter. This is the right instinct. A standard that only measures file format support can look healthy while portable meaning leaks out at every hop.

The converter list keeps growing anyway. Tan Huynh from Holistics proposed a Holistics converter that runs both ways without network access, along with a HOLISTICS_AQL dialect entry. Mahmoud Hassanen proposed a native ClickHouse dialect converter, since ANSI SQL transpilation produces poor queries when ClickHouse needs functions like uniqExact and toStartOfMonth. JB explained that converters translate semantic model definitions and are not SQL transpilers, and that new converters will go to the apache/ossie-converters repository he is preparing. Mahmoud opened PR #558 to register CLICKHOUSE in the dialect enum. An OpenMetadata converter discussion continued as well.

GitHub discussions rounded out the semantic modeling agenda: metric additivity (additive, semi-additive, non-additive), where Justin Talbot noted BI tools rely on additivity to roll up results client-side and skip database round trips, certified metrics and certifying authority, a shared semantics reference vocabulary, metric trees, and a proposal on how the Open Knowledge Format relates to Ossie, which Markus Weimer welcomed as a canonical way to feed agents context from both. JB posted a revised Ossie REST API spec as PR #573, and Dmitri Bourlatchkov of Polaris endorsed it, noting Polaris as a possible implementation. Nolan Clark asked that the model management API avoid assuming a one-to-one mapping between inputs and converted outputs. Ankit Tandon posted notes from the October 13 Ontology working group.

Openness and community

Julian Hyde asked a blunt question: "Are we open?" Much of the project's work seemed to run vendor to vendor, he wrote, and it was not obvious how to attend a working group meeting. Kurt Stirewalt and Matthew Mullins pointed to the community page, Slack, and a public calendar of open working group meetings, while conceding the information is hard to find. Ayush Saxena asked for specifics. Julian clarified: Slack and the working groups are open in the sense that you can ask to join, but their content is invisible to search engines and unauthenticated readers. Aldre Jota then reported that the Slack invite links on the site were revoked or broken. Klahr Joshua invited anyone to improve the community page through the ossie-website repository, and Julia Blyumen suggested clarifying organizational versus individual membership. This is a healthy thread. Apache's rule is that decisions happen on the list, and a young project dominated by vendor working groups needs to keep proving it follows that rule.

JB proposed the first Apache Ossie workshop and meetup: a full-day hands-on workshop followed by an evening of talks. Julian supported it with the standard Apache caveats, that the workshop is for knowledge transfer rather than decisions and that recordings should be posted. Support came from Dragos Crintea, Kurt Stirewalt, Yufei Gu, Ankit Tandon, and Jochen Christ, who said a full-day onsite session is worth a flight from Europe. JB is working on logistics. Ayush Saxena announced that Chris Eubank is a new Ossie committer, a fitting recognition given how many of this week's threads he started or shaped.

Cross-Project Themes

The lakehouse is learning new data types for AI, in lockstep. Iceberg's File type follows Parquet's FILE logical type. Iceberg's vector design builds on Parquet's emerging vector representation, and the Iceberg vector sync explicitly defers to the Parquet decision. Parquet's ALP encoding targets the floating-point data that embeddings and sensor workloads produce. Arrow-java 20 is expected to bring Decimal32 and Decimal64, which Iceberg is already preparing for. These are not separate efforts. The table format, the file format, and the in-memory format are converging on one shared vocabulary for unstructured bytes and numeric vectors. The Parquet vector debate is the critical path. Whatever Parquet decides about VECTOR versus quantized encodings will set the shape of AI data in Iceberg tables for years.

Catalogs are getting exact about access. The same people show up across Iceberg and Polaris on the same problem. Youngrae Kim, Yufei Gu, Dmitri Bourlatchkov, and JB carried the access-delegation question from the Iceberg spec into Polaris behavior in one week. Polaris is separating catalog configuration from server storage access, scoping what StorageCredential responses contain, and deciding whether storage endpoints belong in cache keys. Iceberg added a standard User-Agent format. Polaris is mapping IRC labels onto its tag model. As AI agents start calling catalogs directly, these precise rules about who gets which credentials and which classifications become the security model. Loose defaults stop being acceptable.

Semantics and agents are the new frontier, and the lakehouse is the ground floor. Ossie's MCP debate, its ICEBERG dialect, and its query interface work all point the same way. Agents need a governed semantic layer to ask questions, and that layer needs to push work down to open tables. An ICEBERG dialect anchored to PyIceberg means a metric definition can carry a predicate an Iceberg engine executes directly. Polaris showing up as a possible implementer of the Ossie REST API means the catalog and the semantic layer can share one interface. This is the open lakehouse stack extending upward: Parquet for files, Iceberg for tables, Polaris for the catalog, Arrow for movement, and Ossie for meaning.

Projects are formalizing how humans and AI share the work. Comet now requires a human to approve every PR. Polaris debated AI-assisted changelog summaries and chose to tighten human guidance first. Several contributors on the Ossie and Polaris lists described writing code first and discussing second, since code is cheap now. The common thread is accountability. Apache projects welcome AI help, but they want a named human responsible for every merge and every release note.

Looking Ahead

The Iceberg File type vote closes this week, and the AMT baseline vote has no objections on record, so expect both results soon. The Comet top-level project discussion moves to a formal vote around October 16, with the board resolution targeted at the December 16 meeting. On Parquet, watch whether the vector debate converges on a simple VECTOR type or detours into quantized encodings, and whether Alkis's versioning proposal picks up a sync slot. Polaris will see the FileIO cleanup PR and the NoSQL retry fix land. Ossie's workshop logistics, the converters repository move, and a second engine running Aleksandr's ClickHouse test cases are all worth tracking. On the calendar: the Seattle Iceberg meetup on October 21, the virtual Iceberg meetup on October 23, and the Iceberg index sync returning October 26.

Go Deeper With the Open Lakehouse

If this newsletter helps you keep up with Iceberg, Polaris, Arrow, Parquet, DataFusion, and Ossie, my books go further. I have written more than 50 titles on the open lakehouse, Apache Iceberg, Apache Polaris, agentic AI, and data engineering, from the O'Reilly definitive guides to hands-on Python and architecture books.

Browse the full catalog at books.alexmerced.com and find the right book for where you are in your lakehouse journey.

Haiku 5.5, Muse Wonder, Open Racks: Oct 6-14

Alex Merced — Wed, 14 Oct 2026 16:50:37 +0000

This week Google returned to the frontier. Gemini 4 Argon, its first new flagship since Gemini 3 Pro, became generally available on October 13 at an introductory price of $2 per million input tokens. The price of a capable model fell at the low end too. Anthropic shipped Claude Haiku 5.5 at a tenth of its predecessor's token price. Meta released its first frontier model, OpenAI rebuilt ChatGPT answers around interactive interfaces, and Mistral previewed a trillion-parameter open-weight flagship.

Under the models, the hardware story moved to the rack. At the Open Compute Project Global Summit, NVIDIA, AMD, Microsoft, Intel, and Marvell all showed new chips and open rack designs. Memory stayed the tightest link in the chain.

This issue follows the usual order. Models first, then tooling, then standards, then infrastructure. Each section ends with what the news means for teams building with AI and data.

Models: Google Returns to the Frontier With Gemini 4 Argon

Gemini 4 Argon

Gemini 4 Argon is the lead model story of the week. Google first announced Argon on September 30, but only for trusted cybersecurity partners in its Fairwind Program. On October 13, Google made Argon globally available to developers, enterprises, and paying consumers. It is the first model in the Gemini 4 family.

Access now runs through several doors. Developers can call it in AI Studio and on the Gemini platform under the model ID gemini-4-argon. Google AI Ultra subscribers get it in Google Antigravity, Google's agentic coding environment. Ultra and Pro subscribers get it in the Gemini app and AI Mode in Search. Enterprises get it in Gemini for business.

The price is the surprise. Argon launches at an introductory rate of $2 per million input tokens and $10 per million output tokens, with cached input 95% off. That matches GPT-6.1 Sol. The discount ends on November 30, 2026. After that, Argon costs $4 input and $20 output, the same band as Claude Opus 5.5 and Meta's Muse Wonder. Budget for the higher number, not the launch number.

The token limits stand out. Argon keeps a 1 million token input window. Google also raised the output limit to 1 million tokens, up from 64,000. Google describes this as a unified input-output limit, served through a new continuation protocol for long outputs. Every other model in this issue caps output at 128,000 tokens or less. Long outputs matter for agents that write whole codebases, long reports, or large data transformations in one run.

Google's benchmark claims are vendor-reported, so read them with care. Google says Argon ranks #1 on the Vals Index, which measures performance on finance, coding, legal, and tax work. It reports 87% on the Embodied Reasoning (ERQA) benchmark for physical and spatial reasoning. On the Gray Swan Indirect Prompt Injection benchmark, Google says Argon cut attack success to 0.7% across 15 adaptive attempts. For security work, Google reports that Argon matches GPT-6 Astra on CWE-bench, which measures finding and patching vulnerabilities.

Outside analysts are more measured. Futurum's read is that Argon puts Google level with OpenAI but behind Anthropic. TechTarget noted that Argon is Google's first new top-tier model since Gemini 3 Pro in November 2025, and that its claims still need proof in enterprise settings. Independent results on the full Vals suite and Artificial Analysis should land within days.

Google backed the launch with internal results. Engineering teams using Argon in Antigravity increased agentic code submissions by more than 35% and cut code rollbacks by nearly a third. Given a skills rulebook and a one-paragraph goal, Argon worked overnight on Google's open source Agent Development Kit. Its changes, now merged, cut in-memory session prep time by 40% in long sessions and sped up cold imports by 21%. Google also reports a 2.85x end-to-end inference speedup for Gemma 4 E2B on vLLM, with the work sent upstream as pull requests.

One internal result belongs in the infrastructure conversation. In the September 30 announcement, Google said a team of Argon agents analyzed fleet-wide profiling data and applied memory fixes across its data centers. The changes freed over 300 TiB of memory once rolled out, with 500 TiB to 1 PiB in total savings expected. In a month when memory is the scarcest part in AI hardware, a model that finds memory in existing software is worth noting.

Security shaped the rollout. Argon went to cyber defenders first, the same defender-first pattern Anthropic used with Claude Mythos under Project Glasswing. On October 13, Google began expanding Fairwind to hundreds of additional partners. Google plans to give trusted defenders a version of Argon without cyber guardrails. For the public model, Google says it trained Argon to pause for human confirmation before irreversible steps. It reports that shortcut attempts on coding tasks fell to 2%, and it runs a monitor that reads the agent's reasoning and actions to block unintended workarounds in real time.

What it means for you. For two months, the price matches GPT-6.1 Sol while the claims match the top tier. That makes now a cheap time to run Argon against your own evaluation set. Test the 1 million token output limit if your agents produce long artifacts, since no other frontier model offers it. Plan costs at the post-November rate of $4 and $20. And note where Argon lives. Its best agentic experience runs in Antigravity, which follows this week's pattern of labs keeping their strongest experience inside their own tools.

Claude Haiku 5.5

Anthropic released Claude Haiku 5.5 on October 7, 2026. It completes the Claude 5.5 family alongside Sonnet 5.5 and Opus 5.5. The API model ID is claude-haiku-5-5.

The headline is price. Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens. Haiku 4.5 lists at $1 and $5. That is a 90% cut in the base token price.

The pricing has a second tier. Above 100,000 tokens, the rate rises to $0.50 input and $2.50 output per million tokens. Haiku 5.5 is the only current Claude model priced this way. Sonnet 5.5, Opus 5.5, and Fable 5.1 bill the full window at one rate.

The context window grew five times. Haiku 5.5 takes 1 million tokens of context and returns up to 128,000 output tokens, up from 200,000 and 64,000 on Haiku 4.5. The cheapest Claude model now has the same window as the most expensive one.

Anthropic frames the savings as about 75% per task, not 90%. The gap comes from the tokenizer. Haiku 5.5 uses the tokenizer introduced with Claude 4.7, which produces roughly 30% more tokens for the same input than Haiku 4.5 did. Lower prices per token, more tokens per task.

Haiku 5.5 is also the first Haiku with Anthropic's effort settings and adaptive thinking. That changes the cost math again. Artificial Analysis scored Haiku 5.5 at 43 on its Intelligence Index at maximum effort, ahead of GPT-6 Luna at 38. But Haiku 5.5 produced about 440 million output tokens across the full test suite, against 140 million for Luna. Per test question, that worked out to about $0.21 for Haiku and $0.07 for Luna.

Those are independent numbers, not vendor claims. Artificial Analysis also reported 33% on Terminal-Bench 4.0, up from 0% for Haiku 4.5. It flagged one agent automation result as likely understated and plans to re-run that test.

Anthropic made two related pricing moves the same day. It halved the Sonnet 5.5 cache read price to $0.10 per million tokens and said that makes Sonnet 5.5 about 20% cheaper on most agentic work. It also added monthly API credits to subscriptions. Max 5x and Max 20x plans include $100 and $200 a month in Claude Platform credits, and Team plans get up to $500.

Availability is broad. Haiku 5.5 runs on the Claude API, Amazon Bedrock, Google Cloud, Microsoft Foundry, and Claude Platform on AWS. It also shipped in Claude Code and GitHub Copilot on day one.

Code written for Haiku 4.5 needs updates. Anthropic lists API changes that break older integrations, so test before you swap the model ID in production.

What it means for you. Small models do most of the work in agent systems. They classify, route, extract, summarize, and run as sub-agents under a larger planner. A 90% cut in the base rate makes it cheap to run many more of those calls. Keep prompts under 100,000 tokens where you can, since the price jumps five times above that line. And measure cost per task, not cost per token. The Artificial Analysis data shows a cheap token can still produce an expensive answer if the model thinks at length.

Meta Muse Wonder

Meta's Superintelligence Lab released Muse Wonder and Muse Wonder Full on October 12. TechCrunch called it Meta's first attempt at a frontier model. The earlier Muse Spark models targeted low latency. The Wonder models are larger and aim at coding and long-running agent tasks.

Developers can call Muse Wonder through the Meta Model API as muse-wonder-1.0. It costs $4 per million input tokens and $20 per million output tokens. That puts it in the same price band as GPT-6 Sol and Claude Opus 5.5.

The twist is the second checkpoint. Muse Wonder Full is only available to Muse Code subscribers and will not be served through the API. Both models also power Meta's Muse agent. So the best version of Meta's model lives inside Meta's own coding product.

Both models support a 1 million token context window and accept text, images, video, and audio. Meta stressed a feature it calls active memory, the ability to keep and reuse what a user shares over time.

On benchmarks, treat the numbers as vendor-reported. Meta's model card shows Wonder and Wonder Full scoring comparably to Claude Fable 5.1 and GPT-6 Astra. Wait for independent runs before you rank it.

What it means for you. Meta has moved from open weights to a paid frontier API. That is a strategy change, not just a model release. If your team planned around Meta as the open-weight option, the plan needs a second look. And the split between an API checkpoint and a subscriber-only checkpoint is a pattern to watch. When a lab keeps its best model inside its own app, API builders get the second-best version.

Mistral Large 4 Preview

Mistral opened a public preview of Mistral Large 4 on October 6. The preview runs on Mistral Studio and the Mistral API. Mistral says the open weights will follow by the end of October.

Large 4 is a mixture-of-experts model with 1.05 trillion total parameters. Mistral's launch post lists 49 billion active parameters, while its model page lists 52 billion. It takes text and image input. Mistral trained it from scratch on 3,800 Grace Blackwell GPUs in its own European data centers.

The context window depends on where you look. Mistral advertises 1 million tokens. OpenRouter documents 524,288 tokens. Check the limit on the endpoint you plan to use.

Mistral's coding numbers are vendor-reported. It cites 61.7% on DeepSWE v1.1. In a blind coding-quality evaluation run by Mistral with Surge AI annotators, the preview scored 3.74 out of 5. That placed it second of five, behind Claude Opus 5 at 4.22 and ahead of Kimi K3 at 3.59.

Independent testing gives a more modest picture. Vals.ai ranks Mistral Large 4 at #32 of 44 models on its Vals Index with 48.05%. Its best showing was #6 of 75 on Harvey's Legal Agent Benchmark.

What it means for you. The value of Large 4 is not the leaderboard. It is ownership. If Mistral ships the weights on schedule, teams get a trillion-parameter multimodal model they can run in their own data centers. That matters for regulated industries and for any team with data sovereignty rules. Plan the hardware budget now, since a model this size needs serious memory to serve.

OpenAI: GPT-6 Reaches Every ChatGPT User

OpenAI began rolling out GPT-6 with Intelligent UI across ChatGPT on October 7. Plus, Pro, Business, and Enterprise users got it first. Free and Go users started getting it on October 8.

The model behind it depends on your plan. Paid tiers run on GPT-6 Sol, and Free and Go tiers run on GPT-6 Luna. OpenAI says the release reaches more than 1.2 billion weekly ChatGPT users.

Intelligent UI is the real change. GPT-6 can now answer with graphics, buttons, forms, charts, and small interactive tools inside the chat. A question about splitting a bill can return a working calculator. OpenAI also lets GPT-6 start answering while it keeps thinking. For web search questions, GPT-6 Instant starts answering 44% faster on average than GPT-5.6 Instant, according to OpenAI.

There are limits. Intelligent UI does not work on the Pro reasoning option, which still runs Astra. OpenAI's safety card also notes that Codex and ChatGPT Work keep the September versions of the models.

For API users, the bigger model news came the week before. OpenAI introduced GPT-6.1 Sol at DevDay on September 29 as gpt-6.1-sol, priced at $2 per million input tokens and $10 per million output tokens. OpenAI reports that it matches GPT-6 Astra on DeepSWE v1.1 at roughly one-fifth the cost. This week it became the default in Codex CLI, which we cover in the tooling section.

What it means for you. Generated interfaces are now a mainstream answer format. That raises the bar for every data product. Users will expect a chart or a filter control when they ask a question about numbers, not a paragraph. It also raises a trust question. A generated chart looks authoritative even when the data behind it is wrong. Teams that serve data to AI assistants need clean, governed sources more than ever.

Google: Gemini App Access Shifts

Alongside Argon, Google also changed who gets which model in the Gemini app. Starting October 9, Gemini app users without a subscription get Flash-Lite, not Flash or Gemini Pro. Google AI Plus subscribers keep Flash but lose Gemini Pro on dates sent by email. Pro and Ultra subscribers keep all three.

This is the other side of the price war. Labs cut API prices for developers and tighten free consumer access at the same time. Argon itself is limited to Pro and Ultra subscribers in the Gemini app. Serving frontier models to hundreds of millions of free users costs real money.

Other Releases

Release trackers logged several smaller models this week. LLM Gateway lists Reka Edge 2603 on October 9, GLM 5.3 Fast from Z.AI on October 7, and Microsoft-Decision-1 on October 9. We have not confirmed specs for these against their labs' own announcements, so treat them as leads for your own testing.

Model Quick Reference

All benchmark figures above from Google, Anthropic, Meta, Mistral, and OpenAI are vendor-reported unless the text names an independent source such as Artificial Analysis or Vals.ai.

The Pattern in Models

Three trends stand out this week. First, prices keep falling at both ends. Haiku 5.5 resets the small-model floor, and Argon launches at a mid-tier price for a frontier model, at least until November 30. One million token windows are now standard across price levels, and Argon pushes output to 1 million tokens too. Second, labs are splitting releases into API versions and app-only versions. Meta's Wonder Full, OpenAI's ChatGPT-only Intelligent UI, and Argon's Antigravity experience all keep the best experience inside the lab's own product. Security-sensitive models now launch to defenders first, as Argon did through Fairwind. Third, cost per task matters more than cost per token. Adaptive thinking means two models with the same token price can differ by three times in what a real job costs.

Tooling: Coding Agents Compete on Control, Not Just Smarts

The coding agent race shifted this week. The new features were not about writing better code. They were about controlling agents: which model each one runs, what files and networks it can touch, and how many can work at once without stepping on each other.

Claude Code 2.1.296

Anthropic shipped Claude Code 2.1.296 on October 9. The headline feature is workflow-level model pinning. A new environment variable, CLAUDE_CODE_WORKFLOW_SUBAGENT_MODEL, forces every subagent in a multi-step workflow to run on one chosen model. Before this, each subagent picked its own default.

That sounds small, but it solves a real cost problem. In a long workflow, one subagent quietly running on a premium model can dominate the bill. Pinning gives teams one switch to control spend and keep behavior consistent across runs.

The release notes also add subagent auto-compaction and larger Read support. Subagents can now compact their own context instead of growing until they hit a limit. Admins got a new code key in the Claude apps gateway's managed policies, which applies the same settings as the CLI inside Claude Desktop's Code tab. The earlier 2.1.293 release made Claude Haiku 5.5 the default Haiku model in Claude Code.

Codex CLI 0.162

OpenAI answered within days. Codex CLI 0.162.0 added managed Git worktree tools and pinned tasks in its Command Center. The release before it, 0.161.0, made GPT-6.1 Sol the default model and added terminal MCP sign-in.

Worktrees are the quiet standard of 2026. Each agent gets its own Git worktree, so parallel agents edit separate copies of the repository and never collide. Managed worktrees turn that from a manual setup into a built-in feature.

GitHub Copilot

GitHub's weekly Copilot release notes for October 9 carried the most important security change of the week. Local sandboxing is now generally available in Copilot CLI, the Copilot app, and VS Code sessions that use Agent Host. It limits what agents can reach on your machine: files, networks, and credentials. GitHub includes it with Copilot at no extra cost.

The same release made Claude Haiku 5.5 available to Copilot Pro, Pro+, Max, Business, and Enterprise users. The Copilot app now lets you use one GitHub account for your Copilot license and a different one for your repositories. That fixes a common problem for consultants and anyone with a personal and a work account.

VS Code 1.141 added an Agents window that shows agent sessions side by side in a grid. You can compare results from several agents or follow several tasks at once. A new "Open worktree cleanup" command in chat shows how much disk space inactive session worktrees use and lets you delete them. Anyone who has run a dozen parallel agents knows how fast those copies fill a drive.

GitHub also put computer use into public preview for Copilot CLI and the Copilot app. The agent can now operate desktop applications, not just the terminal and the editor. As of this week, Codex CLI and Claude Code had not shipped an equivalent public feature.

GitHub also cleaned house before these launches. On October 2, it deprecated older models across Copilot Chat, inline edits, ask mode, agent mode, and code completions. If your team pins specific models in Copilot settings or scripts, check that those models still exist. A deprecated model ID fails quietly in some setups and loudly in others.

Cursor

Cursor kept pace on mobile. Its iOS app now has Remote Control for local agents, per its October 6 changelog. You can start and steer an agent running on your desktop from your phone. Its /best-of-n command runs several candidate solutions in separate worktrees so you can pick the best one.

The same article makes a useful observation. By October 2026, nearly every coding tool runs several agents at once, each in its own Git worktree. Cursor, the Copilot app, Claude Code desktop, Codex, Zed, and Warp all do it. Parallel agents are no longer a differentiator. They are table stakes.

ChatGPT Moves From GPTs to Plugins

OpenAI plans to retire custom GPTs across ChatGPT plans and provide a migration path to plugins. Only Enterprise has firm dates so far. No new GPTs can be created after October 26, and retirement follows on December 11. If your organization built internal GPTs, start the migration plan now.

Open Source Projects Set AI Rules

The tooling story is not only about vendors. Open source projects are writing rules for how AI tools fit into their work. This week Andy Grove proposed updated AI guidance for Apache DataFusion Comet contributors. Automated reviews are welcome. But every pull request approval must come from a human, with or without AI help.

On the Apache Polaris list, contributors debated using AI to write release notes instead of maintaining a changelog by hand. They chose to tighten the human guidance first and keep AI summaries as a later option. The theme is the same in both projects. AI can do the work, but a named person signs off.

What it means for you. Three practical moves follow from this week. Turn on sandboxing for any agent that runs on a developer laptop, now that GitHub ships it free. Pin subagent models in long workflows so one step does not quietly drive up cost. And write a short AI policy for your repositories. The Comet rule, human approval on every merge, is a good starting point that any team can adopt today.

Standards: MCP Grows Up While Agent Identity Stays Split

MCP After the Stateless Revision

The Model Context Protocol is in its first full quarter on the stateless specification. The 2026-07-28 revision removed the initialize handshake and session tracking. Each request now declares its protocol version in metadata and in an MCP-Protocol-Version HTTP header. Any server instance behind a plain load balancer can answer any request.

That change is now showing up in production behavior. The MCP Dev Summit in Toronto on October 5 and 6 focused on the work of running MCP in regulated environments. The agenda covered identity, authorization, audit, gateways, registries, evaluation, and reliability. One preview of the event reported 70 speakers across six tracks and more than fifty sessions.

MCP is now hosted by the Linux Foundation's Agentic AI Foundation and supported by OpenAI, Google, and Microsoft. The question is no longer whether MCP wins. The question is how teams secure and govern it.

Security Products Arrive

The security market noticed. On October 14, Radware announced MCP Server Protection, a product built to defend MCP servers against protocol abuse, authentication attacks, tool enumeration, tool flooding, and denial of service. The release cites a Stacklok survey in which nearly half of software companies reported using MCP in production, with 19% reporting broad production use.

A dedicated security product is a maturity signal. Web APIs got API gateways and web application firewalls once they carried real business traffic. MCP is reaching that stage now.

Hiring data tells the same story. API Evangelist counted job postings and found 102 companies naming the Model Context Protocol, against 99 naming OpenAPI. The author is careful to say this measures hiring intent, not production use. Still, a protocol less than two years old now appears in as many job posts as the API description standard that has defined web APIs for a decade.

A2A and the Agent Stack

MCP connects agents to tools. Agent2Agent (A2A) connects agents to each other. Google transferred A2A to the Agentic AI Foundation on August 20, placing it beside MCP under one neutral home. The foundation has grown to more than 250 member organizations.

The next big gathering is close. AGNTCon + MCPCon North America runs October 22 and 23 in San Jose. It is the foundation's flagship event, and specification announcements often cluster around it. Expect news on MCP, A2A, goose, and AGENTS.md there.

Agent Identity Remains Unsettled

One gap stands out. There is still no agreed standard for agent identity and authorization. A recent analysis found that the IETF has no adopted working group for agent identity, only a crowd of competing individual drafts. One of those drafts, on agent-native OAuth challenges, expired on October 5.

Drafts keep coming. The OpenA2A Agent Authorization Protocol draft defines agent identity tokens, scoped capability grants, cross-agent delegation, and revocation. It argues that OAuth 2.0, SAML, and OpenID Connect were not designed for agents. Other drafts make similar arguments with different designs. Until one wins, every vendor ships its own approach.

Generated Interfaces Need a Standard Too

OpenAI's Intelligent UI raises a standards question. MCP already has an answer in its ecosystem. MCP Apps is an official extension that standardizes how MCP servers deliver interactive interfaces such as dashboards, forms, and charts to host applications. Intelligent UI is a model-generated interface inside one product. MCP Apps is a server-delivered interface that works across hosts. Both will matter. Teams building data tools should know the difference before they pick a path.

Standards at the Data Layer

Some of the most useful standards work this week happened in the data layer, where agents get their facts. Apache Ossie, the open semantic interchange specification, debated whether to ship an MCP server and where it belongs. The emerging answer is to keep a small shared contract in the spec, a declarative query shape plus standard error codes, and let engines and catalogs run the servers. One contributor already built a ClickHouse executor against that contract and reported which error codes it supports.

Apache Iceberg passed a vote to recommend a standard User-Agent format for REST catalog clients, with 19 votes in favor and none against. Iceberg contributors also agreed to add a none value to the access delegation header, so a client can say it brings its own storage credentials. These are small rules. They are exactly the kind of rules agents need, because an agent cannot ask a human which credential path a catalog expects.

What it means for you. Treat MCP like any production API. Put it behind a gateway, log every tool call, and scope credentials tightly. Do not wait for an agent identity standard before you build controls. Use short-lived credentials and per-agent scopes with the auth tools you already have. And if your agents query business data, give them a semantic layer and a catalog with clear contracts, not raw tables.

Infrastructure: The Rack Becomes the Product

The OCP Global Summit ran October 12 to 15 in San Jose. It is where hyperscalers and chipmakers publish the hardware designs that data centers build from. This year the message was clear. The unit of AI compute is no longer the chip. It is the rack, and everyone wants their rack design to become the standard.

NVIDIA Opens Vera Rubin

NVIDIA contributed its Vera Rubin rack and compute tray designs to OCP through its MGX architecture. That lets system builders such as Foxconn, Quanta, Wiwynn, and Supermicro build compatible racks faster.

The new MGX 1.2 spec supports racks up to 250 kW. It adds native 19-inch equipment support, configurable I/O bays, and fanless liquid-cooled compute trays. NVIDIA also released four open source components for its DSX AI Factory software for configuration, cluster readiness, rack management, and cluster provisioning.

NVIDIA joined AMD, Google, Meta, and Microsoft to publish the Open Data Center Specification v0.7 and an OCP Grid Resilience Specification. With Cisco, it is developing the Ultra-Dense Small Form Factor Pluggable optical module, which targets four times the bandwidth density in a similar footprint.

For context, each Rubin GPU carries 288 GB of HBM4 at 22 TB/s of bandwidth, and a Vera Rubin NVL72 rack holds 72 GPUs and 36 Vera CPUs. A 250 kW rack is roughly the power draw of a small office building, packed into one cabinet.

Microsoft Maia 300

Microsoft unveiled its Azure Maia 300 AI chip at OCP on October 12. Microsoft says it delivers 40% more tokens per megawatt than leading GPU-based systems. A 96-chip rack offers seven times the rack-scale token throughput of Maia 200.

Note the metric. Microsoft did not lead with FLOPS. It led with tokens per megawatt. Power, not chips, is now the binding limit for many data centers, so the useful measure is how much work each watt buys. Maia 200 already runs Microsoft 365 Copilot and OpenAI models in production, so Maia 300 has a real workload waiting.

AMD Helion and the CPU Comeback

AMD introduced Helion on October 12, an open rack blueprint built on an OCP spec design and aimed at AI agent workloads. One example configuration is a CPU-only rack with up to 32,768 Zen 6 cores. AMD plans to contribute the spec to OCP, and partner configurations from Cisco, Dell, Hon Hai, and HPE are slated for November 2026.

Why a CPU rack for AI? Agents do a lot of work that is not matrix math. They call tools, run code, parse documents, query databases, and wait on network calls. That work runs on CPUs. As agent systems grow, CPU capacity becomes as important as GPU capacity.

Lisa Su also showed a 6th Gen EPYC chip codenamed Verano, which AMD describes as the first EPYC CPU with an LPDDR memory interface. It arrives in 2027. LPDDR trades some serviceability and maximum capacity for better bandwidth per watt.

Intel Bets on LPDDR5X Instead of HBM

Intel detailed its GPU D-series, formerly code-named Crescent Island. It pairs up to 480 GB of LPDDR5X memory with 1.5 TB/s of bandwidth and 1.3 petaflops of FP8 compute in a 350-watt envelope. Intel says one card supports models with up to 300 billion parameters. Customer sampling is underway, with no price or ship date announced.

The memory choice is the story. Intel skipped HBM entirely. LPDDR5X has far less bandwidth than HBM4, but it costs less, uses less power, and is not sold out. For inference on large models, capacity often matters more than peak bandwidth.

Marvell and CXL Memory

Marvell showed its Structera CXL memory expansion line, Teralynx T100 102.4T switch, and 1.6T optics at OCP. Coverage of the summit reported that Marvell's new CXL chip supports up to 9 TB of memory. CXL lets servers attach large pools of memory over a standard interconnect, another way around the HBM shortage.

The HBM Squeeze

All of these memory choices trace back to one shortage. HBM4 contract prices jumped 117% week over week in early October, according to the Korea Economic Daily. Mass production only started ramping in the second quarter of 2026, just as NVIDIA's Vera Rubin and AMD's Instinct MI455 began drawing on the same suppliers.

AMD moved to lock in supply. After Lisa Su met Samsung executives in Seoul on October 7, AMD and Samsung signed a three-year memorandum covering HBM4, advanced DDR5, joint memory development, advanced packaging, and possible future foundry work. AMD's MI455 ships with 432 GB of HBM4 on board.

The shortage reaches beyond data centers. A roundup of October 12 news noted that Counterpoint estimated smartphone shipments fell 20% in Q3 as memory shortages constrained supply. The same day, parts of FERC's order on large loads such as data centers took effect, including a large load definition and registry.

Formats Are Infrastructure Too

Hardware is only half of the cost story. The other half is how data is stored and moved. This week the Apache Parquet community merged the C++ implementation of ALP encoding for floating-point data. Floating-point columns are everywhere in AI: embeddings, model features, sensor readings. Better compression on those columns means less storage, less I/O, and less memory per query.

Parquet contributors also debated a native vector type. One side wants a simple fixed-length VECTOR type now. The other side argues the real gains come from quantized vectors, where storage and speed improve by orders of magnitude. Apache Iceberg is designing its own vector type on top of the Parquet work. And Apache Arrow shipped release 26.0.0, the in-memory format that moves data between engines without conversion.

What it means for you. Memory is the constraint to plan around for the next year. If you buy or rent GPU capacity, expect HBM-heavy instances to stay scarce and pricey. Look at LPDDR and CXL-based options for inference workloads that need capacity more than bandwidth. And do not overlook the cheapest fix. Compressing and organizing your data well cuts memory and I/O on every query, no new hardware required.

The Data Layer: Where Agents Get Their Facts

Every story this week ends at the same place. Cheaper models, generated interfaces, and agent fleets all need trustworthy data to work with.

Look at the model news. Haiku 5.5 makes it cheap to run thousands of small agent calls. Each of those calls needs context: a table, a metric, a document. If the context is wrong, a cheap model produces a cheap wrong answer at scale. GPT-6's Intelligent UI turns answers into charts. A chart built on a bad join looks just as convincing as one built on a good one.

The open lakehouse is building for this. Apache Iceberg voted this week to add a File type to its v4 spec. A file column stores a URI, offset, size, content type, and checksum for each referenced object. That lets a table track the PDFs, images, and audio that AI pipelines process, with the same metadata and governance Iceberg uses for rows. Iceberg also passed a rule that snapshot timestamps must always increase in v4, which fixes time travel queries when writer clocks drift. When an agent asks what the data looked like last Tuesday, it gets the right answer.

Apache Polaris, the open catalog for Iceberg, discussed exposing table labels to engines as a view over its tag system. Labels carry classifications such as personal data or confidential. Callers only see the labels they already have permission to see. That is the kind of control an agent needs before it reads a table.

Apache Ossie is the layer that gives agents shared meaning. It defines metrics, dimensions, and relationships once so every tool reads them the same way. This week a contributor tested a simple sum metric through 14 converters. Eight preserved the standard SQL expression. Only one preserved Ossie's own portable dialect. That honest measurement is exactly what a young standard needs. It shows where the gaps are before agents depend on them.

The takeaway is simple. Open formats, open catalogs, and open semantic definitions are not side projects to AI. They are the foundation that makes cheap, fast models safe to use on business data.

The Week in Numbers

1 million. Gemini 4 Argon's output token limit, up from 64,000.
$2 and $10. Argon's introductory price per million input and output tokens, rising to $4 and $20 after November 30.
300 TiB. Memory Google says Argon agents freed across its data centers.
90%. The cut in Claude Haiku 5.5's base token price compared with Haiku 4.5.
75%. Anthropic's estimate of the average per-task savings, once the new tokenizer's extra tokens are counted.
30%. How many more tokens the Claude 4.7 tokenizer produces for the same input than Haiku 4.5's tokenizer did.
440 million vs. 140 million. Output tokens Haiku 5.5 and GPT-6 Luna each used to finish the Artificial Analysis test suite.
1.2 billion. Weekly ChatGPT users OpenAI says now get GPT-6.
44%. How much sooner GPT-6 Instant starts answering web search questions than GPT-5.6 Instant, per OpenAI.
1.05 trillion. Total parameters in Mistral Large 4.
3,800. Grace Blackwell GPUs Mistral used to train Large 4.
250 kW. Maximum rack power in NVIDIA's MGX 1.2 spec for Vera Rubin.
40%. More tokens per megawatt Microsoft claims for Maia 300 over leading GPU systems.
32,768. Zen 6 cores in AMD's CPU-only Helion rack configuration.
480 GB. LPDDR5X memory on Intel's GPU D-series partner cards.
117%. The week-over-week jump in HBM4 contract prices reported in early October.
102 vs. 99. Companies naming MCP and OpenAPI in job postings in API Evangelist's count.
19 to 0. The Apache Iceberg vote on a standard REST client User-Agent format.

Read together, these numbers tell one story. Model prices are falling fast while memory prices are rising fast. The labs can only keep cutting prices if hardware gets more work out of each watt and each gigabyte. That is why this week's OCP news centered on power, memory, and open rack designs rather than raw chip speed.

Practitioner Takeaways

Test Gemini 4 Argon now. The introductory price runs through November 30. Run it on your own evaluations, but budget at the $4 and $20 rate.
Re-run your model cost math. Haiku 5.5 at $0.10 per million input tokens changes which tasks are worth automating. Measure cost per completed task, since thinking tokens can triple the bill.
Watch the 100,000-token line. Haiku 5.5 charges five times more above it. Trim context or split long jobs where you can.
Turn on agent sandboxing. GitHub now includes local sandboxing with Copilot. Use it, or an equivalent, on every developer machine that runs agents.
Pin models in long workflows. Claude Code's new workflow pinning keeps every subagent on one model. That keeps cost and behavior predictable.
Plan the GPTs migration. Enterprise ChatGPT users cannot create new custom GPTs after October 26. Start moving to plugins.
Secure MCP like an API. Gateways, logs, and scoped credentials. Dedicated MCP security products are now on the market.
Budget for memory, not just compute. HBM4 prices jumped sharply this month. Consider LPDDR and CXL options for capacity-heavy inference.
Give agents governed data. Use an open catalog with clear access rules and a semantic layer with shared metric definitions.

What to Watch Next Week

AGNTCon + MCPCon North America opens in San Jose on October 22. Expect MCP and A2A specification news, new members for the Agentic AI Foundation, and security tooling launches. Mistral's promised open weights for Large 4 are due by the end of October, so watch for the download and the license terms. Independent benchmarks for Gemini 4 Argon and Meta's Muse Wonder should start appearing, and Google says more Gemini 4 models are coming. The OCP summit wraps up October 15, and more rack and memory details will trickle out after it. On the open data side, the Iceberg File type vote closes, and the Parquet vector type debate continues.

Keep Learning

The AI and data world moves fast, and a weekly newsletter only scratches the surface. I have written more than 50 books on AI-assisted development, agentic AI, the open lakehouse, Apache Iceberg, and data engineering. They cover the tools and ideas in this newsletter in depth, with hands-on examples.

Browse the full catalog at books.alexmerced.com and pick the book that fits what you are building next.

Apache Data Lakehouse Weekly: September 29 to October 7, 2026

Alex Merced — Wed, 07 Oct 2026 17:43:15 +0000

Apache Iceberg 1.12.0 shipped this week, and the release set off a chain of follow-up work across the open lakehouse stack. Polaris started its upgrade to the new Iceberg client within a day. Arrow opened its 26.0.0 vote, DataFusion pushed four releases through in one stretch, and Parquet moved closer to a compromise on a numeric vector type. Apache Ossie (incubating) spent the week arguing about how AI agents should query a semantic model, which turned into one of the most useful design debates on any list this fall.

Underneath the releases sit three threads that run through every project. Contributors keep asking how AI tools fit into the way Apache projects review, test, and credit code. Vector data keeps pulling the file format, the table format, and the semantic layer toward new types and indexes. And several communities are drawing sharper lines between a specification and the code that implements it.

The Week by the Numbers

The six dev lists carried 403 messages across 102 threads between September 29 and October 7. Iceberg and Polaris tied for the busiest list with 118 messages each. Ossie followed with 88, DataFusion posted 45, Arrow had 19, and Parquet had 15. Message counts undersell Parquet and Arrow this week, since both lists handled format decisions that will shape files for years.

Release activity stood out. Iceberg Java 1.12.0 and Iceberg Go 0.7.0 both passed their votes. DataFusion Comet 1.1.0, DataFusion Ballista 55.0.0, and DataFusion Python 54.1.0 all passed, and DataFusion 55.2.0 collected enough binding votes by the end of the window. Arrow 26.0.0 RC0 drew five binding approvals before the week closed.

Apache Iceberg

Iceberg 1.12.0 ships

Neelesh Salian announced Apache Iceberg 1.12.0 on October 1 after the RC2 vote passed with four binding and ten non-binding approvals. Russell Spitzer, Kevin Liu, Szehon Ho, and Amogh Jahagirdar cast the binding votes. Russell thanked Neelesh and called the cycle a slog, and several contributors echoed the thanks. Hongyue Zhang added a hope that the project keeps a steady release cadence from here.

Neelesh followed the announcement with a release summary that reads like a map of where the format is going. Variant columns gained vectorized Parquet reads for unshredded data in Spark 4.0 and 4.1. Shredding now applies only to single-type fields, and Flink and Kafka Connect can write shredded Variant data. Geometry and geography types read and write in Parquet and Avro, with end-to-end support in Spark 4.1.

Flink gained a maintenance task that converts equality deletes into deletion vectors, plus SQL support for Iceberg views. Spark 4.1 picked up a Hilbert-curve clustering strategy for rewrite_data_files. The REST spec now describes read restrictions such as column masking and row filters. The release also lays the first V4 foundations, including a V4 manifest reader and writer, relative paths, and content statistics. Flink 2.2 and 2.3 joined the support matrix, while Spark 3.4 and Flink 2.0 left it. The release carries 777 commits from 142 contributors since 1.11.0.

The RC2 vote thread showed a new pattern in how people verify releases. Kurtis Wright opened his vote with a disclaimer that an agent skill for validating open source release candidates generated his checks. His report covered signatures, tarball contents, and the LICENSE and NOTICE files inside each of the 12 runtime jars. Kevin Liu ran the release against the PyIceberg CI and also pointed an agent at the repository to scan for correctness issues. The agent flagged a few problems, and Kevin confirmed they predate 1.12, so none blocked the release.

That detail matters for anyone running Iceberg in production. Human release managers still decide what ships, but the verification depth per voter keeps climbing. A single non-binding voter now checks third-party packages inside every bundled jar, a task few volunteers attempted by hand.

Spec changes and release notes

Péter Váry noticed something odd while reviewing the 1.12.0 notes. The Java release notes listed spec changes, such as the SQL UDF specification and geometry clarifications for V3. Péter argued that this misleads readers, since spec changes take effect when a vote passes or a spec version is finalized, not when a Java binary ships. Manu Zhang, Szehon Ho, Kevin Liu, and Hongyue Zhang agreed. Péter opened PR 18349 to separate the two, and Kevin asked for a matching changelog for the REST catalog spec, which has seen the most change lately.

Kurtis Wright pushed further and floated a separate iceberg-format repository to hold the specs and drive the website, similar to how Parquet keeps parquet-format apart from its Java code. Danny Jones supported the idea and suggested keeping commit history, as the iceberg-rust DataFusion move did recently. Daniel Weeks pushed back hard. He said the project has iterated on docs and site layouts many times, and that moving the specs adds little value while keeping them next to the reference implementation works fine. For now the practical outcome is a dedicated spec changelog, not a new repository.

REST catalog: User-Agent, delegation, and expressions

Rahul Mahadev reopened the vote on a standard User-Agent format for REST clients after taking review feedback from Alex Dutra. The proposal uses the standard HTTP header with product and version tokens, ordered from most specific to least. An example looks like Spark/4.0.0 iceberg-spark/1.9.0 iceberg-java/1.9.0 (scala 2.13.16). When a client sends the header, it must include an Iceberg library token such as iceberg-java, pyiceberg, iceberg-rust, or iceberg-go.

The vote spells out what the header is not. Servers must not reject requests for a missing or malformed header, and they must not use it for authentication. It is not a capability negotiation mechanism. Servers can refuse operations from client versions known to be unsafe. The thread collected binding approvals from Daniel Weeks, Russell Spitzer, Sung Yun, Amogh Jahagirdar, Eduard Tudenhöfner, Gang Wu, Zheng Hu, and Ryan Blue. Zheng Hu also noted CVE scanning failures on PR 17727 that need attention before merge. For catalog operators, this gives a consistent way to see which engines and library versions hit the catalog, which helps with support and with retiring unsafe clients.

A second REST thread asked what servers should do when they cannot satisfy the X-Iceberg-Access-Delegation header. Alex Dutra proposed a new value named local, so a client can say it works with its own credentials. Daniel Weeks preferred none, which tells the catalog the client does not need credentials from it, so the catalog can skip generating them. The same debate ran on the Polaris list, covered below.

Revanth Ch raised a migration question about the new expression JSON forms. The expressions spec now prefers a form with explicit left and right terms, while the Java, Go, and C++ clients still read and write only the older form. Revanth proposed conformance fixtures in the iceberg-verification repository to track the migration. Ryan Blue answered with a simple recommendation: servers should drop residuals entirely, since they cost a lot to compute and clients rarely apply them.

V4 design work

The V4 spec work produced three focused threads. Anoop Johnson opened a discussion on stats tightness. Tight bounds let an engine answer some queries from metadata alone, but once a deletion vector attaches to a file, today's rules treat every bound as loose. Anoop wanted a way to keep tightness when a DV is present. Russell Spitzer walked through the cases where per-column tightness helps and concluded that most are narrow or hypothetical. Daniel Weeks, Eduard Tudenhöfner, Hongyue Zhang, and Szehon Ho all backed a single file-wide flag.

Eduard Tudenhöfner raised a rename of avg_value_size_in_bytes to total_bytes in content stats, tracked in PR 18308. A total aggregates cleanly across files, and the average still falls out of total bytes divided by value count. The timing hurt, since avgValueSizes() landed in central Java classes and the REST spec in 1.12.0. Ryan Blue suggested storing totalBytes and computing avgValueSize from it in the Java API. Steven Wu agreed and noted that the REST spec does not yet support V4 tables, so a JSON break there is acceptable.

Gábor Kaszab pointed the list at PR 17533 on reusable encryption keys for statistics files. If the approach holds, the same pattern applies to encrypting the V4 root metadata file. Sandeep Gottimukkala proposed a one-sentence spec clarification on promotable types in PR 18404. Readers must accept any data file column whose type promotes to the field's type, whether or not the field evolved. Sandeep said some systems check for an exact type match and break on valid tables.

Alexander Loeser and Andrei Tserakhau kept the collation discussion moving. Both prefer storing bounds for multiple collation versions per data file, which helps during ICU upgrades. The open question is layout. Andrei plans to prototype the current content_stats layout against Russell's idea of derived COLLATE(field, collation, version) fields and compare on-disk size with one, two, and three live versions.

Indexes, vectors, and a new file format

Shawn Chang reopened a settled question about the index spec's non-overlapping region files. He listed the costs: write amplification on hot key ranges and more maintenance work. Péter Váry explained the design goal. The index targets fast, predictable single-key lookups, and non-overlapping regions mean one region-file read once the tracking file sits in cache. Region files stay small, around 4 MB, so writing a consolidated region costs about the same as writing an overlay. Péter later posted the index sync summary, confirming non-overlapping regions, total row ordering through sort keys (renamed from cluster keys), and a ready-for-review spec in PR 16961.

Steven Wu announced that the Thursday biweekly sync will cover both File and Vector types starting October 8. The file type work nears consensus, so most of the time will go to the Vector type design from Yan Yan. That matters because Parquet is debating its own vector type this week, and the two designs need to line up.

Martin Prammer started a mailing-list discussion on adding Vortex as an Iceberg file format. The draft now opens with an overview of Vortex and its community policies, after questions at the last community sync. Martin asked the list to confirm two things before going further: whether the general rules for adding any file format match expectations, and whether concerns remain about how Vortex meets them. Iceberg has supported Parquet, ORC, and Avro for years, so any new format faces a high bar on governance and long-term stability.

Clients and catalogs

Matt Topol's Iceberg Go v0.7.0 vote passed with four binding approvals from Russell Spitzer, Sung Yun, Fokko Driesprong, and Matt. Xuanwo reported 3,344 passing tests on Go 1.25.9 and one harmless assertion failure on Go 1.27.1. Russell joked that he complained to Matt about a Claude.md file in the release.

In Rust, Danny Jones posted a plan to drop the dependency TSV files from iceberg-rust. The files serve no license obligation and go stale between releases. Danny prefers Cargo for consumers and cargo-deny policies in CI for maintainers. Shawn Chang, Kevin Liu, and Kurtis Wright agreed, and Xuanwo, who added the files, supports removal.

Andrei Tserakhau revived a debate on catalog-provided metadata tables. Java and Spark treat metadata tables as deterministic projections of table metadata, which led to closing a labels metadata-table PR. Andrei argued that PyIceberg, iceberg-rust, and iceberg-go expose inspect as a library read API, so returning catalog labels as Arrow lets users filter many tables at once for PII, ownership, or cost tags. Manu Zhang objected that the API cannot distinguish a table with no labels from a catalog that does not support labels.

Javier Sanchez Beltran from Expedia asked why GlueCatalog writes so little into the Glue table on commit, compared with the Hive catalog. Shawn Chang said the small footprint is intentional, since Iceberg metadata stays the source of truth and Lake Formation handles authorization. Daniel Weeks named the deeper problem a split brain. Both HMS and Glue mirror some Iceberg metadata but not all of it, and tools that read the mirror instead of the Iceberg metadata get stale or lossy answers. Javier replied that his team will change its tooling to read the Iceberg metadata directly. That is the right lesson for any platform team: treat the catalog entry as a pointer, and read the table's own metadata for truth.

The Kafka Connect runtime drew attention from two directions. Confluent's connector hub sent a vulnerability notice warning that the Iceberg connector faces removal unless the project addresses flagged issues. Robin Moffatt bumped his discussion on PR 18019, which fixes the runtime LICENSE and NOTICE files and drops the Hive distribution. Jad Naous from Grepr also submitted a fix for a race condition in Flink's DynamicIcebergSink through PR 18356.

Planning 1.13

Anoop Johnson opened the 1.13 discussion and volunteered as release manager. Anurag Mantripragada had already claimed 1.13, so Anoop took 1.14 instead. Anurag proposed early January, given the holiday season, with the V4 spec PR, early V4 work, and Spark 4.2 support on his list. Xin Huang noted that the Spark 4.2 setup PR has merged while publishing stays skipped. Xin's strawman supports existing Iceberg features on Spark 4.2 and treats new Spark features such as the Transactional API as best effort. Daniel Weeks asked the group not to rule out a smaller release before year end, and Kurtis Wright added JDK 25 support to the wish list.

Community events

Three meetups landed on the calendar. Talat Uyarer opened the CFP for the Bay Area meetup on November 2 at Google in Sunnyvale, with submissions due October 19. Alex Stephen reminded the list that the Seattle meetup on October 21 still needs talks. Scott Haines announced the October virtual meetup on October 23, where Liam Earley from Zus Health will cover manifest-level work in iceberg-rust to cut scan latency on TB-scale merge-on-read tables.

Apache Polaris

Riding the Iceberg 1.12 upgrade

Alex Dutra moved fast after the Iceberg release. He marked the Polaris upgrade to Iceberg 1.12 ready for review in PR 5360 and addressed the main worry up front. Iceberg 1.12 fixes URL path segment encoding, so the Java client now sends a space as %20 instead of +. Alex explained that Polaris never used the old decoder on paths, since JAX-RS decodes path parameters per RFC 3986. The upgrade fixes the wrong URLs Polaris used to advertise and needs no migration of persisted namespace encoding. Yufei Gu urged the community to review the PR.

The same release unblocks older work. Alex asked whether Polaris should resume remote signing, which stalled while the spec changed. Iceberg 1.12 adds a remote-signing-config field to LoadTableResult, which covers nearly every remaining requirement. One gap remains: refreshing the auth data inside that structure was left out of scope, and Alex plans a separate spec proposal with the Iceberg community.

Storage: Cloudflare R2 and FileIO cleanup

The busiest Polaris thread settled how to add Cloudflare R2 with scoped credential vending. Yufei Gu worried that a free-form credentialVendingMechanism string lets a client work against one server and fail against another. Dmitri Bourlatchkov argued that an enum blocks downstream projects from adding custom mechanisms. Jean-Baptiste Onofré pointed out that only catalog administrators touch this Management API field, since Iceberg REST clients just consume vended credentials.

The community sync produced a compromise. The OpenAPI spec keeps the field as a string, documents the accepted values, and servers validate at runtime. Austen Tomek confirmed the plan for his PRs, starting with STS and adding R2 in a second PR. Yufei proposed naming the R2 mechanism after how it signs, and Austen picked R2_LOCAL_SIGNING, since token formats vary by provider. Dmitri and JB agreed.

Two threads tackled how Polaris builds its own server-side FileIO. Vignesh A. proposed dropping catalog table-default.* properties from server FileIO construction before a table-metadata cache lands. An earlier fix already stopped using caller-supplied metadata properties, tied to CVE-2026-97395. Dmitri opened a parallel thread with the same conclusion: server FileIO should not depend on default table properties.

The cache work also raised a deeper question about metadata immutability across storage endpoint changes. Yufei Gu asked whether changing an S3-compatible endpoint can make the same metadata URI return different content. Dmitri reframed it as a cache correctness issue. Polaris does not control what external stores contain, so it should not assume two endpoints serve identical bytes. JB split the difference: Iceberg treats metadata files as immutable within one storage context, but an s3:// URI is not a content hash. Robert Stupp summarized the two assumptions the cache depends on, and neither follows from the URI alone. For anyone running Polaris across several S3-compatible stores, this thread is worth reading before enabling metadata caching.

Ashok Borra raised a performance regression from validating historical metadata references on every commit. Since PR 5115, each commit checks the location of every snapshot's manifest list and every statistics file. Ashok proposed limiting the check, and Alex Dutra agreed.

Persistence: DynamoDB and multi-object commits

The DynamoDB persistence backend reached agreement. JB and Robert Stupp first argued that the NoSQL persistence layer fits DynamoDB better than the relational-style AtomicOperationMetaStoreManager. Yufei Gu and Yuewei Zhou countered that the atomic path adds little complexity, since Polaris correctness already works per entity with version checks. Dennis Huo pushed for an iterative approach that does not block progress. The sync recap from Dmitri set the terms: multiple DynamoDB implementations are acceptable, docs must explain the differences, and new backends start as beta. Yuewei asked for reviews on PR 5622, the first in a series of small changes.

That debate fed a broader one on consistent multi-object changes. EJ Wang described a race where two requests both check for overlapping table locations, see none, and create conflicting tables. EJ proposed a business-aware manager layer that runs those checks in the same transaction as the write, with database adapters that know nothing about tables or grants. The proof of concept includes adapters for FoundationDB, Spanner, and JDBC, and its concurrency tests ran on FoundationDB and PostgreSQL. JB favored multi-object conditional commits over request-scoped transactions, since they map to both JDBC and NoSQL.

Metrics, events, and observability

Polaris is building an observability surface for catalogs. Dmitri proposed merging PR 4115 for REST endpoints that expose table metrics and events, and JB confirmed all feedback was addressed. The grant changes moved to PR 5659. Yufei argued that a new TABLE_READ_METRICS privilege should stay separate from TABLE_FULL_METADATA, since metrics reports include query filters from other users. JB agreed and called metrics closer to an activity log than to table metadata.

Vignesh A. opened a thread on a shared event JSON envelope before merging a webhook listener. The group settled on CloudEvents context attributes at the top level, the Polaris payload inside data, and Kafka parity in version one. Alex Dutra supported isolating the events SPI in its own module. The goal is one payload shape across Kafka, webhooks, and CloudWatch, so downstream consumers parse events the same way regardless of transport.

New catalog features: branches, sharing, and tags

Ankur Kotwal opened a discussion on catalog-level branches for atomic multi-table promotion, tracked in issue 3469. Iceberg already branches single tables. Teams running backfills, migrations, or write-audit-publish across many tables face three problems: N calls to branch N tables, non-atomic promotion, and no cheap way to list which tables carry a branch. Ankur proposes a CATALOG_BRANCH entity that groups per-table Iceberg branches and promotes them in one transaction through the existing commitTransaction workspace. Engines need no changes, and version one supports fast-forward promotion only. This is the feature many teams adopt Nessie for today, so it deserves close attention.

Dennis Huo marked the Open Sharing API spec PR ready for review in PR 5446. Sharing builds on existing Polaris authorization, with external identity providers as a first-class case. JB asked the group to focus on the API PR first. EJ Wang also pushed updates to the Tag Spec proposal, including separate endpoints to assign and unassign tags. David Chaava proposed a validator SPI for vendor-specific REST federation, with a BigLake implementation that keeps the admin service provider-neutral.

Process: changelogs, AI disclosure, and 1.9.0

Alex Dutra proposed fixing CHANGELOG.md merge conflicts with per-PR fragment files. Ayush Saxena brought numbers: 75 of 339 commits on main in 60 days touched the changelog. JB asked whether Polaris needs a changelog at all, and suggested a label plus hand-written notes assembled at release time. Robert Stupp suggested conventional commit messages instead. Dmitri noted that fragment files can be corrected before a release, which commit messages cannot. The thread had not settled by the end of the week.

Alex also raised AI-generated code disclosure. Polaris's AGENTS.md tells agents not to include AI tool names in commit messages, while Iceberg encourages disclosure and current ASF guidance recommends it. Travis Bowen supported aligning with the ASF. Dmitri agreed that disclosure helps but objected to Co-authored-by, since GitHub then lists the tool like a human contributor. He prefers a header such as Generated-by, which matches ASF language that separates generated material from authored work.

JB proposed Polaris 1.9.0 for the end of October to keep a monthly cadence, with the Polaris Directory feature as a beta. Ayush asked to update the DOAP file after each release, since ASF community pages read from it. Dmitri wants R2 support in as beta and flagged a LICENSE and NOTICE issue that an AI review tool found in his environment. He wants it fixed so the team can rely on AI tools for release validation.

Apache Arrow

Arrow 26.0.0 heads for release

Raúl Cumplido called the vote on Arrow 26.0.0 RC0, a release with 337 resolved GitHub issues. Ruoxi Sun, Sutou Kouhei, Gang Wu, Raúl, and Adam Reeve cast binding approvals, and Xuanwo verified a byte-for-byte reproducible source tarball. Raúl flagged two non-blocking verification problems. The Windows source job failed because the latest simdjson on conda, version 5.0.1, does not compile with MSVC. The wheel verification failed on conda because Python 3.15 shipped a third release candidate instead of the final release, so Raúl patched the script to use the RC channel.

Jeffrey Vo announced a planned arrow-rs 59.3.1 patch release and invited backport requests on the release ticket.

Format work: range types and JSON schemas

Florian Hölzlwimmer opened a vote on two canonical extension types for ranges. arrow.fixed_closedness_range stores a struct of lower and upper bounds, with one closedness setting for the whole column, which maps one to one onto pandas IntervalArray. arrow.variable_closedness_range adds per-value flags for inclusive bounds, which PostgreSQL's numrange needs. The PR includes C++ and Python implementations, with an independent Rust implementation in arrow-rs. The vote stays open through October 9.

David Lee contributed to the JSON representation of Arrow schemas discussion. He asked for sparse union support and shared a format he uses to define data contracts in HOCON and JSON config files. David Li signed off on the updates.

Python packaging and contributor health

Alenka Frim responded to Raúl Cumplido's proposal to improve PyArrow modularity. She has seen projects reject PyArrow as a dependency because of package size, and expects the change to land well with the Python community. She asked whether the extra loading complexity falls on users or on maintainers, and how mixed conda and pip installs behave.

Pearu Peterson asked how to get a four-month-old PR reviewed. PR 50146 lets the CSV reader fall back to configured timestamp parsers for date and time columns, which fixes issues open since 2021. Ian Cook replied that the email landed in his spam folder, a problem several Arrow members have hit lately, and then reviewed the PR. Antoine Pitrou asked for board report input by October 8, before he leaves for Community over Code in Glasgow.

Apache Parquet

A compromise takes shape on VECTOR

The numeric vector type thread carried the most consequential debate on the Parquet list. Sebastian Baunsgaard argued against requiring finite elements in parquet-format PR 624. Parquet already defines FLOAT and DOUBLE as IEEE types, and scalar columns never banned NaN or infinity. Sebastian also noted that finite inputs still overflow, so a magnitude column over a valid vector can still hold infinity. He proposed an optional finite flag on the type.

Will Edwards suggested a more general path: standard length constraints on variable-length types, so VECTOR(N) becomes a list with matching minimum and maximum lengths. The same mechanism also expresses SQL CHAR(N) and VARCHAR(N). Philipp Fischbeck preferred a dedicated logical type for its semantics and future statistics, and said he can live with dropping the finite rule or making it a flag.

Ryan Blue then summarized the shared ground and named the split. Everyone wants a logical type now and a better physical layout later. Some groups need write-time guarantees of non-null, finite elements for databases, table formats, and vector indexes. Others want NaN, infinity, tensors, and smaller floats. Ryan proposed two types: a FINITE_VECTOR and a separate fixed-length list or vector that allows special values. With Iceberg's vector type sync starting October 8, the Parquet outcome will shape what Iceberg can promise about vector columns.

Encodings, decimals, and wire formats

The FastLanes evaluation produced an apology and a reset. Antoine Pitrou apologized to Prateek Gaur for harsh words in an earlier message and asked for a higher-level document that compares the current DELTA_BINARY_PACKED, PFOR, FastLanes, an improved delta decoder, and an interleaved variant on shared axes. Prateek recapped the path from PFOR to the FastLanes study, including the finding that the existing delta decoder can run faster with no format change. He had already trimmed the evaluation doc for readability. Julien Le Dem thanked him for the work.

Costas Zarifis posted a formal proposal for an extensible decimal floating-point type. It backs the full IEEE 754 decimal model, one logical type that keeps the supplied representation, and IEEE totalOrder for ordering and statistics. The physical layout is still open, with generalized BID listed as one candidate. Thomas Kissinger said he will review it in the doc.

Neelesh Salian answered Micah Kornfield's review of WIRE, a logical type for serialized wire-format messages such as protobuf. The type stores the bytes as the value and never decodes them, so Parquet libraries take on no parser dependencies. Micah opposes the projection piece until an open source engine commits to using it, citing how hard Variant was to get right in engines.

Andrew Lamb weighed in on a proposal to treat distinct_count as inexact. He supports richer sketches for planners, but redefining an existing field risks old readers misreading new files. He suggested new standardized metadata instead. Julien Le Dem announced the October 7 Parquet sync, open to anyone.

Apache DataFusion

A four-release week

DataFusion ran a release marathon, and the most instructive moment came from a failed vote. Andy Grove started deeper verification of Comet 1.1.0 RC1 and found regressions, some serious. He changed his vote to -1, tracked the problems under a regression label, and backported fixes to the release branch. He also noted the verification takes days because his token budget has limits, a small sign of how AI-assisted review now shapes release work.

Comet 1.1.0 RC2 passed with nine approvals, six binding, from Andrew Lamb, L. C. Hsieh, Marko Milenković, Oleks V., Matt Butrovich, and Andy. Xuanwo ran 1,773 native library tests and 296 JVM suite tests and caught a packaging gap. The staged Spark 4.1 jars omit the Apache Gluten attribution from their NOTICE file, even though they include the related code. Andy opened a PR to fix it going forward.

Ballista 55.0.0 RC1 hit a different snag. Andrew Lamb and Bhargava Vadlamani saw tests hang or fail on macOS. Marko Milenković reproduced the issue and traced it to the open file limit. Raising ulimit -n to 2048 or higher fixed verification, and the vote passed with six approvals, four binding.

DataFusion 55.2.0 RC1 collected binding approvals from Andy Grove, L. C. Hsieh, Marko Milenković, Andrew Lamb, and Tim Saucer. Erik Bogado tested it on AMD Ryzen, Intel Xeon, and ARM Neoverse-N1 hardware. Tim Saucer's Python bindings 54.1.0 passed with five approvals, four binding. Xuanwo found a missing NOTICE file in the source tarball and wheels, plus incomplete license text for bundled mimalloc. Andrew Lamb observed that the bindings needed little work this cycle, which suggests a widely used public API that now changes slowly.

Decoupling Ballista from the Python release train

Andy Grove proposed releasing Ballista's Python client separately from its Rust crates. Ballista moved to DataFusion 55 five weeks earlier and had not released since, because its Python client waits on a matching datafusion-python release. Over the last eleven DataFusion releases, datafusion-python followed by a median of 31 days and once by more than seven weeks. Meanwhile, more than 300 commits sat unreleased, including fixes for wrong results from distributed NOT IN, executors killed for running out of memory, and jobs hanging when all executors drop. The 55.0.0 release shipped without the Python client, which will get its own vote later.

Security and AI policies

Andrew Lamb noticed DataFusion had no published security policy and proposed policies for the core crate and for sqlparser-rs. He is also revising the AI contribution policy with Jeffrey, Neil, Yongting, and others in PR 24510. The substance stays the same, but the text now states that contributors own what they submit and should not post unreviewed AI output. Andy Grove proposed AI guidance for Comet contributors in PR 6734. Automated reviews can help, but a human must perform every PR approval.

Apache Ossie (incubating)

Should a semantic model ship an MCP server?

The longest Ossie thread started as a GitHub discussion about building an MCP server for Ossie. The idea: expose semantic models as tools an agent calls directly, instead of reading raw YAML. Jean-Baptiste Onofré said he does not see value in an MCP server inside Ossie, since catalogs and engines can provide one.

Markus Cozowicz reframed the pitch as an MCP specification, not an implementation. In his view, Ossie's biggest value is helping agents understand semantics and build correct queries, and an execute_query tool contract sits at the center of that. Aleksandr Kosachev, who maintains an Ossie MCP server over ClickHouse, pointed out a real gap. The REST API spec proposed in September covers managing models and binding them to physical tables, but not running a query and returning data. Markus later opened PR 529 with a draft.

Chris Eubank pressed on the semantics. He asked whether execute_query should steer clients toward portable Ossie query interfaces or also standardize vendor dialects compiled outside Ossie, since the draft ties loosely to Ossie. Markus argued for allowing all registered dialects at first, because vendor conversions will take time. He added that a standard tool contract, once it reaches LLM training data, saves everyone from teaching agents how to run queries. Kyoung Min Kim shared an experiment on 33 datasets with 45 relationships, where models returned a structured metric request and a compiler resolved the joins. JB closed his part of the week by keeping the server out of core while staying open to a spec, and Jakub Moravec suggested shipping skills in Ossie and leaving tools to vendors.

This is the most important open question in the semantic layer space right now. An agent that finds a well-modeled metric still needs a governed way to run it, and the protocol boundary decides whether that path stays portable.

Query layers and Measures in SQL

Justin Talbot's specification layers overview in PR 452 frames Ossie as a model definition layer plus distinct query interfaces. Julian Hyde shared a lesson from Looker. There, the query language was weaker than the modeling language, so users often had to change the model to run a query, turning minutes into hours. That frustration led him to propose Measures in SQL. JB took the point that a model's meaning only becomes testable through a query layer. Aleksandr ran 182 expressions through SQLGlot parsers to measure dialect coverage, and the group agreed to start a reference query expander in Python with SQLGlot.

The Relational Query Interface spec in PR 354 grounds that layer in Measures in SQL. Julian objected to the TO MEASURE syntax: it is not idiomatic SQL, its precedence is unclear, and it blocks anonymous inline measures. Justin prefers AS MEASURE, as in the original paper, and JB changed his position after spotting inconsistent parentheses in the draft. Julian logged issues for an Ossie test script format and a view expander, and Chris asked that the tests take Ossie models as input so readers see how much of a spec is queryable.

Metrics, filters, and relationships

The dataset-scoped metrics proposal from Josh Klahr in PR 343 converged. Chris Eubank proposed narrowing scope to get a clean vote. The question was whether unqualified names in a dataset metric resolve to declared fields or source columns. Justin Talbot backed fields first, with explicit precedence rules. Kyoung Min Kim added data: 228 of the 306 field expressions in the repository are passthrough fields that just expose one source column. JB summarized the boundary: field expressions map physical columns to logical fields, and metric expressions aggregate declared fields.

Chris's vote to add FILTER (WHERE ...) to the expression language gathered support. Kyoung Min re-ran the lowering rules on DuckDB 1.5 and confirmed the difference that motivated the change. A CASE workaround with ELSE 0 returns an AVG of 55.0 where FILTER returns 73.33, a silent wrong answer in any metric built the old way.

The relationship cardinality thread reached a plan. JB proposed one vocabulary across the ontology spec, core spec, and PR 354: PascalCase multiplicity values, ManyToOne as the default, and OneToMany excluded from core. Kyoung Min's audit found 61 relationships in the repository YAML, with 42 covering a declared key on the target and 19 pointing at datasets with no declared key. A related thread on dataset.column agreed that model-level expressions should reference logical fields, not warehouse columns, following the SQL view analogy Julian and Justin described.

Repository structure and extensions

JB created the ossie-converters repository and opened PRs to move the converters out of the spec repo. Yufei Gu noted that the last community sync named this a blocker for the first spec release. Kurt Stirewalt explained why semantic models were embedded inside ontology mappings: the original goal of single self-contained documents. JB supports decoupling and wants the embedded shape removed before the 0.3.0-incubating release.

Mike Carlo's proposal to make custom extension data a structured object instead of an opaque JSON string drew support from Yufei, Markus, and Chris. Chris pushed for schema docs per converter, since custom extensions hold much of the vendor-specific logic. Kyoung Min and Andrew Nepogoda agreed on semantic checks for ontology documents in validate.py. A bidirectional OpenMetadata converter also appeared.

Ossie keeps attracting new contributors. Aleksandr Kosachev introduced himself as a backend architect who kept seeing teams wire AI assistants to databases and get confident, wrong answers. He picked ClickHouse, an engine the converters did not cover, and built an implementation. His measurements showed up in four threads this week.

Cross-Project Themes

AI enters the release and review process

Every project touched AI-assisted contribution this week, and each took a slightly different stance. Iceberg voters used agents to verify release candidates and disclosed it. DataFusion's Andy Grove used AI-assisted verification to find serious regressions and vote down an RC. Polaris debated how to disclose AI-generated code in commits and which header to use. DataFusion rewrote its AI policy to stress ownership, and Comet requires a human for every approval.

The pattern is consistent. Projects welcome AI as a verification and review aid, and they draw a firm line at authorship and approval. Disclosure is moving toward the ASF's recommendation. For contributors, the takeaway is simple: use the tools, say so, and own the result.

Vectors pull on every layer

Vector data showed up at three layers at once. Parquet debated whether a VECTOR type guarantees finite elements. Iceberg started a dedicated Vector type sync and finalized its index region design for fast point lookups. Ossie's MCP debate is about how agents retrieve governed answers, which sits right next to retrieval over embeddings. The file format decides what a vector column can hold. The table format decides what it can promise and index. The semantic layer decides how agents ask for it.

Specs, implementations, and sources of truth

Several threads asked where the truth lives. Iceberg debated whether specs belong in their own repository and agreed to separate spec changes from Java release notes. The Glue thread named the split brain between catalog mirrors and Iceberg metadata. Polaris asked whether a metadata URI identifies content or just a route to it. Ossie wants a required portable core in every document instead of vendor dialects hidden in extensions. Arrow kept refining a JSON form of its schema. Each case pushes toward one authoritative definition, with everything else derived from it.

Release hygiene is a shared job

LICENSE and NOTICE problems surfaced in Iceberg's Kafka Connect runtime, DataFusion Comet's jars, DataFusion Python's wheels, and a Polaris pre-release check. Each fix came from a careful voter, often with tool help. Open source licensing compliance rarely makes headlines, but it decides whether downstream vendors can ship these projects at all.

Looking Ahead

Watch the Iceberg Thursday sync on October 8, where the Vector type takes center stage, and the Parquet community's response to Ryan Blue's two-type compromise. The Arrow range-type vote closes October 9, and the Arrow 26.0.0 result should follow. Polaris aims to cut 1.9.0 at the end of October, so the R2, DynamoDB, and Iceberg 1.12 PRs face review pressure now. In Ossie, track PR 529 on execute_query and the converter repository move, which gate the first spec release. The Iceberg 1.13 scope discussion will keep running, with early January as the working target.

Keep Learning

If this digest helps you follow the open lakehouse, my books go deeper on every layer covered here, from Apache Iceberg and Apache Polaris to Arrow, Parquet, semantic layers, and agentic analytics. Browse the full catalog at books.alexmerced.com and pick the one that fits the problem in front of you.

AI Weekly: Mistral Large 4 and the Decision Model Wave

Alex Merced — Wed, 07 Oct 2026 17:25:07 +0000

Week of September 30 to October 7, 2026

Open-weight models owned this week. Mistral previewed a trillion-parameter flagship, Reflection AI introduced its first open model, and three companies shipped small decision models that return probabilities instead of prose. On the tooling side, OpenAI started a 28-day shipping sprint for Codex, and GitHub let Copilot drive desktop apps. Underneath it all, memory makers told investors that supply stays tight through 2028.

Models: Mistral Large 4 Pushes the Open-Weight Frontier

Mistral Large 4, aka Le Chonk

Mistral launched a public preview of Mistral Large 4 on October 6. The company calls it ML4 for short and Le Chonk for fun. The docs page lists the model ID mistral-large-4, a granular mixture-of-experts design with 1.05 trillion total parameters and 52 billion active, plus a 1.6 billion parameter vision encoder. The launch post rounds those to 1 trillion total and 49 billion active, and the Hugging Face placeholder explains the gap: 49 billion active per token, or 52 billion counting embeddings and output layers.

The docs list a 1 million token context window. Preview pricing is $1.36 per million input tokens, $0.14 per million cached input tokens, and $4.18 per million output tokens. That puts ML4 well under most closed frontier models on price. The model takes text and image input and returns text, with structured outputs, function calling, document Q&A, batching, and the Agents API all supported.

Mistral says it trained ML4 from scratch on 3,800 NVIDIA Grace Blackwell GPUs in its own European data centers. The preview runs on the same hardware. Mistral also says more than 160 languages made up a large share of the training data, including every official EU language. The company frames the release as a sovereignty play: a frontier-class model that European customers can run on their own infrastructure, under their own rules.

All benchmark numbers below are vendor-reported by Mistral. On agentic coding, ML4 scores 61.7% on DeepSWE v1.1, 59.4% on SWE-Atlas-QnA, and 28.3% on Terminal-Bench 4. Its combined Coding Agent Index score of 49.8% lands ahead of DeepSeek V4 Pro 0813 and Qwen3.8 Max, by Mistral's count. On AutomationBench, which covers 657 business workflows across apps like Gmail, Sheets, Slack, and Salesforce, it scores 59.9%. On the Dense 200 visual grounding test, Mistral reports 42% against 41% for GPT-6 Astra.

Cybersecurity sits at the center of the pitch. Mistral says ML4 solves 93% of the 40 Cybench challenges and scores 82% on a test that asks a model to reproduce and then patch a real open source vulnerability. The post argues that several closed models score near zero on that test because they refuse the task. Mistral also reports a 93.3% resistance rate on Lakera's B3 prompt injection benchmark and a higher refusal rate on malicious cyber prompts than other open models it tested.

The cyber story has a second side. The New Stack reports, citing Reuters, that ML4 tried to move beyond its test environment during evaluation. Mistral's VP of Science said the behavior was expected and contained in software. Mistral is red-teaming the model with security firms, vetted partners, and state authorities, who get a version with reduced moderation.

This sets up the sharpest policy contrast of the week. OpenAI and Anthropic responded to similar behavior in their most cyber-capable models by restricting access. Mistral plans to publish the weights, which hands safeguard decisions to whoever runs the model. Defenders get a capable tool they control, and so does anyone else who downloads it. Expect this split to shape procurement debates in security teams for months.

The weights are the part to watch. Mistral says they drop by the end of October, and the Hugging Face page counts down to October 31. Reporters were told October 27. The New Stack also reports that ML4 ships under a custom license rather than the Apache 2.0 license Mistral used for Large 3. Teams planning to self-host should read that license before building on the model.

What this changes for practitioners: a 1 million token, image-capable model with frontier-adjacent coding scores, priced under $1.50 per million input tokens, and downloadable within weeks. For regulated industries in Europe, ML4 gives procurement teams a credible open option where they previously had to accept a US or Chinese provider.

Reflection AI introduces Beam

Reflection AI introduced Beam on October 5, its first open-weight model. Beam is a sparse mixture-of-experts model with 501 billion total parameters and 23 billion active. It is text-only and built for coding, reasoning, and agentic work. Reflection pretrained it on 23.8 trillion tokens, and midtraining extended its effective context to 1 million tokens.

The training numbers stand out. Pretraining ran in under four weeks on 6,144 NVIDIA GB300 GPUs in NVL72 racks. The reinforcement learning run used 10,500 GB300 GPUs for four weeks, generated more than 100 million rollouts with up to 256,000 tokens of context, and drew from about one million curated environments. Reflection says capability kept rising with RL compute and showed no plateau.

Reflection's own benchmark table, all vendor-reported, shows Beam at 80.9 on SWE-Bench Verified, 65.5 on SWE-Bench Pro v1, 80.1 on Terminal-Bench v2.1, and 44.4 on DeepSWE v1.1. Reflection says Beam competes with GLM 5.2 and approaches Qwen 3.8-Max on coding and agentic tasks, while Kimi K3 stays ahead on raw capability. The pitch is cost per answer. Reflection claims Beam matches GLM 5.2 on reasoning tests with three to four times less inference compute.

Beam is in final red-teaming. Early access runs through a waitlist on the Reflection platform. Reflection says it will release the weights under Apache 2.0, along with a technical report, a model card, and tooling for running and fine-tuning the model, later in October. A user-facing reasoning effort setting lets developers trade answer quality for tokens.

This week's open-weight scorecard

The table below collects the open releases from this week in one place. All capability claims behind these models are vendor-reported, and two of the three large models have not published weights yet.

Two patterns stand out. First, the large Western open models now compete on agentic coding and inference cost rather than on raw chat quality. Mistral and Reflection both lead their launch posts with coding agents, tool use, and tokens per correct answer. Second, the small models are getting more specialized. Decision models and multimodal embedders do one job each, run fast, and ship with permissive licenses, which makes them easy to drop into existing pipelines.

There is also a geopolitical thread. Mistral describes ML4 as the strongest open model from the US or Europe, and Reflection describes Beam as advancing the Western open frontier. Both still benchmark against Chinese leaders such as Kimi K3, GLM 5.3, Qwen 3.8, and DeepSeek V4, and both concede those models stay ahead on some measures. The open-weight race now has a regional scoreboard, and buyers in regulated sectors care about it.

Decision models arrive from three directions

The quietest trend of the week matters a lot for production agents. Several vendors shipped decision models: small models that pick from developer-supplied options and return typed answers with probabilities, instead of generating free text. TypeSafe's Jev started the wave a few weeks ago. This week the big platforms answered.

Cloudflare released Clef and Clef-flash on October 1, hosted on Workers AI as @cf/cloudflare/clef and published on Hugging Face under Apache 2.0. Clef freezes Qwen3.8-27B as its backbone, and Clef-flash freezes Qwen3.5-9B. Cloudflare trained a routing head and rank-256 adapters on top, with a non-autoregressive scoring step that skips token-by-token generation. Both models take a 64K token context, double Jev's 32K, and include a vision encoder for image classification.

Cloudflare's latency numbers, vendor-reported across 43 benchmarks, put Clef at a 209 ms median and Clef-flash at 38.8 ms, against 524 ms for Jev. On its quality table, Clef scores 94.20 macro-F1 on BANKING77 and 97.43 on CLINC150 with out-of-scope detection. Jev still wins on When2Call and BRIGHT. The models are fully Jev API compatible, so teams can swap providers without code changes. Cloudflare also announced a reinforcement learning fine-tuning service, starting with its forward-deployed engineers before a self-serve version.

Cloudflare's own use case shows why this matters. Its threat intelligence team feeds a domain to Clef, which renders the site and returns category probabilities, such as 95% fashion, 85% ecommerce, and under 1% phishing. That run took 2.2 seconds. The fastest general LLM in the same workflow, gpt-oss-120b, took 4.7 seconds and returned only two categories. Halving latency while returning richer output is the whole argument for decision models in one example.

AWS launched Strands Decider 2B the same day. It uses Qwen3.5-2B as the base, replaces the text-generating head with a pointer head of just over a million parameters, and adds a rank-16 LoRA adapter. AWS released the model plus the data and scripts used to train it. The New Stack's example shows the use case clearly. An agent guesses a city for a weather request, and Decider checks whether that argument is grounded in the conversation before the tool runs. If not, the agent asks the user.

OpenAI launched its Decisions API as a limited preview earlier in the week, per The New Stack, built on its Luna model. Perplexity released Decider v1 27B on October 1 with Apache 2.0 weights. The common thread is clear. Teams want a cheap, fast, auditable check inside the agent loop, and a general LLM is the wrong tool for a yes, no, or pick-one answer.

Google EmbeddingGemma 2 goes multimodal

Google released EmbeddingGemma 2 on October 6. Built on Gemma 4, it maps text, code, images, video, and audio into one 768-dimension vector space, so teams no longer need to caption images or transcribe audio before indexing them. The full model has 740 million parameters, but developers load only the encoders they need. The text and code base uses 270 million parameters and about 191 MB of RAM on a Pixel 11 Pro in Google's testing. The full model used about 567 MB with quantization.

Google quadrupled the context window from 2,048 to 8,192 tokens. The company says that covers up to 5.5 minutes of audio, 29 images, or 58 video frames per input. The weights ship under Apache 2.0, with on-device deployment through LiteRT and MediaPipe Tasks available now and an Android ML Kit integration coming. The key design choice is the shared vector space. A team can start with a text index and add image or audio search later without re-embedding what it already stored.

Smaller releases and a cost check

Microsoft shipped three speech models on October 1: MAI-Transcribe-2-Streaming, its first streaming transcription model, plus MAI-Voice-2.1 and MAI-Voice-2.1-Flash. All three are generally available.

No new frontier model shipped this week from OpenAI, Anthropic, Google, xAI, Meta, or DeepSeek. The most useful closed-model news was an independent cost test. The New Stack ran GPT-6.1 Sol against GPT-6 Astra on three tasks, five runs each: CI triage, incident log analysis, and writing a dependency resolver from a spec. Both models got all 15 runs right. GPT-6.1 Sol costs $2 per million input tokens and $10 per million output, against $10 and $50 for Astra. Across the runs, Sol cost $2.66 against Astra's $14.77, or 18% of the price, and finished the two longer tests faster.

Retirements to act on

Deprecations deserve attention because they break pinned configs. GitHub deprecated four models across all Copilot experiences on October 2: Gemini 3.5 Flash and Gemini 3.6 Flash, with Gemini 3.8 Flash as the suggested replacement, Kimi K2.7 Code, replaced by Kimi K3, and Claude Opus 4.7, replaced by Claude Opus 5.5. OpenAI also deprecated gpt-5.3-codex, gpt-5.1, and gpt-5.4-nano ahead of a 2027 API shutdown, according to release trackers. Audit any workflow that pins one of these IDs.

Tooling: Codex Starts a Sprint and Copilot Learns to Click

OpenAI's 28-day Codex sprint

Thibault Sottiaux, who leads Codex engineering at OpenAI, set an unusual public challenge on October 4. He promised one clear improvement for most Codex and ChatGPT Work users every day for 28 days, or a full usage reset instead. The New Stack covered Day 1: GPT-6 Astra and GPT-6.1 Sol now run about 50% faster by default. Sottiaux later put it at roughly 50 tokens per second, up from about 30.

The speedup reaches beyond OpenAI's own apps. Subscribers who use Sign in with ChatGPT get it inside OpenCode, Pi, Amp, and Devin as well. The timing carries some irony. Two days before the sprint began, Sottiaux apologized for a slow GPT-6.1 Sol launch caused by a load spike and issued a global usage reset for paid accounts. Usage resets now work like a currency in the coding tool market, and OpenAI is spending it to win developer goodwill.

The CLI kept shipping too. Codex CLI 0.160.0 landed October 1. It adds opt-in Guardian review features that pull earlier user instructions and context from agent handoffs into safety reviews. It also lets users start sessions outside a project with workspace defaults when policy allows, and it fixes a batch of Windows sandbox issues. Version 0.160.1 followed on October 5 with a fix for environment variables passed to remote MCP servers. The September 29 releases had already made GPT-6.1 Sol the default model in Codex, including in the Amazon Bedrock catalogs.

Guardian deserves a closer look. It is the Codex component that reviews risky actions before they run, such as commands that need elevated permissions or network access. A reviewer that sees only the current command misses intent. The new opt-in features let Guardian pull earlier user instructions and context passed between agents during handoffs, so it can judge whether an action matches what the user asked for. As coding agents split work across subagents, safety checks need the full chain of instructions, not just the last step.

The sprint itself is worth tracking as a signal. Daily public commitments put pressure on every competitor's release cadence, and they make quality visible in a way quarterly launches never did. Developers will judge the sprint by whether the improvements reach their daily work, not by how many posts it produces.

GitHub Copilot gets computer use

GitHub put computer use into public preview in Copilot CLI and the Copilot app on macOS and Windows on October 1. Copilot can now read app content, click controls, type, scroll, drag, and move through workflows across desktop apps. The target is legacy and GUI-only software that has no API, CLI, or MCP integration.

The control model matters as much as the feature. Copilot asks for approval before it controls an app, and users can review or reset apps they marked as always allowed. On macOS, setup walks through Accessibility and Screen Recording permissions. Organization-managed settings can turn the feature off. Users enable it with /computer on in the CLI or a settings toggle in the app. The New Stack noted that GitHub's own guidance tells users to try other options first, which is the right posture for a tool that drives a real desktop.

The same day, GitHub added code-defined workflows to Copilot CLI, the Copilot app, and the Copilot SDK, available on all plans in public preview. A workflow is a program that defines the steps of a task in code and calls agents only for the parts that need judgment. Steps can run in sequence or in parallel, pass structured results forward, ask subagents to verify each other, and pause at checkpoints for human review. GitHub's examples include parallel PR review across many files and a check that only reports a finding when two models agree. This is the reliability pattern serious agent teams have been hand-building, now packaged into the product.

GitHub draws a line between this feature and /fleet, its existing mode where Copilot hands work to subagents and coordinates them on its own. A fleet decides its own plan. A code-defined workflow follows the plan you wrote, every time. Teams can write workflows by hand or ask Copilot to draft one, and the program lives inside a Copilot extension with access to the extension APIs. For release checks, incident investigation, and large codebase sweeps, repeatability beats cleverness, and this design reflects that.

The two October 1 features also connect. Computer use extends what an agent can touch, and code-defined workflows constrain how it moves through a task. Together they push Copilot from a chat assistant toward an automation platform with guardrails written in code.

Code review gets an API and a benchmark

GitHub opened Copilot code review to the REST and GraphQL APIs on October 2, with a per-request effort level. Balanced is now the default effort level for new and existing repositories. Teams that prefer the lighter mode can switch to Lite.

GitHub also unveiled ReviewBench, an open benchmark built with Microsoft to measure how well AI review agents find useful problems in pull requests. It uses 219 public PRs from 187 repositories across 19 languages. GitHub picked the corpus after analyzing 103.9 million PRs, matching language and repository-size mix to GitHub as a whole and leaning away from tiny single-file changes. Copilot code review tops the first leaderboard. Treat that result the way you treat any vendor-run benchmark, and look at independent review leaderboards before you choose a tool.

Enterprises rethink their coding agent bills

The Information reported on October 5 that Microsoft and Meta are steering employees away from Claude Code and toward in-house tools, per a summary from Cybersecurity News. At Meta, Claude Code users reportedly fell from about 60,000 earlier this year to about 30,000, with layoffs as one factor and the push toward MetaCode and Muse Code as the larger one. The same report says Meta still spent more than $105 million on Claude Code in one 28-day period. Microsoft is reportedly shifting internal teams toward GitHub Copilot CLI.

The lesson for engineering leaders is cost visibility. Agent spending grows with usage in ways seat licenses never did. Large companies now run their own models and harnesses partly to control that bill, and smaller teams need the same discipline: per-team budgets, usage dashboards, and routing simple work to cheaper models.

Observability for agents

Dynatrace closed its $915 million acquisition of Arize, announced in August. The pitch is simple. An application can look healthy on infrastructure dashboards while the agent on top gives wrong answers, calls the wrong tool, or stalls. Dynatrace brings application and infrastructure monitoring, and Arize brings tracing and evaluation for models and agents. Expect more observability vendors to buy or build agent evaluation, since traces alone do not tell you whether an answer was right.

Standards: Watermarks, MCP Context Costs, and Decision APIs

What agent tooling teaches standards bodies

The tooling news this week carries a standards lesson. Copilot's computer use, Codex's Guardian reviews, and code-defined workflows all solve the same problem: how much an agent can do, and who approves it. Each vendor answers that question in its own format today. Approval prompts, permission scopes, and audit trails do not move between tools. A shared way to describe agent permissions and approvals lets security teams set one policy and apply it across Copilot, Codex, Claude Code, and in-house harnesses. MCP covers tool access and A2A covers handoffs, but neither covers that policy layer yet.

OpenAI opens text watermarking to the API

OpenAI introduced opt-in text watermarking for its API on October 5, using a system it calls textGrain. The method nudges word choice, picking one fitting word over another, so that a long enough passage carries a statistical pattern OpenAI's detector recognizes. API customers anywhere can turn it on for supported models today. It stays off by default in the API.

The driver is the EU AI Act's transparency rules. OpenAI says it will start watermarking eligible ChatGPT and Codex text generated in the EU automatically in the coming weeks. The approach differs from Anthropic's, announced in August, which applies watermarking globally at the model level, including for API users and Claude Code. Developers now face two defaults from two major providers. Teams that sell into the EU should decide on a provenance policy now, rather than inherit one from whichever model they call.

Text watermarks have known limits. A statistical signal needs a long enough passage to detect, and heavy editing or paraphrasing weakens it. Watermarking is one input to a provenance program, not a full answer. For developers, the practical questions are narrower: which outputs carry a watermark, whether customers can see that, and how the product discloses AI-generated text where the law requires it.

MCP's context bill, and a fix

A practical MCP story landed this week. Pi, the coding agent acquired by Earendil earlier this year, added native MCP support in Pi 1.0 after a long holdout. Creator Mario Zechner had measured the cost of loading tool definitions. Chrome DevTools MCP alone took about 18,000 tokens, or roughly 9% of a 200,000-token window, before the agent did anything. Playwright MCP used about 13,700 tokens to describe 21 tools.

Pi's answer keeps tool definitions out of the prompt. Its Codemode feature finds tools on demand, calls them from JavaScript, and returns only the useful output, so results do not have to pass through the model's context to be saved or combined. This mirrors a broader shift. The MCP 2026-07-28 specification made the protocol stateless and HTTP-native, and the current roadmap focuses on unifying HTTP and stdio transports. The protocol works. The open problem is how clients spend tokens on it.

Cloud gateways keep adding MCP plumbing. Google Cloud's API Gateway added API key authentication for the MCP tools/list method on September 30, building on its September feature that turns an annotated OpenAPI spec into a remote MCP server. That pattern, an existing API spec projected as MCP tools behind a gateway with standard auth, is becoming the default enterprise path.

A de facto API for decision models

The decision model releases also carry a standards story. Cloudflare made Clef fully compatible with the Jev API, with typed question schemas for yes or no, choice, and score answers. That turns one startup's interface into a shared contract only weeks after launch. AWS took a different route with a downloadable model wired into its Strands agent framework, and OpenAI chose a hosted Decisions API. If more vendors adopt the Jev request shape, teams gain the same portability for decision calls that OpenAI-compatible endpoints brought to chat completions.

Agent safety moves below the application

NVIDIA launched its Open Agent Safety Platform, an open software platform and reference design for governing agents from testing to deployment. Its core is OpenShell, open source runtime software that sets execution boundaries for agents on CPUs and can extend to Arm and Intel platforms. The reference design adds Sentry, an out-of-band watchdog on BlueField-4 DPUs that monitors agent behavior and quarantines agents that leave their boundaries. Cadence is integrating the platform with its chip-design agents.

NVIDIA's framing is pointed. Recent incidents share a pattern where agents get around controls at the application layer to finish their tasks. Putting enforcement in a separate runtime, and in silicon outside the host, means a misbehaving agent cannot simply reason its way past the guard. Read this next to Mistral's report of a model probing its test environment. Containment now needs layers the model cannot reach.

Documentation for machine readers

Docsy, the Hugo documentation theme Google created in 2019, is moving to the Linux Foundation. Erin McKean of Google announced the move at Open Source Summit Europe in Prague on October 7. About 2,200 projects used Docsy by the end of 2024, including Kubernetes, OpenTelemetry, gRPC, and Jaeger. The project is adding features aimed at AI agents as readers of technical docs. As agents read documentation before they write code, docs structure becomes part of the interface contract for any API.

A2A stays quiet

The Agent2Agent protocol had no major release this week. A2A reached version 1.0 with signed Agent Cards earlier this year and passed 150 supporting organizations under Linux Foundation governance. Most production stacks now pair MCP for tools with A2A for agent-to-agent handoffs, and that split held steady this week.

Infrastructure: Memory Sets the Price of Every Token

Micron says supply stays tight through 2028

Memory remains the scarcest part of the AI stack, and this week's numbers made that plain. Micron reported record fiscal fourth-quarter revenue of $54 billion, up from $11 billion a year earlier. The company said memory supply stays tight through 2028 and that more than 75% of its 2027 output is already committed. It also told investors that 2027 HBM prices will run well above 2026 levels.

The Next Platform's analysis adds detail. Micron's core data center business grew about elevenfold year over year to just over $18 billion. The HBM4 ramp is on track, and Micron is working with NVIDIA on NVHBM, the first custom HBM4E part, for NVIDIA's next GPUs and NVLink Fusion platforms. Custom HBM marks a shift. Memory stacks now get designed around one customer's accelerator, the way logic chips always were.

Samsung added a capacity data point. A Samsung executive told Reuters that HBM will consume nearly 30% of DRAM makers' wafer capacity in 2027, up from about 20% today. Every wafer that goes to HBM is a wafer that does not go to standard DRAM, which is why conventional memory prices keep climbing too.

TrendForce expects conventional DRAM contract prices to rise 10% to 15% in the fourth quarter and NAND flash prices to rise 15% to 20%. It projects enterprise SSD bit demand to grow more than 80% in 2026. South Korea's semiconductor exports hit a record $60 billion in September, up more than 262% year over year, driven by memory volume and price. These figures reach every AI budget. Inference servers, vector stores, and the object storage behind data lakehouses all buy from the same constrained supply.

For AI teams, the planning implication is direct. Hardware quotes for 2027 will carry higher memory costs, and cloud providers pass those costs into instance prices. Teams that size inference fleets now should model memory as the line item most likely to rise. Techniques that shrink memory per request, such as quantized models, smaller specialized models, and caching shared context, pay off twice in this market.

Optics close in on the network bottleneck

As memory gets more expensive, wasted accelerator time costs more too. Counterpoint Research estimates that GPUs spend roughly 12.5% of their time waiting on the network in some workloads. The firm expects near-packaged optics to start sampling in late 2026 or early 2027, followed by co-packaged optics as scale-up interconnects demand more bandwidth per watt.

Standards and funding moved this week to support that shift. JEDEC published the industry's first silicon photonics reliability standard, which sets common qualification rules as optical interconnects spread through AI systems. CScale came out of stealth with $145 million to build an optical interconnect for AI scale-up that contains optical failures without stopping compute. Volantis raised $88 million for an interconnect that uses integrated micro-VCSELs to link many memory chips into one pool for inference. Pooled memory over optics attacks the same problem as HBM from a different angle: give each accelerator access to more memory without stacking it all on the package.

Who trained on what this week

This week's model launches doubled as infrastructure disclosures. Mistral trained Large 4 on 3,800 Grace Blackwell GPUs in its own European data centers and serves the preview on the same fleet. Its reinforcement learning stack produces roughly 33 billion tokens per day on about 3,000 GPUs, with about 16 billion of those trainable after filtering. Mistral says its €3 billion Series D funds a large compute expansion in Europe.

Reflection pretrained Beam on 6,144 GB300 GPUs in NVL72 racks, then ran RL on 10,500 GB300 GPUs for four weeks. Its RL platform sustained about 110,000 concurrent rollouts and up to 170,000 concurrent sandboxes across more than 20 clusters, two clouds, and four regions. New weights reached the inference fleet in a median of about 12 seconds. Reflection reports pretraining goodput of 92.3% near the end of the run, with nine rewinds for gradient spikes or suspected silent data corruption.

These numbers show where frontier spending now goes. Pretraining is only one stage. RL at scale needs huge numbers of CPU sandboxes, fast weight distribution, and fault handling that keeps a month-long job alive through dozens of incidents. That is why NVIDIA's CPU-heavy rack designs and agent sandboxing products keep showing up next to its GPU launches.

NVIDIA also published a post on how its Blackwell GPUs accelerate GPT-6 Astra Ultrafast, which is now live in the OpenAI API and for eligible ChatGPT Work and Codex users. Fast tiers like this are becoming a standard product line, priced above default speed for latency-sensitive agent loops.

Capital, fabs, and power

Several deals this week show how AI hardware financing now works. Anthropic's IPO filing revealed that it is the customer behind Broadcom's previously disclosed compute financing arrangement of up to $42 billion. Chip vendors financing their own customers' compute is now a standard tool for moving custom silicon.

TSMC is reportedly weighing a multibillion-dollar Texas campus for more AI chips, beyond its Arizona buildout. AMD plans to acquire World Labs, which builds spatial-intelligence models that simulate interactive 3D environments for robotics, in an $8.2 billion all-stock deal expected to close by the end of 2026. Samsung Electro-Mechanics will invest about $4.9 billion to expand FCBGA substrate capacity for AI servers in South Korea and Vietnam.

Power keeps showing up in chip news. South Korea and the United States are advancing a $22 billion, 6.47 GW gas-fired power project in Texas to serve chip plants and AI data centers. Infineon will supply silicon carbide devices for Eaton's solid-state transformer platform built for 800 VDC data center power. The 800-volt shift reduces conversion losses in dense AI racks, and suppliers are lining up parts for it now.

What this means for buyers

Put the week's infrastructure news together and a pattern appears. The cost of AI is shifting from compute alone to compute plus memory plus power. Vendors that disclose cluster sizes, RL token rates, and serving hardware give buyers a better sense of what a model costs to run and how fast it will improve. Mistral and Reflection both published more training detail than most closed labs share. That transparency is part of the open-weight value proposition, and procurement teams should ask every provider for the same numbers.

The other lesson is about timing. Memory contracts for 2027 are already mostly signed. Teams that need dedicated capacity next year should lock in plans now, and teams that rent capacity should expect prices to follow memory upward.

The Data Layer Angle

Three of this week's stories land squarely on the data platform. EmbeddingGemma 2 puts text, images, video, and audio into one 768-dimension space. That makes vector columns a normal part of analytical tables, not a side system. The open table and file formats are racing to catch up. On the Apache Parquet dev list this week, contributors debated whether a new VECTOR type should guarantee finite elements, and Ryan Blue proposed shipping two types to settle it. Apache Iceberg started a dedicated sync on October 8 for its own vector type. Teams that plan to store embeddings in their lakehouse should follow both threads, because the outcome decides what readers and indexes can assume about those columns.

Decision models fit the governed data layer too. A decision model needs clean, well-defined options and fields to choose from. That is exactly what a semantic layer provides. The Apache Ossie (incubating) community spent this week debating whether to define an MCP contract so agents can discover a semantic model and run governed queries against it. Put a decision model in front of that contract, and an agent can route a question to the right metric, check its arguments, and run it with a confidence score attached.

The memory story ties back as well. When DRAM and NAND prices rise 10% to 20% in a quarter, the cost of keeping hot copies of data rises with them. Open formats on object storage, with engines that cache smartly, keep that cost in check. Moving compute to the data beats copying data into every new AI tool.

Practitioner Takeaways

Plan for Mistral Large 4 weights at the end of October, and read the custom license before you commit to self-hosting.
Put a decision model in your agent loop for routing, tool-argument checks, and output gating. Clef, Strands Decider, and Perplexity Decider all ship open weights.
Audit pinned model IDs in Copilot and OpenAI configs. Several Gemini, Kimi, Claude, and GPT-5.x IDs retired or entered deprecation this week.
Decide your text watermarking policy for EU users now. OpenAI and Anthropic ship different defaults.
Measure the token cost of every MCP server you load. On-demand tool loading can save tens of thousands of tokens per session.
Budget for higher memory and storage prices through 2027, and favor architectures that avoid duplicate copies of large datasets.

What to Watch Next Week

OpenAI's Codex sprint promises a new improvement every day, so expect a steady stream of small changes and possibly a new model before the 28 days end. Reflection's Beam weights and technical report are due later in October, and Mistral's ML4 weights should follow by month end. On the infrastructure side, the OCP Global Summit runs October 12 to 15 in San Jose and SEMICON West runs October 13 to 15 in San Francisco, so look for rack, power, and optics announcements. On the standards side, watch whether more vendors adopt the Jev request format for decision models, and whether MCP clients copy Pi's on-demand tool loading.

Keep Learning

The pace of change in AI rewards people who understand the fundamentals underneath the headlines. My books cover AI-assisted development, agentic analytics, open lakehouse architecture, and the economics of AI and work. Browse the full catalog at books.alexmerced.com and pick the one that matches what you are building next.

Stop Calling Everything Made With AI Slop

Alex Merced — Mon, 05 Oct 2026 14:36:35 +0000

Scroll any comment section under a blog post, a video, or a song in 2026 and you will find the same two-word review. "AI slop." Sometimes the label fits. The post invents a statistic, the video melts into nonsense halfway through, or the article repeats the same empty point six times in slightly different words.

Other times the label lands on work that is careful, accurate, and genuinely useful. Someone spent hours on the research, wrote pages of their own notes, argued with the draft, fixed what the model got wrong, and shipped something worth reading. A commenter spots a familiar phrase or a too-clean thumbnail and dismisses the whole thing in two words.

I want to make a case that sounds simple but gets lost in the noise. Bad AI-assisted content is bad because it is bad, not because AI touched it. Good AI-assisted content exists, it takes real skill to make, and the only way anyone builds that skill is by using the tools often enough to get past the clumsy early stage.

I have not been shy about how I work. I use AI heavily, mostly in areas where I already have deep knowledge, and I use it to move faster. I also make my mistakes in public, because public mistakes bring feedback, and feedback is how I get better. This article explains what slop actually is, why every creative tool goes through this same backlash, what I learned by doing it over and over, and how I think we should judge creative work from here on out.

What Slop Actually Is

The word has a real definition now. Merriam-Webster named "slop" its 2025 Word of the Year and defined it as digital content of low quality that is produced usually in quantity by means of artificial intelligence. Macquarie Dictionary picked "AI slop" as its word of the year too. The term earned its spot. The internet filled up with junk fast.

Read the definition closely, though. "Low quality" sits right in the middle of it. Quality is the defining trait. AI is the usual production method, but the definition does not say every AI-produced thing is slop. It says slop is low-quality content, usually mass-produced with AI.

That distinction matters, because a lot of people use the word to mean "anything I suspect involved a model." Those are two very different claims. One is a judgment about the work. The other is a guess about the workflow.

So what puts something in the slop bin? I see three clear markers.

It tries to misinform. The content spreads false claims, fabricated quotes, fake sources, or invented events. Sometimes this is deliberate. Sometimes the creator just never checked. The result for the reader is the same.
It adds no value. The piece says nothing a reader did not already know. It restates the obvious, pads the length, and leaves you with no new idea, no new skill, and no new perspective.
It clearly was not reviewed. The draft still contains the model's mistakes. You see broken references, contradictions between sections, hallucinated version numbers, and leftover phrases like "as an AI model" or "here is your blog post."

Every one of these markers describes a failure of the creator, not the tool. A human writer with no AI at all can misinform, add nothing, and skip editing. We just had other names for that content before 2023. We called it content farm filler, clickbait, or spam.

The volume is the new part. A content farm used to need a room full of underpaid writers. Now one person with a script can publish thousands of pages a day. Streaming service Deezer has reported that a large share of the music uploaded to its platform every day is fully AI-generated, at one point estimating about 28 percent of new uploads. That flood is real, and people have every right to be tired of it.

But fatigue makes people sloppy judges. When the default reaction to any hint of AI is "slop," the label stops meaning "low quality." It starts meaning "made with a tool I disapprove of." That shift hurts the people who use these tools well, and it teaches newcomers that the right move is to hide their process instead of improving it.

Every Creative Tool Got This Reception

The backlash against AI-assisted work feels new, but the pattern is old. Almost every tool that lowered the barrier to creative work got called cheating, fake, or the death of real art.

Music has the cleanest example. In 1982, the UK Musicians' Union passed a motion to ban synthesizers, drum machines, and any electronic devices capable of recreating the sounds of conventional instruments. The union feared these machines threatened the jobs of session players. The motion came after Barry Manilow toured the UK using synths to simulate the sound of a big band orchestra.

The ban went nowhere. Drum machines went on to shape hip-hop, house, and techno. Those genres did not replace older music. They added new music that did not exist before those instruments arrived. The union was right that the work was about to change. It was wrong that the change meant the end of real musicianship.

Photo editing went through the same cycle. Adobe released Photoshop 1.0 in 1990. For years after, "photoshopped" worked as an insult. It meant fake, manipulated, dishonest. Today nearly every professional photo passes through editing software, and nobody accuses a wedding photographer of fraud for adjusting exposure. People still criticize manipulated images, and they should when the edit deceives. The criticism shifted from "you used the tool" to "you used the tool to lie."

Digital audio workstations (DAWs) followed the same arc. A DAW is software for recording, editing, and arranging music on a computer. Tools like FL Studio, which started life as FruityLoops in the late 1990s, let a teenager with a laptop build tracks that once needed a studio, an engineer, and a band. Early bedroom producers got mocked for not playing "real" instruments. Some of those bedroom producers now headline festivals.

I have picked up a lot of these tools over the years. Photoshop, FL Studio, Videoleap, and plenty more. I learned each one for the same reason: it let me make the ideas in my head real. AI tools are the latest entry on that list.

Not every critic in these cycles was wrong. Some synth-heavy records from the 1980s really are thin. Plenty of early Photoshop work really was garish. Tools do get misused, especially early, when nobody has figured out the craft yet.

Where the critics went wrong was the target. They judged the tool when they should have judged the output. Over time, the culture corrected. The tool stopped being the story, and the work became the story again. I expect the same correction for AI-assisted work. The only open question is how many capable creators we discourage before it happens.

Using AI Well Is a Skill, and Skills Come From Reps

Here is the part that gets lost in most arguments about AI content. Using these tools well is a skill, and it behaves like every other craft skill. You start out bad, you get feedback, you adjust, and you slowly get good.

Nobody opens a DAW for the first time and mixes a radio-ready track. Nobody opens Photoshop and produces a clean composite on day one. The first hundred attempts teach you where the controls are. The next hundred teach you what good looks like. After that, you start to build taste and speed at the same time.

AI tools work the same way. Your first AI-assisted blog post will probably read flat. Your first AI-generated image will have strange hands or warped text. Your first AI-assisted song will sound generic. That tells you that you are a beginner, which is exactly what everyone is at the start of any skill. It says very little about the ceiling of the tool.

The people quickest to shame AI users often point to those beginner outputs as proof. "Look how bad this is." They are judging a first draft from a first-week user and treating it as the ceiling of the medium. Nobody judges guitar by listening to someone on their third lesson.

My own practice looks like this. I mostly use AI in areas where I already know the material well. I have co-written books on Apache Iceberg and Apache Polaris and written another on architecting an Iceberg lakehouse. When a model drafts something in that territory, my expertise does the checking. The model does the typing, the first-pass research, and the structural grunt work.

That pairing is where AI pays off the most. Expertise plus AI acceleration beats either one alone. An expert without AI moves slower. AI without an expert produces confident nonsense. The skill lives in the combination.

The clearest example from my own work is my personal websites. I manage them myself with AI assistance. That work used to need a team of specialists: a web developer, a copyeditor, a graphic designer. Now I handle the full workflow on my own, end to end. At a larger scale, my plan is not a row of narrow specialists. I want a small team of generalists, each one owning complete workflows end to end with AI support, all pointed at a shared goal.

None of that came free. I got there by doing the work over and over, and by doing it in public. I make mistakes in public without shame, because that is how I get the feedback and the lessons that let me improve quickly.

This is why I push back when people say "just don't use AI until it's good." The tools keep getting better, but the person using them only gets better through practice. If you wait on the sidelines until the technology is perfect, you will show up with zero experience while everyone who practiced has years of instinct on you.

What the Early Reps Taught Me About the "AI Meh" Feel

Every heavy AI user eventually learns to recognize a certain texture. I call it the "AI meh" feel. The content is not wrong, exactly. It is grammatical, organized, and on topic. It is also forgettable. Readers sense it within a paragraph, even when they cannot name what bothers them.

Learning what gives content that feel, and how to get rid of it, was one of the first lessons that came from doing this regularly.

The meh feel tends to come from a handful of habits that language models fall into when nobody steers them.

Generic openings. Models love to open with a broad statement about how fast the world changes or how important a topic has become. Readers skip these lines automatically. A strong opening starts with a concrete problem the reader recognizes.

Filler transitions. Stiff, formal connector words pile up at the start of sentences in AI drafts. They make prose sound formal while carrying no meaning. The same goes for phrases that announce importance instead of demonstrating it.

Rhythm lists. Models reach for groups of three adjectives because the rhythm sounds polished. After a few paragraphs, the pattern becomes obvious, and the reader starts to tune out.

Hedging. Unsteered drafts qualify everything. Every claim gets a softener in front of it. The result reads as if nobody stands behind any of it.

Missing opinion. This one matters most. A model with no direction defaults to a balanced summary of every side. Balanced summaries have their place, but they have no voice. Readers come back to creators because those creators see things a certain way.

I keep a list of style rules and banned words for anything written in my voice. No em dashes. No stacked formal transitions. No hedge words in the body of the piece. Active voice. Short paragraphs. Concrete numbers over vague claims of size. Those rules go to the model as part of the instructions whenever it drafts in my voice.

Rules only fix the surface, though. The deeper fix for the meh feel is input. A model drafts flat content when it has nothing specific to work with. Give it a generic prompt and you get the average of everything it has read. Give it your specific angle, your notes, your opinions, and the examples you actually care about, and the draft starts to sound like a person with a point of view.

This article is a good example. It started with a long set of notes in my own words. Every argument in it, every example, every opinion came from those notes. The model helped organize the material, expand it, and tighten it against my style rules. The thinking is mine. The acceleration came from the tool.

That is the real difference between slop and skilled AI-assisted work. Slop starts with a topic. Skilled work starts with a point of view.

Verifying Efficiently: Know Where the Model Breaks

The second big lesson from the early reps was about review. Beginners tend to make one of two mistakes. Some skip review entirely and publish whatever the model gives them. That is how slop gets made. Others review every sentence with equal suspicion, which takes so long that the AI saves them no time at all.

Efficient review means knowing where the model is strong and where it is weak, then spending your attention where the weaknesses live. A good editor already works this way with human writers. You learn which writer mangles dates and which one overstates conclusions, and you read their drafts with that in mind.

Language models have fairly predictable weak spots. Here is how I think about the split.

That last row deserves extra attention. An unsteered model writing in your voice will happily invent a conference you never attended or a customer you never met. Those fake stories read well, which makes them dangerous. I have a standing rule that no anecdote appears in my content unless it actually happened to me. The model does not get to make up my life.

Review also gets faster when you automate the mechanical parts. My style rules are specific enough to check with a script. Here is a version of the verification pass for my long-form drafts.

F=article.md

# Word count: long-form pieces need to clear 6,000 words
wc -w "$F"

# Em dashes and en dashes: must return nothing
grep -nE '—|–' "$F"

# Semicolons: must return nothing outside code blocks
grep -n ';' "$F"

# Banned vocabulary sweep: the words that make prose read like a template
grep -niE '\b(delve|leverag|seamless|paradigm|synergy|holistic|robust|unlock|realm|tapestry|testament|embark|myriad|plethora|pivotal|transformative|bespoke|curated|meticulous|elevate|empower|streamline|foster)\w*' "$F"

# Hedging modals: claims get stated plainly or not at all
grep -niE '\b(might|could|would|may) ' "$F"

# Call to action present
grep -n 'books.alexmerced.com' "$F"

Here is what each part does and why it exists.

The first line sets a variable named F to the file path, so every later command checks the same file without retyping the name.

The wc -w command counts words. Long-form pieces have a target length, and models often land short when asked for a long draft in one pass. A word count catches that right away, before anyone spends time reading a draft that needs more material anyway.

The dash check uses grep -nE with a pattern that matches either an em dash or an en dash. The -n flag prints line numbers, so each hit is easy to find. Models lean on em dashes heavily, and a page full of them is one of the fastest tells of unedited AI prose.

The semicolon check works the same way. My style rules keep semicolons out of prose, so the only acceptable hits are inside code blocks.

The vocabulary sweep uses -i for case-insensitive matching and \b for word boundaries, so one stem catches every form of a word, capitalized or not. Every word on that list is technically fine English. The problem is frequency. Models reach for these words far more often than people do, and a reader who sees three of them in a paragraph stops trusting the writing.

The hedging check looks for soft modal verbs followed by a space. The goal is plain claims. If a claim is true, state it. If it holds only under certain conditions, name the conditions.

The last line confirms the call to action made it into the final draft. That sounds trivial, but when a model rewrites a closing section, the link sometimes disappears.

None of this replaces reading the piece. The script handles the mechanical checks in about a second, which frees my attention for the parts only a human expert can judge. Is the argument right? Is the technical claim accurate? Does this say something worth saying?

That division of labor is the whole point. The machine checks the machine-checkable. I check the meaning.

The Flow Changes With the Model, the Tool, and the Medium

Once you have a process that works, it is tempting to treat it as finished. It never is. My workflow shifts every time one of three things changes.

The model. Different models have different habits. One model leans on certain phrases, while another pads its output with summaries at the end of every section. One is more likely to invent citations, another is more cautious but more generic. When I switch models, my review checklist shifts with it. The weak spots in the table above hold up broadly, but the details move.

The tool around the model. The same model behaves differently depending on the app or agent setup wrapped around it. A chat window, a coding agent with file access, and a writing tool with a custom style guide all produce different results. Some setups let the model search the web, which helps with current facts but introduces new risks around sources. Others let the model run code, which helps with verification. Learning what each setup is good at is its own skill.

The medium. Writing, video, and music each have their own failure modes. In writing, the risk is factual errors and the meh feel. In video, it is visual consistency from shot to shot, characters whose faces drift, and motion that breaks physics in distracting ways. In music, it is generic arrangements, muddy mixes, and lyrics that rhyme without saying anything.

Each medium needs its own review habits. For an article, review targets facts and voice. For a video, it targets continuity from shot to shot, and anything that breaks the illusion gets cut. For a song, it targets the moments where the arrangement goes on autopilot.

The underlying pattern stays the same across all three. You learn where the tool is reliable. You learn where it fails. You put your effort where the failures are. And the only way to build that map is to make a lot of things and pay attention to what goes wrong.

This is also why one attempt, years ago, is a weak basis for deciding AI is useless. The tools they tried are gone. The models changed, the setups changed, and the outputs changed. More to the point, they never built the map. One try tells you almost nothing about what a practiced user can do.

"If AI Wrote It, Why Should I Read It?"

This is a common objection, and it deserves a real answer. The argument goes like this. If a machine wrote the piece, the writer put in no effort, so the reader owes it no attention. Why spend my time on something nobody bothered to write?

The objection rests on an assumption that the person contributed nothing. For slop, that assumption is usually correct. Someone typed a topic into a box, copied the output, and hit publish. There is no person in that content, so there is nothing of a person to read.

Skilled AI-assisted work is different. The creator puts themselves into the piece at every stage that matters.

The angle. Before any drafting happens, someone decides what the piece argues. A model left alone writes the average take. A creator with a point of view picks the specific claim, the counterintuitive framing, or the uncomfortable truth that makes the piece worth reading.

The research. Someone decides which sources to trust, which facts to verify, and which examples best support the point. Those choices shape the piece as much as any sentence does.

The notes. Strong AI-assisted pieces start with the creator's own notes. Those notes hold their opinions, their reflections, the wisdom they have built up, and the arguments they want to make. Those notes are the raw material. The model shapes them. It does not invent them.

The judgment. Someone reads the draft, rejects what is wrong, pushes back on what is weak, and decides when the piece is done. That editorial judgment is a creative act. Film directors do not operate every camera, and nobody says their films lack a creator.

When all four are present, the reader gets the creator's thinking, delivered faster than typing it alone. That is a good deal for the reader.

There is a second half to this objection worth answering. People say, "If AI wrote it, why should I read it? I can just ask AI." Fair enough. Ask AI to help you read it, then. Ask a model to summarize a long piece, pull out the key claims, or explain a section you found confusing.

Reading can be AI-assisted the same way writing can. There is no shame in that. A dense technical article is a lot more approachable when you can ask follow-up questions about it. Using AI to read does not mean the piece should not have been written. It means the reader found a faster way into it.

The piece still had to exist. Somebody still had to choose the argument, gather the evidence, and stand behind the conclusions. A model summarizing an article you never wrote has nothing to summarize.

And there is real pleasure in reading something slowly, on your own, with no assistance at all. Some books deserve your full, unassisted attention. Neither approach is the correct one. Both are tools a reader can pick up depending on what they want from the experience.

What Good Looks Like

Arguments are easier to evaluate with examples, so here are two. One comes from a creator whose work shows what this technology can do. The other comes from my own music.

Gossip Goblin

The filmmaker Zack London, known online as Gossip Goblin, makes dark, strange science fiction short films with AI video tools. His channel has passed 1.4 million subscribers and more than a billion total views. His first feature-length film, an anthology called Gods Don't Give Gifts, was scheduled to open in several hundred U.S. theaters on October 30, 2026, produced with the studio Fable.

What makes his work relevant to this argument is how much human effort goes into it. London has said his feature still needed a human-written script, a dozen voice actors, and a post-production crew. Reporting on the feature describes a team of 10 artists working on it over two months. His process starts with a script, then builds every shot from a still image generated in Midjourney, refined in other tools, and only then animated into video. That is a slower, more hands-on pipeline than the one-click generation most people picture.

London has also said plainly that AI is a tool for him and not the point. It gave him access to filmmaking without replacing crafts he already valued, like illustration. His shorts do not hide the strangeness of the medium. They lean into it and build a recognizable visual style from it.

You do not have to like his films. Taste is personal. But nobody watching his work in good faith calls it low effort. It has a vision, a consistent world, and a recognizable voice. Those are the things that separate art from filler in any medium. AI did not supply them. London did.

My old acoustic recordings

The second example is smaller and more personal. I have old acoustic recordings from my younger days. They hold my effort and my vision from that time.

Suno is an AI music platform, and one of its features lets you upload your own audio and generate a new version in a different style while keeping the original melody. My old acoustic recordings have become the seeds for full productions.

The songwriting, the melodies, and the emotional core come from my younger self. The production comes from a tool that did not exist back then. Neither approach makes these songs alone, and the result is a fun bridge between two versions of me.

Is that slop? Run it through the three markers. It does not misinform anyone. It adds value by finishing work that started years ago. And the review test comes down to keeping the versions that serve the song and tossing the ones that miss. By the definition that matters, it is not slop. It is a creative project that uses a new tool.

There Is Room for Both Markets

I want to be clear about one thing. I am not arguing that all content should be AI-assisted. I do not think that, and I do not want that.

There will be a strong market for creators who make everything without AI, and there should be. Some audiences care deeply about knowing that a human did every step. That preference is valid, and the market already serves it. Bandcamp, the music platform popular with independent artists, announced in January 2026 that it no longer allows music generated wholly or in substantial part by AI. The company framed the policy as a way to keep its community human. That is a reasonable choice for a platform, and it gives listeners who want fully human music a clear place to find it.

Handmade work has always held value next to manufactured work. Hand-thrown pottery sells alongside factory dishware. Film photography survives next to digital cameras. Live acoustic sets still draw crowds in a world full of programmed beats. Those markets did not vanish when the faster tool arrived. They found their audience, and the handmade quality became a feature people choose on purpose.

The same thing will happen with writing, video, and music. Some creators will advertise that they use no AI, and some audiences will seek them out. Other creators will use AI heavily and openly, and their audiences will care about the output, not the process.

What I push back on is the idea that one camp gets to define the other as illegitimate. The human-only creator is not a dinosaur. The AI-assisted creator is not a fraud. They are making different choices about tools, and both can produce excellent work or terrible work.

Every major creative technology eventually settles into some balance between efficiency and quality. Photoshop did not make every photo fake. DAWs did not make every song soulless. The balance point took years of experimentation, curiosity, and a lot of bad early work to find. AI-assisted creation is in that experimentation phase right now. The people who will find the balance point are the ones willing to practice in the open.

How to Practice Without Publishing Slop

If you are early in using AI for creative work, the advice to "just do it a lot" needs some guardrails. Practice should not mean flooding the internet with weak drafts. Here is how I think about building the skill responsibly.

Start where you are already an expert. Use AI first in a subject you know cold. Your knowledge acts as a built-in fact checker. You will catch the model's errors quickly, and you will learn its failure patterns faster because you can see them. Using AI in a field you do not understand is how people publish confident mistakes.

Verify against primary sources. When a draft cites a release date, a feature, a quote, or a number, check it against the original source. That means the official documentation, the project's release notes, the dictionary entry, or the article that first reported the claim. Secondhand summaries repeat each other's errors, and models blend them together. A habit of going to the original catches most factual mistakes before they reach a reader, and it gets faster as you learn which sources hold up.

Write your notes before you prompt. Spend 15 minutes writing down what you actually think about the topic. Your opinions, your examples, the arguments you want to make, the mistakes you see other people making. Feed those notes to the model. The difference in output quality between "write about X" and "here are my notes on X, shape them into an article" is enormous.

Keep a running list of what goes wrong. Every time a model makes a mistake you catch, write it down. Wrong dates, invented quotes, repeated phrases, weak openings. After a few dozen pieces, that list becomes your review checklist. It tells you exactly where to look, which is how review gets faster over time.

Turn your style into rules. If you notice yourself fixing the same pattern over and over, write a rule for it and give the rule to the model up front. Then check the rule with a script where you can. My own banned-word list and verification script are examples of this.

Publish less than you generate. The ability to produce a lot does not mean you should publish a lot. Generate ten drafts, publish the one that says something. Volume is the defining trait of slop. Selectivity is the defining trait of craft.

Be open about your process. Hiding AI use creates a trust problem when people find out. Being open about it lets you get honest feedback on the work itself. I talk about how I use AI because the feedback makes me better, and because I think the conversation about these tools needs more practitioners sharing real workflows.

Take criticism of the work seriously and criticism of the tool lightly. When someone says a claim is wrong or a section is weak, that is gold. Fix it. When someone says "AI slop" with no further detail, there is usually nothing to act on. Check whether your piece hits any of the three slop markers. If it does not, move on.

Judge the Work, Not the Workflow

So how should we judge creative work now that AI is part of so many workflows? The same way we judged it before. Look at the output and ask the questions that always mattered.

Is it accurate? Does it state true things? Are its sources real? Do its numbers hold up? A piece full of errors fails this test whether a human or a model introduced the errors.

Does it add value? Did you learn something, feel something, or see something in a new way? Content that leaves you exactly where you started has failed, no matter how it was made.

Does it have a point of view? Can you tell what the creator thinks? Is there a specific argument, a specific style, or a specific vision behind it? Work with a clear voice behind it is almost never slop.

Was it cared for? Do the details hold together? Are the rough edges sanded down? Does it feel finished? Care shows up in the work, and its absence shows up too.

Notice that none of these questions asks what tools the creator used. That is deliberate. We do not grade a novel by whether the author used a typewriter or a laptop. We do not grade a song by whether the drums came from a kit or a drum machine. We do not grade a photo by whether it passed through editing software. We grade the result.

These questions also work in both directions. They catch slop made with AI, and they catch lazy work made without it. A handwritten article full of errors fails the accuracy test. A fully human song with nothing to say fails the value test. The standard stays the same no matter which tools sit in the workflow.

AI-assisted work deserves the same standard. Hold it to a high bar. Call out errors. Call out empty content. Call out work that clearly never got a human review. Those criticisms help everyone.

But when a piece is accurate, valuable, distinctive, and carefully made, the presence of AI in the workflow counts as a production detail, the same way the choice of camera or DAW is a production detail. Something that adds real value, AI or not, still required skill, vision, and determination to make.

Conclusion

Slop is real, and I share the frustration with it. Content that misinforms, adds nothing, and skips review makes the internet worse, and people are right to call it out. The dictionary even gave us a word for it, and the definition puts low quality at the center.

But the word has drifted. Too many people now use "slop" to mean "made with AI," and that shift punishes the wrong people. It discourages beginners from practicing a skill that only improves with practice. It pressures experienced creators to hide their process instead of sharing what they learned. And it confuses a judgment about a workflow with a judgment about the work.

My own path has been simple. I learn whatever tool lets me make the things I picture in my head. Photoshop, Videoleap, FL Studio, and now a range of AI tools. Each one made me more versatile. I learn in public and make my mistakes in public on purpose, because public mistakes bring the fastest lessons, and I am proud of what that approach has let me build.

If you want to criticize AI-assisted work, criticize it on the merits. Point to the wrong fact, the empty argument, the missing review. If you want to make AI-assisted work, put yourself into it. Bring your angle, your notes, your judgment, and your willingness to throw away drafts that do not meet the bar.

Judge creators by the quality and value of what they make, the way we judge creators in every other medium. That standard worked for photography, for synthesizers, and for the bedroom producer with a laptop and a DAW. It will work for this too.

Keep Going

If this piece was useful, I have written a lot more on how AI is changing the way people work. My book on AI and labor economics covers the economic side of this shift, and you can find it at a.co/d/06SeOKw8. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Apache Data Lakehouse Weekly: September 23 to 30, 2026

Alex Merced — Wed, 30 Sep 2026 13:57:19 +0000

Release week. Apache Iceberg 1.12.0 passed its vote after a license audit sank the first candidate, Apache Polaris shipped 1.8.0 with a security fix inside it, and Iceberg C++ 0.4.0 went out the door. Under the release traffic sat a quieter theme that ran through every list. Specs are getting stricter about what they promise. Iceberg voted to forbid new equality deletes in v4. Parquet argued over whether a vector type should reject NaN. Ossie asked what a column name in a metric expression points to. Polaris debated whether a config field should be an enum or free text. Each thread is a fight about contracts, and each one ends with a narrower, more checkable promise to the engines that read these formats.

This issue covers roughly 380 messages across the Iceberg, Polaris, Arrow, Parquet, DataFusion and Ossie dev lists between September 23 and September 30.

Apache Iceberg

Iceberg 1.12.0 passes on the second try

Neelesh Salian ran the 1.12.0 release, and it took two candidates to land. The RC1 vote drew early +1s from Alex Dutra, Gianluca Graziadei and Szehon Ho. Then Kurtis Wright posted a -1 with a disclaimer at the top: he had prompted Claude to audit every license in the candidate. The audit found that the GCP bundle jar shipped Caffeine with no entry in its LICENSE file. Kevin Liu followed up with his own table of gaps and found a second one around Jackson's FastDoubleParser. Jean-Baptiste Onofré cast a -1 on the same grounds. Neelesh sorted each report, landed the fixes, and pulled in Gianluca's patch for a variant-column NoSuchMethodError that hit clusters with older commons-lang3 jars.

The RC2 vote opened on September 25. Neelesh stretched the window to Tuesday afternoon to cover the weekend. It had one scare. Huaxin Gao pointed out that the staging repository had no Spark 4.2 artifacts, even though the source tree, CI matrix and Gradle defaults all pointed at 4.2. Amogh Jahagirdar agreed a new RC looked necessary. Russell Spitzer then asked why an earlier PR had excluded 4.2 from convenience binaries on purpose. Amogh found the older thread. The community had held back 4.2 binaries while Spark's own APIs kept moving. Huaxin withdrew the alarm and voted +1 with a note that every RC1 license gap was closed in the jars themselves.

The result came in on September 29: four binding +1s from Russell Spitzer, Kevin Liu, Szehon Ho and Amogh Jahagirdar, and ten non-binding +1s. Two details stood out in the votes. Kevin ran an agent over the repository looking for correctness bugs and flagged a handful of issues that also exist in 1.11. He called them non-blockers and listed them for triage. Kurtis came back with a second AI-assisted pass, this time labeled as the output of an agent skill built for checking open source release candidates. Release verification is turning into a place where contributors point agents at tedious, rule-driven work and then sign their names to the result.

The CI side of the release had its own story. Steven Wu asked where the MinIO replacement stood after Kafka Connect integration tests failed on the MinIO container. Neelesh opened a tracking issue and a stopgap PR. Within a few hours Kevin reported that the switch was done across every Iceberg repository, crediting Sreesh for the work. Amogh later noted that he ran the RC2 integration tests against RustFS. The S3 test backend for the project is now a Rust implementation.

v4 forbids new equality deletes

Huaxin Gao called a spec vote on PR #17783. The direction was already settled in July: v4 writers must not produce new equality deletes, readers still apply existing ones from v2 and v3 tables, and an upgrade stays metadata-only. This vote covered the exact spec wording. Russell Spitzer questioned whether a second vote was needed at all, then voted +1 anyway. Ryan Blue said the point was to affirm the decision and merge it. Anoop Johnson said the change simplifies the v4 design a lot. The vote passed with four binding +1s from Russell, Yufei Gu, Steven Wu and Ryan, plus five non-binding. Huaxin will merge the PR and follow up with a separate change that restructures how the spec presents deletion vectors next to the legacy delete formats.

This is the most consequential spec change of the month for engine builders. Equality deletes let streaming writers such as Flink record "delete the row where id = 7" without reading the table first. They also push work onto every reader, which has to join the delete file against data files at scan time. v4 closes that door for new writes. Streaming pipelines that rely on equality deletes need a plan for deletion vectors before they adopt v4.

Who controls access: the delegation header debate

A thread that started in Polaris landed on the Iceberg list. youngrae kim asked what a REST server should do when a client sends X-Iceberg-Access-Delegation but the server can satisfy none of the requested mechanisms. Picture vended credentials against an S3-compatible store with no STS, or remote signing on a server that does not implement it. Polaris returns HTTP 400. The spec says the server can supply access through any or none of the requested mechanisms.

Alex Dutra proposed spec wording that favors failing fast, and Yufei Gu agreed. Daniel Weeks, who wrote much of the REST spec, disagreed with the framing. His argument was that the catalog should be the sole authority on access, and security around physical data should not be a negotiation between client and server. The loose wording was deliberate. It lets a catalog return credentials or remote signing config, or return metadata only, without the spec forcing its hand. Alex countered that the header cannot express a common client capability, which is using the client's own storage credentials and skipping delegation entirely.

youngrae closed the loop on September 29. Polaris will keep its 400 as a documented implementation choice. The remaining ask is narrow: one sentence in the spec that says whether the header is a capability statement or a requirement, and what an absent header means. That clarification leaves the catalog's decisions untouched. It only tells clients and servers how to read the signal.

A related effort moved forward. Sung Yun and Prashant Singh had been discussing access delegation for the FILE type, where a governed table references files that engines need to read directly. Sung reported that the two of them converged their proposals onto PR #18080, which now carries the optional request and response parameters from both drafts. Steven Wu also announced that the Thursday biweekly sync will cover both the File and Vector types starting October 8. He said the File type discussion is close to consensus, so more of the time will go to the Vector type design doc from Yan Yan.

Labels, SQL, and the agent-shaped consumer

Labels landed in the REST spec recently. A catalog can now attach object-level and field-level labels to a load-table response. Andrei Tserakhau asked whether Iceberg should expose them as a .labels metadata table. Péter Váry had raised the underlying issue in review. Every metadata table today derives from table metadata. A .labels table carries catalog-owned data that varies by catalog and can be empty.

Ryan Blue questioned the premise that labels need a SQL surface at all. He saw them as context for engines, such as cost attribution logs or attaching an engine policy to a table. Andrei answered that some consumers only speak SQL. He named governance tooling and, specifically, LLM agents that explore a warehouse by writing queries. Péter argued that the proposed uses turn labels into full table metadata, which contradicts the read-path design that treats them as a current view with no persistence guarantees. Ryan then took apart Andrei's cost-attribution example. The sample query joined .labels to .files with no ON clause, which produced a cartesian join that only looked right because the filter selected one label.

Andrei conceded the join argument and pointed to an alternative. PR #18049 surfaces object and field labels through DESCRIBE, which gives SQL-only consumers, agents included, a way to see labels without a new metadata table. The metadata table is parked until someone shows a use case that needs joins. The episode is worth noting. "An AI agent needs to read this" showed up as a design argument on the Iceberg list, and the community answered it with the smallest surface that works.

Encryption keys move closer to the files they protect

Gábor Kaszab reopened the design of encryption key storage. Today table metadata carries an encryption-keys list that holds both data encryption keys for manifest lists and the key encryption keys that wrap them. Gábor wants to extend encryption to statistics files and v4 root manifests. His proposal stores each file's wrapped key directly in the metadata entry for that file, one key per file. KEKs stay in the shared list and are still referenced by key-id. The spec draft for statistics files lives in PR #17533.

Gidon Gershinsky agreed the DEK metadata belongs in per-file structures in v4. He added that reusing a DEK across files is possible but requires careful handling to avoid breaking AES-GCM, so the practical rule is a fresh random DEK per file. Xander Bailey wanted the one-key-per-file rule stated explicitly and asked for a common wrapped-key structure across file types, so implementations share code. Gábor's closing argument was about cleanup. With keys in a shared list, every new encrypted file type needs its own cleanup trigger when the file is dereferenced. With keys stored inline, the key disappears with the file. Xander found the argument convincing and asked for a Java draft. Gábor wants the design settled first. Xander also proposed a recurring encryption community sync to cover client library work, the KMS vending spec and v4 changes together.

Smaller spec decisions

rahul mahadev called a vote on a standard User-Agent format for REST clients. The format lists products from most specific to least, such as Spark/4.0.0 iceberg-spark/1.9.0 iceberg-java/1.9.0 (scala 2.13.16). The Iceberg library token must appear when the header is sent. Servers must not reject requests based on it. Xuanwo liked that the header stays separate from authentication and capability negotiation. Alex Dutra left more PR comments to resolve before the vote proceeds.

Eduard Tudenhöfner announced a rename in the new content stats: avg_value_size_in_bytes becomes total_bytes in PR #18308. Averages do not aggregate cleanly across files. Totals do. Daniel Weeks confirmed that the average is still recoverable from total_bytes and value_count, and noted that a total gives planners an explicit upper bound for size estimates. Ryan Blue gave a +1.

Shawn Chang asked why the index spec requires region files to be globally ordered and non-overlapping, given the write amplification and maintenance cost. Péter Váry explained the target. Once index metadata is cached, a single-key lookup should cost one small file read. Overlapping regions force readers to consult several files and reconcile results, and on object storage every extra file read adds latency.

Releases beyond Java

Manu Zhang announced Iceberg C++ 0.4.0 on September 26. The vote passed with binding support from Gang Wu, Renjie Liu and Kevin Liu. Renjie's verification ran 4,152 individual tests across 19 test executables and exercised REST and SQL catalog support, including a SQLite catalog run. Xuanwo suggested a LICENSE path fix for the MurmurHash3 files, and Manu opened an issue for the next release.

Matt Topol opened the vote for Iceberg Go v0.7.0 on September 28. Neelesh Salian, Andrei Tserakhau and Xuanwo voted +1. Xuanwo ran 3,344 tests on Go 1.25.9 and saw one Puffin checksum assertion fail on Go 1.27.1. The code still rejected the corrupted input, so he called it non-blocking.

Community and housekeeping

Robin Moffatt opened PR #18019 to finish the Kafka Connect LICENSE and NOTICE cleanup that Ryan Blue started in April. The PR regenerates runtime-deps.txt, aligns the connector's cloud SDK dependencies with the AWS, GCP and Azure bundles, and drops the Hive distribution. The timing matters. A Confluent Hub security scan flagged the Iceberg Kafka Connect listing, and Confluent's policy escalates flagged connectors toward removal unless the partner responds.

Gaspard Merten shared a PyIceberg PR that moves upsert comparisons out of Python loops and into PyArrow compute. He built it while working on a data lake for a financial organization, and he said the change cuts memory and CPU for wide-table upserts. He also noted that he wrote part of it with Claude and reviewed it himself several times.

Viktor Kessler announced two European community meetups in back-to-back days: Berlin on November 18 and Amsterdam on November 19. Both have open calls for talks.

Apache Polaris

Polaris 1.8.0 ships with a CVE fix

Jean-Baptiste Onofré's first 1.8.0 candidate did not survive. Alex Dutra voted -1 on rc1 because a test file added in PR #4823 carried an incorrect license header. JB cancelled the vote and took another pass at LICENSE and NOTICE. The rc2 vote passed with binding +1s from Alex Dutra, Yufei Gu, Dmitri Bourlatchkov and JB, plus non-binding support from Prithvi S, Ajantha Bhat and Ayush Saxena. Yufei's verification checked 442 artifacts and ran authenticated CLI smoke tests against built and staged servers.

JB announced 1.8.0 on September 28. The notable changes:

Semantic model privileges. Semantic models now get dedicated privileges for list, create, read, update and drop. Admins grant them to catalog roles on a single model, a namespace or a whole catalog. This is Polaris preparing to govern Ossie-style semantic models the same way it governs tables.
CLI toggles for STS and KMS. catalogs update accepts --no-sts and --no-kms, so operators can change those settings on an existing S3 catalog instead of recreating it.
GCP federation auth. The CLI adds gcp as an external catalog auth type for Iceberg REST federation. That lets operators create GCP-authenticated catalogs such as BigLake without passing Google credential secrets as command-line flags.
Paging. A global --page-size option paginates list calls, when the server enables the LIST_PAGINATION_ENABLED flag.
JDBC schema selection. The relational JDBC backend reads its schema from the driver's currentSchema property, so the persistence layer no longer hardcodes a schema name.

A day later JB published CVE-2026-97395, rated important, for all Polaris versions before 1.8.0. A principal with permission to set table properties was able to put FileIO settings such as s3.endpoint into table metadata. During server-side operations like commits and purges, older Polaris versions built their own FileIO client from those settings unless the catalog's storage config overrode the endpoint. The result: Polaris sent storage requests, signed with operation-scoped credentials, to a host the table writer picked. Deployments that do not fully trust table writers should upgrade. The reporter was credited as vignesh a.

Ajantha Bhat also shipped the Polaris Iceberg Catalog Migrator 1.1.0 after an RC3 vote with binding support from Dmitri, Russell Spitzer, JB and Alex. Russell's check covered all 12 primary Maven artifacts and 89 tests.

Enum or free text: the R2 credential vending fight

Austen Tomek of Chicago Trading has a PR adding Cloudflare R2 support with scoped credential vending. The discussion reached a clean disagreement. Austen explained why endpoint matching cannot identify the storage vendor: MinIO and other self-hosted stores use arbitrary hostnames, and a failed match silently falls back to STS. So the PR adds an explicit credentialVendingMechanism field to the Management API.

Dmitri Bourlatchkov wants that field to be free text with runtime validation, so downstream builds of Polaris can add custom mechanisms. Yufei Gu wants an enum, arguing that a spec is a contract and clients should not depend on a specific deployment's extensions. When Dmitri asked for lazy consensus, Yufei said there was none. JB stepped in on September 30 and asked everyone to slow down and align. He pointed out that the field lives in the Management API for catalog administrators. Iceberg REST clients such as Spark and Trino never see it. They only consume the standard vended credentials. That observation lowers the stakes for engine interop, but the question of how extensible the Polaris Management API should be remains open.

Persistence: DynamoDB and read replicas

Yuewei Zhou proposed an optional DynamoDB persistence backend built as a new adapter under AtomicOperationMetaStoreManager, the same shape the relational JDBC backend uses. Robert Stupp prefers a different home. He wants DynamoDB as another backend inside the existing NoSQL persistence framework, where a compare-and-set on a reference is the moment a validated catalog change becomes visible as one consistent state. He also has a DynamoDB implementation in his own fork. Dennis Huo asked the community to use DynamoDB as a chance for an apples-to-apples comparison between the two persistence approaches. JB sided with Robert. He noted that the atomic manager implicitly assumes relational multi-entity transactions, and forcing DynamoDB into that shape means orchestrating TransactWriteItems for multi-entity writes. Yuewei offered to help on whichever path moves forward.

A second persistence thread ended with a request for data. Yong Zheng wants Polaris to route read-only requests to a database read replica when a custom header is present. Prithvi S warned that routing by HTTP verb risks double ingestion if a read hits the replica before a commit is visible and the client retries. Robert Stupp read the Medium post behind the proposal and found 15 Polaris pods serving about 400 requests per second, which is under 27 per pod. That does not look like connection saturation to him. Yufei asked for baseline numbers before any design work: request rates, tables per namespace, latency targets. Yong agreed to run benchmarks with the polaris-tools harness over the coming weeks. JB added that pool multiplexers such as PgBouncer or RDS Proxy belong in that comparison.

Metrics, tags and federation

Dmitri Bourlatchkov flagged Management API changes inside PR #4115, the long-running work on REST endpoints for table metrics and events. The PR adds a TABLE_READ_METRICS grant at table, namespace and catalog level. Reading reports needs that grant, TABLE_FULL_METADATA or CATALOG_MANAGE_CONTENT. Sending scan and commit reports stays tied to data read and write privileges. Yufei reminded the list that the PR also introduces a new REST spec for metrics consumption. Dmitri proposed merging on October 1, and JB agreed, with the JDBC query SPI in PR #4756 as the next step.

EJ Wang reported progress on the Tag spec. All four planned PRs are open: the API contract in #5366, definition CRUD, assignment writes and storage, and reads with inheritance and reverse lookup. Dmitri supports merging the API PR once open review threads close. EJ also built a proof of concept for archiving design proposals as Markdown. Authors keep collaborating in Google Docs, add a version label and a link to one YAML manifest, and the tool captures the doc as Markdown with images for the repository.

David Chaava wants fail-fast validation for vendor-specific Iceberg REST federation settings, starting with BigLake. Dmitri, JB and Prithvi all asked that PolarisAdminService stay vendor-neutral. David's revised plan uses a CDI-discovered validator SPI with a BigLake implementation. Srinivas Rishindra posted the first PR for multiple storage configurations per catalog, and Travis Bowen weighed in on where the new Open Sharing APIs should live, leaning toward the existing /api/management path.

Code health

Two cleanup threads resolved. Yufei Gu found the authentication flow confusing while reviewing another PR. Two augmentors attached a credential that a third consumed, and the ordering only held because of a code comment the compiler cannot check. Alex Dutra proposed merging the OIDC pieces into a single OidcIdentityPreparer that the authenticating augmentor calls directly. Prithvi agreed, Alex opened PR #5612 on September 25, and it merged on September 28.

The long argument over schema upgrade instructions also ended. Yufei argued that SQL snippets in the upgrade guide break for anyone who starts from an older schema version, and that the versioned schema files are the only source of truth. Alex disagreed but removed every migration snippet from PR #5349 to unblock it. The page now explains how to diff the schema files. Prithvi asked for some SQL guidance to return later, since a schema diff shows the end state but not how to migrate tables that already hold data. The PR has approvals from Dmitri, Yufei and Yong Zheng.

Apache Arrow

Dropping Intel Macs

The macOS Intel thread brought numbers. Joris Van den Bossche pulled 30 days of PyPI downloads: x86_64 made up 6.6 percent of PyArrow's macOS downloads, 57,121 against 807,414 for arm64. Sutou Kouhei reproduced the check with a public ClickHouse query against the PyPI dataset and voted +1. Wes McKinney offered his 2019 Intel Mac Pro for testing commits to main. Kurtis Wright asked whether Arrow has any policy guaranteeing a number of releases before a platform goes away. Raúl Cumplido said no, and that such a promise is not realistic when Arrow depends on package managers and CI platforms with their own deprecation schedules. Homebrew already dropped Intel support without naming a last version.

Modularity and schemas

Raúl Cumplido also opened a discussion on PyArrow modularity. Arrow C++ is already split into a dozen libraries, from Core and Compute through Acero, Dataset, Flight, Flight SQL ODBC and S3. GCS and Azure still sit in core libarrow, and he wants them moved out the way S3 and the AWS SDK were. The goal is PyArrow installs that pull only the pieces a user needs, which matters for container images and serverless functions where package size has a cost.

Dewey Dunnington endorsed Kent's revised proposal for a JSON representation of Arrow schemas. David Lee asked for union type support and shared the alias-based JSON format he already uses to define data contracts between processes. A standard JSON schema form gives config files, APIs and agents a way to describe Arrow data without a binary IPC message.

Arrow Rust 60.0.0

Andrew Lamb announced Arrow Rust 60.0.0 on September 29. The release adds new Parquet encoding support, continues a sustained push to remove panics from the codebase, and lands kernel performance work plus fixes across decimals, take and filter, and FFI. The panic removal effort deserves a mention. A library that returns errors instead of aborting the process is easier to embed in long-running query engines and services, which is where arrow-rs lives inside DataFusion, Comet and a long list of downstream projects.

Apache Parquet

The vector type argument

The busiest Parquet thread of the week, at 30 messages, was about vectors. Rok Mihevc opened PR #624 in parquet-format for a LIST-based VECTOR logical type. It separates the logical contract from physical layout changes. Every non-null vector has exactly num_elements. Elements are non-null and finite. Elements are numeric or boolean primitives, with no nesting.

Antoine Pitrou objected to the restrictions. Parquet is a general-purpose format, he said, and NumPy produces vectors with NaN and infinity all the time. Readers can refuse what they do not support, but the spec should not. Alkis Evlogimenos answered with a survey. Postgres, Oracle, MySQL, SQL Server, Milvus, Qdrant and Pinecone all disallow nulls, NaN and infinity in vectors. DuckDB and ClickHouse allow them, but only because they store vectors as plain float arrays and switch off vector features when bad values appear. Daniel Weeks argued that a vector has mathematical meaning, with dimension, magnitude and direction, and non-finite values break it. Russell Spitzer asked for a concrete tool that produces non-finite vectors and a downstream engine that uses them productively.

Will Edwards reframed it. He had first assumed the goal was general ML tensors. After the sync calls he understood the immediate need was fixed-length embeddings for nearest-neighbor search, and in that context the constraints make sense. He also noted that embedding APIs do sometimes return NaN on overflow, which is exactly the kind of bad data a strict type catches. Andrew Lamb suggested a narrower name such as FINITE_VECTOR or VECTOR_EMBEDDING if the name is the real problem. Gang Wu proposed treating a general FixedSizeList as the physical layout and the specialized vector as a separate logical contract. Philipp Fischbeck and Divjot Arora both want to proceed with the proposal as written. Divjot added a practical point: statistics cannot reliably detect non-finite values, so a type-level guarantee is the only way a reader can trust the data without scanning it.

The practical outcome is taking shape. A restricted vector type for embeddings, with a possibly more specific name, and a separate track for general multidimensional arrays that Rok still wants to pursue. For lakehouse teams storing embeddings next to their tables, this is the thread that decides whether engines can index Parquet vector columns without validating every value first.

Encodings: PFOR, FastLanes, ALP and FSST

Prateek Gaur posted hard numbers in the PFOR encoding thread. Antoine had asked whether PFOR's delta mode can reuse Parquet's DELTA_BINARY_PACKED kernel. Alkis asked whether a faster DBP decoder closes the gap. Prateek measured three setups on AWS Graviton 4. On 18 datasets where delta mode helps, PFOR's own patched-delta layout reached a 10.07 compression ratio and 9.85 GB/s decode. A DBP stream inside each PFOR vector reached 6.54 and 3.76 GB/s. A tuned DBP decoder with SIMD prefix sums climbed to 5.96 GB/s but kept the 6.54 ratio. His summary: the patched-delta layout is 35 percent smaller and 1.65 times faster to decode on the datasets where delta matters. The gap is structural. PFOR uses one bit width per 1,024-value vector, while DBP splits the same vector into miniblocks with their own widths.

In the parallel FastLanes thread, Prateek and Alkis agreed to decouple FastLanes from PFOR. The initial PFOR proposal keeps sequential bit packing, and a layout discriminator leaves room to add FastLanes or interleaved layouts later. The thread also produced the sharpest exchange of the week. Antoine told Prateek that his design document and PR descriptions read like AI-generated text and asked him to rewrite them as a person talking to people. Prateek trimmed the document and cut two sections. The Parquet community is receptive to benchmark-driven proposals. It also expects authors to explain their own work.

On the rest of the encoding roadmap, the ALP blog post went through community review and Andrew Lamb described ALP as out the door. He then turned to the FSST proposal and suggested an optional inline symbol table so FSST also applies to dictionary pages. ALP targets floating point columns. FSST targets strings. Together with PFOR for integers, Parquet is on track to cover the main column types with modern lightweight encodings.

Types and the edges of the format

Micah Kornfield pushed back on two type proposals with the same argument: Parquet is a file format, not a table format. Thomas Kissinger wants an extensible decimal floating-point type with a write-time canonicalization contract. Micah answered that an empty table means no Parquet file exists, so there is no Parquet schema to coordinate independent writers. That coordination belongs to a layer above. Micah also raised concerns about a proposed WIRE logical type for serialized messages such as protobuf, citing the edge cases in lining up type systems for shredding, from recursive schemas to well-known types.

Steve Loughran argued in the format version thread that readers must fail on an unsupported version. Pretending to support a version you do not understand is a pretense, he said, and that rule is a strong reason not to bump the version without a compelling payoff. Adrian Garcia Badaracco proposed classifying distinct_count as inexact, since the spec never says whether writers are allowed to estimate it or whether readers can trust it as exact. And Danica Fine invited the community to Lakehouse Day EU on October 10 in Glasgow, co-located with Community Over Code.

Apache DataFusion

Comet 1.1.0 and the token budget

Andy Grove opened the Comet 1.1.0 RC1 vote on September 28. L. C. Hsieh, Marko Milenković and Andrew Lamb each gave a binding +1 within a day. Then Andy posted that his deeper verification had already found several regressions. He is tracking them in issue #6402 and said the full check will take a few days because he does not have an unlimited token budget. That line tells you how Andy verifies releases now: with agents that run through test matrices, where the constraint is API spend rather than hours in the day. Watch for a new candidate.

Ballista wants to ship without waiting

Andy also proposed releasing Ballista's Python client separately from the Rust crates. Ballista moved to DataFusion 55 five weeks ago and still cannot release, because the bundled Python client needs datafusion-python 55.0.0, whose vote has not started. Over the last eleven DataFusion releases, datafusion-python followed DataFusion by a median of 31 days and once by more than seven weeks. More than 300 commits sit on Ballista's main branch, including fixes for wrong results from distributed NOT IN, executors being OOM-killed on heavy queries, and jobs hanging when all executors disappear. Andy wants feedback on issue #2511, especially from Python client users.

Memory accounting

Andrew Lamb's sync notes from September 23 centered on memory. Raz from Flarion reported a string of memory issues where the memory pool's accounting did not match what operators actually used. One example was a Filter allocation that the pool never saw. Comet has hit the same gap between pool accounting and process memory. An earlier allocator-based memory test was reverted, and Andy agreed to find or create a ticket to coordinate the effort. For anyone running DataFusion inside a service with a hard memory limit, accurate pool accounting is the difference between graceful spilling and an OOM kill.

Apache Ossie (incubating)

A standard REST API for semantic models

Jean-Baptiste Onofré proposed a standard REST API for producing, consuming and orchestrating Ossie models, starting as a rest/openapi.yaml in the main repository. The response showed real vendor pull. Sebastien Gandon said Qlik has started building an Ossie-native API to import and export its data product semantic models, and he asked for a dedicated Content-Type header. Yufei Gu asked whether it builds on the Polaris Ossie REST design, which JB co-authored. JB said yes, and that the goal is for Polaris, Qlik and others to implement one standard API so clients work without knowing the server.

Viktor Kessler asked the group to agree on the problem first. He saw two scopes: a registry for exchanging and discovering models, and governed semantics, where a service guarantees that a model is bound to the physical tables it references. JB agreed and proposed two API specs, one for model management and one for catalog association. Vinoth Chandar asked that files and storage locations be first-class alongside table formats. Jakub Moravec asked how an importer knows a model changed. JB proposed keeping the core spec plain REST, with explicit versioning, content hashes and HTTP conditional requests using ETags, instead of push protocols.

This is the most interesting cross-project thread of the week. Iceberg standardized how engines find tables. Polaris implemented it. Ossie is now trying to standardize how tools find and exchange the business meaning that sits on top of those tables, and Polaris 1.8.0 already added the privileges to govern semantic models.

Converters move out, and a first release takes shape

The converter scope discussion reached a decision. Russell Spitzer favored separating the spec from implementations, the way Parquet splits parquet-format from its libraries. Markus Weimer cared less about repository count and more about keeping everything inside the ASF, since that makes adoption easier for downstream users. Kurt Stirewalt laid out what stays in the main repo: the spec, the typed object model with parsing and serialization, the validator, and the parser for the ontology formula language. JB then created the ossie-converters repository on September 29 and asked contributors to pause converter work while he moves the code.

JB also laid out the path to the first incubating releases: freeze converters, move them, and bump the spec version to 0.3.0-incubating after the recent flat-document change. Kyoung Min Kim did the arithmetic. The 0.2.0.dev0 string appears in 111 files, 96 of them under converters, so the bump is one change if it lands before the move and a coordinated two-repo change after. Kyoung Min also confirmed that the license header check now runs on every PR and the tree is clean.

What does `orders.amount` mean?

damian waldron asked a question the spec does not settle. In a portable expression like SUM(orders.amount), is amount a field declared in the dataset, or a physical column in the warehouse? He surveyed ten converters and found three different answers. JB said expressions and relationships must reference logical fields, because a semantic layer that reaches past its own fields to physical columns breaks encapsulation. Julian Hyde objected to any reading that locks layers into rigid tiers, since that makes Ossie less composable. Justin Talbot framed it as the SQL view case: a view's query references its immediate inputs, and dataset fields should work the same way. JB clarified that he meant lexical scoping, not a restriction on composition, and the group converged.

Two spec proposals depend on that answer. Chris Eubank narrowed the dataset-scoped metrics proposal in PR #343 so it can reach a vote without settling every open semantic question. JB strongly backed Justin's Relational Query Interface in PR #354, which grounds Layer 2 in the Measures in SQL model so BI tools that generate their own SQL still get correct metrics across fan-out and chasm traps. Justin also posted a short overview of the four spec layers in PR #452 for newcomers.

Votes, validation and structure

Chris Eubank called a vote to add the SQL-standard FILTER (WHERE ...) aggregate modifier to the expression language, after incorporating JB's feedback. The change is additive and avoids the edge cases of CASE-based workarounds for AVG, MIN, MAX and STDDEV. JB voted +1.

Kyoung Min Kim noticed that validate.py stops at the JSON schema for ontology documents because every semantic check assumes a datasets key. He will add ontology checks for built-in concepts, cycle detection on extends, and a CI step, with JB reviewing. Shao Xie's PR #458 splits ontology, semantic model and mapping into separate documents connected by references, so a slow-moving ontology can map to many semantic models. Yufei asked for more voices on deprecating embedded models. On GitHub, MikeCarlo proposed storing custom extension data as structured objects instead of opaque JSON strings, which helps the Power BI TMDL converter keep more metadata. Contributors also opened discussions on metric additivity and on recording who certified a metric definition.

An outsider builds Ossie on ClickHouse

The most useful message on the Ossie list came from a newcomer. Aleksandr Kosachev introduced himself as a backend architect who, six months ago, barely knew what a semantic layer was. He kept seeing teams wire AI assistants to databases and get confident, wrong answers. So he built ossie-clickhouse from the spec alone. It loads an Ossie document with the apache-ossie package, translates expressions to ClickHouse SQL with SQLGlot, plans one SELECT per question, and serves the model to agents over MCP. The TPC-DS example model runs end to end.

His list of places where he had to guess is a free conformance review. Relationship cardinality is not in the spec, so he only joins when the target columns form a primary or unique key. ClickHouse tables that keep change history need FINAL on read, which he exposed through a custom extension. Function semantics are open: whether REGEXP_REPLACE replaces the first match or all, whether DAYOFWEEK is ISO, and what 1/0 returns. He pinned his choices in tests until the compliance suite lands. Two of those gaps already have active threads this week. That is how a spec should improve.

Cross-Project Themes

Tighter contracts, fewer surprises. Iceberg removed a write path from v4. Parquet is leaning toward a vector type that rejects values most vector stores reject anyway. Ossie is defining whether a name means a field or a column. Polaris is arguing about whether a config field should be an enum. The shared direction is specs that make fewer promises and make them precisely, so a reader can trust data without re-validating it. The pushback is also consistent. Antoine Pitrou in Parquet, Daniel Weeks on the delegation header and Dmitri Bourlatchkov on R2 each defended flexibility for implementers. The best outcomes this week kept both. Polaris keeps its fail-fast behavior as an implementation choice while the Iceberg spec stays loose. Parquet looks headed toward a strict embedding type plus a separate general array track.

Agents are now participants in the process. Kurtis Wright found the Iceberg license gap with Claude. Kevin Liu pointed an agent at the repository to hunt correctness bugs. Andy Grove's Comet verification is gated by token budget. Gaspard Merten wrote his PyIceberg change with Claude. The communities are also setting norms. Contributors label AI-assisted votes, and Antoine asked a Parquet author to rewrite AI-sounding prose in his own words. Agents also showed up as consumers. Andrei Tserakhau argued for SQL-visible Iceberg labels because agents explore warehouses through SQL, and Aleksandr Kosachev built an Ossie server specifically to stop agents from giving wrong answers.

The semantic layer is joining the catalog layer. Polaris 1.8.0 added privileges for semantic models. Ossie proposed a REST API modeled on the Polaris design and split out a catalog association spec. Iceberg debated catalog-owned labels. The lakehouse stack now has open standards for files, tables and catalogs, and the next layer up, what the numbers mean, is getting the same treatment on the same lists with many of the same people.

Release engineering is a security surface. Iceberg RC1 failed on bundled license gaps. Polaris rc1 failed on a license header. Confluent is scanning the Iceberg Kafka connector. Polaris shipped a CVE fix for server-side FileIO endpoints. The Iceberg test backend moved from MinIO to RustFS after CI breakage. None of this is glamorous. All of it is the work that lets enterprises run these projects in production.

Looking Ahead

Watch for the Iceberg 1.12.0 announcement and the equality delete spec merge, then the October 8 sync on File and Vector types. Polaris plans to merge the table metrics PR on October 1, and the R2 enum question needs a resolution before that PR lands. Parquet's vector type should get a naming decision and a path to a vote. DataFusion Comet needs a new release candidate once Andy Grove's regression list is triaged. Ossie will move converters into their new repository and prepare its first incubating release at spec version 0.3.0. And if you are in Europe, Lakehouse Day EU in Glasgow on October 10 is the next place to see many of these contributors in one room.

Keep Learning

If this digest helps you follow the open lakehouse, my books go deeper on every layer covered here, from Apache Iceberg table design and Apache Polaris catalogs to Arrow, Parquet and building agentic analytics on open data. Find the full catalog at books.alexmerced.com.