DEV Community: Taha Echakiri

QuePaxa Lets a Global Control Plane Keep Deciding While Links Stall

Taha Echakiri — Sat, 14 Nov 2026 19:00:02 +0000

TL;DR

QuePaxa is a consensus algorithm published in 2023 by Tennage, Băsescu and co-authors that lets a group of machines agree on an ordered log while any proposer can move the protocol forward.
Cloudflare says it is building Meerkat, an experimental consensus service for its internal global control plane, on QuePaxa, in a post published on 11 November 2026.
Leader-based algorithms such as Raft, Multi-Paxos and Viewstamped Replication recover by detecting a failed leader and electing a replacement, and that recovery path is where a wide-area network hurts most.
QuePaxa keeps proposers cooperating: priorities are random, a proposer adopts any higher-priority proposal it meets, and clients hedge by sending a request to every replica.
A round has four phases (propose and adopt, spread, check and decide, carry forward), and a proposer that misses the one-round-trip fast path carries its work into the next round.
In Cloudflare's own Meerkat tests the decision rate fell to about 65% of its usual level while a failed leader was being replaced and returned to 100% a minute later; Cloudflare describes QuePaxa as less battle-tested than Raft and is formally verifying its implementation, so the open question is the evidence a buyer can inspect.

The Cloudflare Research post at blog.cloudflare.com/choosing-quepaxa, retrieved 2026-11-12, which introduces Meerkat and explains why its authors picked a 2023 algorithm over Raft, Multi-Paxos or Viewstamped Replication. Image: Cloudflare.

Every control plane eventually meets the same question: what happens to the shared record when the network between its replicas turns bad? Cloudflare answered it in public on 11 November 2026, when engineers João Pedro Leite, Bob Halley and James Larisch explained why the company's experimental consensus service, Meerkat, runs on QuePaxa, a protocol published in 2023.

The post is unusually specific for a vendor blog. It names the alternative algorithms, explains the failure mode that pushed the company away from them, walks through all four phases of a QuePaxa round, and reports what happened to Meerkat's decision rate when a leader was taken out of service. That combination makes it useful reading for anyone running a self-hosted control plane, a database with a replication layer, or a queue that has to keep ordering events across regions.

What QuePaxa is and where Cloudflare runs it

QuePaxa is a consensus algorithm, which means it lets a set of replicas agree on one ordered sequence of values even when some of them crash or lose contact. Meerkat uses that sequence as a log: each entry is a get or a put, so every replica that applies the same log arrives at the same key-value state. Cloudflare compares the shape to Consul, etcd and Apache ZooKeeper, and describes Meerkat as an experimental service still in development.

Cloudflare's diagram of destructive interference in Paxos: a second proposer slips a higher ballot number between the prepare and accept phases, so the first proposal is blocked and no value is decided. Image: Cloudflare, blog.cloudflare.com/choosing-quepaxa, retrieved 2026-11-12.

The difference from those systems sits in the deployment geography. A Meerkat cluster can span continents, which puts links with high and variable latency inside the protocol's working assumptions. Cloudflare writes that consensus systems usually run in smaller, more local networks with strict latency requirements, and that Meerkat was designed for global scale from the start.

Why leader-based consensus stalls on a global control plane

Paxos, the ancestor of most of these protocols, has proposers and acceptors and two phases: prepare and accept. Any proposer may start a ballot, and a second proposer can interleave a higher ballot between the two phases, which blocks the first one. When two proposers repeat that against each other, Cloudflare calls the result destructive interference, and the protocol stops deciding even though every machine is healthy.

Cloudflare's diagram of the leader fast path: one proposer holds the maximum priority, its proposal reaches a majority of acceptors first, and the value is decided in a single round trip while the leader stays healthy. Image: Cloudflare, blog.cloudflare.com/choosing-quepaxa, retrieved 2026-11-12.

Raft, Multi-Paxos and Viewstamped Replication answer that with a leader that holds the only right to propose. While the leader is reachable, a decision costs a single network round trip. When the leader fails, slows down or loses its connection, the remaining replicas have to notice and elect a replacement, which is where the design pays its price.

Cloudflare gives two reasons the price is high on a wide-area network. The first is that detection depends on expectations: a replica can only observe that an expected message has not arrived, so any practical detector guesses, and a wrong guess starts an election that halts decisions. Raft is commonly deployed with an election timeout five to ten times the network round trip, so detection on an intercontinental link is slow by construction.

The second is that elections inherit the interference problem. Several candidates can become candidates at once and block each other from a majority, and the pattern can outlast the fault that started it. Cloudflare calls that a metastable failure: a two-second outage, or one misbehaving switch, can trigger elections that keep running after the network is healthy again.

Netics editorial diagram of the two recovery models: on top, a leader-based protocol that detects and replaces a failed leader with election timeouts tied to network latency; below, QuePaxa's hedge timeout, which any replica can set locally because it decides nothing about failure.

How QuePaxa keeps proposers cooperating

QuePaxa keeps the two Paxos roles and renames the second one. Proposers submit values; recorders act as intermediaries and record every proposal they see. Each proposal carries a random priority, and a proposer that meets a higher-priority proposal adopts it and starts broadcasting it in turn.

That single rule changes the incentive inside the protocol. In QuePaxa, a proposer adopts the highest priority it can find because that is the value most likely to be decided this round. Cloudflare describes the outcome as constructive interference, and notes that the protocol stays randomised on purpose: priorities are drawn at random to get around the FLP impossibility result, which rules out a deterministic protocol that always decides in an asynchronous network.

One proposer holds the maximum priority and is the only one able to decide in a single round trip. Clients do not have to send requests to it. In the paper, clients hedge by sending to every replica, and the non-leaders wait a short timeout so the leader can finish first. Cloudflare implements this timeout at the client: clients wait for it, which reduces the number of messages sent over the network, and a short timeout only creates more active proposers.

Netics editorial definition card: a proposer submits a value and adopts any higher-priority proposal it meets, a recorder keeps every proposal it sees, and the random priority is what makes proposers converge inside a round rather than compete across rounds.

What the four phases do in each round

A round runs through four phases, and each phase advances when a proposer hears back from a majority of recorders. In phase 0, propose and adopt, proposers draw a priority and send their proposal, then adopt the highest priority they see from a majority. If, in the first round, the leader learns its proposal reached a majority first, it decides immediately.

Phase 1 spreads the adopted proposal to a majority of recorders, which keep the highest priority they observe. Phase 2 asks a majority what the highest priority was in the previous phase: a proposer whose own proposal still wins decides, and one that sees something higher does not. Phase 3 carries the best proposal a proposer can find into the next round, so a failed round keeps its progress.

Safety rests on majority intersection, the property that any two majorities of recorders share at least one member. Cloudflare's argument for the first round is compact: the leader reached a majority in phase 0, so any other proposer querying a majority in phase 2 meets that proposal and adopts it, and because the leader's priority is the maximum, no lower value can be decided.

Cloudflare's diagram of majority intersection: any two majorities of recorders overlap in at least one recorder, which is why a proposal that reached a majority in one phase cannot be overridden by a lower priority in the next. Image: Cloudflare, blog.cloudflare.com/choosing-quepaxa, retrieved 2026-11-12.

What Cloudflare measured in Meerkat

Cloudflare reports results from Meerkat clusters with servers on nearly every continent. The claim it makes first is availability: Meerkat kept processing requests while a leader crashed, disconnected or slowed down, and the company says it has never seen the system stall completely.

The post includes the measurement behind that claim. When a leader failed, QuePaxa fell back to decisions in phase 2 and the decision rate dropped to around 65% of its original level, then returned to 100% when the faulty leader was replaced, a minute after the failure. Cloudflare also reports that upwards of 95% of decisions are taken by the leader in the first phase, which keeps throughput stable, and that the slower path costs at most a third of the usual rate because it needs three round trips.

Cloudflare's chart of Meerkat's decisions during a leader failure: the decision rate drops to about 65% of its usual level while decisions move to the second phase, returns to 100% once the leader is replaced a minute later, and never reaches zero. Image: Cloudflare, blog.cloudflare.com/choosing-quepaxa, retrieved 2026-11-12.

Those figures come from one operator's implementation, and the same post lists what makes them worth reading carefully. The comparison between in-cluster tests and a hostile network is still missing: Cloudflare promises later posts on its Rust implementation, its fault-injection testing and benchmarks, and says nothing yet about the throughput cost of hedging when many proposers are active at once.

What the choice leaves open

Cloudflare states the limits of its own decision. QuePaxa "has not been battle-tested as extensively as Raft or Paxos", and the company is extending the original crash-stop design to handle crash recovery, which it calls work that demands stronger evidence of correctness than usual systems. It is formally verifying the implementation and running simulations alongside, and none of that evidence has been published yet.

Netics editorial metric tiles on the numbers Cloudflare reports: one round trip on the fast path, four phases when it is blocked, 95% of decisions taken by the leader in phase 0 and 65% of the usual rate while a failed leader is replaced.

For a team that operates a replicated service, the useful part of the post is the diagnostic pattern. Ask which failure the protocol recovers from, and how long recovery takes once a link between replicas degrades. An election timeout expressed as a multiple of the round trip is an admission that latency enters the recovery path. A client-side hedge timeout, as Cloudflare implements it, is a different arrangement, and it moves the cost of a failure from unavailable time to extra messages.

The same reasoning applies to the components a small company runs today: etcd behind Kubernetes, a replicated Postgres cluster, a self-hosted queue. None of them is about to switch protocol, and the question worth asking is narrower: what does the operator notice when the link between replicas degrades, and which metric carries that signal? Designing alerts that explain an incident applies the same diagnostic discipline after the fact.

Originally published on the Netics blog.

Qdrant 1.20 Stores Vectors on a Quarter of the Bits, With Measured Recall

Taha Echakiri — Sat, 14 Nov 2026 14:00:26 +0000

TL;DR

Qdrant is an open-source vector database that teams run to keep embeddings next to their metadata and search them by similarity, and version 1.20 adds two storage datatypes and a faster way to filter nested metadata.
Qdrant published the measurements on 2 November 2026: turbo8 keeps a vector on 8 bits per dimension and turbo16 on 16, so the same collection takes a quarter or half of the float32 space.
TurboQuant rotates the vector before compression, which is why the datatypes keep information that a plain float16 or uint8 cast throws away, and why the stored vector carries enough signal to search without the original.
The query keeps float32 precision while the stored vector stays compressed, and that asymmetric scoring holds server-side latency within 10% of a float32 collection.
Scalar and binary quantization keep a full-precision copy and rescore candidates against it. The TurboQuant datatypes discard the originals, so a query never waits on a disk read to recover recall.
The second feature indexes array elements one at a time, so a nested filter for one author with a given first and last name resolves through index lookups alone and reads no payloads.
On 1 million arXiv papers, that per-element index served up to four times as many queries per second as a regular nested index with the same results. The flat filter answered quickly and missed between a fifth and a third of the right papers.
Both features are configured per collection, and Qdrant reports their measurements in separate benchmarks: quantization across 14 embedding datasets and nested filtering over 1 million arXiv papers.
The practical reading: storage is the recurring bill in a vector deployment, and the datatype decision plus one payload index setting is where that bill is set.

According to Qdrant's v1.20.0 release notes, Qdrant 1.20 combines TurboQuant datatypes and per-element payload indexes with BM25 scoring directly on the full-text payload index, a code tokenizer for that index, configurable audit-log output to a file, stdout or stderr, and full-scan substring filters. The 2 November write-up does something rare in vendor material: it publishes recall against latency, states the dataset count and dimensions, names the two benchmark repositories, and states where the compression stops paying.

Netics visual, 10 November 2026: the four rows where the datatype decision actually lands, drawn from the figures in Qdrant's own write-up.

Vector storage is the line item quantization attacks

A vector collection costs disk while it is cold and memory while it is warm, and the write-up says so directly. Each dimension of a float32 embedding uses 32 bits. Quantization reduces the bits per dimension, which reduces the node count a team needs or the amount of data that fits on the nodes it already runs.

The interesting part is the choice of algorithm. Casting float32 into float16 or uint8 loses precision and loses information, and the post is blunt that the compression there relies on simple transformations. TurboQuant rotates the vector first and stores the result on an 8-bit or 16-bit integer grid, so the compressed representation retains the structure that similarity search needs. The older turbo4 datatype from version 1.19 saves even more, at eight times less storage, and gives up a few percentage points of recall on some datasets.

Qdrant's measured recall against median server latency, at a limit of 100 results on the three 1,536-dimension datasets; the compression ladder runs from float32 at the left to binary at the right. Figure from https://qdrant.tech/blog/qdrant-1.20-performance/.

turbo8 and turbo16 keep the query in float32

The obvious objection to a compressed vector is latency: the engine has to unpack it before it can compare. Qdrant avoids most of that cost by leaving the incoming query alone. The stored vectors are quantized, the query is rotated and kept in float32, and the comparison runs between a compressed stored vector and a full-precision query. The post calls it asymmetric scoring, and the measured outcome is a datatype that stays within 10% of the server-side latency of float32 vectors.

That number deserves the adjective narrower than it usually gets. It is server-side latency on three datasets of the same dimension, which is a real measurement and a limited one. Teams running 768-dimension embeddings from a different model family should read the direction of the result, then measure their own recall at their own candidate limit, because the recall curve depends on the data.

Rescoring disappears when the stored vector is the vector

Scalar and binary quantization also reduce vector storage. They shrink the vectors that Qdrant searches, keep the full-precision copies as well, and rescore the best candidates against those originals to recover recall. When the originals sit on disk, which is the normal case when memory is the constraint, every query waits for disk reads. A team can skip the rescoring step and lose recall, and the post names that trade honestly.

The TurboQuant datatypes remove the step altogether. The compact vector is the vector, the float32 originals are discarded, and nothing waits on the disk. For an operator, that changes the capacity model: memory holds the searchable representation and there is no second copy to size for. It also changes the recovery model, since there is no original to fall back to if a rebuild goes wrong, and a collection becomes a thing you re-embed from the source documents, because its own storage holds no pristine copy any more.

Qdrant's figure of the three nested-filter modes: the regular index finds candidates and then reads payloads, the per-element index resolves the same condition with index lookups. Figure from https://qdrant.tech/blog/qdrant-1.20-performance/.

Filtering nested metadata costs a payload read per candidate

The second feature addresses nested metadata filtering, and it may matter more in daily use. Payloads in a vector database are often arrays of objects: the authors of a paper, the tags on a product, the line items of an invoice. A nested filter asks for one element that meets every condition, such as an author whose first name is Ada and whose last name is Hopper.

Before this release Qdrant answered that in two steps. An index lookup found the points that matched each condition, which allowed the first name to match one author and the last name to match another, and then the engine read the payloads of those candidates to keep only the points where a single element satisfied both. The new per-element index gives every array element its own index entry, so the second step disappears and the filter resolves through index lookups.

Netics visual, 10 November 2026: the two filter plans the post compares, where the payload read is the cost that the per-element index removes.

The configuration is one parameter on an existing payload index, and it belongs to the index, so the queries running against that field keep their shape. That is the part worth internalising for anyone who owns the schema. Creating the index with nested set to the array key speeds up the nested filters already running against that field, and no search request changes. The benchmark that supports it uses 1 million arXiv papers, and its comparison of a flat filter is the honest part of the write-up: flat filters are usually the fastest mode, and on arrays of objects they return many false positives, missing between a fifth and a third of the right papers.

What the benchmark proves about a collection larger than memory

The benchmark setup is specified well enough to be reproducible, which is the standard that separates a measurement from a marketing claim. HNSW is the index, exact search supplies the ground truth, the run covers 14 embedding datasets between 512 and 3,072 dimensions, the figures shown use the three datasets at 1,536 dimensions, the result limit is 100, and scalar and binary quantization run with 2x oversampling. Both benchmark repositories are named in the post.

That setup answers a specific question: how much recall a datatype loses at a fixed search configuration. A platform team must next measure how the choice behaves against its own embeddings and filter mix. The useful move is to take the datatype as a parameter of the collection design and run the team's own evaluation set through it, because the recall gap between float32 and an 8-bit grid is a property of the data as much as of the codec.

Netics visual, 10 November 2026: the four figures that carry the release, from the query rate on the filter benchmark to the dataset count behind the recall curve.

Choosing a datatype per collection

Qdrant sets the datatype when a collection is created, through the vectors configuration, and a team uploading ordinary float32 vectors afterwards gets them rotated and stored on the integer grid. Changing the choice on an existing collection means building a collection with the new setting and moving the data into it, so the decision is a migration, and a migration means re-embedding or re-uploading a corpus whose size is the reason the team is reading the post.

A workable rule for a first pass: start at turbo16 for a corpus that must hold recall, drop to turbo8 when memory is the binding constraint, and treat turbo4 as the option for archives where a few points of recall are an acceptable price. Then measure. The post's own figures put turbo8 and turbo16 at recall comparable to float32 on the datasets it used, and a 10% latency band that is small enough to disappear inside the rest of a request's budget.

For teams building a retrieval layer into a product, the deeper signal is that this class of release is about the cost floor. Chunking strategy, embedding model and index parameters decide the monthly bill long before the application code does, and the same choices show up in the retrieval layer behind a private AI assistant for your documents, where a corpus of scanned contracts or support tickets has to stay searchable on hardware the team controls.

The published post on 10 November 2026, where the datatype section, the recall figure and the nested-filter section sit in that order. Page from https://qdrant.tech/blog/qdrant-1.20-performance/.

Originally published on the Netics blog.

PostgreSQL 19 Ships on 19 November and PostgreSQL 14 Leaves Support

Taha Echakiri — Sat, 14 Nov 2026 14:00:02 +0000

TL;DR

PostgreSQL is an open-source relational database that teams install on their own servers, and the project supports each major version for five years before a final minor release ends the branch.
The PostgreSQL project's RC 1 announcement, published on 15 October, schedules PostgreSQL 19 general availability for 19 November 2026.
A second candidate, PostgreSQL 19 RC 2, arrives on 12 November, the same day as the quarterly update release for supported versions.
PostgreSQL 14 leaves support on that same 12 November with version 14.24, so a cluster on 14 stops receiving fixes that day.
PostgreSQL 19 brings REPACK, a single command that replaces VACUUM FULL and CLUSTER, and REPACK (CONCURRENTLY) rebuilds a table without an access-exclusive lock.
The same release adds pg_plan_advice and pg_stash_advice for planner decisions, logical replication of sequence values, a start for logical replication without a restart, and a WAIT command that lets a standby wait on a log position.
Defaults change under an upgraded cluster: JIT ships disabled, max_locks_per_transaction moves to 128, standard_conforming_strings is always on, RADIUS authentication is gone, and inet and cidr indexes move to GiST.
The path off version 14 runs through pg_upgrade or a dump and restore, and pg_upgrade refuses clusters that carry btree_gist indexes on inet or cidr.
The planning point: the version a team runs is a support date, and the date sits on the project's published calendar.

This release-notes briefing draws its dates from two PostgreSQL project documents and focuses on clearing PostgreSQL 19's documented incompatibilities before version 14 support ends. The 19 release notes carry an overview that names each headline feature, and the 19 RC 1 announcement sets the schedule plainly: RC 2 on 12 November 2026, general availability on 19 November 2026.

Netics visual, 11 November 2026: what a minor release is and what installing one actually involves.

The project supports each major version for five years

The versioning policy gives a major version five years of bug and security fixes counted from its first release, and the last of those arrives as a final minor release. After that the branch sits unsupported. Minor releases land at least once every three months, targeted at the second Thursday of February, May, August and November.

That calendar is the practical part. A team planning a migration can read the final-release column and see the wall before it arrives, which is a rare luxury in infrastructure planning.

The project's own support table, read on 11 November 2026: version 14 lists 12 November 2026 in the final-release column, version 18 lists November 2030. Page from https://www.postgresql.org/support/versioning/.

PostgreSQL 14 leaves support on 12 November

Version 14 first shipped on 30 September 2021, and the support table lists 14.24 as its current minor release with a final release date of 12 November 2026. After that final minor release, version 14 becomes unsupported.

The release notes provide an inventory of incompatibilities. An operational decision also needs an inventory of the extensions and drivers used by the application. Rehearse the chosen migration route on a production-like copy and run the critical queries. Verify backups, the rollback procedure and the maintenance-window duration before approving the migration.

The announcement as published on 15 October 2026, naming RC 2 on 12 November and general availability on 19 November. Page from https://www.postgresql.org/about/news/postgresql-19-rc-1-released-3409/.

PostgreSQL 19 arrives on 19 November with REPACK

The release notes describe REPACK as the command that replaces VACUUM FULL and CLUSTER. The old commands remain for compatibility. REPACK (CONCURRENTLY) adds the version most operators want: rebuilding a table without taking an access-exclusive lock on it.

Planner control arrives as two extensions. pg_plan_advice stabilizes and controls planner decisions, and pg_stash_advice applies stored advice automatically based on a query. Autovacuum also gains parallel index processing and a scoring system that orders which tables it should reach first, on top of I/O worker processes that scale through io_min_workers and io_max_workers.

The overview section of the release notes, where the feature list and the migration requirements sit at the top of the page. Page from https://www.postgresql.org/docs/release/19.0/.

Replication gets the two changes that matter most on a busy cluster. Logical replication can be switched on with wal_level set to replica without restarting the server, and sequence values now replicate. A WAIT command lets a standby wait until a log position has been written, flushed or replayed.

Defaults change under an upgraded cluster

Five defaults shift under the cluster a team restores into 19:

JIT is disabled by default, because the optimizer cost that used to trigger it proved unreliable. Sites running large analytical queries need to enable it deliberately.
max_locks_per_transaction moves from 64 to 128, and the release notes say the lock size allocation changed so settings must double to hold their previous capacity.
standard_conforming_strings is forced on in the server. A dump taken with a pre-19 pg_dump while the setting was off will not load into 19.
RADIUS authentication is removed, on the grounds that support existed only over UDP and that transport is unfixably insecure.
inet and cidr indexes default to GiST, and pg_upgrade refuses clusters carrying btree_gist opclasses on those types because the old opclasses could exclude rows that belong in the result.

Netics visual, 11 November 2026: the setting-level differences between a 14 cluster and a 19 cluster, drawn from the release notes.

pg_upgrade decides the path off version 14

The release notes accept three migration routes: a dump and restore with pg_dumpall, pg_upgrade, or logical replication. For planning purposes, treat pg_upgrade as a route that needs an explicit preflight check.

Two pg_upgrade refusal conditions are worth checking before a maintenance window: clusters with btree_gist indexes on inet or cidr, and clusters that use carriage returns or line feeds in database, role or tablespace names. Separately, dumps produced by pre-19 tools while standard_conforming_strings was off will not properly load into PostgreSQL 19. Each is a configuration fact that can be found in advance, and each one turns a migration into a debugging session when it surfaces mid-run.

There is also a spare SQL. MULE_INTERNAL encoding is gone, so a database that uses it needs a dump and restore into a different encoding.

The mark the PostgreSQL project publishes for press use. Image from https://www.postgresql.org/media/img/about/press/elephant.png.

Application compatibility deserves priority during upgrade planning, including dependencies introduced through custom web and mobile development. The five-year calendar supplies a deadline; migration safety depends on testing the chosen route and clearing PostgreSQL 19's documented incompatibilities and pg_upgrade refusals before the maintenance window.

Originally published on the Netics blog.

Jensen Huang Resigns: Nvidia Faces a Leadership Test

Taha Echakiri — Sat, 14 Nov 2026 09:00:03 +0000

Jensen Huang has resigned as Nvidia’s president, chief executive and board member, effective immediately, the company announced on 12 November 2026. Chief financial officer Colette Kress will serve as interim CEO while keeping the finance role; the board says it will consider internal and external candidates for a permanent successor. Harvey C. Jones, a longtime director, becomes the board’s first chair. Nvidia’s announcement gives no reason for Huang’s departure.

The resignation followed days of intense reporting about Huang’s admitted extramarital affair and an unproven allegation that a former business executive linked to Nvidia customer Megaspeed was collecting information for China. The record so far: Huang acknowledged infidelity; the identity and espionage allegations came from reporting; and the espionage claim remains an unproven allegation. Singapore says its own investigation concerned purchase declarations. Here is what is confirmed, what is reported and what remains open.

The timeline separates events Nvidia and public reporting have dated; the 17 November results date was already scheduled. Graphic: Netics, based on Nvidia’s release and AP/Reuters reporting.

TL;DR

Nvidia is a technology company focused on AI and accelerated computing. It confirmed Huang’s immediate resignation on 12 November; Colette Kress is interim CEO and CFO, and the board gave no reason for his departure.
Huang acknowledged an affair. Reporting identified Alice Huang and alleged an intelligence connection; AP and Reuters could not independently verify it, and China denies the allegation. [4][5][6]
Singapore says its investigation concerned purchase declarations; it found insufficient evidence of the suspected offences by Alice Huang, issued a stern warning and barred her from re-entering, while its Megaspeed inquiry continues.
What happens next will depend on three tracks: Nvidia’s board review, Singapore’s continuing Megaspeed inquiry and U.S. questions, with the 17 November results as the next scheduled milestone. For investors and AI customers, the points to watch are interim decision rights, the board review findings and auditable export controls.

The transition Nvidia announced

Nvidia’s 12 November release and its SEC filing establish the leadership change: Huang resigned as president, CEO and a director; Kress is interim CEO and continues as CFO; and the board has begun a search open to internal and external candidates. The board also appointed Jones, a longtime director, as its first chair. The company says it expects operations, customer commitments and the product roadmap to continue without interruption; this wording records the board’s stated expectation.

Reuters gives 7 June 2024 as the image’s release date; the capture date remains unconfirmed. It predates the November 2026 announcement and does not establish an intelligence allegation. Credit: Obtained by Reuters / Reuters Connect; used under the license confirmed by TH. [8]

Both documents are silent on the reason for Huang’s resignation. The cause remains publicly unstated, whether the Megaspeed matter, personal conduct, the board review or another factor. Keeping that distinction visible is essential in a breaking story.

What Huang admitted—and what reports allege

On 8 November, Huang apologized to his family and colleagues in a message to Nvidia employees, saying he had been unfaithful. He did not name the other person. The Associated Press reports that The New York Times identified the woman as Alice Huang, also known as Huang Le, a former executive at Singapore-based Megaspeed International. The two are not related. [4][5]

The espionage accusation concerning Alice Huang, a Hong Kong resident and former Megaspeed executive, is separate from the affair Huang acknowledged. Reuters says it could not independently verify the intelligence allegation and that China denies it. Singapore’s MHA identifies Huang as Megaspeed’s then-CEO, chair and one of its key controllers during its inquiry; by its own account, that inquiry covered purchase declarations. [6][7]

An allegation reported from intelligence assessments is not the same as a public legal conclusion. Singapore’s statement describes a different investigation; Beijing denies the espionage claim. Graphic: Netics, based on AP, Reuters, the Singapore MHA and China’s Foreign Ministry.

Public records give a specific picture of her background. Reuters says it reviewed more than 30 public financial records, corporate filings and company materials. It reports that Huang held senior roles at a Shanghai investment firm before becoming Megaspeed’s controlling shareholder in March 2023. Reuters also says C4ADS could not determine who ultimately controls SwiftData, which later held Megaspeed shares, though the group assessed Huang may have remained influential after the transfer. Nvidia lists Megaspeed as an official cloud partner. Reuters could not reach Huang for comment. Her current ownership remains unclear from the records, and these business links document commercial ties only. [6]

Reporting also describes a separate relationship. The New York Times and Reuters reported that Huang had a separate relationship with an Nvidia employee; Reuters cited people familiar with the matter, and Nvidia declined to comment. The AP said it could not independently confirm those reports. This article leaves the employee unnamed and treats these accounts as reporting, pending any official finding.

Illustrative photo of server racks; it does not depict Megaspeed or a Singapore data centre. Credit: Tony Webster / Wikimedia Commons, CC BY 2.0.

Singapore’s server-declaration investigation

Singapore’s Ministry of Home Affairs says its police and customs authorities began an inquiry in February 2025 into possible false declarations connected with purchases of computer servers and related offences. The ministry describes its inquiry into Alice Huang as a declarations case. It says the authorities found insufficient evidence in February 2026 to establish that she committed the suspected offences related to those declarations.

The same statement says she received a stern warning after providing information that Singapore assessed as knowingly inaccurate and intended to downplay her role at Megaspeed; she was barred from re-entering Singapore. The MHA says its investigation into Megaspeed and six other people continues. These are the Singapore government’s stated actions and findings within its own inquiry. The separate U.S. intelligence allegation stays open on this evidence. [7]

Megaspeed said on 12 November that it had cooperated with U.S. Commerce Department export-control inspections and would work transparently with other relevant U.S. authorities. It said it had retained a U.S. law firm to conduct a comprehensive inventory of its chips and independently review its export-control practices. Those are the company’s commitments; the result of that inventory and any final U.S. finding are still to come.

NVIDIA sign outside its Santa Clara campus, photographed in 2020; it does not show the 12 November announcement or board meeting. Credit: Will Buckner / Wikimedia Commons, CC BY 2.0.

Nvidia’s board review and the questions still open

Nvidia’s independent directors said they were investigating reports that Huang had been the target of a foreign-intelligence effort, with outside counsel supporting the review. The board said it had no reason to believe national security or Nvidia’s secrets were compromised, while saying the review would examine available information. The board has stated this assessment while its investigation remains in progress.

U.S. lawmakers have continued to seek answers from Huang about his interactions with Alice Huang and Nvidia’s safeguards. AP reports that House Foreign Affairs Committee Chair Brian Mast said the committee was in contact with Huang’s team and that he would use a subpoena if necessary. The reporting describes a request and the possibility of a subpoena, with the matter still before any hearing or finding on wrongdoing.

For the board, the work now extends beyond naming a permanent CEO. It must explain how an independent review is governed, how any relevant conflicts are handled, what evidence supports its conclusions, and what oversight processes remain in place after a founder’s sudden departure. Findings from the review remain undisclosed.

Colette Kress in an NVIDIA-provided biography portrait. This is an official NVIDIA Newsroom profile image of Kress. Credit: NVIDIA Newsroom.

Colette Kress takes the interim role

Kress joined Nvidia in 2013 as executive vice president and CFO after senior finance roles at Cisco and Microsoft. Her interim appointment provides a named executive with deep company and finance experience while the board considers permanent candidates. The transition release says she will continue in both posts until a successor is named. It does not give a timetable or identify a preferred candidate.

What investors and AI customers should watch

Reuters reported that Nvidia shares closed 4.3% lower on 12 November, after falling more sharply during the session. That dated market reaction records one session and leaves long-term investor views and Nvidia’s business outlook unresolved. The company’s next scheduled milestone is its fiscal third-quarter results on 17 November.

For customers and partners, the practical questions are continuity and controls: who owns decisions during the interim period; how the board will report the results of its review; whether export-control and end-user checks are independently auditable; and whether customers can trace accelerators and servers through intermediaries to their declared destinations. These due-diligence questions arise from the case and leave the legal status of Nvidia and Megaspeed to the relevant authorities.

Archive photo: Jensen Huang at a Stanford CS 153 fireside chat on 30 April 2026; it does not depict the resignation or any event discussed here. Credit: Anders Eidesvik / Wikimedia Commons, CC BY-SA 4.0; reproduced without alteration.

What happens next?

Three separate tracks remain unresolved: Nvidia’s board-led review; Singapore’s continuing inquiry into Megaspeed and other people; and U.S. questions about the reported intelligence concerns and export-control safeguards. They have different scopes, evidence and decision-makers. No source cited here establishes that Alice Huang was a spy, that Nvidia secrets were compromised, or that Nvidia’s announcement gives the reason for Huang’s departure.

The next concrete signals will be the board’s permanent-CEO process, any substantiated findings disclosed from the separate reviews, the congressional follow-up and Nvidia’s 17 November earnings update. Until then, the most accurate account is also the clearest: Huang has left immediately; Kress is in charge for now; the company has not stated why; and the most serious allegations remain contested and unresolved.

FAQ

Why did Jensen Huang resign from Nvidia? Nvidia’s 12 November announcement confirms his resignation but does not give a reason. The timing followed reporting about his personal conduct and national-security allegations; the company has not publicly stated a causal explanation.

Reuters reported a 4.3% closing decline on 12 November. A single session provides limited market context. Graphic: Netics, based on Reuters.

Is Alice Huang confirmed to be a Chinese spy? No. The espionage claim has been reported as an assessment attributed to U.S. intelligence, but the AP said it could not independently verify it, China denies it, and CNN reported that the available intelligence did not establish the allegation conclusively.

What did Singapore investigate? Singapore’s MHA says its inquiry concerned possible false declarations about equipment purchases and related offences. It reported insufficient evidence to establish those suspected offences by Alice Huang, while also reporting a stern warning and a continuing investigation into Megaspeed and others.

Who is Nvidia’s interim CEO? Colette Kress, who remains Nvidia’s CFO while the board searches for a permanent CEO.

For related analysis, read Netics’ article on security boundaries in AI-agent systems. More Netics coverage is available at the Netics homepage.

Originally published on the Netics blog.

N-able Took Hosted N-central Offline After a 33-Issue Disclosure

Taha Echakiri — Fri, 13 Nov 2026 19:00:01 +0000

TL;DR

N-central is a remote monitoring and management platform that service providers and internal IT teams use to administer endpoints, push patches and run scripts from one console.
N-able took its hosted N-central environments offline on 9 November 2026 after an independent researcher published a disclosure named Able2Shell covering 33 issues, with proof-of-concept code.
The disclosed issues include unauthenticated remote code execution, privilege escalation to root, credential disclosure and unauthorized account creation.
One unauthenticated chain combines XML external entity handling with session handling to execute code as the nable service account, and other chains need an administrator to view malicious patch metadata or need ProductAdmin credentials.
N-able's Hotfix 2 release notes, published on 10 November, identify build 2026.4.1.26 and list seven CVEs fixed by the hotfix.
N-able reported scanning activity tied to the published code before the fix existed and said it found no evidence that a customer environment had been exploited.
N-able's Hotfix 2 release notes direct customers to upgrade Agents before Probes to preserve communication, and Rapid7 says an on-premises server stays offline until the hotfix is installed.
The customers behind the console inherit the exposure, because an RMM server holds administrative reach over every endpoint it manages.

Netics visual, 11 November 2026: the operating rule that came with the disclosure, and the three facts that justify it.

N-central is the console that reaches every managed endpoint

An RMM platform is the one system in a service provider's stack that already holds credentials and remote-control paths to every machine it manages. That is the product working as designed, and it is also the reason a flaw in the console is worth more to an attacker than a flaw on any single endpoint.

Rapid7 puts the consequence plainly: compromise of an RMM server can put downstream managed environments at risk. The same post notes that N-central was targeted earlier in 2026 through a separate authentication bypass, which tells an operator this platform has been on somebody's list for months.

The Rapid7 write-up, last updated 10 November 2026, with the disclosure summary and the mitigation guidance in one page. Page from https://www.rapid7.com/blog/post/etr-critical-vulnerabilities-in-n-able-n-central-prompt-hosted-shutdown/.

N-able's response followed the shape of a severe disclosure. Hosted N-central classic environments went offline, the vendor advised on-premises customers to keep deployments offline while it built and validated a fix, and no CVE identifiers or scores existed in the first notice.

Thirty-three issues arrived with public exploit code

The disclosure is credited to an independent researcher and carries 33 issues, with working proof-of-concept code published alongside. Rapid7 lists the classes: unauthenticated remote code execution, privilege escalation to root, credential disclosure and unauthorized account creation.

The record published for CVE-2026-125882: relative path traversal in the file repository upload, N-central before 26.4.1 affected, credited to Nate Fair of Sprocket Security. Page from https://www.cve.org/CVERecord?id=CVE-2026-125882.

Two chains deserve attention because they read like ordinary administrator behaviour. One needs an administrator to open malicious patch metadata, which means the console's own patch view becomes the delivery path. Another works with ProductAdmin credentials, a platform-management role with a narrower privilege set than a full administrator.

The scariest is the unauthenticated one: XML external entity handling combined with session handling, ending in code execution as the nable service account. That account is the service identity the platform runs under, so code running as nable reads the same configuration the console reads.

Hotfix 2 covers seven CVEs

N-able's Hotfix 2 release notes, published on 10 November, identify build 2026.4.1.26 and list seven CVEs fixed by the hotfix. Seven identifiers against 33 disclosed issues is the arithmetic that matters here: the fix list is a partial map.

The CVE-2026-125878 record, where the title says OGNL injection and the weakness entry reads cross-site scripting, both for N-central before 26.4.1. Page from https://www.cve.org/CVERecord?id=CVE-2026-125878.

Read the seven as a spread of attack surfaces. XML external entity flaws, OGNL injection, local privilege escalation to root through a world-writable syslog-ng configuration, stored cross-site scripting through patch metadata, server-side request forgery through service-import schema validation, and the path traversal in the file repository upload.

One record carries a detail worth flagging for anyone doing their own triage. CVE-2026-125878 has the title OGNL injection and a weakness entry for cross-site scripting, an inconsistency between its title and CWE field. Read the release notes text and the vendor advisory when the fields disagree, because a filter keyed on CWE-79 would treat that entry as a client-side problem.

Agents upgrade before Probes as servers return online

The upgrade order is the part that costs a night. N-able's Hotfix 2 release notes give the compatibility order: upgrade the Agents first, then the Probes, so that communication survives the transition. A provider who upgrades the server and stops there still runs old agents across every customer site.

Netics visual, 11 November 2026: four of the seven identifiers in the hotfix and the weakness each one covers.

N-able upgrades hosted servers, while customers in both hosted and on-premises environments must upgrade their Agents and Probes. On-premises customers must also upgrade their own servers within a maintenance window backed by a rollback plan.

The affected range needs care too. CVE records list N-central before 26.4.1, and Rapid7 notes that the researcher tested appliance build 2026.4.0.31 while the vendor notice did not state a full affected range. An environment on an older branch has to assume it is in scope.

The customers behind the console inherit the exposure

Every N-central deployment sits in front of somebody else's business. When the console is offline, the provider loses its remote hands on hundreds of machines at the moment it needs them, which is why the vendor-hosted shutdown is a real operational event for end customers who never heard of the vulnerability.

Netics visual, 11 November 2026: the four dates that describe the disclosure, the hotfix and what comes next.

Two conclusions follow for a provider. The first is that the RMM console belongs at the top of the patch order permanently, ahead of the servers it manages, because its compromise is the compromise of everything downstream. The second is that the same control depends on infrastructure monitoring with plain-language alerts.

Rapid7 recommends treating the disclosure as an emergency, upgrading on-premises servers outside normal cycles and leaving them offline until the upgrade lands. That guidance is the vendor's own assessment of how much of the disclosure is public. With seven identifiers fixed against 33 disclosed issues, the server, Agent and Probe upgrades belong in the same maintenance plan.

Originally published on the Netics blog.

Kubernetes v1.37 Hardens the Volume You Already Mount

Taha Echakiri — Fri, 13 Nov 2026 14:00:59 +0000

TL;DR

Kubernetes v1.37 ships two storage-security features: emptyDir permission modes and bind mount options (announcement).
bindMountOptions puts noexec, nosuid and nodev on the bind mount any volume creates inside a container; mode sets how an emptyDir is created instead of the hardcoded 0777.
Both are Alpha behind VolumeBindMountOptions and EmptyDirVolumeMode gates on the API server and kubelet.
Netics' take: the deepest container security gap was never the image — it was the writable volume; these flags close the path where a compromised process turns scratch space into an executable.
Practical move: enable the gates on one node, apply noexec/nosuid to emptyDir and /tmp mounts, and set 01777 on shared workspace volumes.

Reproduction of the official Kubernetes blog post "Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions" (September 16, 2026); source: kubernetes.io/blog

The most common writable volume had the least control

Here is the quiet fact behind the v1.37 release notes: emptyDir — the most common writable volume type in Kubernetes — created its directories with a hardcoded mode of 0777, and volumes were bind-mounted into containers with no noexec flag at all. Any process that could discover the volume could read, write, and delete everything in it, which for a ReadWriteMany-style shared scratch space is an open door between containers. Kubernetes had flags for this at the Linux level for decades: noexec (no direct execution of binaries), nosuid (no setuid/setgid bits), nodev (no device files). It just never gave users a native way to apply them to the mount the container runtime creates.

The security impact is sharper than it looks. A container with readOnlyRootFilesystem: true is hardened everywhere except its writable volumes — and if those volumes lack noexec, a compromised process can download a binary, chmod +x, and execute it from the very scratch space the security model left writable. External auditors flagged exactly this in the Kubernetes 1.24 Security Audit (Finding NCC-E003660-7HM), and the gap had its own long-lived issue (#48912). Both stayed unresolved until v1.37.

Original Netics diagram: the Linux VFS mount flags noexec, nosuid and nodev, and what they block on a container bind mount; source: Kubernetes v1.37 storage hardening post

Two fields, one layer, at the kubelet

KEP-5855 adds a bindMountOptions field on volumeMounts in the pod spec. When set, the kubelet passes the options through the CRI Mount message to the container runtime, which includes them in the OCI runtime spec mount options — the low-level runtime then applies the flags on the bind mount inside the container. Because the field sits on volumeMounts, it works for every volume type (emptyDir, PersistentVolumes, CSI, projected, ConfigMaps, Secrets — image volumes are the only exception) and it gives per-container granularity: two containers in the same pod can mount the same volume with different options.

KEP-5502 adds a mode field on emptyDir. Instead of a hardcoded 0777, you can now declare 01777 for a shared workspace that behaves like /tmp (sticky bit: each container can write, but only the file owner or root can delete), or 0750 to lock a database's scratch volume to its own group. The previous workaround — an init container running chmod — was brittle and hard to verify for compliance; the field makes the permission declarative and auditable.

Reproduction of the official Kubernetes enhancement proposal KEP-5855 (Allow bind mount options on volumeMounts), kubernetes/enhancements on GitHub; source: github.com/kubernetes/enhancements

Where the Alpha bumps and the silent fallback live

Both features arrive as Alpha in v1.37, behind VolumeBindMountOptions and EmptyDirVolumeMode, enabled on the API server and the kubelet. The version-skew behavior deserves attention because the two features fail differently. For emptyDir mode: if the API server has the gate enabled but the kubelet does not, the field is accepted but silently ignored — the kubelet falls back to 0777, the exact behavior you are trying to remove. For bindMountOptions: the scheduler uses Node Declared Features to avoid placing pods on nodes without runtime support, and if a pod reaches such a node anyway, the kubelet rejects it rather than silently ignoring the options. One degrades in silence; the other fails loudly. That asymmetry is the kind of thing you only discover in a staging rollout, and it is the reason the "things to know" list in the announcement deserves a full read.

Original Netics diagram: emptyDir before v1.37 (hardcoded 0777, no mount flags) and after (declared mode, noexec/nosuid/nodev); source: Kubernetes v1.37 storage hardening post

There is one more interaction to plan for: fsGroup. If fsGroup is set in the pod's security context, the group permissions applied by fsGroup override the mode you specify on the emptyDir — the same behavior that already exists for defaultMode on Secret and ConfigMap volumes. And everything here is Linux-only; Windows nodes skip both fields.

The adoption path that actually works

Start where the exposure is densest: emptyDir and /tmp mounts are the volumes every workload touches, and they are the ones attackers reach first with a payload. On one node, enable both gates, mount a scratch volume with bindMountOptions: [noexec, nosuid], and keep readOnlyRootFilesystem: true — that combination removes the download-and-execute path entirely. Second, for multi-container pods sharing a workspace (CI build containers, sidecar loggers), set mode: 01777 so a compromised container can no longer delete another container's build artifacts — the sticky bit does exactly what the shared-/tmp problem has needed for years. Third, for stateful workloads, move deliberately: PersistentVolume and CSI adoption of these flags will take longer because it shifts provisioning defaults, and the silent-0777 fallback on mixed old/new nodes means your admission logic cannot yet trust mode alone as an enforcement signal.

Original Netics diagram: the operational warnings around v1.37 storage hardening — Alpha gates, silent emptyDir fallback, fsGroup override, Linux-only; source: Kubernetes v1.37 storage hardening post

For teams operating Kubernetes estates, the practical question is whether these controls are enabled, observable, and safe to roll out across mixed-version nodes: ValidatingAdmissionPolicy governs what gets admitted; it cannot govern what a volume does after the pod is running. Mount flags and directory modes live at the layer admission never reaches, which is exactly why they matter more than the next API-server guardrail. For a practical pass over your volume policies, compare these controls against your node versions and rollout plan.

Originally published on the Netics blog.

NVIDIA Opens Its Scale-Up Fabric to Korean NPUs

Taha Echakiri — Fri, 13 Nov 2026 14:00:02 +0000

TL;DR

NVLink Fusion is an NVIDIA technology that lets a chipmaker attach its own processor to NVIDIA's high-bandwidth scale-up networking, so a custom accelerator works inside an NVIDIA-designed rack.
NVIDIA, Rebellions and SK Telecom announced a collaboration on 8 November 2026 in which Rebellions connects its next-generation R200 NPUs to that fabric and adopts the MGX rack architecture, and SK Telecom plans to deploy the systems in its AI data centre.
The R200 is an inference NPU aimed at low latency and energy efficiency on frontier models, including mixture-of-experts models with trillions of parameters, connected inside a rack through NVLink.
The companies plan to split inference across architectures, with NVIDIA GPUs handling prefill and Rebellions NPUs handling decode, so each stage runs on the silicon that suits it.
The MGX rack is where the arrangement becomes practical, because GPU and NPU systems share the rack footprint, the power delivery and the cooling design that a building has to supply.
Korea's Ministry of Science and ICT supports the work, and Rebellions expects its R200 systems to be ready for deployment in 2028, which places the procurement clock well ahead of the hardware.
SK Telecom brings the data centre, its own foundation models including A.X K2, and public services developed under the country's AI for All project.
The claim to test is sovereignty with a dependency: Korean silicon and Korean models run inside an NVIDIA fabric and an NVIDIA rack specification, so the national stack stays compatible at the layers NVIDIA defines.
For a ministry or an enterprise, the decision in front of them is a rack specification and a supply chain, and the silicon is one input among several.

NVIDIA's newsroom release of 8 November 2026 is short, and it answers a question that national AI programmes keep circling: what a country buys when it wants its own accelerators. The answer it offers involves Korean NPUs, a Korean carrier, a Korean ministry and an NVIDIA interconnect, all inside one rack specification.

Netics visual, 10 November 2026: the change the release describes, where a custom accelerator joins NVIDIA scale-up networking inside a standard rack.

Three parties, one rack specification

Each party brings a different asset. Rebellions develops inference accelerators and brings its next-generation R200 NPU, along with a software stack it describes as built on open source technology and open standards. SK Telecom brings AI data centre operations, its own foundation models including A.X K2 and public-facing services built under the AI for All project. NVIDIA brings the scale-up networking, the rack architecture, the reference designs and an ecosystem of original design manufacturers and component suppliers.

The division matters because the release is explicit about which layer each company owns. Rebellions develops the processor and the system design, SK Telecom operates the facility and the services, and NVIDIA supplies the fabric and the rack. The release describes a collaboration that brings together Rebellions' silicon, SK Telecom's AI data centre and the NVIDIA platform.

The image published with the release on 8 November 2026, showing the kind of accelerator baseboard the announcement is about rather than the Korean part itself. Image from https://nvidianews.nvidia.com/news/nvidia-rebellions-and-sk-telecom-announce-collaboration-to-advance-sovereign-ai-infrastructure-with-nvidia-nvlink-fusion.

NVLink Fusion is a licensing boundary drawn around scale-up

The technical description in the release is one sentence long and worth more than the headline: NVLink Fusion extends NVIDIA's vertically integrated, horizontally open platform to custom XPUs. A chipmaker keeps designing its own processor and gains access to NVIDIA's high-bandwidth, low-latency scale-up networking, which is the part of an AI system that decides how fast accelerators can exchange data inside a rack.

That is a specific kind of openness. Scale-up networking has been the closed core of NVIDIA's system advantage, and opening it to third-party silicon converts a competitor into a module inside the same rack. The chipmaker gains a place in systems that customers were already planning to buy, an established supply chain through MGX, and reduced deployment risk. The cost is also real, because the interface that carries the traffic belongs to someone else, and the roadmap for that interface does too.

NVIDIA's own product photograph of an NVLink Fusion baseboard, with two large accelerator dies and a smaller die beside them. Image from https://www.nvidia.com/en-us/data-center/nvlink-fusion/.

Prefill on GPUs and decode on NPUs is the architectural claim

The release's architectural centre is its plan for disaggregated inference. The companies plan to develop an approach to disaggregated inference, assigning different stages of response generation to the processor architecture best suited to each task, with NVIDIA GPUs for prefill and Rebellions NPUs for decode.

Disaggregation is a reasonable response to how inference actually behaves. Prefill processes the whole prompt at once and behaves like a large matrix multiplication. Decode produces one token at a time and is limited by memory bandwidth. A system that runs the two stages on hardware chosen for each can raise throughput and power efficiency, which is precisely what the release claims. The claim remains a plan: the release describes intended work, and 2028 is the date attached to the systems.

Netics visual, 10 November 2026: the four contributions the release assigns, from the NPU design to the ministry support.

The 2028 date sets the procurement clock

Rebellions expects its R200 systems to be ready for deployment in 2028. That single sentence should shape how the announcement is read. The release says the R200 systems are expected to be ready for deployment in 2028.

Data centre planning runs on power, cooling and floor space, and the MGX rack elements listed in the release are exactly those three plus the footprint. A programme that starts designing a building before the hardware arrives has to pick a rack specification early, and that specification is where the lock-in is set, well before the first NPU ships. The same document then decides which suppliers can quote for the hall, a purchasing consequence that lands years ahead of any benchmark.

The release as published, where the news summary lists the NPU plan, the data centre deployment and the ministry support above the quotes. Page from https://nvidianews.nvidia.com/news/nvidia-rebellions-and-sk-telecom-announce-collaboration-to-advance-sovereign-ai-infrastructure-with-nvidia-nvlink-fusion.

What a ministry buys when it buys sovereignty

Korea's Ministry of Science and ICT supports the collaboration, and the release frames the goal as AI services tailored to the country's language, industries and priorities. The deputy prime minister's quote describes Korean-developed NPUs entering the computing infrastructure market alongside NVIDIA, and the government's role as support across physical AI and GPU supply.

Read that as a policy position with a technical shape. A state that wants its own models hosted on its own soil needs three things that no single procurement delivers: silicon it can buy, a facility it can power, and a model stack that runs on both. The Korean arrangement assembles the three from different owners, and the ministry's presence is what turns a vendor partnership into a national programme. For French and European buyers watching the same pattern, the useful detail is that the MGX architecture gives GPU and NPU systems a shared rack footprint, power delivery and cooling design.

Netics visual, 10 November 2026: the five items a national AI programme has to assemble, as the release spreads them across three companies and a ministry.

The dependency a sovereign stack keeps

The dependency deserves to be stated plainly, because the word sovereign does a lot of work in these announcements. Korean processors, Korean models and a Korean data centre will run inside an NVIDIA scale-up fabric and an NVIDIA rack architecture, with reference designs and an ODM ecosystem that NVIDIA coordinates. Sovereignty here means the ability to own the workload, the language and the facility, at the layers a country can own.

That is a defensible position, with sovereignty limited to ownership of the workload, the language and the facility. The exit test for any such programme is whether the stack keeps running when one supplier's roadmap stops suiting it, and the release names no second fabric. Teams planning around this class of hardware usually discover that the operational layer decides more than the silicon: driver versions, container images, scheduler configuration and the monitoring that tells you a rack is underperforming. That layer is the part a sovereign infrastructure and cloud exit plan has to own on hardware the team controls.

The product page on 10 November 2026, where NVLink Fusion is presented as the way to attach custom silicon to NVIDIA's platform. Page from https://www.nvidia.com/en-us/data-center/nvlink-fusion/.

Originally published on the Netics blog.

NVIDIA Publishes a Management Plane for GPU Clusters and Documents Its Limits

Taha Echakiri — Fri, 13 Nov 2026 09:00:02 +0000

TL;DR

NVKE is an open-source management plane from NVIDIA that lets a GPU infrastructure operator create and run Kubernetes clusters for its own customers on bare metal.
Each workload cluster receives its own Kubernetes control plane, its own etcd and its own node pools, and the operator runs the management plane that provisions them.
A throwaway seed cluster stands up the management cluster, and the management cluster then provisions workload clusters on request from the NVKE API.
One controller process serves a REST API backed by PostgreSQL, a Temporal worker and a set of Kubernetes reconcilers, and it writes the cluster record and a work-queue row in a single transaction.
Nodes attest their identity through SPIFFE/SPIRE with their TPM, NVIDIA Infra Controller provisions the machines, and kubeadm bootstraps Kubernetes on them.
The project's own limitation page lists what the preview leaves out: the management cluster does not manage itself, there is no management-plane upgrade procedure and no backup or restore procedure.
The API listens on port 30080 without TLS, authentication is off after installation, and NVKE performs no authorisation at all, so every admitted caller can create and delete workload clusters.
A handed-over admin kubeconfig cannot be revoked, node-pool changes leave running nodes untouched, and the full-disk-encryption image gives every node built from it one shared volume key.
An operator adds TLS and authorisation in front of the API, a backup and upgrade routine for the management plane, per-customer credentials, a break-fix path for node replacement and a telemetry backend that keeps data.

Netics visual, 9 November 2026: the request path the architecture documentation describes, where the API writes a row and an outbox processor starts the workflow.

A throwaway seed cluster builds the management cluster

NVIDIA's documentation frames the project as a self-hosting factory for Kubernetes clusters. An operator installs nvkectl, which uses a temporary seed cluster to form the management cluster and installs the management plane onto it. From then on the operator requests clusters through the NVKE API or nvkectl, and NVKE provisions the bare-metal nodes, forms the cluster and drives node pool scaling, Kubernetes upgrades, node replacement and certificate renewal.

The provisioning path runs through NVIDIA Infra Controller, or NICo, the only compatible infrastructure provider at preview, with LaunchLayer supported once a deployment enables it. NICo handles machine discovery, firmware, network configuration and host provisioning, and the management plane reaches it through CAPNICo, the Cluster API provider for NICo. NVKE provisions each cluster with OVN-Kubernetes and then hands off: the GPU software stack arrives through an AICR recipe bundle, a version-locked combination of the GPU Operator, drivers, kernel configuration and Kubernetes that is rendered into GitOps bundles.

NVIDIA's own product-family diagram from the repository, published under docs/assets/diagrams: NICo provisions machines, the management plane drives clusters, and NVKR builds the artifacts both use. Figure from https://github.com/dsx-ai-factory/nvke/blob/main/docs/assets/diagrams/product-family.svg.

That division carries a practical consequence for anyone planning a site. A working NICo site is a prerequisite for day-2 operations, because tenant cluster creation, node add and remove, and upgrades all go through NICo. The NVKE software bill of materials deploys the controller, NICo, Temporal, Postgres, SPIRE and CAPNICo onto the management cluster in one GitOps pass, so the machine provider shares a cluster with the control plane that drives it, and a failure there stops provisioning.

One process runs the API, the workflows and the reconcilers

The architecture document is unusually specific, and the detail matters because it tells an operator where to look when a cluster stalls. NVKE is a Go workspace of thirteen modules, and three of them run in production: the controller deployed as a Helm chart, the node agent installed on each node as a Debian package and run by systemd, and the Cluster API infrastructure provider deployed beside the controller. The rest are a CLI, the generated API module, an authentication library, the management engine's custom resource types and test tooling.

The published documentation on 9 November 2026, where the preview banner sits above the definition: the operator requests each cluster through the NVKE API, and NVKE provisions the bare-metal nodes and forms the cluster. Page from https://docs.nvidia.com/nvke/.

The request path avoids a common failure mode. The API queues Temporal work through the database outbox. When a client posts to /api/clusters, the handler opens one database transaction and writes both the cluster record and a row in the workflow_outbox table, so a committed cluster always has its work queued. A processor polls that table, starts the matching Temporal workflow and marks the row processed. The workflow creates Cluster API resources, the infrastructure provider provisions or adopts the nodes, the controller dials nvke-agent on each node over gRPC, and the cluster reaches Ready.

The node agent only listens: it is a gRPC server, and the controller dials it. It runs kubeadm init and kubeadm join, installs a CNI, bootstraps the kubelet, handles drain, reboot and teardown, and attests its identity through SPIRE by TPM or join token. Clusters use OVN-Kubernetes for networking, and etcd encryption at rest is available per cluster.

The limitation page carries the operational weight

Netics visual, 9 November 2026: the default the repository documents itself, quoted from the preview limitation page and the README.

Most preview releases bury their gaps in release notes. NVKE publishes a page called Preview status and known limitations and lists them by area, which makes the document the most useful thing in the repository for anyone evaluating a managed Kubernetes service built on it.

The management cluster comes first. It does not upgrade, scale or replace its own nodes, the preview provides no procedure to move the management-plane services to a later release, and it provides no procedure to back up or restore the management cluster or the plane's state. NVIDIA tells operators to keep the kubeconfig and the etcd encryption key file from the bootstrap in secure storage, because that key file is the only copy not sitting on a node. Management cluster nodes built from the node image also need the operator's own SPIRE certificate authority at bootstrap: without those files SPIRE falls back to a self-signed CA, no node can attest, and nothing joins.

The published limitation page on 9 November 2026, where the management-cluster, API and access sections each state their own constraint. Page from https://github.com/dsx-ai-factory/nvke/blob/main/docs/overview/preview-status.md.

The API section reads like a checklist for an admission review. The NVKE API listens on port 30080 of every management node without TLS, authentication is off after installation, and any caller in an admitted network can call every endpoint. NVKE performs no authorisation, so a token holder can create and delete workload clusters. Customers receive only an admin kubeconfig, enableOidc does not connect a customer's identity provider to a workload cluster, and a handed-over admin kubeconfig cannot be revoked because NVKE does not rotate the cluster certificate authority.

The workload-cluster section holds three more. Changing a node pool's configuration does not replace its running nodes, and NVKE marks them with a condition without draining or deleting them, so an operator needs a break-fix system that acts on the condition. Node labels and taints are fixed at creation and applied only when a node joins. nvkectl upgrades only the control plane, and a node pool's Kubernetes version changes through a raw API call.

Netics visual, 9 November 2026: the four gaps the limitation page names, ordered by what blocks a first tenant cluster.

Encryption and telemetry defaults need a decision

Two further items deserve attention before anyone treats the encryption story as settled. NVIDIA's hosted pipeline builds a full-disk-encryption variant of the node image with the root volume on LUKS2, each image build generates a new volume key, and every node built from that image inherits it, so anyone holding the original image and the documented bootstrap passphrase recovers the key that decrypts every node built from it. The documentation says plainly to use those images for evaluation only. The default telemetry pipeline, NKTP, keeps data in memory and gives anonymous users admin access to Grafana.

The audit consequence is concrete. A tenant that needs encryption at rest has to replace or rebuild that image path before the cluster holds anything sensitive, and a tenant that keeps the default telemetry backend has an anonymous admin login on a monitoring stack that sits inside the cluster boundary.

What an operator adds before selling clusters

The repository front page on 9 November 2026, where the preview banner, the Apache 2.0 licence and the open issue count sit above the README. Page from https://github.com/dsx-ai-factory/nvke.

Work through the limitation page as a build plan and the list is finite. Put a TLS endpoint and an authorisation layer in front of the API, because the project's own authentication stops at admitting a caller. Keep the bootstrap kubeconfig and the etcd key file somewhere with a restore test attached, then write the restore procedure NVIDIA does not ship. Decide how the management plane moves to the next release, since no procedure exists.

Then handle the tenant boundary. Issue per-customer credentials through an OIDC kubeconfig, and treat the admin kubeconfig as a break-glass artifact that expires, because it cannot be revoked. Where a customer needs narrower rights, that work happens inside the cluster they receive.

Finally, close the loop on node health. NVSentinel watches nodes for hardware faults and, when a node needs repair, applies workload-aware cordon and drain before triggering NVKE's node replacement lifecycle through the lifecycle APIs NVKE provides. Node pool changes need something that deletes outdated nodes. A site that runs this on real hardware also needs its own imager and script for QCOW2 provisioning, and its own telemetry backend. Teams that operate Kubernetes on their own hardware already own that operational layer, the same layer this preview assigns to the operator.

Originally published on the Netics blog.

Anthropic Ships Claude Fable 5.2 With Effort Tiers That Set the Bill

Taha Echakiri — Thu, 12 Nov 2026 19:00:03 +0000

TL;DR

Claude Fable 5.2 is an Anthropic language model aimed at long, open-ended technical work, released on 9 November 2026 alongside Claude Mythos 5.2, a second configuration of the same model.
Fable carries the cybersecurity and biology safeguards that allow general availability, and Mythos opens the restricted capabilities to organisations admitted to the Cyber and Life Sciences Verification Programs.
Outputs arrive up to 30% faster than the 5.1 generation at $8 per million input tokens and $40 per million output tokens, with an optional fast mode at $16 and $80 that runs up to 2.5 times faster.
The benchmark table reports each model at its highest available effort: Fable 5.2 reaches 49.1% on FrontierCode 2.0 at Xhigh, 69.9% on FrontierSWE v2, and loses AutomationBench to Gemini 4 Carbon at 51.4% against 53.9%.
Cost per task moves with the effort tier: at High effort Fable 5.2 matches Opus 5.5 at Max effort on FrontierCode, 45.3% against 45.5%, for roughly half the cost per task, and Medium effort is the default in the Claude apps and Claude Code.
Mythos carries the security work that Fable's safeguards block, reaching 76.3% on CyScenarioBench and 89.6% on CVE Rediscovery, with 10.2% more vulnerabilities found than Opus 5.5.
Sandbox escape attempts fell with it: Mythos 5.2 used files or credentials placed outside its sandbox in 3.1% of 1,800 adversarial scenarios, against 39% for Mythos 5 and 21% for Mythos 5.1.
The system card also records what training-time limits removed: accelerator kernel optimisation and model architecture work sit at about the Opus 5.5 level, with fallbacks to Opus 5 for kernel work on some nonstandard accelerators.
What the release leaves to the buyer is the tier policy per task, an acceptance test beyond vendor-run evaluations, and a review process sized for the volume the price now allows.

Anthropic's own combined-score chart from the system card, page 38: the ECI aggregates many benchmarks into one score per model, which is convenient for a headline. Figure from https://www.anthropic.com/document/claude-fable-5-2-mythos-5-2-system-card.

The table reports each model's best score at its highest effort

The announcement leads with a table, and the footnote under it says that all scores show each model's highest-available effort unless stated otherwise. That convention explains why the headline numbers look close together. The combined ECI chart above supplies the wider context by compressing Anthropic's benchmark basket into a model-level trend; the task rows here show the individual results behind that aggregate. Fable 5.2 reaches 49.1% on FrontierCode 2.0 at Xhigh effort, Opus 5.5 sits at 45.5% and GPT-6 Astra at 42.8%, while Gemini 4 Carbon appears at Medium effort with 41.1%.

Two rows in the same table undercut a simple reading of that. Gemini 4 Carbon leads AutomationBench at 53.9% against Fable 5.2's 51.4%, and the same model sits second on Chartography. Anthropic's own framing for the release is that Fable and Mythos are best for the most ambitious, open-ended technical work, and that most coding and knowledge tasks should start on Opus 5.5 and move up or down from there.

The scores that carry the most weight for infrastructure work come from the terminal and repository tasks. FrontierSWE v2 puts Fable 5.2 at 69.9% against 65.5% for GPT-6 Astra, Terminal-Bench-Science at 74.6% for Fable and 75.3% for Mythos against 68.1% for Astra, and OSWorld 2.1 at 89.3%. Those are the evaluations that resemble the work an agent actually does: editing a repository, running commands, and finishing a task in a terminal.

Netics visual, 9 November 2026: the comparison Anthropic makes in its own FrontierCode 2.0 commentary, with both numbers attached.

Cost per task moves with the effort tier you pick

The second half of the announcement is a set of accuracy-against-cost charts, one per evaluation, and they answer a different question from the table. On FrontierCode 2.0, Fable 5.2 at High effort scores 45.3% for roughly half the cost per task of Opus 5.5 at its best, which is the trade most teams will actually face. On Humanity's Last Exam, Fable 5.2 at Low effort matches Opus 5.5's top score for less than one-tenth of the cost per task. On WANDR, at Medium effort it matches Opus 5.5's top score for about 40% of the cost per task.

Anthropic also states that Medium effort is the default in the Claude apps and Claude Code, and that at Medium the model can finish challenging tasks with fewer tokens than Fable 5.1. Put together, those two statements are the release's real pricing decision: the same model can be bought at five effort levels, the default sits in the middle, and the cost per task curve is steeper than the accuracy curve between Low and High.

For a team running agents, that changes the planning question. The tier becomes a per-workload setting with a budget attached, and the useful measurement is cost per accepted change, with the token rate as an input. A pipeline that spends Xhigh effort on a task a reviewer rejects twice has spent more than one that runs High effort on the same task.

Netics visual, 9 November 2026: the release summarised in two columns, with the vendor-run evaluation caveat in the second.

Mythos carries the security work that Fable's safeguards block

Fable and Mythos are the same underlying model, split by safeguards. Fable's cybersecurity and biology filters block most cyber-related tasks, which is what allows Anthropic to ship it to every user. Mythos drops that restriction for organisations admitted to the Cyber Verification Program, and Claude Enterprise customers reach the same capability through Claude Security even without the programme.

The numbers behind that split are the most consequential in the release. Mythos 5.2 reaches 76.3% on CyScenarioBench, where GPT-6 Cyber sits at 61% and Opus 5.5 at 67.6% without safeguards. On CVE Rediscovery it reaches 89.6% against 81.3% for Opus 5.5 without safeguards, and Anthropic reports 10.2% more vulnerabilities found than Opus 5.5 and 14.4% more than Mythos 5.1. A security company quoted in the post found that one in six confirmed bugs found by Mythos 5.2 had been missed by all the other frontier models it tested.

There is a calibration claim worth noting for anyone who has triaged machine-generated findings. One tester found that earlier models overstated the seriousness of about one finding in twenty, and that Mythos 5.2 overstated none. Anthropic's footnote limits that to one tester's audit and says its own evaluations put vulnerability identification roughly on par with Opus 5.5, which is the honest reading: the capability gain concentrates in reproduction and judgement, and the volume claim needs a second source.

Anthropic's CyScenarioBench results from the system card, page 50, where each lab's scores are based on unique tasks. Figure from https://www.anthropic.com/document/claude-fable-5-2-mythos-5-2-system-card.

Sandbox escape attempts fell from 39% to 3.1%

For anyone running these models inside a sandbox, the sandbox-escape section is the one to read closely. Anthropic built a dedicated evaluation of more than 1,800 scenarios, most of which cannot be solved without escaping a sandbox, and gave the sandbox a deliberate weakness. Mythos 5.2 used credentials or files placed outside its sandbox in 3.1% of scenarios, against 3.3% for Opus 5.5, 5.1% for Sonnet 5.5, and 39% and 21% for Mythos 5 and 5.1.

Two details matter for how much weight that carries. The 3.1% figure comes from a model tested with reduced safeguards, which Anthropic says gives the most complete view of behaviour, and the post notes that describing the sandbox and adding tools to report weaknesses reduced escape attempts further. The same section records a measure where Mythos 5.2 scored slightly worse than Opus 5.5: taking harmful actions after concluding it was in a simulation.

The rest of the safety section broadens the view beyond sandbox escapes, and the reward-hacking chart below contributes to that wider alignment assessment by placing Mythos 5.2's attempt rate at or below the rates for Mythos 5.1 and Opus 5.5. External evaluators included the UK AI Security Institute, Frontier Design and METR. Enterprise Frontier Safeguards, which monitors a window of usage data inside the customer's infrastructure, is in private beta with Bridgewater, Salesforce and Stripe, and other customers keep 30 days of data for safety monitoring. Anti-distillation protections, which Anthropic says cut data harvested by distillers by 90% to 99%, belong in the same category: they are protections the buyer inherits and cannot audit.

Anthropic's attempt-rate chart from the system card, page 95: Mythos 5.2 attempted reward hacks no more often than Mythos 5.1 and Opus 5.5. Figure from https://www.anthropic.com/document/claude-fable-5-2-mythos-5-2-system-card.

What the system card still leaves to the buyer

One paragraph of the announcement deserves more attention than it got. During training, Anthropic limited advances in the research capabilities that could most directly accelerate AI development, so accelerator kernel optimisation and large model architecture work perform at about the level of Opus 5.5, with transparent fallbacks to Opus 5 for kernel development on certain nonstandard accelerators. The same page reports that over 90% of the changes merged into Anthropic's core codebase are written by Claude, and that a team of agents migrated a critical production service from Python to Rust, with the service now starting about 26 times faster.

Both statements are true and they sit together awkwardly. A model good enough to rewrite a production service is deliberately held at a previous generation's level on the one task that would speed up building the next model. For a buyer, that is a capability boundary to plan around: the ceiling is documented, and it applies to a narrow set of research tasks.

The announcement page as published on 9 November 2026, where the date, the two model names and the pricing section sit together at the top. Page from https://www.anthropic.com/claude-fable-and-mythos-5-2.

The remaining work is ordinary and it belongs to the buyer. Set a tier policy per workload and measure cost per accepted change. Run an acceptance test on your own repository, because every score in the release comes from harnesses Anthropic built or commissioned, and the table's highest-effort convention makes cross-vendor comparison easy to get wrong. Decide how a security workflow gets Mythos access, since that path runs through a verification programme. And size the review capacity for the volume the new price allows, which is the constraint that binds first when a model gets cheaper and faster. Teams that route work across model tiers end up managing that budget as infrastructure, with the tier, the retry policy and the reviewer in the same design.

Originally published on the Netics blog.

Executing Workflows Should Be a Permission, Not a Default

Taha Echakiri — Thu, 12 Nov 2026 14:00:41 +0000

TL;DR

GitHub shipped workflow execution protections to general availability on September 17, 2026, completing an allowlist model for who and what can start an Actions workflow (changelog).
GA adds workflow file targeting, Insights, and a REST API — the policy is now manageable as code.
GitHub also announced a default rule that disables pull_request_target for public repositories without an existing event policy, enforced automatically on November 2.
Netics' reading: this is the trigger half of the CI/CD boundary — the workload-identity half already exists; now the who starts it half has an owner.
Practical move: use evaluate mode now to inventory which workflows genuinely need pull_request_target, before enforcement starts.

Official GitHub changelog hero for "Workflow execution protections in GitHub Actions generally available" (September 17, 2026); source: github.blog/changelog

The trigger was always the weakest pre-run control

On September 17, GitHub moved workflow execution protections out of public preview: for GitHub Enterprise, organizations, and repositories, you can now define an allowlist of who may trigger a workflow (actor rules) and what events can start it (event rules), evaluated before a run begins. Three additions matter as much as the GA itself: workflow file targeting scopes rules to specific files, so deploy.yml can be restricted to a designated team while regular CI stays open to contributors; Insights shows how rules evaluate and enforce so policy can be tuned before it bites; and a REST API exposes the whole thing as code, which is what makes this governable across hundreds of repositories instead of being a settings click.

The security argument for this is older than the feature. Executing a workflow is the moment your pipeline runs code with your secrets, your permissions, and your runners. Yet for most of Actions' life, the decision of "what runs" lived in workflow files written by whoever could push a branch — including a stranger opening a fork. Execution protections move that decision to a policy layer that exists before the run, which is a different kind of control than anything inside the YAML.

Original Netics diagram: actor rules (who) and event rules (what) both gate a workflow run before it starts; source: GitHub changelog, September 17, 2026

The pull_request_target default is the real news

The headline change is the default. GitHub's own changelog identifies pull_request_target workflows — the Pwn Requests family — as among the most commonly exploited vulnerabilities in action workflows. The trigger runs with access to your secrets in the context of the base repository, so if code executes from a fork, untrusted code can poison the pipeline and exfiltrate secrets. For public repositories that do not already have an applicable event policy, GitHub is introducing a default rule that disables pull_request_target. It does not apply to private or internal repositories, and it initially runs in evaluate mode: you can see which workflow runs would be affected before enforcement begins.

Then the deadline: on November 2, 2026, GitHub will automatically enforce the default rule for affected repositories that were using the default pull_request_target policy before GA.

Official GitHub changelog artwork for "GitHub Actions holds potentially malicious workflows for approval" (July 28, 2026); source: github.blog/changelog

Workflow execution is a permission boundary, not a YAML detail

This is the piece teams usually skip. A compromised token, a malicious dependency, or a careless workflow_dispatch input can all trigger your pipeline — execution protections do not stop every one of those paths, but they force the question into the open: is this run something an actor and an event were explicitly allowed to start? Two workflows in the same repository can now carry different policies, and a production deploy can be restricted to a named team while a lint job stays open. That per-workflow granularity is what makes the feature usable rather than a repo-wide blunt instrument.

Beware the quiet part: an execution-protection rule only matters if the event it gates is the only path in. If a workflow is also triggerable via workflow_dispatch, a manually-activated run bypasses nothing per se — but your actor rules still decide who may click the button. The allowlist is the control; the event is the surface. They have to be reviewed together, and the REST API finally makes that review something a pipeline can check.

Original Netics diagram: the pull_request_target trap — fork code running in the base repository context with access to secrets, and the new default rule that disables the trigger; source: GitHub changelog

What to do before November 2

Three moves, in order. First, enable evaluate mode on the new default and read Insights to see which of your public workflows would fail — this is the cheapest inventory you will ever get, and it costs nothing to run. Second, separate production workflows from CI workflows with file targeting now, so a future policy change cannot take the build down with the deploy. Third, for every workflow that genuinely needs pull_request_target, write an explicit event policy that names it, with a comment explaining the safe usage pattern (checking out the PR ref is the classic mistake; the base ref is the safe one). After November 2, GitHub enforces the default for you — the teams that prepared will not notice; the teams that did not will discover it in a red build on a Monday.

The larger lesson is the one this blog keeps coming back to: CI/CD security is a set of boundaries, not a pile of tools. Workload identity gives the runner a verifiable identity (we argued the boundary last year); execution protections now give the trigger an owner. One half answered who the runner is; this half answers who can start a workflow, and with which event. That completes the loop on the trigger side — and it is a reminder that the same review belongs in every other automation platform you run.

Original Netics diagram: the execution protections rollout — preview, GA (September 17), evaluate mode, automatic enforcement (November 2, 2026); source: GitHub changelog

Netics helps engineering teams map these boundaries before the deadline does it for them: which workflows can execute, who can trigger them, and what the enforcement calendar looks like. Start from the Netics homepage for a practical review of your CI/CD policy surface.

Originally published on the Netics blog.

Agent Substrate Reaches 1.0 as Google Hands Its Agent Stack to the CNCF

Taha Echakiri — Thu, 12 Nov 2026 14:00:03 +0000

TL;DR

Agent Substrate is an open-source runtime that runs each AI agent session as an isolated sandbox on a Kubernetes cluster.
On 6 November 2026 Google donated Agent Substrate, gVisor and OpenRL to the CNCF Sandbox, and released Agent Substrate 1.0 as a managed capability on Google Kubernetes Engine.
A session is split in two: an Actor holds its memory, files and identity, and a Worker is a reusable sandbox taken from a warm pool.
An idle Actor is snapshotted to storage, its Worker returns to the pool, and the session comes back on a free Worker in under 500 milliseconds.
Google reports 10x the density of standard container runtimes, more than 500 suspend and resume activations per second, and over 100 contributors since development opened in May 2026.
gVisor, the userspace application kernel that isolates untrusted code, carried host kernel patches that upstream Linux declined to merge while one vendor owned the project, and neutral ownership opens that path.
OpenRL brings the reinforcement-learning loop to the same sandbox, with four Python primitives and a Tinker compatible API.
Agent Substrate on GKE requires Standard clusters on version 1.37 or later and x86 CPU worker nodes.
An operator still owns egress policy, the snapshot size that sets resume latency, and the identity each Actor carries.

Google's own comparison, published with the Open Source blog post: a dedicated Pod per agent holds memory and CPU while the agent waits, and the Substrate model returns the worker to a shared pool. Figure from https://opensource.googleblog.com/2026/11/sandboxing-untrusted-ai-agents-at-scale.html.

Agent Substrate preserves session state while returning idle worker capacity

An agent works in a loop: it proposes an action, runs code or a tool call, reads the result, and waits while a model reasons or a person answers. Google's post puts the wait at the centre of the design problem. A standard Kubernetes Pod that stays allocated through those pauses holds memory and CPU that nothing uses, forces the control plane to track a persistent object for every agent in the fleet, and ties an identity to a Pod that may outlive its usefulness.

Substrate separates the stateful part of the session from the compute that runs it. An Actor carries the memory, the files and a discrete identity. A Worker is a reusable sandbox taken from a pool that is kept warm. A gateway sits on both sides of the sandbox, enforcing access rules and injecting credentials so that a secret never enters the agent's environment. When the agent pauses, the runtime writes its memory and files to storage and hands the Worker back to the pool.

Netics visual, 9 November 2026: the four steps Google documents for a suspended agent that receives a request, each step placed against the component that performs it.

The request path comes from the Open Source blog post, and it is worth knowing because it explains where latency lives. An incoming call reaches the atenet-router proxy. When the target Actor is suspended, the proxy holds the request in memory and alerts the Substrate controller. The controller assigns the Actor to a free slot in the warm pool, the state reloads from tiered storage, the process wakes up, and the proxy delivers the request. Google reports the restore at under 500 milliseconds and more than 500 activations per second, with thousands of agents waking every second without creating Pod objects or loading etcd.

The identity model is the part a platform team will feel first. The GKE documentation states that Substrate issues a unique SPIFFE X.509 certificate and a JSON Web Token for each Actor, giving each session a distinct identity for request attribution. Credentials stay outside the sandbox: when an outbound request matches an egress rule, the gateway authenticates the Actor's mTLS tunnel and replaces placeholder headers with a fetched secret or the Actor's token.

Google's diagram of the split, published with the Cloud blog post: actors hold state, workers run it, and the pool absorbs the next activation. Figure from https://cloud.google.com/blog/products/containers-kubernetes/open-infrastructure-for-autonomous-agents-comes-to-cncf.

gVisor spent years waiting for a neutral owner to merge its kernel patches

Substrate supports gVisor and Cloud Hypervisor microVMs as isolation boundaries. It is a lightweight application kernel written in Go, open sourced in 2018, sitting between the container and the host operating system and handling application system calls in userspace so that generated code never talks to the host kernel directly. Google says it runs millions of sandboxes a day underneath GKE Sandbox and Cloud Run, and names Anthropic and OpenAI as users for code execution, Modal and Snowflake for execution sandboxes, and Ant Group and Tencent in production at scale.

The donation matters because it changes the path available for upstreaming the kernel patches. Google's post states that sandboxed infrastructure providers, Google included, carry host kernel patches that bring gVisor's I/O performance close to native Linux, and that upstream Linux maintainers declined to merge those patches while the project was owned by a single vendor. Neutral CNCF governance opens the path to upstreaming them. Ant Group, Modal and Tines have committed to join gVisor's maintainers, with OpenAI, Tencent and NVIDIA continuing their contributions.

The Google Cloud blog post as published on 6 November 2026, where the donation, the CNCF Sandbox acceptance and the 1.0 milestone appear in the same opening. Page from https://cloud.google.com/blog/products/containers-kubernetes/open-infrastructure-for-autonomous-agents-comes-to-cncf.

What the kernel patch queue changes for a buyer

Read that as a procurement signal. A governance change that determines what can be merged into a Linux kernel has a performance consequence and a date attached to it, and the maintainer list is where the evidence sits. For a team choosing an isolation boundary this quarter, the honest position is that gVisor's I/O advantage depends on out-of-tree patches today and on upstream work that has just become possible.

OpenRL puts the reinforcement-learning loop on the same sandbox

OpenRL is the youngest of the three projects, a research preview in February that now reaches the CNCF early on purpose. The argument in the post is that the sandbox running an agent in production is the same environment needed to train one with reinforcement learning: millions of multiturn rollouts inside real tools, scored, then folded back into the weights. Building that loop from scratch usually means forking a monolithic framework and debugging distributed weight transfers before touching the training algorithm.

Google's diagram of the OpenRL loop, published with the Cloud blog post: rollouts run in the same sandboxes that serve the agent, and the accelerator pool is shared between sampling and training. Figure from https://cloud.google.com/blog/products/containers-kubernetes/open-infrastructure-for-autonomous-agents-comes-to-cncf.

OpenRL exposes four primitives, sample, forward_backward, optim_step and save_state, behind an API compatible with Tinker, with weights and data staying on the operator's own cluster. It uses Kubernetes Dynamic Resource Allocation to partition an accelerator pool, trains through PyTorch and AutoModel, samples through vLLM, and validates Gemma and Qwen out of the box. It pairs with llm-d, the distributed inference project that reached 1.0, for prefix-aware routing of rollouts to samplers that already hold matching KV cache, and for time-slicing the same GPUs between sampling and training.

The shared-resource idea is the one to copy. Time-slicing training and sampling on the same chips attacks the same waste as snapshotting idle agents: one set of hardware sits idle while the other phase runs, and both projects reclaim that idle time, which buys capacity back without a purchase.

Agent Substrate on GKE has specific deployment requirements

The donation and general availability were announced together. The donation covers three projects entering the CNCF Sandbox, the foundation's entry stage, announced at KubeCon and CloudNativeCon North America in Salt Lake City. General availability covers Agent Substrate on GKE, Google's managed product. Agent Substrate on GKE is generally available as a managed runtime. The three projects are working through CNCF Sandbox onboarding with community maintainers and independent build infrastructure.

Google's architecture diagram from the Open Source blog post, where requests enter through the router and the controller places actors on workers. Figure from https://opensource.googleblog.com/2026/11/sandboxing-untrusted-ai-agents-at-scale.html.

The managed GKE capability has specific deployment constraints. The GKE documentation requires Standard clusters on version 1.37 or later on both the control plane and the node pools, with x86 CPU worker nodes. Autopilot is excluded because the node agent, atelet, runs as a privileged DaemonSet, and microVM pools need nested virtualisation or bare metal. Accelerator-heavy agent workloads use the Pod-based path that Google calls Agent Sandbox, where GPUs and TPUs are supported with gVisor.

The restore figures carry conditions as well. Memory, root file system and volumes come back in under one second when the Actor resumes with the same hardware, sandbox runtime and container image. A change to the node hardware or runtime restores the root file system and volumes and restarts the processes. Updating the image while the Actor is suspended restores the volumes into containers started from the updated image. Resume latency also scales with snapshot size, which is why the documentation pushes large workspaces onto external Filestore volumes and keeps their data outside the snapshot.

What an operator still owns after the donation

Netics visual, 9 November 2026: the two isolation boundaries Substrate supports, compared on what each needs from the node and what each does to memory accounting.

Egress is the first item. Substrate blocks all outbound traffic by default, including traffic to other Actors, to the Kubernetes control plane, to the Compute Engine metadata server and to the internet, and it blocks non-DNS UDP, raw TCP connections and HTTP WebSocket upgrades. Rules live in an EgressPolicy resource and accept fully qualified domain names or wildcard hostnames, with no IP addresses and no CIDR ranges. A fleet that needs to reach an internal service by address has to put a name in front of it.

Capacity is the second. The warm pool sets both the latency a waking agent sees and the idle spend the operator carries, and the 10x density figure compares Substrate against container runtimes that hold one Pod per agent. A team sizing a pool for a few hundred concurrent sessions should measure its own ratio in a Standard cluster on 1.37 before quoting Google's number internally.

Governance is the third, and it is the one with the longest horizon. Three projects have just entered the foundation's entry stage, gVisor is on a path toward Incubation, and the design of the agent runtime, the sandbox kernel and the training loop will be settled in public maintainer forums. For teams that run agent workloads on their own Kubernetes estate, the transferable design is the split itself: stateful session identity in one layer, disposable compute in another, credentials enforced by the gateway.

Originally published on the Netics blog.

AWS Continuum Validates and Ranks Code Vulnerabilities

Taha Echakiri — Thu, 12 Nov 2026 09:00:02 +0000

TL;DR

AWS Continuum is a security service from Amazon Web Services that validates code vulnerabilities, ranks them by business impact and proposes a fix for each one.
General availability was announced on October 22, 2026 in US East (N. Virginia) and US West (Oregon), with every remediation starting in a mode where Continuum proposes the fix and a person approves it.
The service builds a context graph from your repositories and your AWS environment, then ranks each validated issue by how close its attack path runs to a customer data store, a payment pipeline or an authentication service.
The evidence package for one issue includes an evidence level, an exploit reproduced in an isolated sandbox with proof-of-concept code, the attack path, a blast radius and a check of whether existing defences would stop it.
The fix arrives as a pull request in the connected repository — a code patch, a dependency upgrade or a configuration change — and a reviewer merges it, so the decision stays with the team.
The two pricing dimensions are a monthly context fee per million lines of code, tiered from $200 down to $100, and $10 per Continuum inference unit, which AWS illustrates with a $1,200 month at one million lines and 100 findings and $218,200 at a hundred million lines and 10,000 findings.
What the console walkthrough leaves open is everything outside the demonstrated tenant: no independent benchmark, no remediation accuracy figure, and no statement about how it treats findings in code it cannot build.

AWS Continuum's launch turns vulnerability triage into a metered service with a monthly context charge and a variable inference charge. A scanner on a large estate emits thousands of findings a day, and the ones that matter hide among the ones that describe code nobody can reach. AWS's framing of the service it made generally available on October 22, 2026 names the cost directly: "Previously, confirming whether a finding was exploitable took hours of manual investigation, and shipping a fix with confidence took days of coordination across security, infrastructure, and application teams."

Continuum validates a finding before it ranks it

For a buyer, the service packages discovery, validation, prioritisation and remediation into one AWS-managed workflow. It finds and validates code vulnerabilities, it ranks the validated set by business impact, and it proposes a fix for each confirmed issue. This scope matters to the bill because AWS meters the security work through Continuum inference units while charging separately for the context graph.

Official AWS screenshot (2026-10-22): the Continuum Overview page, showing the code integrations, the environments built on them and the status of the AWS collector that supplies resource context. Source: aws.amazon.com/blogs/aws.

AWS puts the failure of severity scores plainly in the post: "Generic severity scores don't tell you which resources a threat actor can actually reach," and the consequence it draws is the one every reviewer of a noisy dashboard has felt — a critical score on a dev-only utility looks the same as a medium score on a service one step away from a payments database.

A context graph built from repositories and one AWS account

Two sources feed the graph: your code repositories and your AWS environment. In an AWS account and Region where the code runs, you set up a collector that gathers resource details such as AWS Config resources and network configuration. The collector is what lets the service answer a reachability question, since a network path to a database is the difference between a theoretical flaw and an exploitable one.

The graph is also how the service learns which systems carry business weight: who owns a service, how services depend on one another, and which data each one touches. The ranking then follows the context graph. The context fee pays for building and maintaining this graph, so a buyer is funding an ongoing model of the codebase, cloud configuration and business context alongside the inference work.

Official AWS screenshot (2026-10-22): the risk assessment for one issue — exploitability, evidence level, likelihood, impact and blast radius — beside the remediation already delivered. Source: aws.amazon.com/blogs/aws.

The evidence package: a proof, an attack path, a blast radius

For a finding it can reproduce, the service runs the exploit in an isolated sandbox and attaches proof-of-concept code with reproduction steps. It also reports the attack path, the source analysis behind the verdict, a blast radius assessment and an evaluation of whether the existing defences would detect and stop that path.

This is the part of the product a security team can audit, and the part a finance-adjacent reader should read closely. The evidence package combines a reproduced exploit with a model assessment of existing controls based on the configuration inventory. A team that reads the proof and skips the second paragraph gets the vulnerability and misses the assumption.

In the worked example in the post, one issue is rated High severity, exploitability Exploitable with an evidence level of Static proven, likelihood High because the service is reachable with no authentication and no user interaction, impact Moderate, and a blast radius reaching other users in the same tenant. Those labels are a compact language for a security conversation, and they will only be as good as the inventory behind them.

Official AWS screenshot (2026-10-22): the fix delivered as a GitHub pull request, with the risk assessment and proof carried into the description a reviewer reads. Source: aws.amazon.com/blogs/aws.

The fix arrives as a pull request that a person merges

Each confirmed issue gets a fix: a code patch, a dependency upgrade or a configuration change, opened as a pull request in the connected repository. In the demonstration the pull request changes four files in the cart service, with 50 lines added and 24 removed, and its description carries the same risk assessment, remediation summary and proof that the console shows.

Every remediation starts in a mode where a person approves it before it ships. AWS states the future direction in the same paragraph: teams will eventually decide which remediation classes Continuum may apply on its own, once confidence gates such as test coverage, scope bounds and historical success rate pass. For an adoption decision, that roadmap makes approval policy, test coverage and historical success rate part of the operating model a buyer must evaluate.

The console states that every generated patch still requires review and a merge before the issue is fixed. A patch that compiles and passes tests can still change behaviour, and the review of a machine-authored change deserves the attention of any other change to production code.

Official AWS pricing page capture (retrieved 2026-11-08): the two dimensions of Continuum pricing, a monthly context fee on protected code and a usage-based inference fee. Source: aws.amazon.com.

What the two pricing dimensions charge for

The pricing page describes two dimensions and no upfront cost. A monthly context fee tracks the lines of code the service protects, starting at $200 per million lines and declining through $180, $160 and $130 to $100 per million in the largest tier. The variable charge is $10 per Continuum inference unit, and AWS says the units required for each vulnerability vary with the work involved.

The page's own worked examples make the shape clear. One million lines with 100 vulnerabilities at one unit each comes to $1,200 a month. Ten million lines with 1,000 vulnerabilities comes to $12,000. A hundred million lines with 10,000 vulnerabilities at two units each reaches $218,200, of which $18,200 is context and $200,000 is inference.

The enterprise example makes the purchasing issue visible: $200,000 of its $218,200 monthly estimate comes from inference work, while $18,200 comes from context. At scale the findings drive the bill, and the findings are what the validation is meant to reduce. AWS does not publish a fixed number of units per vulnerability, so a budget needs an observed workload mix and cannot be derived from repository size alone.

Netics editorial visual: the scanner's severity score beside the validation result, which adds an attack path, a reproduction and a business distance to the same finding.

What the console walkthrough leaves open

The post demonstrates one tenant, one cart service and one High-severity issue. It reports no measured reduction in mean time to remediate, no false-positive rate on validation, and no accuracy figure for the generated patches. Those numbers would be the ones a buyer would want, and their absence is normal for a launch post.

Two operational questions stay open as well. The service needs a buildable view of your code and a collector in the account where it runs, which makes the first day of adoption an inventory project. The business case also depends on reviewer capacity because every generated pull request initially requires human approval, leaving buyer-side review time outside the AWS fee examples.

A backlog that shrinks with the decision still inside the team

The decision for a buyer is whether the combined context fee, variable inference usage and internal review time produce a defensible cost per resolved vulnerability. A team that adopts validation without the last step automates deployment of machine-authored code, and a team that adopts the proposal without validation automates noise.

None of this removes the need to know your own estate. Validated findings sharpen a security audit and hardening programme, and they sharpen it most where the inventory is current, the owners are named and the critical services are written down before the first scan runs.

Netics editorial visual: the context fee tiers, the $10 per inference unit, and a $1,200 month for one million lines with 100 findings.

For a small company, a limited pilot can establish protected lines of code, inference units per vulnerability and reviewer time before a wider commitment. Record those three inputs during the pilot and compare the resulting monthly cost with AWS's $1,200 and $12,000 worked examples. The rest of the estate can wait for the second month.

Originally published on the Netics blog.