DEV Community: NeticsLabs

DuckDB 2.0.0 LTS Ships Its Biggest Feature Release on a Support Clock

Taha Echakiri — Sat, 24 Oct 2026 19:00:05 +0000

TL;DR

DuckDB is an open-source analytical database that runs inside one process, and version 2.0.0 LTS, codenamed Cinnamon Teal, was released on 21 October 2026 with the Quack protocol, asynchronous I/O, a first-class VARIANT type, agent mode, triggers and an extensible SQL parser.
The release is built from close to 20,000 commits since DuckDB 1.5, and the project calls it the biggest feature release so far, in contrast with the 1.0 line that concentrated on stability.
The support label carries a date: 2.0 is supported until at least October 2027, with the calendar page listing an end of life of 22 October 2027.
Three version lines run in parallel until 17 November 2026, when community support for 1.4 ends, and two lines remain until March 2027, when the extended support for 1.5 ends.
The storage format moves in one direction now, since releases before 2.0 cannot open files written by 2.0 while existing files that 2.0 reads and writes stay readable by older releases.
A migration plan for a small data team is a shadow system, a pinned client library and a decision about whether to wait for the first patch release in November.

What an LTS label buys a data team

DuckDB's release calendar explains the label better than the announcement does. Starting with version 1.4.0, every other release is a Long-Term Support edition, and the community support period for an LTS version currently runs a year after its release. The 2.0.0 entry shows the release date of 21 October 2026 and an end of life of 22 October 2027, which is the one date a team should copy into its own calendar.

That calendar matters more than usual for a tool that lives inside an application. An embedded analytical database is linked into a Python service, a CLI script, a scheduled job or a notebook, and each of those places upgrades on its own schedule. A support window turns an indefinite promise into a deadline somebody can plan against, and the project publishes the next release on the same page, with 2.0.1 tentatively dated 18 November 2026.

The version policy underneath is ordinary semantic versioning, with larger new features arriving in minor versions and patch versions mostly carrying bugfixes. What the LTS cadence adds is a promise about how long a given minor version will keep receiving those patches, which is the question a team asks when a pipeline that runs nightly depends on the answer.

The DuckDB announcement post (duckdb.org/2026/10/21/announcing-duckdb-200, retrieved 2026-10-21), whose summary lists the Quack protocol, asynchronous I/O, a first-class VARIANT type, agent mode, triggers and an extensible SQL parser.

Quack, VARIANT and the features in this release

The feature list is longer than a version bump usually carries. The Quack protocol arrived in DuckDB 1.5.3 in May and let DuckDB run in a traditional client-server setup; 2.0 ships a polished version of it. Asynchronous I/O now covers reads of Parquet and CSV files and of DuckDB's native storage format: the I/O layer scales independently from query processing and keeps multiple requests in flight, which the project says brings far more parallelism to remote reads and faster queries on object storage such as S3.

The other additions change how the engine is used. A first-class VARIANT type gives semi-structured data a native home. Agent mode lets the CLI detect when a coding agent is driving it and switch to compact output, triggers bring CREATE TRIGGER with BEFORE and AFTER triggers at row or statement level, and the extensible PEG-based SQL parser, opt-in since DuckDB 1.5, is now the default. Triggers still have gaps: the announcement lists BEFORE … FOR EACH ROW and UPDATE … FOR EACH ROW triggers as not yet supported, so trigger logic carried over from another database needs checking case by case.

The internals moved as well. DuckDB 2.0 switches to Clang as the default compiler for release builds, with the engine and its core extensions built that way on Linux and macOS, the CLI on Windows, and most Windows core extensions still on MSVC. The ICU library is gone entirely: the icu extension now implements timezones, calendars and collations itself, with timezone data built from the IANA database and compressed to around 45 kB.

Netics editorial scorecard of the support windows that overlap after this release: 2.0.0 LTS supported until at least October 2027, 1.5 extended to March 2027, and 1.4 LTS supported until November 2026.

The storage format now moves in one direction

The breaking change that matters is in the file. DuckDB versions before 2.0 cannot read database files created by DuckDB 2.0, while 2.0 still reads and writes files created by earlier versions, and those older files remain readable by older releases. The trap sits in new files: any database file created by 2.0 cannot be opened by the version a team would roll back to.

The project's own advice follows from that. Test 2.0.0 on a shadow system before switching production over, and if the workload is mission-critical, consider waiting for the first patch release in November. A migration guide accompanies the release, and each service upgrades its client library on its own schedule: once one moves to 2.0, any database file it creates can no longer be opened by the version it ran before.

Snapshots are the cheap insurance here. Any team that exports a copy of its analytical data before the upgrade, in Parquet or in a plain CSV of the tables that matter, converts an irreversible step into a reversible one. Our reading at Netics is that Parquet and CSV copies are the safer insurance, since they sit outside the DuckDB database file format that changed in 2.0.

Netics editorial diagram of the file compatibility change: files written before 2.0 remain readable by earlier releases, while files written by 2.0 cannot be opened by releases before 2.0, which is why a rollback has to be planned before the upgrade.

Three support lines until mid-November, then two until March 2027

Until 17 November 2026 a team can choose between three supported versions, then between two until March 2027, and the choice comes down to risk appetite. Version 1.4 stays supported until late November 2026, which the calendar page states as 17 November 2026. Version 1.5 was extended to March 2027 and, as the announcement puts it, that effectively turns a non-LTS release into an LTS one. Version 2.0 runs until at least October 2027.

The practical reading is that 1.5 is the conservative bridge. A team that needs the fixes but wants a longer runway than 1.4 offers can sit on 1.5 until March 2027, run 2.0 on a shadow system in the meantime, and move once a patch release or two has shipped. The announcement names the storage format as the most likely source of problems.

The calendar also makes the long-term shape visible. Every other release is meant to be an LTS edition, so a team can plan two-step moves and skip the versions in between. The alternation is not strict, since 1.5 gained LTS-length support after its release, and the next patch of the current line carries only a tentative date.

The DuckDB release calendar (duckdb.org/release_calendar, retrieved 2026-10-21), which dates 2.0.1 for 18 November 2026, lists the end of life of 2.0.0 LTS on 22 October 2027 and states that every other release is a Long-Term Support edition.

A migration plan for a small data team

The plan fits on one page. Export the data that matters to Parquet before touching anything. Pin the client library version in the service that reads it, so that no service creates 2.0 database files before the shadow test is done. Run 2.0.0 against the shadow copy and compare row counts and query results on the reports somebody actually reads. Decide on a date, and write down which version line the team will sit on until that date.

The reason to bother is visible in the feature list. Our reading at Netics is that a single-process analytical engine that gains a client-server mode, a semi-structured type and asynchronous I/O can query more data where it already sits. That argument is the same one behind the private assistant that answers from your own documents: keeping the data where it already lives, and adding the compute next to it.

An embedded DuckDB lives inside an application, which is where Netics does custom web and mobile development with small teams. The version choice above is the kind of decision that belongs in a runbook with a date on it, because each support line has an end date.

Originally published on the Netics blog.

The Browser Becomes an AI Distribution Point: What Mozilla and Mistral Actually Changed

Taha Echakiri — Sat, 24 Oct 2026 09:00:03 +0000

TL;DR

On September 16, 2026, Mozilla and Mistral announced that Firefox Smart Window (beta) is now powered by Mistral Small 4, with the United States, Canada, and France at launch and the UK and Germany expected later this year.
France is the first new market for Smart Window with official French-language support — a deliberate multilingual-market move, not an English-first port.
The deal's stated thesis is distribution: "open technology needs open distribution," and a browser "shouldn't be a one-way funnel."
The operational commitments that matter: conversations aren't saved on Mozilla's servers by default, Mistral agrees to zero data retention, and Firefox users keep choosing among multiple models.
What still needs proving: whether switching and zero retention survive the integration's data flows — a promise at announcement is not yet a verifiable control.
Three questions before any fleet rollout: policy-level model switching, verifiable zero-retention traffic paths, and French-language coverage in real workflows — not only in the consumer UI.

Why a model inside a browser is a distribution event

The AI competition has moved from "which model is best" to "which models people can actually reach, through which products." That is the shift Mozilla names directly: "Practically overnight, competition has expanded from 'which AI model is best?' to a new high-stakes battle over which models people can access, through which products, and whether people have meaningful choice about AI at all." A browser is the chokepoint where that battle gets decided, because the browser sits between the user and almost every AI product they touch. Putting Mistral Small 4 into Firefox Smart Window is therefore not a feature integration; it is a distribution decision.

Official Mozilla blog image (2026-09-16): the Mozilla and Mistral partnership header used on both announcement pages.

Smart Window is Mozilla's AI browsing assistant: it makes sense of complex search trails, remembers something important you clicked away from, and presents the sources behind its answers. With this deal, those tasks in the US, Canada, and France run on Mistral Small 4, a model Mozilla says it selected after evaluating multilingual performance for the beta. The detail worth pausing on is the market order. France comes first among new markets, with official French-language support. That is the practical form of the companies' stated approach: "AI optimized for local countries and cultures, not exported to them," with models fine-tuned on regional languages and dialects rather than an English-first experience being adapted afterward.

The one-way funnel test

Mozilla's CEO Anthony Enzor-DeMeo put the frame directly: "With the browser sitting at the heart of the web and online experience, it should be a place where different AI providers can compete and open source has a seat at the table." And the sharper line: "A browser shouldn't be a one-way funnel." The test of that sentence is operational, not rhetorical. Mozilla commits that "across all markets where Smart Window beta is available, Firefox users can still choose from a multitude of other AI models." That keeps the model layer swappable, which is the correct architecture for a distribution point. The moment integration stops being a recommendation and becomes the only path, the funnel is closed regardless of press language.

Official Mozilla blog image (2026-09-16): the Smart Window beta interface in Firefox with Mistral as the active model — the surface where model choice is presented to the user.

The strategic bet is bigger than one model. Mozilla describes a scenario where "a handful of closed-model companies" control the browser, the search engine, the model, and the surrounding services together. Mistral is the counterweight: a frontier European AI company reaching consumers without owning a browser, and a browser integrating a model without becoming a model company. That division of ownership is what makes the partnership structurally interesting — neither party can close the funnel by itself.

Zero data retention is a data-flow claim, not a slogan

The privacy commitments are specific. "Conversations aren't saved on Mozilla's servers by default, and partners like Mistral agree to zero data retention." Treat that as a contract to verify, not a headline. Zero retention only means something if the request path actually avoids logging, caching, and model-provider telemetry, and if the browser's own session stores behave the same way. The architecture should be auditable: where the conversation travels, what the model provider receives, what the browser keeps locally, and what happens to the input when the assistant is disabled. For European users, this intersects with GDPR expectations around purpose limitation and storage limitation — the announcement's commitments align with the regulation's direction, but alignment at announcement time is not enforcement at runtime.

Official Mistral AI news image (2026-09-16): the partnership announcement hero for Mistral and Mozilla bringing open, private, multilingual AI to the browser.

There is an honest counterpoint here. Mistral's own framing of the fourth reason — "putting sovereign AI in everyone's hands" — is consumer-facing language for something the company has mostly sold to enterprises. This deal extends Mistral beyond its business base, which is a real expansion but also a new attack surface: consumer browser behavior is a different trust boundary than a contractual enterprise deployment. The zero-retention commitment is precisely what makes that boundary survivable; it is also what makes it worth verifying.

What Netics would check before rolling this out

For an organization deciding whether Firefox with Smart Window belongs in its fleet, the questions are concrete. First, can the default model be changed or the assistant disabled at the policy level, not just in a per-user setting? Second, can we verify the zero-retention claim in the deployment — does the traffic path show what the announcement promises? Third, does French-language support extend to the organization's actual workflows, or only to the consumer UI?

Netics editorial visual: Mozilla CEO Anthony Enzor-DeMeo's test line — "A browser shouldn't be a one-way funnel" — rendered as the article's closing standard for the partnership.

The distribution thesis is sound: an independent browser carrying an open-weight European model is a meaningful alternative to the closed-funnel scenario Mozilla warns about. But distribution points earn trust through verifiable switches and auditable data flows, not through partnership announcements. Those are separate workstreams, and only one of them is done. Netics' own work on alternative AI architecture choices is summarized on neticslabs.com.

Originally published on the Netics blog.

Eight eMudhra Intermediate CA Keys Rested on Eight Bits

Taha Echakiri — Fri, 23 Oct 2026 19:00:03 +0000

TL;DR

eMudhra is an Indian certificate authority that sells TLS and code-signing certificates under the emSign brand, and an intermediate CA is the certificate that signs the certificates individual websites present to a browser.
A researcher showed the root programs on 17 October 2026 that the private keys of eight eMudhra intermediates were single-byte numbers, one of them literally four, recoverable from the public certificate in under a second on a laptop.
Chrome blocked the six TLS keys at 19:45 UTC that evening and Mozilla finished its OneCRL block for Firefox at 22:59 UTC; eMudhra revoked all eight intermediates at 09:38 UTC on 18 October.
Browsers take revocation from a list they fetch themselves, so the block landed with no update and no restart; every other client trusts a CA bundle on disk and can keep trusting a revoked chain for months.
The trust stores that live in code moved next, with Go 1.27.3 and 1.26.10, NSS 3.131.1, Python certifi 2026.10.19, Rust webpki-roots 1.0.10 and a Homebrew exclusion shipped by 20 October, while a Node.js fix remains pending.
An administrator can check on Monday morning which bundles each fleet ships, then confirm that a test chain through the emSign root now fails.
Mozilla removed emSign ECC Root CA - G3 from NSS on 20 October and Chrome will remove it in root store version 43, and Chrome is reviewing eMudhra's other root, emSign Root CA - G1.

Mozilla's incident ticket for the eight eMudhra intermediates (bugzilla.mozilla.org, bug 2083449, captured 2026-10-20), which records the ticket as resolved and carries eMudhra's own account of the key ceremony in its comments.

A private key of four, and sixteen certificates that passed every check

Scott Helme recovered all eight private keys from their public certificates on a laptop, and the whole exercise took less than a second. The values are 156, 230, 155, 4, 115, 56, 66 and 237, one per intermediate. Helme pulled all sixteen certificates behind them, because each key was used in two: one issued by eMudhra and one issued by Sectigo as a cross-sign. The certificate named EM OV TLS CA - G3C carries a private key equal to the number four.

The arithmetic behind that number explains why it fell so quickly. These are elliptic-curve keys on the P-384 curve, where the private key is an integer d and the public key is d × G. A correct P-384 key is a random 384-bit number, one of roughly 10 to the power of 115 candidates. eMudhra's keys are 47 bytes of zeros followed by a single byte, which reduces the search to at most 255 steps: compute 1 × G, 2 × G, 3 × G, stop when the result matches the certificate.

Nothing else about those certificates looks wrong. A public key derived from d = 4 is a perfectly valid point on the curve, the encoding is correct, the curve is correct, and every structural check passes. The only way to find out is to try, which is what happened in October 2026, seventeen months after the material was created in one ceremony on 30 April 2025 and cross-signed by Sectigo's AAA Certificate Services root in May 2025.

Netics editorial diagram built from the key values published by Scott Helme on 2026-10-20: a correct P-384 key sits in a space of about 10 to the power of 115, while these eight keys sit between 0 and 237.

Why revocation reached browsers in hours and everything else in months

The report reached the root programs on the evening of 17 October and the machines responded at the speed of a push channel. Chrome had the report at 18:05 UTC and a CRLSet update blocking the six TLS intermediates at 19:45 UTC. Mozilla had it at 20:06 UTC and finished a OneCRL block for Firefox installations with standard settings at 22:59 UTC. Neither update needed a restart, because CRLSets and OneCRL are lists the browsers fetch on their own.

The certificate authority followed. eMudhra revoked all eight intermediates at 09:38 UTC on 18 October. Apple blocked them on 19 October, Sectigo revoked its cross-signs the same day, and Microsoft followed on 20 October.

Then the browser story ends and the operational one starts. Every non-browser client trusts a CA bundle on disk, skips revocation checks by default and skips Certificate Transparency, a list that covers curl, Python, Node, Go, Java, git, package managers, and the scripts and daemons around them. Revoking a certificate does nothing for a client that never asks whether it was revoked. Helme tested a forged certificate for api.stripe.com against curl, Python and Node with default settings, and all three accepted the chain. That is the gap between an incident a browser closed within hours and one that lasts until an image is rebuilt.

The Mozilla dev-security-policy thread of 20 October 2026 (groups.google.com, captured the same day), where Mozilla's Dennis Jackson published the affected certificate list and the disclosure timeline.

The trust stores that live in code

Go took the most direct route. Releases 1.27.3 and 1.26.10 reject any chain containing one of the eight keys under CVE-2026-111943, the check sits in the standard library and covers Go's own verifier plus the macOS and Windows system-verifier paths, and the change adds no setting to switch it off. A program picks it up when it is rebuilt with the new release.

The other fixes work by taking the root out of the bundle: NSS 3.131.1 drops emSign ECC Root CA - G3, Python certifi 2026.10.19 drops the same root, Rust webpki-roots 1.0.10 drops it, and Homebrew's ca-certificates excludes it along with emSign Root TLS CA - G3 and AAA Certificate Services. Removing a root also cuts off every legitimate certificate that chains through it, and Mozilla said some servers will become unavailable to non-Firefox users of its trust list until those servers move to new certificate chains.

A Node.js fix is still pending, which means a large population of services keeps the old bundle until a base image is rebuilt.

Netics editorial timeline assembled from the root-program reporting of 2026-10-20: Chrome's CRLSet at 19:45 UTC, Mozilla's OneCRL block at 22:59 UTC, eMudhra's revocation at 09:38 UTC on 18 October, and the NSS root removal on 20 October.

What an administrator can check on Monday morning

Start with an inventory. Every container image, every language runtime and every toolchain that ships its own bundle holds a CA list that ages with the image, and a review that covers certificate handling and trust configuration will surface those lists faster than an incident will; that is the ground security audit and hardening covers in practice. The practical questions are concrete: which base images carry the emSign root, which services are built with a Go release older than 1.27.3, and which machines accept a test chain through EM DV TLS CA - G3B right now.

Then there are the devices nobody can update quickly. Routers, printers, televisions, appliances and industrial controllers with frozen firmware will trust these certificates until the intermediates expire in July 2039, and a forged certificate presented to one of them produces the same padlock a user expects. In our reading, Windows code signing carries the same flaw on a different path: software signed with EM OV CS CA - G3B or EM OV CS CA - G3C looks like signed software from a real publisher to anything that skips revocation.

eMudhra's own public site (emudhra.com, captured 2026-10-20), which presents the company as a digital trust provider and lists the emSign product family behind the affected intermediates.

Where eMudhra's trust goes from here

Mozilla removed emSign ECC Root CA - G3 from NSS on 20 October. Chrome will remove the same root in version 43 of its root store and is reviewing eMudhra's other root in that store, emSign Root CA - G1. The Chrome Root Program said it checked every other CA certificate in its store and found no other instance of this failure mode; that check covers Chrome's own store only, and the review of emSign Root CA - G1 is still open.

The root cause is documented, and the accountability question remains open. eMudhra's account inside Mozilla's ticket describes a bug in the ceremony script that created a second weak key pair on the HSM during a key-wrapping test, and labels similar to those of the intended keys, and the weak keys were the ones used for the certificate requests. A warning existed in May 2025, when the CRLs for the six TLS intermediates began failing signature checks and Sectigo filed a bug about it; eMudhra re-signed those CRLs with the keys matching the certificates and closed the incident as a process error.

For the certificate authority, that decision cost sixteen months of trust, counted from the May 2025 CRL warning; the seventeen-month exposure window runs from the creation of the keys in April 2025. For everyone running a fleet, the bill arrives on a schedule set by the oldest base image in the estate, and a printer whose frozen firmware ships the emSign root can still trust this chain in 2039.

Originally published on the Netics blog.

A Cost Comparison Method for Leaving the Public Cloud

Taha Echakiri — Fri, 23 Oct 2026 14:00:34 +0000

TL;DR

A cloud exit is a project that moves workloads off a provider's rented infrastructure onto servers the organisation operates and pays for directly, and the cost comparison is the arithmetic that decides whether the move pays for itself.
The method has five steps: inventory twelve months of invoices, classify every line as storage, transfer or compute, price each bucket at published rates, add hardware, racks, power and people, then set a break-even date.
Published inter-region rates show how providers meter transfer; the exit itself is priced separately, at the provider's internet egress rate. Cloudflare charges nothing for egress on R2, Microsoft charges $0.02 per GB between regions inside Europe, and Amazon's own S3 example prices inter-region transfer at $0.02 per GB.
The best-documented public example is the projection 37signals published in 2023: $3.2m of cloud spend in 2022, a hardware order of about $600,000, $720,000 a year for eight racks of space and power, and roughly $840,000 a year against $2.3m of cloud servers.
The costs the invoice leaves out — redundancy, spares, patching, on-call and the migration itself — decide whether a defensible saving survives contact with operations.
A break-even date turns the whole argument into one number a finance team can approve or refuse.

The cloud invoice prices units of consumption

A cloud bill is a measurement, and it measures one thing: consumption of units the provider defines. Storage is billed per gigabyte-month, transfer per gigabyte leaving a region, requests per million calls, compute per second of a machine that exists because a tenant asked for it. That structure is useful while the workload stays on the provider's side of the wall, because the numbers arrive already aggregated and already trusted by the finance team.

Microsoft's Azure bandwidth pricing page (azure.microsoft.com/en-us/pricing/details/bandwidth/, retrieved 2026-10-18), which lists data transfer in as free and prices intra-continental transfer between regions within Europe at $0.02 per GB and from Europe to other continents at $0.05 per GB.

The problem with using the same document to price an exit is that it never records what an organisation stops paying for. A server the company owns has no per-second meter, so the cost of running it appears in a hardware invoice once, in an electricity contract every month, in a rack rental every month, and in salary every month. None of those four lines sit on the cloud invoice, which is why the two sides of the comparison are usually assembled by different people at different times with different assumptions.

Reading the invoice as a price list first, and only afterwards as a bill, is the step that makes the rest of the method possible. Every line on it belongs to one of three buckets, and the three behave differently when a workload leaves.

Start with twelve months of invoices

In Netics' analysis, twelve months is the shortest window that contains a seasonal peak, an annual renewal and at least one incident. A quarter of billing data will usually understate the peak that sized the environment in the first place, and the peak is what an on-premises design has to absorb.

Pull the line items, and keep the totals for cross-checking. Providers expose per-service and per-region breakdowns, and the interesting rows are the small recurring ones: snapshots nobody deleted, log storage that grew quietly, cross-region replication for a bucket used by one application, load balancers kept alive for a test that ended. Each of those has a hardware equivalent that is either needed or avoidable, and the answer changes the total.

Group the twelve months by application where the tagging allows it. An organisation that cannot attribute spend to an application cannot price an exit per application either, and the fallback is a single all-or-nothing migration that carries far more risk than moving one workload at a time. Tagging discipline is the cheapest prerequisite in the whole project.

Classify every line as storage, transfer or compute

The three buckets translate to on-premises costs in three different ways, and mixing them produces the arithmetic errors that make exits look either impossible or free.

Storage is the easiest to translate. A gigabyte-month on a provider's object store maps to disks, a filesystem, a replication policy and a backup target. The unit price of the disk will look dramatically lower, and the honest comparison includes the second copy, the off-site copy and the person who tests restores. Cloudflare's own documentation lists standard storage at $0.015 per GB-month, and this article takes that published R2 rate as the yardstick a self-hosted terabyte has to beat once redundancy is counted.

Cloudflare's R2 pricing documentation (developers.cloudflare.com/r2/pricing/, last updated 2026-10-01, retrieved 2026-10-18), which lists egress as free for Standard and Infrequent Access storage and prices Standard storage at $0.015 per GB-month.

Transfer is where published prices diverge most. Microsoft charges nothing for data transfer in, and $0.02 per GB for intra-continental transfer between regions within Europe. Cloudflare states plainly that there are no charges for egress bandwidth on R2 for any storage class. Amazon's S3 pricing example prices inter-region transfer out at $0.02 per GB. These inter-region figures show how each provider meters transfer between its own regions. Data leaving for the organisation's own servers is billed at the provider's internet egress rate, which the migration step below prices. An organisation whose bill is dominated by data leaving a region is comparing against a colocation cross-connect and a bandwidth commit, and that comparison has a completely different shape from one dominated by compute.

Compute is the bucket that needs the most care. A reserved instance, a savings plan and an on-demand hour are three different prices for the same silicon, and the on-premises equivalent is a server amortised over a realistic life with a spare capacity margin. Reading only the discounted rate flatters the cloud; reading only the on-demand rate flatters the exit.

Price each bucket at published rates

Published rates matter because they are the only figures both sides of the argument accept without negotiation. A vendor price sheet can be cited in a board paper; an internal estimate cannot.

The exercise is mechanical once the buckets exist. Take the storage gigabyte-months from the invoice, price them at the cheapest tier the workload actually tolerates, and add the redundancy copies the current design implies. Take the transferred gigabytes and price them at the destination rates the provider publishes for the region pairs in use. Take compute hours at the commitment level the organisation genuinely runs, and note separately what the same hours would cost on demand, because that gap is the first thing a provider will offer to close.

Doing this exercise before any migration decision usually produces one of two findings. Either the bill is dominated by a service with no realistic self-hosted equivalent — a managed queue, a global anycast edge, a database with a control plane nobody wants to operate — in which case the exit discussion narrows to the workloads that can move. Or the bill is dominated by commodity compute and storage, in which case the numbers are large enough that the operational question becomes worth answering.

Netics editorial sequence diagram of the method: inventory twelve months of invoices, classify each line, price each bucket at published rates, add the costs the invoice omits, and set a break-even date before committing.

Either finding is useful, and both are cheaper to reach on a spreadsheet than after a migration.

Add what the invoice leaves out

The exit bill has five components that never appear on a cloud invoice, and each has a real price.

Hardware is the visible one: servers, disks, network gear and spares, amortised over a life the organisation is willing to defend. Racks and power come next, usually as a monthly line from a colocation provider or as an internal charge for space and cooling. Redundancy is the third and the most consistently underestimated, because the provider absorbs a failed disk or server without the tenant doing anything, and in a rack that work falls to the organisation.

People are the fourth. Running infrastructure means patching, capacity planning, certificate renewal, backup verification and on-call rotation, and the honest version of the comparison names the hours. An organisation that already employs the people does not get to price them at zero, because those hours were previously spent elsewhere.

The migration itself is the fifth, and it is a one-off cost with a long tail. Moving data out of the provider to the organisation's own servers counts as egress to the internet, which the inter-region rates quoted above do not cover, so the exit fee has to be priced from the provider's own internet egress rate, sometimes on a very large volume. The tail is the period where both environments run in parallel and the organisation pays for both.

Set a break-even date

A comparison without a date is a debate. With a date, it becomes a decision rule: the move happens if the accumulated saving crosses the accumulated cost within a period the organisation is willing to wait.

The arithmetic is straightforward. Take the annual cloud spend attributable to the workloads being moved, subtract the annual run cost of the replacement, and divide the one-off cost — hardware, migration, parallel running — by that annual difference. The result is a payback period in years, and it should be stated with the assumptions that produced it visible on the same page.

Two sensitivities belong next to that number. The first is hardware life: a four-year refresh shortens the period, a seven-year one lengthens it. The second is the rate at which the remaining cloud spend grows, because a workload that grows ten percent a year changes the crossover date and a shrinking one may never justify the move at all.

Netics editorial diagram drawing the two sides of the comparison: a single monthly cloud invoice covering compute, storage and transfer, against an exit bill spread across hardware, racks, power, redundancy, spares and operating hours.

A payback period of a few years is a decision most finance teams can take. A payback period that depends on optimistic assumptions about both hardware life and growth is a decision to revisit, and saying so is part of the method.

What the arithmetic looks like when a company publishes it

Publicly itemised exits are rare, which makes the 37signals account of February 2023 worth reading as a partial worked example of the method, with two caveats: its figures are the company's own preliminary projection, published before the move, and they cover the run-cost side only. The post gives no migration or parallel-running cost and no break-even date, so the last step of the method has no worked figure here.

The company spent $3.2m on cloud in 2022. Just under a million of that stored eight petabytes of files in S3, fully replicated across several regions; the remaining roughly $2.3m covered application, cache, database and search servers, and that was the part it intended to bring to zero. The hardware order came to somewhere in the region of $600,000, which the post amortises over a conservative five years at $120,000 per year. Eight dedicated racks across two data centres cost about $60,000 per month through a colocation provider, which is $720,000 per year, with enough space and power already provisioned to absorb the new machines.

Add those together and the run cost is about $840,000 per year against $2.3m in the cloud. The post calls the difference a saving of roughly $1.5m per year, sets aside half a million for unforeseen expenses over the period, and arrives at seven million dollars saved over five years in server expenses, with the ops team unchanged in size. Those are the post's rounded figures: $2.3m minus $840,000 is about $1.46m a year, about $7.3m over five years, and about $6.8m once the $500,000 is set aside. The seven-million figure covers the roughly $2.3m of servers only: the post defers the exit of the eight petabytes from S3 to 2024 and gives no cost for moving them out.

The value of the example is that every number has a source and a position in the method, and its weak points are just as visible. Storage was separated from compute. Hardware was amortised across five years. Racks and power were priced as a monthly line. The saving was stated as one rounded figure with a $500,000 contingency, and the five-year window was declared. Two assumptions need the scrutiny the method asks for. The only people cost is an ops team unchanged in size, with no hours named, which is exactly the line the method refuses to price at zero. And the figures are the company's own rough, preliminary calculations, made before the move, so they illustrate the method and leave the outcome unproven. An organisation can disagree with any single assumption and still use the structure to reach its own answer.

Originally published on the Netics blog.

A Vendor Backup Secret Opened Over 300 Microsoft 365 Tenants

Taha Echakiri — Fri, 23 Oct 2026 14:00:03 +0000

TL;DR

ConnectWise Cloud Backup is a hosted backup service, formerly sold as SkyKick, that managed service providers connect to their customers' Microsoft 365 tenants so that mail, files and Teams data are copied somewhere outside the tenant.
A secret belonging to that application was compromised and used to sign in to customer tenants as the backup service itself, and Huntress confirmed unauthorized sign-ins in 162 tenants on 16 October 2026, then counted almost 1,000 Entra tenants with sign-in activity from the attacker's address on 17 October, over 300 of them successful.
The abuse reached further than the backup app: service principals for ConnectWise Cloud Management applications assigned themselves directory roles, mostly Exchange Administrator, and in one tenant an application granted itself Global Administrator and added credentials to two other applications.
Multi-factor authentication had no bearing on these sign-ins, because no user was authenticating; the application presented its own secret and used permissions the customer had already granted.
ConnectWise shut down application access to Cloud Backup and stated it has no indicators that partner backup data was compromised; Microsoft described the activity as an active supply-chain attack.
Disabling the named applications stops new use, and the credentials those applications added stay valid, so containment runs through an audit of every secret on every enterprise application in the tenant.

What a backup application holds inside a tenant

An application that backs up Microsoft 365 has to read what it backs up. To do that it is registered in the customer's tenant as a service principal and granted Microsoft Graph and Exchange permissions that allow it to read mail, files and chat, and in many deployments to write them back during a restore. The grant is made once, by an administrator, during onboarding.

Huntress's investigation page, published 16 October 2026 and updated on 17 October (huntress.com/blog/vendor-credential-enables-unauthorized-access, captured 2026-10-18), listing the compromised application IDs and the containment steps for affected tenants.

That grant is not a session. It does not expire with a working day, it does not ask for approval when the application authenticates, and it does not depend on any person being present. The application authenticates with its own secret, held on the vendor's side, and the tenant accepts the request because the permissions were granted months or years earlier.

Every backup, ticketing, monitoring, HR or signature product onboarded this way carries the same property. The vendor's secret becomes a standing key to the customer's tenant, and the customer's only view of it is a row in the enterprise applications list, usually with the consent screen nobody re-reads.

How the intrusion reached over 300 tenants

The activity was visible to administrators before it was public. A user on Reddit reported a Microsoft Privileged Identity Management notification on 15 October that a SkyKick service principal had assigned itself the Exchange Administrator role, in a tenant that had not used the application for years. Microsoft later confirmed unauthorized authentication attempts on 15 October, and ConnectWise stated in its advisory that it had "recently identified unauthorized access to application identities within the ConnectWise Cloud Backup environment", then shut down application access for partners and clients.

Huntress, which monitors customer tenants, attributed the sign-ins to the legitimate Cloud Backup application ID 506e08f9-6e9a-4da4-942b-a099a03644ae and confirmed unauthorized sign-ins in 162 tenants as of 19:00 UTC on 16 October. Its update of 17 October raised the scale: almost 1,000 Entra tenants showed sign-in activity from the attacker's address, and over 300 of those succeeded. The Hacker News put the figure at 173 tenants on the same day. Both accounts trace the authentication to the address 144.172.64.128, and Huntress linked that address to scripted sign-in attempts on 13 and 14 October using the curl user agent 7.88.1, alongside python-urllib and python-requests agents.

Netics editorial diagram placing the 162-tenant count against the mechanism: an application secret presented directly to the tenant, with no user prompt in the path.

The escalation moved beyond the backup application. Huntress reported that service principals for the Cloud Management applications covering Exchange Online, Teams and Microsoft Graph assigned themselves directory roles, mostly Exchange Administrator, across nearly fifty of the tenants it monitors. In one partner tenant the Cloud Management - MS Teams application, ID 24205314-2b89-4e07-a2a4-579e898a5640, granted itself Global Administrator and then added new credentials to two other applications, including a single-tenant Huntress application. Todyl, analysing intrusions on nine tenants, found only sign-in activity and no evidence of created accounts, changed passwords or persistence through the backup app. Other responders saw persistence. Microsoft observed a backdoor after successful service principal authentication, followed by Microsoft Graph API reconnaissance and attempts to add additional credentials, and in the tenant where the Teams application became Global Administrator, Huntress recorded a Python script making nine changes to the two other applications, six of them successful, with the new secrets set to expire in 2028.

Why multi-factor authentication covered nothing here

Multi-factor authentication verifies a person. The sign-ins in this incident presented a client secret, so there was no person to verify and no prompt to approve. Todyl's account is explicit: the attacker authenticated directly to Microsoft 365 as the backup application using its secret, and because the requests did not involve a user login, protections such as MFA were ineffective.

The permissions already granted did the rest. Directory.ReadWrite.All, which Microsoft's threat intelligence assessed as one of the permissions likely abused, is a directory-wide write permission; in our reading, an application holding it reaches well beyond the mail and files a backup service exists to copy.

Conditional access policies that gate interactive sign-ins therefore leave this path open, and a policy written to require compliant devices for administrators does not touch an application token. The control that covers it is narrower and less comfortable: know which applications hold which grants, and rotate their secrets on a schedule that assumes the vendor's side will eventually be compromised.

ConnectWise's trust advisories page (connectwise.com/company/trust/advisories, captured 2026-10-18), carrying the Cloud Backup, formerly SkyKick, service interruption notice and the statement that application access was shut down for partners and clients.

The containment sequence the responders published

Huntress's guidance orders the work in a way worth copying. Affected organisations should disable two application IDs immediately, Cloud Backup 506e08f9-6e9a-4da4-942b-a099a03644ae and Cloud Management - MS Teams 24205314-2b89-4e07-a2a4-579e898a5640, and consider disabling the other Cloud Management IDs for the duration. The instruction is to disable the applications and keep them: deleting an application removes its consent, so service can only come back through re-consent, and disabling pauses backups while leaving the backups already stored in place.

Disabling stops new use and removes nothing that already happened. Huntress states the consequence plainly: disabling the applications does not remove credentials that were already added, so every credential on every Microsoft 365 application in the tenant has to be reviewed. That review is the part most likely to be skipped under pressure, and it is the part that decides whether the incident ends.

Re-enabling waits on the vendor. Huntress advises keeping the applications disabled until ConnectWise confirms that all credentials have been reset. On 17 October at 5:15 PM ET, ConnectWise communicated that it had disabled the relevant apps, and the vendor's advisory says it has no indicators to date that partner backup data was compromised and that the existing cloud backup archive is not impacted by the service interruption. Its update of 18 October at 01:00 UTC, reported by BleepingComputer, states that scheduled backups are not running and that access to affected backup archives and restoration capabilities remains unavailable, and adds that the existing archives remain encrypted and isolated, with no indication that they have been compromised. For a small organisation, that means no restore from this service until access returns.

A second wave is worth noting for anyone who reads the first one as solved. Huntress identified further attempts from the same address on 15 October against 16 of its protected tenants in the policy, national security, trade and critical mineral sectors, and most of those tenants did not use ConnectWise products, so the attempts failed because the application was not present. Huntress's update of 19 October adds the rest of the picture so far: across more than 10,000 tenants with ConnectWise applications, only 144.172.64.128 and Microsoft-internal ranges produced application sign-ins, while 144.172.64.99, 144.172.64.60 and 144.172.94.243 showed password spraying or failed logins. It found post-compromise changes in a second tenant, made around 03:45 UTC on 15 October like those in the first, placed the last successful sign-in at 08:44 UTC that day, and limited containment to tenants with confirmed unauthorized sign-ins, on the evidence that the activity had stopped.

What a small organisation can check in its own tenant

The first check is an inventory, and it is short. Open the enterprise applications list in the Microsoft 365 admin centre and export it, then for each third-party application look at the permissions it holds and the date consent was granted. Applications that no longer serve a purpose are the cheapest finding on the list, and the Reddit report of a service principal still active in a tenant that had stopped using the product years earlier is the reason to look.

Netics editorial checklist derived from Huntress's published guidance: disable rather than delete, review credentials on every application, rotate secrets, search sign-in logs for the source address, and re-enable only after the vendor confirms the reset.

The second check is the sign-in log. Searching for the address 144.172.64.128 and for the user agents python-urllib, python-requests and curl/7.88.1 over the exposure window costs a few minutes and gives a first answer for the tenant in question. Huntress warns that these user agents are low fidelity on their own and need the address beside them, and that the directory role assignments of 15 October carried kiota-dotnet, an agent ConnectWise's own applications also use. It also notes that Microsoft's audit logs record no source address for app-only actions and that directory reads through Microsoft Graph leave no audit trail, so an empty search proves little; Microsoft recommends reviewing application credential changes, OAuth grants and related Graph activity as well.

The third is secret hygiene on the applications that remain. A vendor application's credentials should have an owner, a rotation date and a record of who can read them on the vendor side, and none of the three is standard practice in small organisations. Where a provider offers certificate-based authentication with a client certificate stored in a managed vault, that is a materially better arrangement than a shared secret.

The fourth is a written answer to a question most vendors never hear: if your environment were compromised tomorrow, which of our credentials would the attacker hold, and how quickly would you tell us. The tenants opened in this incident, more than 300 by Huntress's count of 17 October, were reached through a service the customers had chosen to trust, and the trust was expressed as a permission grant they had not looked at since onboarding.

Netics runs security audit and hardening engagements that can include this inventory and the app-consent review that goes with it.

Originally published on the Netics blog.

Qubic's Wallet Drain Traces Back to Predictable Randomness

Taha Echakiri — Fri, 23 Oct 2026 09:00:02 +0000

TL;DR

Qubic is a layer-1 blockchain that runs fee-free transactions and smart contracts on a consensus mechanism called Useful Proof of Work, which redirects mining energy into training artificial intelligence models.
Between 23:42 UTC on 17 October and 01:26 UTC on 18 October 2026, an attacker drained 1,404 wallets of roughly 2.94 trillion QU, valued at about USD 1.4 million.
The wallets fell because their recovery seeds came from a generator with insufficient randomness. The project states the fault sat in its wallet software and that the chain protocol itself was not compromised.
Affected seeds were created in any browser on an iPhone or iPad, in Safari on a Mac until 18 October 2026, or in the Qubic iOS app before July 2025.
Qubic halted the chain at tick 85,371,949; 87.1 percent of the drained funds stayed on the network and froze in place, and exchange MEXC froze the account that received the other 12.9 percent within about an hour.
Web wallet version 4.26.4, dated 18 October, replaced JavaScript's Math.random() in seed creation with crypto.getRandomValues(). That fixes new seeds, and it leaves every seed already created in the old code as it was.

The 104-minute window and the wallets inside it

The attack ran for 104 minutes. Qubic's own statement puts the first drain at 23:42 UTC on 17 October and the last at 01:26 UTC on 18 October, and the count it published is 1,404 wallets holding roughly 2.94 trillion QU. Cointelegraph reports the haul at about USD 1.4 million.

Qubic's own network explorer (explorer.qubic.org/network, captured 2026-10-19), which reports the chain's current tick and the tick table behind it, the counter Qubic used to name the halt point. Qubic halted the chain at tick 85,371,949.

A rate of roughly thirteen wallets a minute is the detail worth sitting with. As Netics reads it, once a seed can be derived, the work per wallet collapses to a computation. Wallet count then measures the size of the vulnerable population.

Qubic is a layer-1 chain that carries transactions and smart contracts without fees, built on a consensus design called Useful Proof of Work that repurposes mining energy for AI training workloads. Qubic halted the network at tick 85,371,949.

A seed generator built for animation timers

Qubic's account of the cause is one sentence long and it is the whole story: seeds were generated with insufficient randomness. crypto.news reports that the fix in web wallet version 4.26.4 replaced JavaScript's Math.random() in the seed-creation routines with crypto.getRandomValues().

Those two calls look interchangeable in a code review, and their guarantees differ. Math.random() is built for convenience: it produces a number that looks scattered, quickly, and it is specified without a promise of unpredictability. MDN's JavaScript documentation warns that it is not cryptographically secure and should not be used for anything related to security. crypto.getRandomValues() is built for unpredictability. It draws from the platform's cryptographic generator.

The distinction is old and it keeps costing money. crypto.news ties this incident to a class of failure researchers call Milk Sad, a faulty key-generation process that lets an attacker predict private keys, and Cointelegraph places it in a line that runs through Coldcard's July disclosure of wallets weakened by a firmware bug and, in 2023, Trust Wallet's browser extension and the Libbitcoin Explorer tool. Weak-seed failures form a recurring class, and Qubic is the latest case in it.

Netics editorial diagram: the change in Qubic's web wallet release 4.26.4, where seed creation moved from Math.random() to crypto.getRandomValues(), the browser interface built for cryptographic values.

The three environments Qubic named

Qubic listed the affected seeds by where they were created: any browser on an iPhone or iPad, Safari on a Mac until 18 October 2026, and the Qubic iOS app before July 2025. That list is more specific than most incident notices and it says something about how the vulnerability behaved.

If the defect were purely in one function, every environment would be equally exposed and the list would be unnecessary. The list implies a dependency on the platform underneath: the same call may have produced output of different quality depending on the engine that implemented it, and every entry on the list is an Apple platform. This reading is Netics analysis, and no source confirms it. Whether the underlying weakness came from the generator's seeding on those platforms or from the wallet's own use of it, the practical lesson holds. A key generator inherits the entropy of the runtime it runs in, and a browser is the runtime with the least control over that.

For a user, this is the uncomfortable part: the notice does not say which seeds are safe. Consulting the release notes tells you that new seeds are generated correctly. Nothing in the published material so far establishes that seeds created in the old code became safe once the code changed, because a seed is stored data and a code change does not rewrite it. Rotation is the only remedy that does not depend on an assumption.

What a validator halt can and cannot undo

Qubic stopped the chain, 87.1 percent of the stolen tokens stayed where they were and froze, and MEXC froze the account holding the remaining 12.9 percent in about an hour. That is an unusually complete containment for a crypto incident, and it is worth being precise about why it worked.

It worked because the asset never became bearer cash. Theft on a chain is a state change that the network agrees on, so a coordinated halt converts a completed theft into a pending one, and a centralised exchange converts an off-chain exit into a frozen account. Both moves need something that a bearer asset does not have: an authority that can act after the fact. Qubic has a way to halt its chain, and MEXC was able to freeze the receiving account.

The trade-off is on the record too. A chain that can be halted to protect funds can be halted for reasons its holders do not endorse, and the recovery decision now sits with a vote of the Computors. Qubic says a full technical report follows once a recovery plan is settled. Anyone treating this containment as a property of the asset class should read that report before generalising it.

Netics editorial diagram: the published numbers behind the Qubic drain, including the share of funds that stayed on the network and the share that reached an exchange.

What to check before you trust a key generator

The useful question for anyone who ships software that creates keys, seeds or recovery phrases concerns three things: which function produces the value, in which runtime it runs, and whether the answer stays unpredictable to someone who can read the code.

Four checks catch this class of bug before an incident report does:

Name the API in the code review. A seed that comes from Math.random(), a language's default shuffle, a timestamp, or a hash of the device name is predictable by construction, whatever entropy the surrounding system claims to have.
Check the source on every platform. The same call can behave differently across browser engines and OS versions, which is exactly what Qubic's three-environment list describes. Confirm that crypto.getRandomValues() is the source on each target runtime. A distribution test cannot prove that a generator is unpredictable: Math.random() can pass one while staying predictable to anyone who models its internal state.
Treat a generator fix as a rotation event. Patching the code protects future seeds. It says nothing about the keys already written down, exported or stored, and those are the ones an attacker can work on at leisure.
Write down the blast radius before the incident. Qubic could contain 87.1 percent because a halt was available and an exchange answered in an hour. Most systems do not, which makes early detection the only containment they own.

For teams in France and Morocco building products on top of key material, the same four checks map onto a procurement question that is easy to ask a supplier: which API generates our keys, on which runtimes, and how do we know the output is not reproducible. A vendor that cannot answer that in writing is asking you to take on the entropy risk without a description of it. We build and review exactly that kind of boundary in our security work, and the first output of the exercise is usually the answer to a question nobody had asked before.

Qubic's own wallet release notes (github.com/qubic/wallet/releases, captured 2026-10-19), which list version 4.26.4 dated 18 October 2026 and the fix line naming cryptographically secure seed generation: the change that protects future seeds and leaves every earlier one untouched.

Originally published on the Netics blog.

DeepSeek Just Published the Blueprint for Training Agents at Scale

Taha Echakiri — Thu, 22 Oct 2026 14:00:39 +0000

TL;DR

The headline claims are production measurements: about three million sandboxes a day, over 380,000 concurrent, more than 5,000 creations per second, from a scale unit of about 160 nodes.
Four backends behind one libdsec SDK — FnCall, container, microVM, full VM — with different isolation and startup cost.
Most agent sandboxes are nearly idle — about 90% draw no more than 5% of requested CPU — while memory stays pinned and images see low fanout.
Density comes from EROFS layers loaded on demand from 3FS, virtio-pmem DAX for read-only layers, and DAMON balloon free-page reporting.
RL co-design keeps rollouts alive across GPU preemption: pause/resume, agent-built environments, access controls hardened against agents that try to cheat.
The honest costs: heavier VM backends, persistent workspaces, and a residual latency toll — treat the sandbox tier as a platform decision, not glue code.

The 160-node sandbox factory behind the report

On September 19, 2026, DeepSeek published its DSec technical report on arXiv, documenting the production platform that has carried agentic training and evaluation from DeepSeek V3.2 through V4.1. The abstract: supporting such workloads "requires an elastic execution platform rather than a single sandbox runtime".

Our stance is stronger: training agents at scale is an infrastructure problem first, and DSec is the most complete public blueprint yet — elastic backends, on-demand image delivery, high-density memory management, RL co-design.

Netics editorial visual: DSec production scale — about 3 million sandboxes a day, 380,000 concurrent, over 5,000 creations per second, per the DeepSeek report.

Three million short-lived sandboxes a day

One scale unit spans roughly 160 nodes and serves about three million sandboxes a day — over 380,000 concurrent, over 5,000 creations per second — with single jobs requesting up to 32,000 instances in a burst. Median lifetimes: 17.4 minutes for containers, 15.5 for microVMs, p99 over three hours. Per-node peaks: 1,048 containers and 524 microVMs; stable operation runs 3,200 containers or 800 microVMs per node.

Four isolation backends behind one libdsec client

DSec exposes four backends through one Python SDK, libdsec: FnCall for short, stateless tasks in precreated reusable containers; containers for the bulk of engineering workloads; Firecracker microVMs where isolation matters more; full VMs for complete operating systems. Each container or VM sandbox runs a per-sandbox proxy (aether) with chronus instances for commands and filesystem access; FnCall skips the proxy. Network policy is per sandbox too: the documented example allows PyPI but denies NPM.

Official DeepSeek DSec report, Figure 1: DSec architecture — per-sandbox proxies (aether) mediate between container/VM sandboxes and the platform; FnCall follows a separate path. (arXiv:2609.22978, 2026-09-19)

Pinned memory, idle CPUs, and images nobody reuses

The naive just-run-a-container approach dies on this profile. While an agent waits for the model's next action, a sandbox's "CPU usage is sparse and naturally suitable for overcommit" — about 90% of sandboxes use no more than 5% of requested CPU. Yet sandboxes are stateful: memory, page caches, and writable state stay pinned after the CPU goes idle, so resident cost caps capacity. Images add a second constraint — containers show a median fanout of three, microVMs a median fanout of one — and eager pulling concentrates load on the distribution path: the report's ablation shows eager pulling stretches completion time by 1.7× while on-demand loading cuts disk writes by 57%.

EROFS layers, on-demand reads, and a 21.2% memory reclaim

Environments compose into independently versioned read-only EROFS layers, served block-by-block on demand instead of unpacked from tarballs. Image data lives in 3FS, fetched only when accessed and in bulk, while writes stay on local disk; a de-duplicated EROFS set totaling 30 TB covers the files accessed by 70% of container tasks. Memory density gets two mechanisms: virtio-pmem with DAX removes page-cache duplication for read-only layers, and DAMON with virtio-balloon free-page reporting, which the evaluation says "reduces memory consumption by 21.2% without significant CPU overhead". For CPU, sandboxes are latency-sensitive (LS) or best-effort (BE); BE sits under SCHED_IDLE with core scheduling, cutting SMT latency inflation from 45.2% to 17.3%.

Netics editorial visual: the four DSec sandbox backends — FnCall, container, microVM, full-VM — behind the libdsec SDK, with per-sandbox network rules.

Rollouts that survive GPU preemption, agents that try to cheat

DSec "decouples stateful rollout execution from preemptible GPU training". Earlier generations ran the agent loop inside the preemptible GPU pod and recovered via command-log replay; since V4.1, rollout execution lives on DSec — a worker container plus an agent sandbox, both outside the GPU pool — as the single source of truth. On preemption, the framework pauses sandboxes — containers freeze process trees and swap memory; microVMs snapshot, terminate, restore — and rollouts resume. Environments follow the heading "Build environments of Agents, by Agents, for Agents": agents checkpoint sessions via pack_diff, turning incremental snapshots into reusable environments with no separate image pipeline.

The report is equally blunt about untrusted agents: they forge requests to chronus sockets and overwrite /bin/bash; after file and socket access controls landed, "an agent attempted to bypass them using XFS_IOC_SWAPEXT" — an ioctl swapping the data-extent mappings of two files — and the attempt corrupted XFS metadata and forced a filesystem shutdown. Access control is a live security problem, the principle behind our zero-trust boundaries outside the model piece.

Official DSec report, Figure 9: overview of the core system mechanisms — composable layers, high-density resource management, image distribution, and RL co-design (arXiv:2609.22978).

The honest costs of a sandbox factory

The inconvenient parts are in the source, so we state them. Full-VM and microVM backends are heavier to operate, and microVM workspaces persist as task-specific disk images — 53,590 of them in one production week, on top of just two shared base images.

QoS is not free: core scheduling limits SMT-induced latency inflation to 17.3% at 50% best-effort load, yet the report concedes residual degradation from turbo reductions plus "memory bandwidth and shared last-level cache (LLC) contention that core scheduling does not address". The 21.2% reclaim is a measured win, not a zero: resident costs remain, and every mechanism adds surface — placement, admission control, proxies, snapshot plumbing.

Treat the sandbox tier as a platform decision

Four decisions follow for any team running agentic training at real scale: pick an isolation tier per workload, deliver images on demand from shared storage, set an explicit overcommit policy — measurement, reclamation, QoS classes — and co-design the training loop with the sandbox layer so preemption stops destroying rollout state.

Netics editorial visual: what the sandbox-factory approach buys — on-demand image layers, about 21.2% of memory reclaimed by DAMON, RL co-design — and what it costs in operating complexity.

Run the exercise on your own fleet: list the environments, measure the fanout, count the minutes sandboxes actually burn CPU. If your stack still builds a fresh environment per task and pulls the full image first, DSec shows the ceiling. The harness half is being productized fast — our breakdown of OpenAI's Agents API covers it — and the execution tier is now public knowledge too. More analysis lands on neticslabs.com.

Originally published on the Netics blog.

AlmaLinux Hands the Certification Test Kit to Its Users — and Publishes the Results

Taha Echakiri — Thu, 22 Oct 2026 14:00:02 +0000

TL;DR

On September 24, 2026, the AlmaLinux OS Foundation launched software certification for third-party products: results land in the public catalogue at catalog.almalinux.org, and every tester is named.
The alma-certify suite was fully rewritten as a Python tool that runs on the machine under certification, ships as an RPM in the extras repository, and finishes most runs in under 10 minutes.
A catalogue listing is a claim by whoever ran the test — vendor, Foundation, or community member — and validations stack, tracked per AlmaLinux major version.
Compatibility certification is not a security certification: FIPS 140-3 validated modules and a Common Criteria certification for AlmaLinux OS 9.2 are separate paths.
For procurement, trusting a Linux compatibility listing now has a concrete checklist: who tested, against which release, and whether the suite can be re-run locally.
The shift inverts the old vendor-administered certification bottleneck

The vetting bottleneck just changed hands

On September 24, 2026, the AlmaLinux OS Foundation announced software certification for third-party products, and the announcement opens with the argument in full: "Software certification is here, and hardware certification takes under 10 minutes." The mechanics behind that line — the test kit, the catalogue, the named tester — matter more than the headline.

Official AlmaLinux Certification Catalog (catalog.almalinux.org), captured 2026-09-28 — community-validated, AlmaLinux-validated, and vendor-validated listings side by side.

Certification in the Linux ecosystem has historically been a vendor-administered bottleneck: paid programmes, non-disclosure agreements, and closed test cycles decided who earned the badge. The AlmaLinux certification page says plainly this is the problem the programme exists to remove: "Certifying in the Linux ecosystem is too often a tedious, painful process." The response: the test kit is public, the results are public, the tester is named.

The sections below cover what the programme does, what a listing is and is not, and the questions procurement engineers should carry into the next compatibility check.

One catalogue, and every tester named

All of this work lands in the public catalogue at catalog.almalinux.org, covering systems, components, software, benchmarks, and validations — with attribution in one sentence: "Every listing says who did the testing: the vendor, the AlmaLinux OS Foundation, or a community member."

The results are cumulative: "Those validations stack and complement each other, and they're tracked per AlmaLinux major version." Stacking puts a community confirmation alongside a Foundation-validated result. Per-major-version tracking means a 9.x listing does not carry over to 10.x — every major upgrade resets the question.

alma-certify: the test kit runs on the machine it tests

The engine is a full rewrite — "It replaces the Ansible-based suite that we used to provide" — running as a Python tool "directly on the machine being certified, and produces a report the catalog can ingest", delivered as a single package: "It ships as an RPM in the extras repository." Nothing is hidden: "As always, it's fully open source."

The open-source alma-certify suite, fully rewritten as a Python certification tool that runs on the machine being certified. (github.com/AlmaLinux/alma-certify, captured 2026-09-28)

Speed is a stated goal: "for the overwhelming majority of systems a certification run now finishes in under 10 minutes." The vendor loop is short — "you just list your product, cite the releases it supports, and submit" — and the community route shorter: "Just sign in to catalog.almalinux.org and confirm that a product works on the release you're running."

A named tester beats a logo — and a listing stays a claim

What the catalogue does not do is certify on anyone's authority but the tester's own. A community confirmation is a 'works for me' report: useful evidence, not a guarantee. The model surfaces disagreement instead of burying it — two listings for the same product, one green and one not, are information rather than a bug.

That is the distance from a vendor-run programme: the old style resolved conflicts inside a closed process; this one publishes both claims and lets the next tester — or the customer — re-run the suite and decide. Reproducibility is the audit: point the tool at the machine you actually run.

Netics editorial visual: the certification journey — list the product, run alma-certify on the machine, community confirmation, published in the catalogue with the tester named.

Compatibility is not security: FIPS and Common Criteria stay outside the catalogue

The most important sentence in the programme's own documentation is the shortest: "Certification here is a statement about compatibility." It means AlmaLinux runs correctly on a piece of hardware, or software runs correctly on AlmaLinux — nothing beyond that. The page is explicit: "This is not a regulatory or security certification."

AlmaLinux OS 9.2 carries those assurances separately: "FIPS 140-3 validated cryptographic modules and a Common Criteria certification." Separate programmes, separate evidence chains — no catalogue listing substitutes for either. If a vendor's slide blends 'certified to run on AlmaLinux' into 'already FIPS-ready', the correction is above.

What procurement should check before trusting a listing

For an enterprise engineer the catalogue turns a compatibility claim into an inspection checklist, in order: who ran the test; which major version — a 9.x listing says nothing about 10.x; which tool version produced the report; and whether the suite can be re-run locally, so a modest lab can repeat the test instead of taking the listing on faith.

Netics editorial visual: three ways a product becomes 'certified' — the closed vendor queue, the public community catalogue, and the separate FIPS 140-3 / Common Criteria assurance paths.

This is release change control applied to compatibility: a claim about a specific release needs evidence pinned to that release. Run the suite against the machine you operate, not the marketing slide.

The bar for enterprise Linux just moved

The quiet consequence is competitive. A distribution's certification programme used to be a moat: the vendor controlled who got tested, at what price, under what terms. AlmaLinux dismantled the moat by publishing the tooling and the results, and by treating community validation as first-class evidence that stacks with vendor and Foundation results.

The bar for every distribution claiming enterprise readiness is higher: with an open test kit, public results, and a named tester, a competing claim that cannot say who tested what against which release looks like an assertion without provenance. For EU procurement the catalogue maps onto the supply-chain scrutiny that NIS2-style diligence already demands — without ever claiming to replace FIPS or Common Criteria assurance.

Netics editorial visual: what to check before trusting a Linux compatibility listing — who tested, which major version, re-running the suite, and the boundary between compatibility and security certification.

The philosophy echoes the ownership and change-control debates we covered in Linux ownership governance and Ubuntu 26.10's feature freeze: governance in the open beats governance in a private queue. The kit is in users' hands; the honest move is to run it. New infrastructure analysis lands on neticslabs.com.

Originally published on the Netics blog.

Google Routes Inference on KV-Cache Pressure: The Multi-Cluster GKE Gateway, Measured

Taha Echakiri — Thu, 22 Oct 2026 09:00:02 +0000

TL;DR

On September 21, 2026, Google benchmarked the multi-cluster GKE Inference Gateway at 17,000 compute nodes across the US and Europe, behind one global virtual IP.
The routing signal reads memory, not packets: the EPP exposes KV-cache utilization, and past a 40% threshold traffic spills to the next healthy region automatically.
Measured cost: under 1% overhead at 99.5% of direct local throughput, with near-linear scaling 0.72 → 2.10 req/s across one to three clusters at 99.9% success.
Round-robin treats every request as equal; LLM requests are not. Agentic 100k to 800k+ token contexts exhaust memory before compute — routing must read KV-cache pressure.
Caveats: Google's own benchmark, one MoE model (SGLang) on Google clusters, feature in preview since March 2026 — evidence, not a launch, no SLA.
The reusable model: config cluster outside the request path, KV-cache utilization first-class, LeaderWorkerSet-aware routing to leader pods; claims tagged in the manifest below.

On September 21, 2026, the Google Cloud blog published Global AI routing with <1% overhead on multi-cluster GKE Inference Gateway, signed by GKE product manager Fisayo Feyisetan and software engineer Sina Chavoshi. The headline numbers — 17,000 compute nodes, near-linear scaling across three regions, under 1% routing overhead — deserve attention, but what the gateway routes on matters more. It reads how full each region's KV cache is and sends each request where memory remains, and that choice separates a network load balancer from an inference routing plane.

Official Google Cloud blog figure (2026-09-21): multi-cluster GKE Inference Gateway topology — the config cluster holds routing configuration but sits outside the request path; each target cluster runs its own EPP and reports KV-cache utilization back to the load balancer.

The routing plane is where inference economics are won

Round-robin at the network layer treats every request as equal: "Round-robin load balancing is inadequate for serving LLMs because it treats every request as equal," as the post states. Heavy prompts saturate GPU compute cores, long generations stress memory bandwidth, and long-context conversations eat VRAM until the engine can no longer schedule anything new. Agentic workloads with 100k to 800k+ token contexts consume memory faster than any previous AI traffic, and a routing layer blind to that pressure strands expensive compute.

Our coverage has been building toward this layer: NVIDIA Topograph topology scheduling places work on devices inside a node, while the gateway sits above it, deciding which region takes the request — chasing the same objective as any scheduler: maximize "intelligence per dollar" instead of letting capital idle while requests queue elsewhere.

Three regions, one virtual IP — and a telemetry signal underneath

The deployment spans three clusters: us-east5 as the config cluster, with us-west8 and europe-west4 serving. Clients see none of that geography — requests hit a single global virtual IP, and the gateway decides in real time which cluster serves each one.

Official Google Cloud blog figure (2026-09-21): the measured trade-off — routing through the gateway added less than 1% overhead, delivering 99.5% of a direct local cluster call.

The decision engine is the Endpoint Picker Proxy. The EPP reads the KV-cache token utilization natively exposed by inference engines and emits it as a load-balancer metric; when a region runs hot on that signal, Google writes, "it spills traffic to the next healthy region." The threshold is a parameter you choose — this deployment crossed at 40% KV-cache utilization. Routing can also key off queue depth or running concurrency, while the config cluster holds the routing configuration outside the request path.

The measured trade-off: under 1% overhead for a global tier

Every team evaluating a global routing tier asks the question the post poses: "How much throughput am I giving up for cross-region capability?" Measured answer: routing through the gateway added less than 1% overhead, delivering 99.5% of the throughput of a direct, local cluster call. The number deserves respect and a caveat in the same sentence: it was produced on Google's own deployment, one leading MoE foundation model served with SGLang. Under 1% is real evidence that a global entry point can be nearly free, not a promise for every fleet, model mix, or accelerator. The post names the signal, the threshold, and the measurement, so the result is reproducible on your own hardware.

Near-linear scaling at 17,000 nodes

The harder test is scale. Every client request originated from us-east5, so added clusters sat at maximum distance from the traffic source; if the gateway could not distribute load across that distance, throughput would flatten as hardware grew. Instead, scaling held near-linear:

Official Google Cloud blog figure (2026-09-21): near-linear throughput growth from one to three clusters (0.72 → 2.10 req/s) at a 99.9% success rate under heavy multi-client concurrency.

Fleet topology	Request throughput	Token throughput	Success rate
1 cluster (us-east5-a)	0.72 req/s	2,898 tok/s	99.87%
2 clusters (+ us-west8-a)	1.40 req/s	6,380 tok/s	99.95%
3 clusters (+ europe-west4-b)	2.10 req/s	8,457 tok/s	99.90%

Google reports "a near-linear throughput boost while maintaining a 99.9% success rate under heavy multi-client concurrency." Read it precisely: 99.9% is the success rate observed in this test, not a general service-level agreement.

Why round-robin fails when memory is the bottleneck

Round-robin did not become wrong; it became insufficient for the workloads dominating accelerator fleets. Below memory saturation, network-layer round-robin remains perfectly adequate — an honest counterpoint worth stating before teams over-engineer. Agentic traffic exhausts memory long before compute: LLM requests run for minutes and hold VRAM for their duration, until the engine can no longer schedule anything new.

The mechanics are concrete: as the primary region climbed toward its high-bandwidth memory limits, the gateway detected saturation when the cluster crossed its 40% KV-cache utilization threshold and routed the overflow onward. "No operator intervention was needed." The fleet-wide lesson stands apart from any gateway: plan connection limits and timeouts for AI-scale latency early, or expect aborted connections.

Netics editorial visual: the round-robin assumption versus KV-cache-aware routing — reading memory pressure (40% threshold spillover) instead of treating every request as equal.

Copy the operational model: config cluster, leader pods, telemetry

Distributed inference engines under tensor parallelism serve only from the master; the post puts it plainly: "only the master (rank-0) pod serves the API." GKE already directs local traffic to leader pods with standard Service selectors and LeaderWorkerSet (LWS), and the multi-cluster gateway integrates with that foundation — routing global traffic to the correct regional services so cross-region balancing respects multi-node topologies out of the box. Building custom proxies to replicate that is the work Google argues against.

The March 17, 2026 preview announcement names the resources: InferencePool groups pods sharing hardware and model configuration; InferenceObjective names models and sets serving priorities; GCPBackendPolicy on a GCPInferencePoolImport configures CUSTOM_METRICS or in-flight request limits. For teams standardizing on native Kubernetes constructs, this is the pattern worth copying — the same principle behind our read of Kubernetes 1.37 container storage hardening: keep the control plane small, declarative, and portable while data paths grow.

What Google's benchmark proves — and what it leaves open

Vendor benchmarks are necessary and insufficient, and this one is honest about its shape. The numbers are Google's, at 17,000 nodes, serving one leading MoE model with SGLang. They prove the architecture can behave like one fleet at scale; they do not prove those numbers transfer to heterogeneous fleets running multiple model families on mixed GPU and TPU generations — the post claims runtime-, model-, and accelerator-agnostic architecture while demonstrating a single workload.

Netics editorial visual: network-layer round-robin versus the multi-cluster Inference Gateway — routing signal, scope, measured overhead, and saturation behavior.

Two dates help. The gateway entered preview on March 17, 2026 with model-aware routing already promised; September 21 is a benchmark of that preview, not a launch. The 99.9% success rate is a test outcome, not a contractual SLA, and the 40% threshold is a parameter you choose. The useful takeaway is the instrument list — KV-cache utilization first-class, queue depth, in-flight requests — because memory-aware routing is a property of telemetry quality, and telemetry is a property of your stack. Measure on your own fleet before promising anyone else's numbers. New infrastructure analysis lands on neticslabs.com.

Originally published on the Netics blog.

The Controller That Runs Your AI Data Center Had Hard-Coded Credentials

Taha Echakiri — Wed, 21 Oct 2026 19:00:03 +0000

TL;DR

On September 22, 2026, NVIDIA disclosed CVE-2026-65113, a hard-coded-credentials flaw (CWE-798) in the NVIDIA Infrastructure Controller (NICo) for Linux, rated 9.8 out of 10 on the CVSS v3.1 scale and present in every version from 0 through 1.9.
NICo is not an edge utility: it is the Kubernetes-based control plane that automates bare-metal lifecycle management for AI infrastructure — hardware discovery, firmware validation, DPU provisioning, OS deployment, network isolation, and tenant sanitization, across rack-scale systems including GB200 and GB300.
The advisory is a 14-CVE batch: one critical (the hard-coded credentials), five high (including SQL injection at 8.8, missing authentication at 8.3, improper authentication at 8.2, OS command injection at 8.0), and eight medium. The pattern across all fourteen is a management plane that accumulated authentication and trust-validation weaknesses; no in-the-wild exploitation was confirmed at disclosure, and the exploit-prediction score was low.
The fix is NICo version 2.0 from the official repository. The operational response goes beyond upgrading: isolate the management plane from untrusted networks and audit any hard-coded or default credentials before you trust the controller again.

Why this CVE matters more than its CVSS number

A hard-coded credential in a random web app is a bad day. A hard-coded credential in the controller that provisions every server, DPU, and tenant network in an AI data center is a different class of problem, because of what the controller is allowed to do.

NICo's own documentation describes it as delivering "zero-touch lifecycle automation for bare-metal systems that secures datacenter infrastructure at its foundation." The architecture is a Kubernetes-based control plane with a REST API, site agents, and Temporal workflows, deployed on the bare-metal cluster it manages. It handles hardware discovery, firmware validation, DPU provisioning through the DOCA Platform Framework, OS deployment, network isolation, and tenant sanitization. It is the single authority that prepares a bare-metal machine to become part of a multi-tenant AI cloud.

Official NVIDIA Product Security page (nvidia.com/en-us/product-security/, captured 2026-09-30): the bulletin listing shows the September 2026 NVIDIA Infrastructure Controller advisory (5879), rated Critical, with CVE-2026-65113 first in the list of fourteen.

That centrality is exactly what makes CVE-2026-65113 dangerous. The flaw allows an unauthenticated remote attacker to bypass authentication using the static credentials; NVIDIA's advisory describes a successful exploit as potentially leading to escalation of privileges, data tampering, denial of service, and information disclosure. Applied to NICo, that is not one server: it is the plane from which an attacker can reach every tenant's bare metal, every DPU, and the network isolation the controller is supposed to enforce. NVIDIA's own security guidance in the same disclosure cycle is blunt about the class: restrict access to administrative interfaces and isolate management-plane services from untrusted networks.

Fourteen problems, one pattern

The hard-coded credentials are the headline, but they are not the whole story. CVE-2026-65128 is a SQL injection rated 8.8 that low-privilege attackers can use to reach code execution. CVE-2026-65114 is missing authentication on a critical function, rated 8.3. CVE-2026-65121 is improper authentication enabling privilege escalation, at 8.2. CVE-2026-65130 is OS command injection at 8.0. Add improper certificate validation (7.5 and 6.7), external file/path control, XML injection, workflow-enforcement failures, a separate hard-coded password, uncontrolled resource consumption, and uncleared debug information, and the picture is consistent: a management plane that accumulated authentication and trust-validation weaknesses across its release history, disclosed together because they were only reviewed together.

Official NVIDIA Infra Controller documentation (docs.nvidia.com/infra-controller, captured 2026-09-30): "What is NICo?" — the zero-touch bare-metal lifecycle management controller that the September bulletin's fourteen CVEs address.

The pattern is worth naming: as AI infrastructure consolidated into GPU clouds and large on-prem clusters, the management plane became the crown jewel, and its security did not scale with its privileges. Cloud partners and GPU-as-a-service providers run NICo precisely because it automates provisioning of DPU-equipped racks; that automation is valuable, and it is valuable to an attacker in exactly the same proportion. NVIDIA's documentation lists Vault and external-secrets as part of the reference deployment, which makes the hard-coded credential all the more conspicuous: the surrounding stack attempts to manage secrets properly while the controller itself ships with one baked in.

The operational response

The direct fix is mechanical: upgrade to NICo version 2.0, or clone the latest release from the project's official GitHub repository. That is the precondition, not the whole answer.

Official NICo source repository (github.com/dsx-ai-factory/infra-controller, captured 2026-09-30): the open-source home of the NVIDIA Infra Controller, moved from the NVIDIA organization on September 4, 2026; version 2.0 is the fixed release for bulletin 5879.

First, treat the management plane as what it is — the most privileged network in the estate. NICo, like other infrastructure controllers, should be reachable only from a dedicated management VLAN or jump-host path for operators, never from tenant networks or the internet. The DPUs it provisions are not controllable by tenant host operating systems, which means the controller is the critical point of failure for the hardware-level security model: if the controller goes, so does the isolation story. Second, audit for credentials before re-trusting the upgrade: scan for hard-coded passwords and default accounts across the controller configuration, and rotate anything that looks inherited. Third, if you are a multi-tenant operator, financial or otherwise — this advisory is aimed squarely at AI-cloud and GPU-as-a-service providers — write the management-plane isolation into your change plan, not just the version bump.

Netics editorial diagram grounded in bulletin 5879 guidance: the management plane before the fix — NICo 0–1.9 reachable from tenant networks with hard-coded credentials — and after — NICo 2.0 on a dedicated management VLAN with a completed credential audit.

This is the same lesson the BlueField-4 scale-in analysis drew from a different direction: when control leaves the host and moves into infrastructure hardware, the control plane becomes the trust boundary, and every operator needs to know exactly what that plane can do. Here the plane can provision every tenant's hardware, and until this advisory it answered to a hard-coded credential. Upgrade, isolate, audit — in that order, and before the next batch of GPU capacity goes online.

Originally published on the Netics blog.

Perplexity Red-Teamed Its Agent Sandbox: The VM Held, the Network Didn't

Taha Echakiri — Wed, 21 Oct 2026 14:01:00 +0000

TL;DR

On September 23, 2026, Perplexity's Secure Intelligence Institute published "Escaping SPACE: Part I," a red-team report on SPACE, the microVM sandbox platform behind Perplexity Computer. Nine models received root access inside a guest VM with the explicit objective of crossing its boundaries.
No VM-to-host escape was observed in 108 runs — even with access to SPACE's source code, web search, and software package installation. That boundary held.
The network boundary did not. With limited network access, four of the nine models (Claude Opus 5.0, GPT-5.6 Cyber, GPT-5.6 Sol, Kimi K3) bypassed the egress policy in 11 of 54 partial-network runs, via DNS spoofing or shared-IP routing.
The problem is broader than one vendor: eight of ten third-party sandbox platforms tested showed at least one network-policy bypass. Only Cloudflare Sandbox and NVIDIA OpenShell held in these experiments.
The structural lesson for anyone running agent workloads: treat "sandboxed" as at least two separate claims — isolation of compute, and confinement of network — and test them separately.
The operational change that follows: audit the two boundaries independently, run your own partial-network red team, prefer gateways that terminate TLS and validate SNI against a DNS-attributed allowlist, and assume shared-IP bypasses are permanent.

What the red team actually tested

SPACE is built on Firecracker microVMs. Each sandbox is a guest VM on a host the guest does not control, and a per-sandbox gateway enforces egress policy on everything the guest sends out. The report evaluates two boundaries that are easy to conflate: the VM-host boundary (can an agent that already has root inside the guest reach the host?), and the network boundary (can an agent reach a destination the egress policy blocks?).

The experiments ran nine models as agents under four settings defined by two dimensions: whether the agent had access to SPACE's source code, and whether network access was fully blocked or partially allowed (package repositories plus the Perplexity Search API). The sandbox-escape task asked agents to obtain an AWS honeytoken stored on the host; the network-bypass task asked them to retrieve a flag from a blocked callback URL.

Official Perplexity figure from "Escaping SPACE: Part I" (perplexity.ai, 2026-09-23): the report's own summary of the two boundaries — VM-host isolation with zero verified escapes in 108 runs, and network-policy bypasses under partial network access.

The design matters because it isolates the failure modes. Giving an agent root inside a guest VM is the worst-case assumption for compute isolation: if the hypervisor boundary is sound, root inside the guest should still mean nothing outside it. Giving the agent partial network access is the realistic case for production agents, which almost always need package registries, APIs, or retrieval services to do useful work. A partial-network setting is therefore the setting that should be tested: it is where an agent still does useful work, and it is where policy mistakes surface — a fully blocked network proves nothing about egress controls, and full access tests nothing at all. The two tasks fail in very different ways, and the report's phrasing is exact: "A successful network-policy bypass does not imply a VM–host escape, and the absence of an observed escape is not a proof of isolation."

Why the egress policy failed

Every successful bypass exploited the same structural gap: the policy was expressed in domain names, but enforcement happened against IP addresses. Perplexity describes two concrete methods.

Official Perplexity figure from "Escaping SPACE: Part I" (perplexity.ai, 2026-09-23): bypass method 1 — the agent injects a forged DNS response that maps the allowed pypi.org name to the blocked callback's IP address, so the gateway's name-to-IP policy check is satisfied by a poisoned mapping.

In the DNS-spoofing variant, the agent injected a forged DNS response impersonating the external resolver, mapping the allowed hostname pypi.org to the blocked callback's IP. The gateway then saw a connection to an allowed name, en route to an address it had not decided to block. That is a policy-implementation failure specific to SPACE — the report says it was addressed with source-address validation — and it is the easiest of the two to fix.

The second method is more uncomfortable because it is structural. A single IP address serves many domains: content delivery networks host thousands of unrelated customer domains behind shared IP pools, and Fastly's Fiddle service exists precisely to make server-side fetches on behalf of arbitrary URLs. One run sent a request to fiddle.fastly.dev instructing the service to fetch the blocked callback URL. GPT-5.6 Cyber found another path through images.taboola.com, an image-fetching service reachable through an IP the policy already allowed, chained to thum.io for a screenshot, then recovered the flag from the PNG with OCR. Every hop in that chain went through an IP the gateway already allowed, so the egress policy never saw the final destination — the name-based allowlist was satisfied the whole way, which is exactly why fix belongs at the gateway, not inside the guest.

Official Perplexity figure from "Escaping SPACE: Part I" (perplexity.ai, 2026-09-23): validation results across third-party sandbox platforms — at least one network-policy bypass was demonstrated in eight of the ten platforms tested, and neither bypass succeeded against Cloudflare Sandbox or NVIDIA OpenShell.

Perplexity's own conclusion is the transferable part: "Relying only on the destination IP address at the host to determine whether a connection is allowed is insufficient." Any operator building an egress allowlist — whether for an agent sandbox, a CI runner, or a microservice — is building the same failure mode if the check happens after DNS resolution and trusts the resolver.

Eight of ten platforms failed the same test

Perplexity did not stop at its own sandbox. It ran direct HTTP/HTTPS tests against ten third-party sandbox platforms that advertise hostname- or domain-based egress controls, testing whether authorizing one hostname could authorize a request for a different hostname served by the same CDN or multi-tenant infrastructure. At least one network-policy bypass was found in eight of the ten platforms. The details matter: Deno Sandbox enforced HTTPS policy but allowed arbitrary TCP to unlisted addresses, and several vendors shipped mitigations quickly after disclosure (microsandbox, Daytona, Deno); others documented known limitations rather than reclassifying the behavior. The two platforms where neither tested bypass succeeded were Cloudflare Sandbox and NVIDIA OpenShell.

The vendor-name range is the news: E2B, Vercel, Modal, Daytona, Fly.io, microsandbox, Deno, and others are all mainstream execution surfaces for agent frameworks. An agent whose egress "policy" can be redirected through a shared CDN edge is not actually confined — it is billable. This connects directly to the control-plane argument we made when Cloudflare placed Cursor agents in customer-controlled sandboxes: the sandbox is only as good as the controls the customer can actually verify, and egress is where most platforms still trust the guest's view of the network.

What operators should change

First, stop auditing whether the sandbox is secure as one question. The report's core finding is that compute isolation and network confinement are independent properties with different failure rates — and in this test, very different: 0/108 versus 11/54. Second, test your own platform the way Perplexity tested its own: give an agent partial network access, point it at a blocked destination, and see whether a shared CDN edge or a poisoned DNS answer gets it out. Third, prefer platforms that terminate TLS at the gateway and validate the SNI and HTTP hostname against a DNS-attributed allowlist — that design closes both bypass classes at once, and it is the fix Perplexity shipped. Finally, assume the IP-sharing problem is permanent: it is a property of today's Internet architecture, not of any single vendor, so an egress control that does not look past IP addresses will keep failing regardless of which sandbox product you buy.

Netics editorial diagram grounded in the Perplexity report: eight of ten third-party sandbox platforms showed at least one network-policy bypass, four of nine models bypassed egress in partial-network runs, and only Cloudflare Sandbox and NVIDIA OpenShell held.

The pattern is the same one we flagged in the CVE-2026-80521 container-escape analysis: the boundary that marketing describes and the boundary that attacks actually cross are rarely the same line. Containers held against a kernel hole only until the kernel had a hole; agent sandboxes hold at the hypervisor while leaking at the gateway. Scope the threat model to the boundary that demonstrably fails, and test that one.

Originally published on the Netics blog.

CERN and Mistral Plan Flare, a Foundation Model for Collider Physics

Taha Echakiri — Wed, 21 Oct 2026 14:00:02 +0000

TL;DR

Flare is a foundation model that CERN and Mistral intend to train on billions of simulated proton-proton collisions from the Large Hadron Collider, so that one pretrained network handles several reconstruction tasks that a chain of specialised algorithms performs today. A foundation model is trained once on a large dataset and then reused across many jobs.
CERN and Mistral signed a statement of intent on 15 October 2026, which establishes a long-term strategic partnership for foundation models in science and gives the Paris-based lab access to one of the world's largest scientific datasets.
The model is designed to reconstruct collisions, identify particles, flag unexpected signals and evaluate future detector designs, learning across a whole chain that currently runs as a sequence of separate tools.
CERN holds about one exabyte, or one million terabytes, of LHC experimental data, and the number of collisions is expected to rise tenfold once the HL-LHC starts operating around 2030.
No budget and no timeline were disclosed for Flare; separately, Reuters reported that the open-weight model Mistral unveiled in October will be released to the public on 27 October.
The pattern a smaller organisation can borrow has three parts: a dataset large enough to train on, labels produced by simulation or by an existing pipeline, and an evaluation that a third party can repeat.

CERN's press release page for the CERN and Mistral statement of intent, 15 October 2026 (home.cern, retrieved 2026-10-17), carrying the official CERN/Mistral image.

What CERN and Mistral signed on 15 October

CERN and Mistral signed a statement of intent on 15 October 2026 establishing a long-term strategic partnership focused on foundation models for scientific discovery, according to CERN's announcement. The partnership pairs CERN's work in fundamental physics, scientific data and large-scale computing with Mistral's work in large-scale model training. Reuters reported the same day that the arrangement gives the French company access to one of the world's largest scientific datasets.

A statement of intent is the earliest formal step in a partnership, and CERN and Mistral disclosed neither a budget nor a timeline for the work. Reuters quoted CERN chief information officer Enrica Porcari saying the partnership would "advance AI capabilities that can serve the whole physics community and be shared".

Mistral's chief executive has argued the same point elsewhere. Arthur Mensch wrote in Le Monde that frontier models were "now indispensable to scientific progress" and criticised companies that keep their advantages to themselves, according to Reuters. Our reading is that the partnership and that argument point in the same direction: a lab that wants scientific credibility needs access to data it does not own.

Netics visual: the four stages from detector signal to identified particle, and the single pretrained model that CERN and Mistral intend to train across them.

Flare replaces a chain of specialised algorithms with one model

Reuters reports the model's name, Flare, and quotes CERN describing it as "the first foundation model for collider physics, trained on billions of simulated particle collisions". CERN's own page calls the project a Large Physics Model. The tasks it is built for are reconstruction, particle identification, flagging unexpected signals and evaluating future detector designs.

The architectural argument is worth understanding, because it is the reason a physics laboratory would pursue a foundation model at all. Today, CERN turns detector signals into physics objects through a chain of specialised algorithms, each transforming the output of the previous step, with machine learning improving performance inside several of those steps. A single pretrained model would learn across the whole chain and serve multiple tasks at once. A CERN spokesperson told Reuters the model uses the same idea as the large language models behind generative AI, trained on collision data in place of text.

Netics visual: the stored data volume and the tenfold collision forecast that sets the timing.

Simulation carries the labelling burden. Billions of simulated collisions supply training examples where the correct answer is known by construction, which is a luxury most industrial datasets lack and the reason physics is a good place to test the approach.

One exabyte stored, and ten times more collisions after the upgrade

The scale is the part of this announcement that could not be replicated elsewhere. CERN holds around one exabyte, or one million terabytes, of experimental data from the Large Hadron Collider. The number of collisions is expected to increase tenfold once the High-Luminosity LHC begins operation around 2030. Our reading is that this date sets the real deadline, because a chain of specialised algorithms gets harder to run as that volume arrives.

CERN's chief information officer framed the laboratory's value in terms of testing conditions. High-energy physics, Porcari said, "provides a unique environment in which artificial intelligence can be developed and tested with exceptional rigour". In our reading, rigour here means a measurable ground truth, repeated runs and a physics community that checks published results.

Official CERN/Mistral press image published with CERN's announcement of 15 October 2026 (home.cern, retrieved 2026-10-17).

The dated release sits apart from the statement of intent

Two announcements travelled together and they carry different weights. Reuters reported that Mistral this month unveiled a new open-weight model it says can challenge its rivals, and that the model will be released to the public on 27 October 2026. That release has a date. Flare has a statement of intent, a description and no schedule.

The distinction matters to anyone evaluating Mistral as a supplier. An open-weight release is something a customer can download, run and benchmark on its own hardware. A partnership with CERN is a research commitment whose outputs may never be published as weights, and CERN describes the intent that the capabilities serve the whole physics community and be shared. Both can be true at once, and a procurement decision should rest on the dated item.

The partnership also continues a direction Mistral has been building. The Next Web reported that the company acquired Emmi AI, a physics AI startup, and launched an industrial engineering product, and that Johannes Brandstetter, now Mistral's vice president of AI for engineering, co-founded Emmi AI.

What an SME can copy from the pattern

A small company will not train a physics model, and the structure of this project is still worth copying, because it describes how a domain model gets built.

Netics visual: the quotation from Enrica Porcari, CERN Chief Information Officer, on high-energy physics as a test environment for AI.

Start from a task chain, and count the steps. CERN's reconstruction pipeline is a sequence of specialised tools, and the value of one model comes from learning across the boundaries between them. The equivalent inside a company is the path from an incoming invoice to a posted accounting entry: several tools, several handoffs, and errors that appear at the joins.

Then ask where the labels come from. Physics simulates its training data. A company can simulate a smaller version of the same thing by replaying historical documents whose correct outcome is already recorded, and by generating edge cases deliberately. Without labels the project has no evaluation, and without evaluation it has no way to show that the model is better than the chain it replaces.

Evaluation is the third part. Our test, borrowed from how physics results are checked: a third party should be able to repeat the measurement and get the same answer.

Data at this scale also moves through infrastructure somebody controls. The terms of Mistral's access to CERN's data were not disclosed. Netics runs sovereign infrastructure and cloud exit engagements for European organisations that need compute next to data they are unwilling to hand over.

Flare has no published schedule. The announcement gives no benchmark, no evaluation against the existing reconstruction chain and no word on whether the weights will be published.

Originally published on the Netics blog.