DEV Community: Taha Echakiri

OpenCode's Breach Shows What a Stored Provider Key Costs

Taha Echakiri — Sun, 25 Oct 2026 14:00:47 +0000

TL;DR

OpenCode is an open-source AI coding agent that developers run from a terminal, with a hosted Console that manages accounts, workspaces and the provider keys some users link to it.
On 17 October 2026 the company emailed affected Console users under the subject "Action Required: Potential provider key breach", saying an attacker had reached a legacy system it was decommissioning and that third-party keys saved in a workspace, including OpenAI and Anthropic keys, may have been exposed.
OpenCode put the impact at 1.4 percent of Console users, described the incident as a large-scale distillation attack, and asked affected users to revoke and replace the keys at the provider, update the applications that used them, and check recent usage.
OpenCode has not said whether the exposed keys were stored in the legacy system itself. Our reading is a general risk: a system kept running while it is retired can keep its access to stored credentials, and here one compromise put keys tied to customers' own provider accounts at risk.
The bill for a stolen key arrives on the customer's account, because the key carries the customer's provider identity, quota and payment method. The only stop that works after the fact is revocation at the provider.
On 19 October, two days after the notice, OpenCode's status page showed its Console API component, which covers dashboard, workspaces and keys, as operational; a reader looking there for a security signal would have found only an ongoing slow-response incident.
Four controls survive a vendor breach: one key per tool, a spend limit at the narrowest scope the provider offers, log retention long enough to reconstruct usage, and a revocation runbook that does not wait for the vendor's confirmation.

What the notice said, and what it left open

The email that went out on 17 October had a subject line that leaves no room for interpretation: "Action Required: Potential provider key breach". The body explained where the access came from, in a sentence that has since been quoted widely: "We identified unauthorized access to a legacy system that we were in the process of decommissioning."

The next line is the one that matters to anyone who has used the product. "Third party provider keys (eg OpenAI, Anthropic) saved in your workspace may have been exposed."

OpenCode's own product site (opencode.ai, captured 2026-10-19), which presents the agent as an open-source coding tool run from a terminal. The Console it advertises is the system whose stored provider keys the notice concerned.

Those two sentences answer different questions. The first describes a perimeter failure in a system on its way out. The second names the data at risk: credentials customers had saved in a workspace, issued by other companies. The notice does not say whether those credentials were stored in the legacy system itself.

By 18:08 UTC the company had confirmed the incident publicly, describing it as "a large scale distillation attack" and putting the impact at 1.4 percent of OpenCode Console users. The affected group, per OpenCode, is a subset of users who had linked external providers. The company says it emailed everyone affected and that a user who received no email was not affected, and that it is working with Anthropic, OpenAI and Google on remediation. The distillation label is OpenCode's own choice; the write-up's author offers a reading of it and calls that reading a hypothesis until OpenCode publishes a follow-up.

What a system being retired can still hold

The open questions are worth writing down because they decide how much this incident tells us. OpenCode has not said how many accounts 1.4 percent represents, how the attacker entered the legacy system, how long the access lasted, or whether every key stored there was taken. One outlet quoted in the write-up makes the point directly: the figure measures neither exposure nor loss.

The notice establishes that the legacy system was still reachable while it was being decommissioned. It does not say which data that system held, or how it connected to the workspaces where keys were saved.

This is the part of the incident that generalises. In our reading (OpenCode has not described its own retirement plan), a system can be retired in stages: traffic moves first, then the database, then the credentials, and the credentials can stay in place after the traffic has gone. A system in that middle state is at its most dangerous, since it has lost the attention that a production system gets while keeping the access that made it worth a target.

Netics editorial diagram: the trade-off of linking a provider key to a vendor workspace, weighed as convenience in billing and rotation against what a leak transfers to an attacker.

The remedy OpenCode published is the only one available at that point. Users were asked to "revoke and replace those keys through the relevant provider's dashboard, update any applications using them, and review recent usage for unexpected activity."

Read that instruction as a description of where the control actually lived. The customer's credential could only be invalidated at the provider, by the customer. Every hour between the breach and the customer reading the email is an hour in which the credential worked.

Why the bill lands on your account

A provider key saved in a vendor's workspace still belongs to the customer's provider account. It carries the customer's identity at OpenAI, Anthropic or Google, which includes the quota attached to it, the payment method behind it and whatever guardrails the account has configured.

That is why the aftermath of this class of breach tends to show up as an invoice. The write-up notes user reports of bills in the hundreds of dollars, and it labels them as user reports with no figure from OpenCode behind them, which is the right way to read them today. The mechanism is simple: an attacker holding a valid key is, to the provider's billing system, the customer. In our reading, nothing in the request distinguishes a distillation run from a legitimate batch job.

The distinction between the two custody models is worth pricing before an incident. As a general pattern, and without any claim about how OpenCode bills its own traffic: when a vendor resells model traffic, the provider key in its systems is its own, and a leaked key draws on the vendor's provider account. When you link your own key, you keep the direct provider relationship, and you take on the identity risk. Both arrangements are legitimate; only one of them makes your provider account the blast radius of someone else's perimeter.

What the console status page showed on 19 October

OpenCode publishes a status page, and on 19 October its component list showed the Console API, described as covering the dashboard, workspaces and keys, as operational. The open item on the page was different: a slow-response incident that began at 16:44 UTC and was still marked ongoing 6 hours 48 minutes later, when the page was captured, with a note that pay-as-you-go traffic on one model was seeing slower responses.

That capture shows no security incident. A status page is built to track availability, and OpenCode disclosed this incident by email and on social media. It is also a reminder about the signal. Teams that monitor a vendor's status page for evidence of a security event are monitoring a page that was never built to carry one.

What did carry the signal, in this case, was a researcher posting a screenshot at 15:30 UTC, and the company confirming at 18:08 UTC. The screenshot showed the customer email itself, so the notice was already in affected users' inboxes when the first public post appeared.

OpenCode's own status page (status.opencode.ai, captured 2026-10-19), which lists the Console API covering dashboard, workspaces and keys as operational, with an ongoing slow-response incident as the open item. Security incidents are disclosed by email and social media, so this page carries no signal about them.

Four controls that survive a vendor breach

Every control here assumes the vendor will be breached at some point, and asks what the customer still holds when that happens.

One key per tool, issued from a named account. A shared key copied into three tools makes revocation a coordination project; a key per tool makes it a quick action with a known blast radius.
A spend limit at the narrowest scope the provider offers, per key where it allows one. The limit converts an unlimited exposure into a capped one, and it caps the loss until revocation lands.
Usage logs kept longer than the incident. Reconstructing what a stolen key did requires provider-side usage history that reaches back to the first unauthorised request, so check the provider's retention period before you need it.
A revocation runbook that runs before confirmation. Each hour spent waiting for a vendor statement is an hour in which the key still works at the provider.

There is a contract question underneath all four, and it costs nothing to ask a supplier: which credentials do you store on our behalf, where do they live, how long do they stay after we stop using the feature, and what is your deletion evidence. A vendor that answers with a retention policy and a deletion record has thought about the decommissioning window. One that answers with a security page has not. We run that review as part of our security work.

Netics editorial diagram: the decommissioning window as the exposure, where a system that has lost its operational attention can still hold and validate the credentials stored in it.

Originally published on the Netics blog.

Two Critical PHP Bugs Leaked Early, and Four Branches Now Carry the Fix

Taha Echakiri — Sun, 25 Oct 2026 14:00:02 +0000

TL;DR

PHP is the scripting language that runs a large share of the web, and on 21 October 2026 two critical remote code execution vulnerabilities in it were fixed in versions 8.5.12, 8.4.27, 8.3.36 and 8.2.35.
The pair is known as php2shell, both identifiers score 9.2, and the bugs sit in memory management and error handling, code that nearly every request touches.
The first bug interrupts a request at the memory limit and turns that interruption into code execution; the second leaves stale state in a worker that a later request reuses.
The embargo broke about two days before the planned 22 October release: the CVE identifiers circulated at around 15:00 UTC on 20 October with a wrong 10.0 score, PHP moved its release forward by one day, and a distributor published fix images at 07:41 UTC on 21 October.
Four versions carry the fix, released at 13:30 UTC on 21 October, and distribution packages follow on each distributor's own schedule.
Where the update waits, PHP's own advice is to remove the memory limit through php_admin_value, lower post_max_size and enforce request limits at the proxy.
The first signs of the memory bug surfaced during a 2024 external audit, its full impact became clear only in August 2026, and the PHP Foundation calls the delay a structural failure caused by the security team's decision process and its lack of resources.

What php2shell breaks

CISA calls the pair php2shell. Both vulnerabilities end in the same place, which is arbitrary code running inside a PHP worker with the permissions that worker already holds, reached by a request that carries no credentials at all. Every PHP installation that answers web requests is exposed, and that set includes PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed.

PHP's own notice ranks mod_php as the highest risk. Under mod_php a demonstrated exploit fires before any application code runs, so a framework patch, an application firewall rule inside the app, or a hardened CMS plugin never gets the chance to look at the request. AWS described the blast radius as bounded to the affected worker and its application permissions, with no cross-customer propagation established at the time of its bulletin, which is the reassuring half of the statement and also its limit.

The php.net release archive (php.net/archive/2026.php, retrieved 2026-10-21), which lists PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 as the versions fixing two critical vulnerabilities allowing unauthenticated remote code execution.

A memory limit that turns into code execution

CVE-2026-109618 is remote code execution through out-of-memory handling. PHP enforces the memory_limit setting by aborting a request that exceeds it, and that abort can arrive while internal request data is only partially initialised, for example during the parsing of a form-based file upload. PHP can later treat that partially initialised data as valid, which corrupts memory.

The interesting part for anyone running shared hosting or a multi-tenant platform is that the trigger is a resource limit, where a limit normally exists to protect the server from a greedy script. Here the path that enforces it becomes the attack surface itself, which is why PHP's first mitigation for organisations that cannot update immediately is to remove the trigger by setting memory_limit to -1 through php_admin_value in the pool or the Apache configuration.

That mitigation buys time and carries its own cost. An unlimited memory limit on a PHP worker means one badly written request can consume the machine, so the second half of PHP's advice matters as much as the first: set post_max_size to 8M where uploads are unnecessary or 32M where they are needed, keep upload_max_filesize below it, and enforce decoded request size, field count, multipart part count, header count and nesting depth at the web server or the proxy.

Stale state in the same worker

CVE-2026-109619 is remote code execution through stale global state, and its shape is different. Internal state owned by one request survives after the memory behind it has been freed, which PHP names as happening after a fatal error. The worker that handled the failed request goes on to serve the next one, and that later request can use the stale state before normal initialisation resets it.

The PHP Foundation's post reports that a use-after-free and a worker crash were confirmed, and that the reporter demonstrated remote code execution against a real-world application running on Apache with mod_php. The detail that matters operationally is the word worker: the Foundation writes that a later request handled by the same worker could then use the freed memory, and that remote code execution was also confirmed on PHP-FPM through application-level code paths.

Netics editorial timeline of the disclosure: the CVE identifiers circulating at 15:00 UTC on 20 October with a wrong 10.0 score, the pre-announcement at 19:30 UTC, a distributor publishing fix images at 07:41 UTC on 21 October, and the release at 13:30 UTC.

The embargo broke two days early

Coordinated disclosure works on a timetable that distributors need in order to have packages ready on the day. Here the identifiers leaked about two days before the planned release date, and the release itself moved forward by one day. The PHP Foundation's account places the first break at around 15:00 UTC on 20 October, when the two CVE identifiers began circulating on forums and social media with a wrong release time and a wrong CVSS score of 10.0. The real score is 9.2.

The project responded by publishing its pre-announcement at 19:30 UTC and by moving the release from 22 October to 21 October at 14:00 UTC, because distributors were still packaging the fix and an immediate release would have left users without an update path. The second break followed right after the pre-announcement, when Calif, one of the reporting firms, published a proof-of-concept video by mistake at around 20:00 UTC on 20 October and copies spread on social media.

The third break came from inside the process: at 07:41 UTC on 21 October, more than six hours before the coordinated release, Chainguard published built images containing the fix. Anyone could study the fix in binary form while every other PHP user was still waiting. PHP monitored the proofs of concept that appeared and reported no functional exploit at the moment of release, which is a narrow statement covering a short window. The case for publishing minor vulnerabilities in the open is covered in a separate Netics article. Both policies share the same scarce input, which is a maintainer with enough hours to answer.

What to patch today, and the four versions that fix it

The fixed versions are PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35, released at 13:30 UTC on 21 October 2026 according to CERT-EU, half an hour before the 14:00 UTC slot the PHP Foundation had announced. The affected ranges are 8.5.0 to 8.5.11, 8.4.0 to 8.4.26, 8.3.0 to 8.3.35 and 8.2.0 to 8.2.34. Distribution packages follow on each distributor's own schedule, so the package on a managed server can trail the upstream release, as the AWS timeline below shows.

PHP 8.1 and older sit outside every one of those ranges because the project stopped supporting them. AWS lists PHP 5.4 to 8.1 as having no upstream fix and requiring migration, and notes that its own Amazon Linux packages and WorkMail upgrade were targeted for 22 October while Elastic Beanstalk platform version v4.13.12 shipped on 21 October. PHP also published minimal source backports for 7.4, 8.0 and 8.1 for teams that build from source or have no access to a long-term support vendor.

CERT-EU points at one more category that inventories usually miss: third-party software that embeds PHP and never exposes it as a package, which the NCSC-NL repository tracks. That is the same problem CISA named in its alert, where PHP is often a component of other software and its prevalence may not be directly observable.

Netics editorial banner placing both CVEs at 9.2 and naming the shared mechanism: memory management and error handling, which every PHP request traverses, with mod_php reachable before application code runs.

What to do when the update waits

Some systems cannot take an update the same day, and the interim advice is specific enough to act on. Removing the memory limit through php_admin_value closes the trigger for CVE-2026-109618, and PHP notes that an underlying allocation failure can still reach the vulnerable path, so the mitigation reduces exposure and the installation stays vulnerable until it is patched. PHP also recommends a container or cgroup limit as an availability boundary, and warns against substituting RLIMIT_AS or RLIMIT_DATA, which can cause the vulnerable allocation failure.

The second layer sits in front of PHP. A web application firewall may provide some protection until systems are updated, according to CISA, and PHP's own guidance pushes request limits to the server or the proxy. Both reduce the space an attacker has to work in until the patch is applied.

The third layer is detection, and it is the one that tells you whether the day went badly. CISA asks defenders to watch for PHP worker processes launching shells or downloading tools, which is a short rule to write into whatever monitoring already exists and a reliable signal that a worker crossed from parsing requests into running commands.

Why the engine fix shipped in October 2026

The disclosure history explains the delay better than any severity score. In 2024 PHP went through its first external security audit in years, funded by the Sovereign Tech Agency, organised through OSTIF and carried out by Quarkslab. While fixing one of the issues that audit reported, a core developer contracted by the PHP Foundation found the first signs of the out-of-memory problem. A potential exploit was theorised in 2024, no concrete vector was found at the time, and other issues with clearer impact took the attention.

The picture sharpened through 2026. On 11 August a researcher at Calif reported an unconditional remote code execution, and OpenAI independently reported the same root cause. Further attack vectors came from Wiz on 24 August and from OpenAI on 18 September, and from 11 August the fix became the team's top priority. The Foundation writes that the bugs were hard to fix without breaking compatibility, hurting performance or forcing changes on the large ecosystem of third-party extensions, and it calls the delay a structural failure caused by the security team's decision process and its lack of resources.

The volume is the part that shapes the next year. The security team has published 15 CVEs since 1 July and worked through more than 100 reports that turned out to be invalid, duplicates or not reproducible. AI tooling helps validate reports and group the ones that share a root cause, and a person still makes every call and writes every patch. For an operator the lesson is unglamorous: the runtime under a large share of the web rests on a small team of core developers, several of whom could work on this issue full time only because of Foundation funding, and the day the embargo breaks is a bad day to discover that nobody knows how many PHP installs the company runs.

The PHP Foundation's post of 21 October 2026 (thephp.foundation, retrieved 2026-10-21), which announces the release, describes the two vulnerabilities and records the disclosure timeline that moved the release forward by a day.

Originally published on the Netics blog.

The World Bank Wants Small AI Built for Weak Connections

Taha Echakiri — Sun, 25 Oct 2026 09:00:03 +0000

TL;DR

Small AI is the World Bank's term for a design approach to AI development: narrow models aimed at one well-defined problem, running on the devices people already own, trained on local data and reachable through SMS or voice.
Paschal Donohoe, the Bank's chief knowledge officer, told the Guardian that these models can run on a mobile phone with low connectivity, in the language of the community, and argued that development which took centuries could happen within decades.
The Global AI and Digital Summit was held in Seoul from 19 to 22 October 2026, with a session titled Small AI. Big Impact on the programme for the morning of 20 October, the point at which, in our reading, a phrase can become something governments are asked to fund. This article reads the programme as captured on 19 October, and the signal is the agenda itself, which gave small AI a session of its own.
The numbers behind the agenda are large: more than 2 billion people were offline as of 2025, and the Bank says 1.2 billion young people in the Global South will reach working age over the next decade against roughly 420 million new jobs.
The enablers the Bank itself names are broadband, digital literacy and trusted local institutions, and it warns that without them solutions risk remaining pilots. Its own World Development Report found almost a third of rural schools in sub-Saharan Africa without electricity.
A deployed small AI is a procurement object, and four commitments decide whether it survives contact with users: which languages are tested, on which devices, with which offline behaviour, and who ships the updates.

What the World Bank means by small AI

The Bank's own definition is a design approach, and the word approach is doing the work. Small AI describes narrow, use-case-driven solutions that address a single well-defined problem on accessible hardware: basic phones, sensors or microcontrollers, running models locally.

Netics editorial diagram: the World Bank's definition of small AI, restated as the four properties that make it testable — a narrow task, an accessible device, a local dataset and a delivery channel that works without an app store.

Donohoe's version, given to the Guardian around the World Bank and IMF annual meetings in Bangkok, is the same idea in the language of a development agency. He describes models available in a local setting, with low connectivity and in the local language of the community, and adds that such models tend to be available on a mobile phone.

The example he reaches for is concrete. Farmers in the Bank's AgriConnect agricultural scheme in India needed soil analysis in their own language, and the arithmetic of expert supply rules out the obvious answer. There will not be enough scientists and agricultural experts to do that for every small farm in India, he said. A narrow model that answers one question well, on a device the farmer already owns, can reach the farms that the supply of experts cannot cover.

That framing carries straight into procurement. A soil-analysis model can be specified with a language, a device and an accuracy figure for a specific question, which makes it easier to test and much harder to fake than a general assistant judged on a demo and an average benchmark score.

Seoul puts the phrase on a programme

The summit was held from 19 to 22 October 2026 in Seoul, co-hosted by the World Bank Group and the Government of Korea, and the session names are worth reading as a map of the Bank's priorities. Small AI. Big Impact was scheduled for 9:45am KST on 20 October with Ajay Banga, Bosun Tijani, Hoshik Shin and Kate Kallot, moderated by Mark Tetto.

A second session, Closing the Gap: The Promise of AI in Developing Economies, was scheduled for 21 October with Donohoe alongside Michael Kremer, Gaurav Nayyar and Darlington Akogo. What matters for this analysis is the programme itself, which records the ideas the Bank chose to give a session of their own.

Programmes like this matter beyond their own hour, because a term that appears on a ministerial agenda can become a term that appears in loan documents, tender specifications and national AI strategies. Small AI is moving from an argument made in an interview to a category that vendors will be asked to supply, and the specification for it will be written by whoever shows up with a definition first.

The Bank's own framing for the summit makes the stakes explicit. With 1.2 billion young people entering the workforce in developing countries over the next 10 to 15 years, how AI and digital technologies expand access to jobs is described as a defining challenge. That sentence sits next to a much less comfortable one on the same event pages: as of 2025, more than 2 billion people remained offline.

The World Bank Group's own event page for the Global AI and Digital Summit in Seoul (worldbank.org, captured 2026-10-19), which carries the programme, the session titles and the speakers for 19 to 22 October 2026.

Two billion offline and 420 million jobs

The labour arithmetic is the reason this argument has urgency. Over the next decade, 1.2 billion young people in the Global South will reach working age, and current trends point to about 420 million new jobs, according to the Bank's summit event pages. In our reading, the gap is one of demand, and a cheaper delivery channel can address part of it.

Kristalina Georgieva, managing director of the International Monetary Fund, speaking at the same Bangkok meetings, called on governments to provide the preconditions: access to electricity, access to digital infrastructure, people geared to adoption and government protection against bad actors. With those in place, she said, adoption could be as democratic as the cellphone. She also warned that over time 40 percent of jobs worldwide are likely to be augmented or eliminated by AI.

Donohoe's counterweight to that warning is a development claim. Economic development that could have taken centuries, he argued, now has the potential to happen within decades. He pointed to 120 billion dollars of private capital the Bank mobilised in the past year through guarantees and other instruments.

Both halves belong in the same paragraph. A model that answers a soil question and a financing guarantee address different parts of the same gap, and in our reading what connects them is delivery cost: a phone-based narrow model interests a development bank because it promises to reach more users for each dollar spent. Neither the Guardian interview nor the summit pages give an accuracy, device or cost figure for small AI, so that promise is still untested.

The enablers decide whether a pilot ships

The Bank is unusually direct about the limits of its own idea. Its summit event pages state that small AI still requires foundational enablers: broadband connectivity, digital literacy and trusted local institutions, and that without them solutions risk remaining pilots.

The Guardian's reporting adds the physical picture. The Bank's World Development Report, published in August, found that almost a third of rural schools in sub-Saharan Africa lacked electricity and more than two-thirds did not have a consistent internet connection. A model that runs offline on a phone removes one dependency from that list. The others remain, and a school without power cannot host a deployment of any kind.

The gap between offline capability and those physical enablers is where most small AI projects will be decided. A pilot is funded for a quarter, on one device model, in one district, with a team that includes the person who built it. A service runs for years, across whatever handsets people actually own, maintained by someone who inherited it. The technical work of moving between the two is unglamorous: versioning, logging, evaluation and a rollback plan.

The Guardian's interview with Paschal Donohoe, the World Bank's chief knowledge officer (theguardian.com, 18 October 2026, captured 2026-10-19), which carries Donohoe's remarks and the AgriConnect example cited above.

Four commitments a small AI contract can carry

The outcome turns on the contract. What the buyer writes down decides it, and the four lines below are the ones that separate a demo from something a ministry or a mid-sized company can run.

Languages. Name the languages and dialects covered, and the test set each figure comes from. A single accuracy number across a multilingual population hides the failure that matters, which is the user whose language had only a thin share of the training data.

Devices. State the handset baseline, the operating system version and the quantisation the model ships at, then measure latency and memory use on that baseline. A model that runs on the evaluator's phone but fails on the farmer's is a pilot artifact.

Offline behaviour. Write down what the application answers when there is no network, and how it marks an answer produced without fresh data. The alternative is a user who cannot tell a cached guess from a live result.

Updates and data. Assign one owner for model updates, version numbers and rollback, and state what leaves the phone and where it lands. On a shared device in a low-connectivity setting, the data boundary is a privacy decision before it is an engineering one.

Our approach to delivery work is set out under our build practice. For a company in France or Morocco weighing a narrow on-device model against a hosted assistant, the trade is legibility against cost: in our reading, an on-device model can be audited, tested per language and designed so that less data leaves the phone, and it will never have the range of a hosted model. Small AI is a defensible choice when the task is narrow and the alternative is no service at all.

Netics editorial diagram: the four commitments a small AI contract can carry — which languages are measured and on which test set, the device baseline, who ships model updates, and what leaves the phone.

Originally published on the Netics blog.

DuckDB 2.0.0 LTS Ships Its Biggest Feature Release on a Support Clock

Taha Echakiri — Sat, 24 Oct 2026 19:00:05 +0000

TL;DR

DuckDB is an open-source analytical database that runs inside one process, and version 2.0.0 LTS, codenamed Cinnamon Teal, was released on 21 October 2026 with the Quack protocol, asynchronous I/O, a first-class VARIANT type, agent mode, triggers and an extensible SQL parser.
The release is built from close to 20,000 commits since DuckDB 1.5, and the project calls it the biggest feature release so far, in contrast with the 1.0 line that concentrated on stability.
The support label carries a date: 2.0 is supported until at least October 2027, with the calendar page listing an end of life of 22 October 2027.
Three version lines run in parallel until 17 November 2026, when community support for 1.4 ends, and two lines remain until March 2027, when the extended support for 1.5 ends.
The storage format moves in one direction now, since releases before 2.0 cannot open files written by 2.0 while existing files that 2.0 reads and writes stay readable by older releases.
A migration plan for a small data team is a shadow system, a pinned client library and a decision about whether to wait for the first patch release in November.

What an LTS label buys a data team

DuckDB's release calendar explains the label better than the announcement does. Starting with version 1.4.0, every other release is a Long-Term Support edition, and the community support period for an LTS version currently runs a year after its release. The 2.0.0 entry shows the release date of 21 October 2026 and an end of life of 22 October 2027, which is the one date a team should copy into its own calendar.

That calendar matters more than usual for a tool that lives inside an application. An embedded analytical database is linked into a Python service, a CLI script, a scheduled job or a notebook, and each of those places upgrades on its own schedule. A support window turns an indefinite promise into a deadline somebody can plan against, and the project publishes the next release on the same page, with 2.0.1 tentatively dated 18 November 2026.

The version policy underneath is ordinary semantic versioning, with larger new features arriving in minor versions and patch versions mostly carrying bugfixes. What the LTS cadence adds is a promise about how long a given minor version will keep receiving those patches, which is the question a team asks when a pipeline that runs nightly depends on the answer.

The DuckDB announcement post (duckdb.org/2026/10/21/announcing-duckdb-200, retrieved 2026-10-21), whose summary lists the Quack protocol, asynchronous I/O, a first-class VARIANT type, agent mode, triggers and an extensible SQL parser.

Quack, VARIANT and the features in this release

The feature list is longer than a version bump usually carries. The Quack protocol arrived in DuckDB 1.5.3 in May and let DuckDB run in a traditional client-server setup; 2.0 ships a polished version of it. Asynchronous I/O now covers reads of Parquet and CSV files and of DuckDB's native storage format: the I/O layer scales independently from query processing and keeps multiple requests in flight, which the project says brings far more parallelism to remote reads and faster queries on object storage such as S3.

The other additions change how the engine is used. A first-class VARIANT type gives semi-structured data a native home. Agent mode lets the CLI detect when a coding agent is driving it and switch to compact output, triggers bring CREATE TRIGGER with BEFORE and AFTER triggers at row or statement level, and the extensible PEG-based SQL parser, opt-in since DuckDB 1.5, is now the default. Triggers still have gaps: the announcement lists BEFORE … FOR EACH ROW and UPDATE … FOR EACH ROW triggers as not yet supported, so trigger logic carried over from another database needs checking case by case.

The internals moved as well. DuckDB 2.0 switches to Clang as the default compiler for release builds, with the engine and its core extensions built that way on Linux and macOS, the CLI on Windows, and most Windows core extensions still on MSVC. The ICU library is gone entirely: the icu extension now implements timezones, calendars and collations itself, with timezone data built from the IANA database and compressed to around 45 kB.

Netics editorial scorecard of the support windows that overlap after this release: 2.0.0 LTS supported until at least October 2027, 1.5 extended to March 2027, and 1.4 LTS supported until November 2026.

The storage format now moves in one direction

The breaking change that matters is in the file. DuckDB versions before 2.0 cannot read database files created by DuckDB 2.0, while 2.0 still reads and writes files created by earlier versions, and those older files remain readable by older releases. The trap sits in new files: any database file created by 2.0 cannot be opened by the version a team would roll back to.

The project's own advice follows from that. Test 2.0.0 on a shadow system before switching production over, and if the workload is mission-critical, consider waiting for the first patch release in November. A migration guide accompanies the release, and each service upgrades its client library on its own schedule: once one moves to 2.0, any database file it creates can no longer be opened by the version it ran before.

Snapshots are the cheap insurance here. Any team that exports a copy of its analytical data before the upgrade, in Parquet or in a plain CSV of the tables that matter, converts an irreversible step into a reversible one. Our reading at Netics is that Parquet and CSV copies are the safer insurance, since they sit outside the DuckDB database file format that changed in 2.0.

Netics editorial diagram of the file compatibility change: files written before 2.0 remain readable by earlier releases, while files written by 2.0 cannot be opened by releases before 2.0, which is why a rollback has to be planned before the upgrade.

Three support lines until mid-November, then two until March 2027

Until 17 November 2026 a team can choose between three supported versions, then between two until March 2027, and the choice comes down to risk appetite. Version 1.4 stays supported until late November 2026, which the calendar page states as 17 November 2026. Version 1.5 was extended to March 2027 and, as the announcement puts it, that effectively turns a non-LTS release into an LTS one. Version 2.0 runs until at least October 2027.

The practical reading is that 1.5 is the conservative bridge. A team that needs the fixes but wants a longer runway than 1.4 offers can sit on 1.5 until March 2027, run 2.0 on a shadow system in the meantime, and move once a patch release or two has shipped. The announcement names the storage format as the most likely source of problems.

The calendar also makes the long-term shape visible. Every other release is meant to be an LTS edition, so a team can plan two-step moves and skip the versions in between. The alternation is not strict, since 1.5 gained LTS-length support after its release, and the next patch of the current line carries only a tentative date.

The DuckDB release calendar (duckdb.org/release_calendar, retrieved 2026-10-21), which dates 2.0.1 for 18 November 2026, lists the end of life of 2.0.0 LTS on 22 October 2027 and states that every other release is a Long-Term Support edition.

A migration plan for a small data team

The plan fits on one page. Export the data that matters to Parquet before touching anything. Pin the client library version in the service that reads it, so that no service creates 2.0 database files before the shadow test is done. Run 2.0.0 against the shadow copy and compare row counts and query results on the reports somebody actually reads. Decide on a date, and write down which version line the team will sit on until that date.

The reason to bother is visible in the feature list. Our reading at Netics is that a single-process analytical engine that gains a client-server mode, a semi-structured type and asynchronous I/O can query more data where it already sits. That argument is the same one behind the private assistant that answers from your own documents: keeping the data where it already lives, and adding the compute next to it.

An embedded DuckDB lives inside an application, which is where Netics does custom web and mobile development with small teams. The version choice above is the kind of decision that belongs in a runbook with a date on it, because each support line has an end date.

Originally published on the Netics blog.

A Security Audit Checklist for a 20-Person Company

Taha Echakiri — Sat, 24 Oct 2026 14:00:03 +0000

TL;DR

A security audit is a review that checks whether a company's systems, accounts and backups behave the way the company believes, and it ends in written evidence a client, an insurer or a future enterprise customer can read.
At twenty people the work fits a single week: an inventory, accounts and admin rights, a patching clock, one verified restore, logs, suppliers and an evidence pack.
Two public reference frameworks carry the inventory: CIS Controls v8.1 with its 18 controls and 153 safeguards, where Implementation Group 1 is defined as essential cyber hygiene, and NCSC's 10 Steps with ten named components for organisations that have somebody on security.
Accounts come first, because a list of who holds admin, a second factor on every account that reaches production and a trail for joiners, movers and leavers answers most of a client questionnaire on its own.
Patching runs on a published clock with a named owner, and the reason is timing: when PHP fixed CVE-2026-109618 on 21 October 2026, CISA urged applying the patch immediately, with priority for internet-facing systems.
Backups count on the day somebody restores one from the copy that leaves the building, with the date written into the evidence pack.
Logs earn their place when a dashboard exists that a named person opens, and the archive keeps the one path an investigation would need to reconstruct.
Suppliers hold the data, so every contract gets the same three questions, and the evidence pack gathers the answers in one folder.
Four people can carry the week: one owner, two technicians, and one person from finance or operations who reads the result.

What a security audit produces at twenty people

A company of twenty runs on a laptop fleet, a handful of cloud accounts, a website somebody built, a shared drive and two or three business applications. Nobody in that company spends their day on security, and the honest starting point of the audit is a sentence the founder can say out loud: this is what we own, this is who can reach it, and this is how we would know if something changed.

Netics editorial card counting what the public reference lists contain: 18 CIS controls in v8.1, 153 CIS safeguards, the 10 NCSC Steps and the 3 CIS Implementation Groups.

The output of the audit is a document with rows in it. Each row names an asset or a process, the person who owns it, the state it is in today, and the date somebody checked. A penetration test answers a different question and produces a different artefact, which is why a small company that buys one test a year can still fail a client questionnaire that asks whether the staff laptops are encrypted, whether admin accounts use a second factor, and when a backup was last restored.

Three audiences read that document. A client's procurement team reads it before signing, an insurer reads it when pricing a cyber policy, and an enterprise customer reads it when a twenty-person supplier asks to connect to their systems. Each of those conversations has a shorter, cheaper version once the rows exist, and a longer, more expensive version when they have to be produced under deadline.

The inventory comes before the framework choice

Three public lists cover the same ground, and a company of twenty can use any of them as scaffolding for the rows. CIS Controls v8.1 is the most prescriptive: 18 controls, 153 safeguards, and Implementation Groups that let a smaller organisation start with Group 1, which CIS defines as essential cyber hygiene, the foundational set of safeguards every enterprise should apply against the most common attacks. NCSC publishes the 10 Steps to Cyber Security as guidance aimed at medium to large organisations with a dedicated security role, and says plainly that smaller organisations may be better served by its Cyber Action Toolkit while the ten principles still apply. The NIST Cybersecurity Framework, current version 2.0, works one level up: profiles describe an organisation's own alignment of requirements, risk appetite and resources against the framework's outcomes, and are used to prioritise what to improve.

The practical move is to pick one list as the spine and write the rows against it. A twenty-person company that adopts CIS Control 1 and Control 2 ends up with two spreadsheets: every device and where it lives, and every piece of software with its version and its owner. That second sheet is where the audit starts paying for itself, because the software nobody remembers installing is the software nobody patches. CISA made the same point about PHP during the October 2026 emergency, warning that the language is often a component of other software and that its prevalence may not be directly observable.

The CIS Critical Security Controls list page (cisecurity.org/controls/cis-controls-list, retrieved 2026-10-21), which numbers the 18 controls of v8.1 from Inventory and Control of Enterprise Assets through Penetration Testing.

Accounts, admin rights and a second factor

The account section is the one that produces findings fastest. CIS Control 5 covers account management and Control 6 covers access control management, and both reduce to a question a technician can answer in an afternoon: which identities can reach production systems, and which of those hold administrative rights.

Start with a per-system list. For each business application and each cloud console, write down the administrators, the service accounts, and the accounts that exist because a contractor needed them once. The per-person view follows, and it catches the pattern that small companies share: one shared administrator login, used by three people, with a password that lives in a chat message.

Three checks close the section. The first is the second factor, present on email, on the virtual private network and on every console that can delete data. The second is the departure trail, meaning the written procedure that removes access on the person's last working day. The third is the count: how many administrators exist, and whether that number matches what the founder expected before the list was drawn.

Patching on a published clock

Every reference list treats patching as continuous work, and CIS Control 7 names it continuous vulnerability management. At twenty people the work happens when somebody remembers, which is the failure mode the clock removes. A written rule assigns each severity a deadline, names the person who checks the vendor feeds, and describes what happens when a critical fix lands at the end of a week.

The PHP emergency of 21 October 2026 shows why the rule has to exist before the incident. Two critical vulnerabilities, CVE-2026-109618 and CVE-2026-109619, allowed unauthenticated remote code execution, and CISA urged applying the patch immediately, with priority for internet-facing systems. The fix arrived in PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35, distributions and long-term-support vendors had the patches in advance, and PHP 8.1 and older no longer receive updates from the PHP project, which turns an end-of-life runtime into a migration project. For those branches the PHP security team published minimal source backports for 7.4, 8.0 and 8.1, meant for people building from source or without access to an LTS vendor.

Two rows belong in the evidence pack here. The first lists the runtimes in use and their versions, and the second records the date the last critical fix was applied. CISA notes that a web application firewall may provide some protection until systems are updated, and asks organisations that cannot update immediately, or that still run PHP 8.1 or earlier, to take PHP-based systems offline until they can be patched or updated.

Netics editorial table pairing each reference framework with what it provides and what the company writes down: CIS v8.1 with its 18 controls and 153 safeguards against asset, account and patch rows, the NCSC 10 Steps against one named owner per step, and NIST CSF 2.0 profiles against a current and a target profile.

Backups and one verified restore

CIS Control 11 covers data recovery, and the audit question behind it is narrower than a backup policy: which copy exists, where it lives, who can read it, and when somebody last restored from it. A company that answers those four questions on one page is ahead of most of its size.

The copy that matters is the one that leaves the building. Ransomware reaches network shares and attached drives with the same credentials the staff use, so the row has to name a destination outside that reach, whether that is object storage in a second account or a removable disk in a drawer. Encryption belongs in the same row, because an unencrypted copy of client data in a second location is a second incident.

The verification is a date, and it goes on the calendar. One restore per quarter, on a test machine, of a directory somebody would genuinely miss, with the outcome written down. A restore that fails is the most valuable finding in the whole audit, because it arrives months before a real outage would have delivered it.

Logs, alerts and a dashboard somebody opens

CIS Control 8 covers audit log management, and the small-company version splits into two decisions: which systems keep logs, and which log lines would change somebody's day. Both answer to a named reader, because a log store nobody queries fills a disk and teaches the team to ignore it.

The alert list for a company of twenty is short. Failed sign-ins in volume on the email and remote-access systems, a new administrator account appearing, a large deletion or a bulk file change, and anything that launches a command interpreter from a web server process. CISA's monitoring advice during the PHP emergency names exactly that last pattern, asking defenders to watch for PHP worker processes launching shells or downloading tools, and it works as a template for any web runtime the company runs.

Retention is the second half. An incident discovered in November often started in September, so the archive has to cover the gap between discovery and cause. The design question for the alerts themselves, which is the subject of the monitoring pillar on alert design, sits on top of this: a page that carries the four fields somebody needs is worth ten that carry a metric name.

Suppliers, access and the evidence pack

CIS Control 15 covers service provider management, and NCSC's tenth step covers supply chain security, which puts supplier review inside the same list as the technical work. The three questions are the same for every contract: which company data reaches the supplier, which of the supplier's staff can read it, and what happens to it on the day the contract ends.

Twenty-person companies usually find a handful of suppliers holding more data than expected: the payroll provider, the accountant's portal, the email host, the website agency, the backup service. Each of those belongs on one sheet with the answers and a review date. The sheet doubles as the exit plan, because a supplier that cannot answer the third question is a supplier whose data cannot be removed.

The evidence pack is where everything above lands. One folder, one index, and per section the rows plus the screenshots and the dates: the device inventory, the software inventory, the admin list, the patching record, the restore test, the alert list, the supplier sheet. Filing work that already exists costs an hour; reconstructing it during a client's security review costs a week.

The NCSC 10 Steps to Cyber Security collection page (ncsc.gov.uk/collection/10-steps, published 11 May 2021, retrieved 2026-10-21), whose sidebar lists the ten components from Risk management and Asset management through Logging and monitoring, Incident management and Supply chain security.

Assigning the week across four people

The checklist fits a week when four people share it. One owner holds the document and the deadline, and that person is usually the founder or the operations lead, because every section needs a decision that a technician cannot take alone. Two technicians take the inventory, the account list, the patch record and the restore test. One person from finance or operations takes the supplier sheet and the contract questions, which are administrative work that technical staff tend to defer.

The estimate per section lands near half a day for the inventories, half a day for accounts, two hours for the patch record, two hours for the restore test, two hours for logs and alerts, and half a day for suppliers. That leaves the evidence pack, which takes an hour when the sections are finished and cannot be compressed when they are not.

The cadence afterwards is the part that decides whether the audit was worth doing. Accounts and patching change monthly, so their rows get a short monthly review. Suppliers and backups change slowly, so they get a quarterly one. The audit itself repeats yearly, with the same document, dated, because the second version takes a fraction of the time and turns a one-off exercise into a record that shows improvement.

Originally published on the Netics blog.

The Browser Becomes an AI Distribution Point: What Mozilla and Mistral Actually Changed

Taha Echakiri — Sat, 24 Oct 2026 09:00:03 +0000

TL;DR

On September 16, 2026, Mozilla and Mistral announced that Firefox Smart Window (beta) is now powered by Mistral Small 4, with the United States, Canada, and France at launch and the UK and Germany expected later this year.
France is the first new market for Smart Window with official French-language support — a deliberate multilingual-market move, not an English-first port.
The deal's stated thesis is distribution: "open technology needs open distribution," and a browser "shouldn't be a one-way funnel."
The operational commitments that matter: conversations aren't saved on Mozilla's servers by default, Mistral agrees to zero data retention, and Firefox users keep choosing among multiple models.
What still needs proving: whether switching and zero retention survive the integration's data flows — a promise at announcement is not yet a verifiable control.
Three questions before any fleet rollout: policy-level model switching, verifiable zero-retention traffic paths, and French-language coverage in real workflows — not only in the consumer UI.

Why a model inside a browser is a distribution event

The AI competition has moved from "which model is best" to "which models people can actually reach, through which products." That is the shift Mozilla names directly: "Practically overnight, competition has expanded from 'which AI model is best?' to a new high-stakes battle over which models people can access, through which products, and whether people have meaningful choice about AI at all." A browser is the chokepoint where that battle gets decided, because the browser sits between the user and almost every AI product they touch. Putting Mistral Small 4 into Firefox Smart Window is therefore not a feature integration; it is a distribution decision.

Official Mozilla blog image (2026-09-16): the Mozilla and Mistral partnership header used on both announcement pages.

Smart Window is Mozilla's AI browsing assistant: it makes sense of complex search trails, remembers something important you clicked away from, and presents the sources behind its answers. With this deal, those tasks in the US, Canada, and France run on Mistral Small 4, a model Mozilla says it selected after evaluating multilingual performance for the beta. The detail worth pausing on is the market order. France comes first among new markets, with official French-language support. That is the practical form of the companies' stated approach: "AI optimized for local countries and cultures, not exported to them," with models fine-tuned on regional languages and dialects rather than an English-first experience being adapted afterward.

The one-way funnel test

Mozilla's CEO Anthony Enzor-DeMeo put the frame directly: "With the browser sitting at the heart of the web and online experience, it should be a place where different AI providers can compete and open source has a seat at the table." And the sharper line: "A browser shouldn't be a one-way funnel." The test of that sentence is operational, not rhetorical. Mozilla commits that "across all markets where Smart Window beta is available, Firefox users can still choose from a multitude of other AI models." That keeps the model layer swappable, which is the correct architecture for a distribution point. The moment integration stops being a recommendation and becomes the only path, the funnel is closed regardless of press language.

Official Mozilla blog image (2026-09-16): the Smart Window beta interface in Firefox with Mistral as the active model — the surface where model choice is presented to the user.

The strategic bet is bigger than one model. Mozilla describes a scenario where "a handful of closed-model companies" control the browser, the search engine, the model, and the surrounding services together. Mistral is the counterweight: a frontier European AI company reaching consumers without owning a browser, and a browser integrating a model without becoming a model company. That division of ownership is what makes the partnership structurally interesting — neither party can close the funnel by itself.

Zero data retention is a data-flow claim, not a slogan

The privacy commitments are specific. "Conversations aren't saved on Mozilla's servers by default, and partners like Mistral agree to zero data retention." Treat that as a contract to verify, not a headline. Zero retention only means something if the request path actually avoids logging, caching, and model-provider telemetry, and if the browser's own session stores behave the same way. The architecture should be auditable: where the conversation travels, what the model provider receives, what the browser keeps locally, and what happens to the input when the assistant is disabled. For European users, this intersects with GDPR expectations around purpose limitation and storage limitation — the announcement's commitments align with the regulation's direction, but alignment at announcement time is not enforcement at runtime.

Official Mistral AI news image (2026-09-16): the partnership announcement hero for Mistral and Mozilla bringing open, private, multilingual AI to the browser.

There is an honest counterpoint here. Mistral's own framing of the fourth reason — "putting sovereign AI in everyone's hands" — is consumer-facing language for something the company has mostly sold to enterprises. This deal extends Mistral beyond its business base, which is a real expansion but also a new attack surface: consumer browser behavior is a different trust boundary than a contractual enterprise deployment. The zero-retention commitment is precisely what makes that boundary survivable; it is also what makes it worth verifying.

What Netics would check before rolling this out

For an organization deciding whether Firefox with Smart Window belongs in its fleet, the questions are concrete. First, can the default model be changed or the assistant disabled at the policy level, not just in a per-user setting? Second, can we verify the zero-retention claim in the deployment — does the traffic path show what the announcement promises? Third, does French-language support extend to the organization's actual workflows, or only to the consumer UI?

Netics editorial visual: Mozilla CEO Anthony Enzor-DeMeo's test line — "A browser shouldn't be a one-way funnel" — rendered as the article's closing standard for the partnership.

The distribution thesis is sound: an independent browser carrying an open-weight European model is a meaningful alternative to the closed-funnel scenario Mozilla warns about. But distribution points earn trust through verifiable switches and auditable data flows, not through partnership announcements. Those are separate workstreams, and only one of them is done. Netics' own work on alternative AI architecture choices is summarized on neticslabs.com.

Originally published on the Netics blog.

Eight eMudhra Intermediate CA Keys Rested on Eight Bits

Taha Echakiri — Fri, 23 Oct 2026 19:00:03 +0000

TL;DR

eMudhra is an Indian certificate authority that sells TLS and code-signing certificates under the emSign brand, and an intermediate CA is the certificate that signs the certificates individual websites present to a browser.
A researcher showed the root programs on 17 October 2026 that the private keys of eight eMudhra intermediates were single-byte numbers, one of them literally four, recoverable from the public certificate in under a second on a laptop.
Chrome blocked the six TLS keys at 19:45 UTC that evening and Mozilla finished its OneCRL block for Firefox at 22:59 UTC; eMudhra revoked all eight intermediates at 09:38 UTC on 18 October.
Browsers take revocation from a list they fetch themselves, so the block landed with no update and no restart; every other client trusts a CA bundle on disk and can keep trusting a revoked chain for months.
The trust stores that live in code moved next, with Go 1.27.3 and 1.26.10, NSS 3.131.1, Python certifi 2026.10.19, Rust webpki-roots 1.0.10 and a Homebrew exclusion shipped by 20 October, while a Node.js fix remains pending.
An administrator can check on Monday morning which bundles each fleet ships, then confirm that a test chain through the emSign root now fails.
Mozilla removed emSign ECC Root CA - G3 from NSS on 20 October and Chrome will remove it in root store version 43, and Chrome is reviewing eMudhra's other root, emSign Root CA - G1.

Mozilla's incident ticket for the eight eMudhra intermediates (bugzilla.mozilla.org, bug 2083449, captured 2026-10-20), which records the ticket as resolved and carries eMudhra's own account of the key ceremony in its comments.

A private key of four, and sixteen certificates that passed every check

Scott Helme recovered all eight private keys from their public certificates on a laptop, and the whole exercise took less than a second. The values are 156, 230, 155, 4, 115, 56, 66 and 237, one per intermediate. Helme pulled all sixteen certificates behind them, because each key was used in two: one issued by eMudhra and one issued by Sectigo as a cross-sign. The certificate named EM OV TLS CA - G3C carries a private key equal to the number four.

The arithmetic behind that number explains why it fell so quickly. These are elliptic-curve keys on the P-384 curve, where the private key is an integer d and the public key is d × G. A correct P-384 key is a random 384-bit number, one of roughly 10 to the power of 115 candidates. eMudhra's keys are 47 bytes of zeros followed by a single byte, which reduces the search to at most 255 steps: compute 1 × G, 2 × G, 3 × G, stop when the result matches the certificate.

Nothing else about those certificates looks wrong. A public key derived from d = 4 is a perfectly valid point on the curve, the encoding is correct, the curve is correct, and every structural check passes. The only way to find out is to try, which is what happened in October 2026, seventeen months after the material was created in one ceremony on 30 April 2025 and cross-signed by Sectigo's AAA Certificate Services root in May 2025.

Netics editorial diagram built from the key values published by Scott Helme on 2026-10-20: a correct P-384 key sits in a space of about 10 to the power of 115, while these eight keys sit between 0 and 237.

Why revocation reached browsers in hours and everything else in months

The report reached the root programs on the evening of 17 October and the machines responded at the speed of a push channel. Chrome had the report at 18:05 UTC and a CRLSet update blocking the six TLS intermediates at 19:45 UTC. Mozilla had it at 20:06 UTC and finished a OneCRL block for Firefox installations with standard settings at 22:59 UTC. Neither update needed a restart, because CRLSets and OneCRL are lists the browsers fetch on their own.

The certificate authority followed. eMudhra revoked all eight intermediates at 09:38 UTC on 18 October. Apple blocked them on 19 October, Sectigo revoked its cross-signs the same day, and Microsoft followed on 20 October.

Then the browser story ends and the operational one starts. Every non-browser client trusts a CA bundle on disk, skips revocation checks by default and skips Certificate Transparency, a list that covers curl, Python, Node, Go, Java, git, package managers, and the scripts and daemons around them. Revoking a certificate does nothing for a client that never asks whether it was revoked. Helme tested a forged certificate for api.stripe.com against curl, Python and Node with default settings, and all three accepted the chain. That is the gap between an incident a browser closed within hours and one that lasts until an image is rebuilt.

The Mozilla dev-security-policy thread of 20 October 2026 (groups.google.com, captured the same day), where Mozilla's Dennis Jackson published the affected certificate list and the disclosure timeline.

The trust stores that live in code

Go took the most direct route. Releases 1.27.3 and 1.26.10 reject any chain containing one of the eight keys under CVE-2026-111943, the check sits in the standard library and covers Go's own verifier plus the macOS and Windows system-verifier paths, and the change adds no setting to switch it off. A program picks it up when it is rebuilt with the new release.

The other fixes work by taking the root out of the bundle: NSS 3.131.1 drops emSign ECC Root CA - G3, Python certifi 2026.10.19 drops the same root, Rust webpki-roots 1.0.10 drops it, and Homebrew's ca-certificates excludes it along with emSign Root TLS CA - G3 and AAA Certificate Services. Removing a root also cuts off every legitimate certificate that chains through it, and Mozilla said some servers will become unavailable to non-Firefox users of its trust list until those servers move to new certificate chains.

A Node.js fix is still pending, which means a large population of services keeps the old bundle until a base image is rebuilt.

Netics editorial timeline assembled from the root-program reporting of 2026-10-20: Chrome's CRLSet at 19:45 UTC, Mozilla's OneCRL block at 22:59 UTC, eMudhra's revocation at 09:38 UTC on 18 October, and the NSS root removal on 20 October.

What an administrator can check on Monday morning

Start with an inventory. Every container image, every language runtime and every toolchain that ships its own bundle holds a CA list that ages with the image, and a review that covers certificate handling and trust configuration will surface those lists faster than an incident will; that is the ground security audit and hardening covers in practice. The practical questions are concrete: which base images carry the emSign root, which services are built with a Go release older than 1.27.3, and which machines accept a test chain through EM DV TLS CA - G3B right now.

Then there are the devices nobody can update quickly. Routers, printers, televisions, appliances and industrial controllers with frozen firmware will trust these certificates until the intermediates expire in July 2039, and a forged certificate presented to one of them produces the same padlock a user expects. In our reading, Windows code signing carries the same flaw on a different path: software signed with EM OV CS CA - G3B or EM OV CS CA - G3C looks like signed software from a real publisher to anything that skips revocation.

eMudhra's own public site (emudhra.com, captured 2026-10-20), which presents the company as a digital trust provider and lists the emSign product family behind the affected intermediates.

Where eMudhra's trust goes from here

Mozilla removed emSign ECC Root CA - G3 from NSS on 20 October. Chrome will remove the same root in version 43 of its root store and is reviewing eMudhra's other root in that store, emSign Root CA - G1. The Chrome Root Program said it checked every other CA certificate in its store and found no other instance of this failure mode; that check covers Chrome's own store only, and the review of emSign Root CA - G1 is still open.

The root cause is documented, and the accountability question remains open. eMudhra's account inside Mozilla's ticket describes a bug in the ceremony script that created a second weak key pair on the HSM during a key-wrapping test, and labels similar to those of the intended keys, and the weak keys were the ones used for the certificate requests. A warning existed in May 2025, when the CRLs for the six TLS intermediates began failing signature checks and Sectigo filed a bug about it; eMudhra re-signed those CRLs with the keys matching the certificates and closed the incident as a process error.

For the certificate authority, that decision cost sixteen months of trust, counted from the May 2025 CRL warning; the seventeen-month exposure window runs from the creation of the keys in April 2025. For everyone running a fleet, the bill arrives on a schedule set by the oldest base image in the estate, and a printer whose frozen firmware ships the emSign root can still trust this chain in 2039.

Originally published on the Netics blog.

A Cost Comparison Method for Leaving the Public Cloud

Taha Echakiri — Fri, 23 Oct 2026 14:00:34 +0000

TL;DR

A cloud exit is a project that moves workloads off a provider's rented infrastructure onto servers the organisation operates and pays for directly, and the cost comparison is the arithmetic that decides whether the move pays for itself.
The method has five steps: inventory twelve months of invoices, classify every line as storage, transfer or compute, price each bucket at published rates, add hardware, racks, power and people, then set a break-even date.
Published inter-region rates show how providers meter transfer; the exit itself is priced separately, at the provider's internet egress rate. Cloudflare charges nothing for egress on R2, Microsoft charges $0.02 per GB between regions inside Europe, and Amazon's own S3 example prices inter-region transfer at $0.02 per GB.
The best-documented public example is the projection 37signals published in 2023: $3.2m of cloud spend in 2022, a hardware order of about $600,000, $720,000 a year for eight racks of space and power, and roughly $840,000 a year against $2.3m of cloud servers.
The costs the invoice leaves out — redundancy, spares, patching, on-call and the migration itself — decide whether a defensible saving survives contact with operations.
A break-even date turns the whole argument into one number a finance team can approve or refuse.

The cloud invoice prices units of consumption

A cloud bill is a measurement, and it measures one thing: consumption of units the provider defines. Storage is billed per gigabyte-month, transfer per gigabyte leaving a region, requests per million calls, compute per second of a machine that exists because a tenant asked for it. That structure is useful while the workload stays on the provider's side of the wall, because the numbers arrive already aggregated and already trusted by the finance team.

Microsoft's Azure bandwidth pricing page (azure.microsoft.com/en-us/pricing/details/bandwidth/, retrieved 2026-10-18), which lists data transfer in as free and prices intra-continental transfer between regions within Europe at $0.02 per GB and from Europe to other continents at $0.05 per GB.

The problem with using the same document to price an exit is that it never records what an organisation stops paying for. A server the company owns has no per-second meter, so the cost of running it appears in a hardware invoice once, in an electricity contract every month, in a rack rental every month, and in salary every month. None of those four lines sit on the cloud invoice, which is why the two sides of the comparison are usually assembled by different people at different times with different assumptions.

Reading the invoice as a price list first, and only afterwards as a bill, is the step that makes the rest of the method possible. Every line on it belongs to one of three buckets, and the three behave differently when a workload leaves.

Start with twelve months of invoices

In Netics' analysis, twelve months is the shortest window that contains a seasonal peak, an annual renewal and at least one incident. A quarter of billing data will usually understate the peak that sized the environment in the first place, and the peak is what an on-premises design has to absorb.

Pull the line items, and keep the totals for cross-checking. Providers expose per-service and per-region breakdowns, and the interesting rows are the small recurring ones: snapshots nobody deleted, log storage that grew quietly, cross-region replication for a bucket used by one application, load balancers kept alive for a test that ended. Each of those has a hardware equivalent that is either needed or avoidable, and the answer changes the total.

Group the twelve months by application where the tagging allows it. An organisation that cannot attribute spend to an application cannot price an exit per application either, and the fallback is a single all-or-nothing migration that carries far more risk than moving one workload at a time. Tagging discipline is the cheapest prerequisite in the whole project.

Classify every line as storage, transfer or compute

The three buckets translate to on-premises costs in three different ways, and mixing them produces the arithmetic errors that make exits look either impossible or free.

Storage is the easiest to translate. A gigabyte-month on a provider's object store maps to disks, a filesystem, a replication policy and a backup target. The unit price of the disk will look dramatically lower, and the honest comparison includes the second copy, the off-site copy and the person who tests restores. Cloudflare's own documentation lists standard storage at $0.015 per GB-month, and this article takes that published R2 rate as the yardstick a self-hosted terabyte has to beat once redundancy is counted.

Cloudflare's R2 pricing documentation (developers.cloudflare.com/r2/pricing/, last updated 2026-10-01, retrieved 2026-10-18), which lists egress as free for Standard and Infrequent Access storage and prices Standard storage at $0.015 per GB-month.

Transfer is where published prices diverge most. Microsoft charges nothing for data transfer in, and $0.02 per GB for intra-continental transfer between regions within Europe. Cloudflare states plainly that there are no charges for egress bandwidth on R2 for any storage class. Amazon's S3 pricing example prices inter-region transfer out at $0.02 per GB. These inter-region figures show how each provider meters transfer between its own regions. Data leaving for the organisation's own servers is billed at the provider's internet egress rate, which the migration step below prices. An organisation whose bill is dominated by data leaving a region is comparing against a colocation cross-connect and a bandwidth commit, and that comparison has a completely different shape from one dominated by compute.

Compute is the bucket that needs the most care. A reserved instance, a savings plan and an on-demand hour are three different prices for the same silicon, and the on-premises equivalent is a server amortised over a realistic life with a spare capacity margin. Reading only the discounted rate flatters the cloud; reading only the on-demand rate flatters the exit.

Price each bucket at published rates

Published rates matter because they are the only figures both sides of the argument accept without negotiation. A vendor price sheet can be cited in a board paper; an internal estimate cannot.

The exercise is mechanical once the buckets exist. Take the storage gigabyte-months from the invoice, price them at the cheapest tier the workload actually tolerates, and add the redundancy copies the current design implies. Take the transferred gigabytes and price them at the destination rates the provider publishes for the region pairs in use. Take compute hours at the commitment level the organisation genuinely runs, and note separately what the same hours would cost on demand, because that gap is the first thing a provider will offer to close.

Doing this exercise before any migration decision usually produces one of two findings. Either the bill is dominated by a service with no realistic self-hosted equivalent — a managed queue, a global anycast edge, a database with a control plane nobody wants to operate — in which case the exit discussion narrows to the workloads that can move. Or the bill is dominated by commodity compute and storage, in which case the numbers are large enough that the operational question becomes worth answering.

Netics editorial sequence diagram of the method: inventory twelve months of invoices, classify each line, price each bucket at published rates, add the costs the invoice omits, and set a break-even date before committing.

Either finding is useful, and both are cheaper to reach on a spreadsheet than after a migration.

Add what the invoice leaves out

The exit bill has five components that never appear on a cloud invoice, and each has a real price.

Hardware is the visible one: servers, disks, network gear and spares, amortised over a life the organisation is willing to defend. Racks and power come next, usually as a monthly line from a colocation provider or as an internal charge for space and cooling. Redundancy is the third and the most consistently underestimated, because the provider absorbs a failed disk or server without the tenant doing anything, and in a rack that work falls to the organisation.

People are the fourth. Running infrastructure means patching, capacity planning, certificate renewal, backup verification and on-call rotation, and the honest version of the comparison names the hours. An organisation that already employs the people does not get to price them at zero, because those hours were previously spent elsewhere.

The migration itself is the fifth, and it is a one-off cost with a long tail. Moving data out of the provider to the organisation's own servers counts as egress to the internet, which the inter-region rates quoted above do not cover, so the exit fee has to be priced from the provider's own internet egress rate, sometimes on a very large volume. The tail is the period where both environments run in parallel and the organisation pays for both.

Set a break-even date

A comparison without a date is a debate. With a date, it becomes a decision rule: the move happens if the accumulated saving crosses the accumulated cost within a period the organisation is willing to wait.

The arithmetic is straightforward. Take the annual cloud spend attributable to the workloads being moved, subtract the annual run cost of the replacement, and divide the one-off cost — hardware, migration, parallel running — by that annual difference. The result is a payback period in years, and it should be stated with the assumptions that produced it visible on the same page.

Two sensitivities belong next to that number. The first is hardware life: a four-year refresh shortens the period, a seven-year one lengthens it. The second is the rate at which the remaining cloud spend grows, because a workload that grows ten percent a year changes the crossover date and a shrinking one may never justify the move at all.

Netics editorial diagram drawing the two sides of the comparison: a single monthly cloud invoice covering compute, storage and transfer, against an exit bill spread across hardware, racks, power, redundancy, spares and operating hours.

A payback period of a few years is a decision most finance teams can take. A payback period that depends on optimistic assumptions about both hardware life and growth is a decision to revisit, and saying so is part of the method.

What the arithmetic looks like when a company publishes it

Publicly itemised exits are rare, which makes the 37signals account of February 2023 worth reading as a partial worked example of the method, with two caveats: its figures are the company's own preliminary projection, published before the move, and they cover the run-cost side only. The post gives no migration or parallel-running cost and no break-even date, so the last step of the method has no worked figure here.

The company spent $3.2m on cloud in 2022. Just under a million of that stored eight petabytes of files in S3, fully replicated across several regions; the remaining roughly $2.3m covered application, cache, database and search servers, and that was the part it intended to bring to zero. The hardware order came to somewhere in the region of $600,000, which the post amortises over a conservative five years at $120,000 per year. Eight dedicated racks across two data centres cost about $60,000 per month through a colocation provider, which is $720,000 per year, with enough space and power already provisioned to absorb the new machines.

Add those together and the run cost is about $840,000 per year against $2.3m in the cloud. The post calls the difference a saving of roughly $1.5m per year, sets aside half a million for unforeseen expenses over the period, and arrives at seven million dollars saved over five years in server expenses, with the ops team unchanged in size. Those are the post's rounded figures: $2.3m minus $840,000 is about $1.46m a year, about $7.3m over five years, and about $6.8m once the $500,000 is set aside. The seven-million figure covers the roughly $2.3m of servers only: the post defers the exit of the eight petabytes from S3 to 2024 and gives no cost for moving them out.

The value of the example is that every number has a source and a position in the method, and its weak points are just as visible. Storage was separated from compute. Hardware was amortised across five years. Racks and power were priced as a monthly line. The saving was stated as one rounded figure with a $500,000 contingency, and the five-year window was declared. Two assumptions need the scrutiny the method asks for. The only people cost is an ops team unchanged in size, with no hours named, which is exactly the line the method refuses to price at zero. And the figures are the company's own rough, preliminary calculations, made before the move, so they illustrate the method and leave the outcome unproven. An organisation can disagree with any single assumption and still use the structure to reach its own answer.

Originally published on the Netics blog.

A Vendor Backup Secret Opened Over 300 Microsoft 365 Tenants

Taha Echakiri — Fri, 23 Oct 2026 14:00:03 +0000

TL;DR

ConnectWise Cloud Backup is a hosted backup service, formerly sold as SkyKick, that managed service providers connect to their customers' Microsoft 365 tenants so that mail, files and Teams data are copied somewhere outside the tenant.
A secret belonging to that application was compromised and used to sign in to customer tenants as the backup service itself, and Huntress confirmed unauthorized sign-ins in 162 tenants on 16 October 2026, then counted almost 1,000 Entra tenants with sign-in activity from the attacker's address on 17 October, over 300 of them successful.
The abuse reached further than the backup app: service principals for ConnectWise Cloud Management applications assigned themselves directory roles, mostly Exchange Administrator, and in one tenant an application granted itself Global Administrator and added credentials to two other applications.
Multi-factor authentication had no bearing on these sign-ins, because no user was authenticating; the application presented its own secret and used permissions the customer had already granted.
ConnectWise shut down application access to Cloud Backup and stated it has no indicators that partner backup data was compromised; Microsoft described the activity as an active supply-chain attack.
Disabling the named applications stops new use, and the credentials those applications added stay valid, so containment runs through an audit of every secret on every enterprise application in the tenant.

What a backup application holds inside a tenant

An application that backs up Microsoft 365 has to read what it backs up. To do that it is registered in the customer's tenant as a service principal and granted Microsoft Graph and Exchange permissions that allow it to read mail, files and chat, and in many deployments to write them back during a restore. The grant is made once, by an administrator, during onboarding.

Huntress's investigation page, published 16 October 2026 and updated on 17 October (huntress.com/blog/vendor-credential-enables-unauthorized-access, captured 2026-10-18), listing the compromised application IDs and the containment steps for affected tenants.

That grant is not a session. It does not expire with a working day, it does not ask for approval when the application authenticates, and it does not depend on any person being present. The application authenticates with its own secret, held on the vendor's side, and the tenant accepts the request because the permissions were granted months or years earlier.

Every backup, ticketing, monitoring, HR or signature product onboarded this way carries the same property. The vendor's secret becomes a standing key to the customer's tenant, and the customer's only view of it is a row in the enterprise applications list, usually with the consent screen nobody re-reads.

How the intrusion reached over 300 tenants

The activity was visible to administrators before it was public. A user on Reddit reported a Microsoft Privileged Identity Management notification on 15 October that a SkyKick service principal had assigned itself the Exchange Administrator role, in a tenant that had not used the application for years. Microsoft later confirmed unauthorized authentication attempts on 15 October, and ConnectWise stated in its advisory that it had "recently identified unauthorized access to application identities within the ConnectWise Cloud Backup environment", then shut down application access for partners and clients.

Huntress, which monitors customer tenants, attributed the sign-ins to the legitimate Cloud Backup application ID 506e08f9-6e9a-4da4-942b-a099a03644ae and confirmed unauthorized sign-ins in 162 tenants as of 19:00 UTC on 16 October. Its update of 17 October raised the scale: almost 1,000 Entra tenants showed sign-in activity from the attacker's address, and over 300 of those succeeded. The Hacker News put the figure at 173 tenants on the same day. Both accounts trace the authentication to the address 144.172.64.128, and Huntress linked that address to scripted sign-in attempts on 13 and 14 October using the curl user agent 7.88.1, alongside python-urllib and python-requests agents.

Netics editorial diagram placing the 162-tenant count against the mechanism: an application secret presented directly to the tenant, with no user prompt in the path.

The escalation moved beyond the backup application. Huntress reported that service principals for the Cloud Management applications covering Exchange Online, Teams and Microsoft Graph assigned themselves directory roles, mostly Exchange Administrator, across nearly fifty of the tenants it monitors. In one partner tenant the Cloud Management - MS Teams application, ID 24205314-2b89-4e07-a2a4-579e898a5640, granted itself Global Administrator and then added new credentials to two other applications, including a single-tenant Huntress application. Todyl, analysing intrusions on nine tenants, found only sign-in activity and no evidence of created accounts, changed passwords or persistence through the backup app. Other responders saw persistence. Microsoft observed a backdoor after successful service principal authentication, followed by Microsoft Graph API reconnaissance and attempts to add additional credentials, and in the tenant where the Teams application became Global Administrator, Huntress recorded a Python script making nine changes to the two other applications, six of them successful, with the new secrets set to expire in 2028.

Why multi-factor authentication covered nothing here

Multi-factor authentication verifies a person. The sign-ins in this incident presented a client secret, so there was no person to verify and no prompt to approve. Todyl's account is explicit: the attacker authenticated directly to Microsoft 365 as the backup application using its secret, and because the requests did not involve a user login, protections such as MFA were ineffective.

The permissions already granted did the rest. Directory.ReadWrite.All, which Microsoft's threat intelligence assessed as one of the permissions likely abused, is a directory-wide write permission; in our reading, an application holding it reaches well beyond the mail and files a backup service exists to copy.

Conditional access policies that gate interactive sign-ins therefore leave this path open, and a policy written to require compliant devices for administrators does not touch an application token. The control that covers it is narrower and less comfortable: know which applications hold which grants, and rotate their secrets on a schedule that assumes the vendor's side will eventually be compromised.

ConnectWise's trust advisories page (connectwise.com/company/trust/advisories, captured 2026-10-18), carrying the Cloud Backup, formerly SkyKick, service interruption notice and the statement that application access was shut down for partners and clients.

The containment sequence the responders published

Huntress's guidance orders the work in a way worth copying. Affected organisations should disable two application IDs immediately, Cloud Backup 506e08f9-6e9a-4da4-942b-a099a03644ae and Cloud Management - MS Teams 24205314-2b89-4e07-a2a4-579e898a5640, and consider disabling the other Cloud Management IDs for the duration. The instruction is to disable the applications and keep them: deleting an application removes its consent, so service can only come back through re-consent, and disabling pauses backups while leaving the backups already stored in place.

Disabling stops new use and removes nothing that already happened. Huntress states the consequence plainly: disabling the applications does not remove credentials that were already added, so every credential on every Microsoft 365 application in the tenant has to be reviewed. That review is the part most likely to be skipped under pressure, and it is the part that decides whether the incident ends.

Re-enabling waits on the vendor. Huntress advises keeping the applications disabled until ConnectWise confirms that all credentials have been reset. On 17 October at 5:15 PM ET, ConnectWise communicated that it had disabled the relevant apps, and the vendor's advisory says it has no indicators to date that partner backup data was compromised and that the existing cloud backup archive is not impacted by the service interruption. Its update of 18 October at 01:00 UTC, reported by BleepingComputer, states that scheduled backups are not running and that access to affected backup archives and restoration capabilities remains unavailable, and adds that the existing archives remain encrypted and isolated, with no indication that they have been compromised. For a small organisation, that means no restore from this service until access returns.

A second wave is worth noting for anyone who reads the first one as solved. Huntress identified further attempts from the same address on 15 October against 16 of its protected tenants in the policy, national security, trade and critical mineral sectors, and most of those tenants did not use ConnectWise products, so the attempts failed because the application was not present. Huntress's update of 19 October adds the rest of the picture so far: across more than 10,000 tenants with ConnectWise applications, only 144.172.64.128 and Microsoft-internal ranges produced application sign-ins, while 144.172.64.99, 144.172.64.60 and 144.172.94.243 showed password spraying or failed logins. It found post-compromise changes in a second tenant, made around 03:45 UTC on 15 October like those in the first, placed the last successful sign-in at 08:44 UTC that day, and limited containment to tenants with confirmed unauthorized sign-ins, on the evidence that the activity had stopped.

What a small organisation can check in its own tenant

The first check is an inventory, and it is short. Open the enterprise applications list in the Microsoft 365 admin centre and export it, then for each third-party application look at the permissions it holds and the date consent was granted. Applications that no longer serve a purpose are the cheapest finding on the list, and the Reddit report of a service principal still active in a tenant that had stopped using the product years earlier is the reason to look.

Netics editorial checklist derived from Huntress's published guidance: disable rather than delete, review credentials on every application, rotate secrets, search sign-in logs for the source address, and re-enable only after the vendor confirms the reset.

The second check is the sign-in log. Searching for the address 144.172.64.128 and for the user agents python-urllib, python-requests and curl/7.88.1 over the exposure window costs a few minutes and gives a first answer for the tenant in question. Huntress warns that these user agents are low fidelity on their own and need the address beside them, and that the directory role assignments of 15 October carried kiota-dotnet, an agent ConnectWise's own applications also use. It also notes that Microsoft's audit logs record no source address for app-only actions and that directory reads through Microsoft Graph leave no audit trail, so an empty search proves little; Microsoft recommends reviewing application credential changes, OAuth grants and related Graph activity as well.

The third is secret hygiene on the applications that remain. A vendor application's credentials should have an owner, a rotation date and a record of who can read them on the vendor side, and none of the three is standard practice in small organisations. Where a provider offers certificate-based authentication with a client certificate stored in a managed vault, that is a materially better arrangement than a shared secret.

The fourth is a written answer to a question most vendors never hear: if your environment were compromised tomorrow, which of our credentials would the attacker hold, and how quickly would you tell us. The tenants opened in this incident, more than 300 by Huntress's count of 17 October, were reached through a service the customers had chosen to trust, and the trust was expressed as a permission grant they had not looked at since onboarding.

Netics runs security audit and hardening engagements that can include this inventory and the app-consent review that goes with it.

Originally published on the Netics blog.

Qubic's Wallet Drain Traces Back to Predictable Randomness

Taha Echakiri — Fri, 23 Oct 2026 09:00:02 +0000

TL;DR

Qubic is a layer-1 blockchain that runs fee-free transactions and smart contracts on a consensus mechanism called Useful Proof of Work, which redirects mining energy into training artificial intelligence models.
Between 23:42 UTC on 17 October and 01:26 UTC on 18 October 2026, an attacker drained 1,404 wallets of roughly 2.94 trillion QU, valued at about USD 1.4 million.
The wallets fell because their recovery seeds came from a generator with insufficient randomness. The project states the fault sat in its wallet software and that the chain protocol itself was not compromised.
Affected seeds were created in any browser on an iPhone or iPad, in Safari on a Mac until 18 October 2026, or in the Qubic iOS app before July 2025.
Qubic halted the chain at tick 85,371,949; 87.1 percent of the drained funds stayed on the network and froze in place, and exchange MEXC froze the account that received the other 12.9 percent within about an hour.
Web wallet version 4.26.4, dated 18 October, replaced JavaScript's Math.random() in seed creation with crypto.getRandomValues(). That fixes new seeds, and it leaves every seed already created in the old code as it was.

The 104-minute window and the wallets inside it

The attack ran for 104 minutes. Qubic's own statement puts the first drain at 23:42 UTC on 17 October and the last at 01:26 UTC on 18 October, and the count it published is 1,404 wallets holding roughly 2.94 trillion QU. Cointelegraph reports the haul at about USD 1.4 million.

Qubic's own network explorer (explorer.qubic.org/network, captured 2026-10-19), which reports the chain's current tick and the tick table behind it, the counter Qubic used to name the halt point. Qubic halted the chain at tick 85,371,949.

A rate of roughly thirteen wallets a minute is the detail worth sitting with. As Netics reads it, once a seed can be derived, the work per wallet collapses to a computation. Wallet count then measures the size of the vulnerable population.

Qubic is a layer-1 chain that carries transactions and smart contracts without fees, built on a consensus design called Useful Proof of Work that repurposes mining energy for AI training workloads. Qubic halted the network at tick 85,371,949.

A seed generator built for animation timers

Qubic's account of the cause is one sentence long and it is the whole story: seeds were generated with insufficient randomness. crypto.news reports that the fix in web wallet version 4.26.4 replaced JavaScript's Math.random() in the seed-creation routines with crypto.getRandomValues().

Those two calls look interchangeable in a code review, and their guarantees differ. Math.random() is built for convenience: it produces a number that looks scattered, quickly, and it is specified without a promise of unpredictability. MDN's JavaScript documentation warns that it is not cryptographically secure and should not be used for anything related to security. crypto.getRandomValues() is built for unpredictability. It draws from the platform's cryptographic generator.

The distinction is old and it keeps costing money. crypto.news ties this incident to a class of failure researchers call Milk Sad, a faulty key-generation process that lets an attacker predict private keys, and Cointelegraph places it in a line that runs through Coldcard's July disclosure of wallets weakened by a firmware bug and, in 2023, Trust Wallet's browser extension and the Libbitcoin Explorer tool. Weak-seed failures form a recurring class, and Qubic is the latest case in it.

Netics editorial diagram: the change in Qubic's web wallet release 4.26.4, where seed creation moved from Math.random() to crypto.getRandomValues(), the browser interface built for cryptographic values.

The three environments Qubic named

Qubic listed the affected seeds by where they were created: any browser on an iPhone or iPad, Safari on a Mac until 18 October 2026, and the Qubic iOS app before July 2025. That list is more specific than most incident notices and it says something about how the vulnerability behaved.

If the defect were purely in one function, every environment would be equally exposed and the list would be unnecessary. The list implies a dependency on the platform underneath: the same call may have produced output of different quality depending on the engine that implemented it, and every entry on the list is an Apple platform. This reading is Netics analysis, and no source confirms it. Whether the underlying weakness came from the generator's seeding on those platforms or from the wallet's own use of it, the practical lesson holds. A key generator inherits the entropy of the runtime it runs in, and a browser is the runtime with the least control over that.

For a user, this is the uncomfortable part: the notice does not say which seeds are safe. Consulting the release notes tells you that new seeds are generated correctly. Nothing in the published material so far establishes that seeds created in the old code became safe once the code changed, because a seed is stored data and a code change does not rewrite it. Rotation is the only remedy that does not depend on an assumption.

What a validator halt can and cannot undo

Qubic stopped the chain, 87.1 percent of the stolen tokens stayed where they were and froze, and MEXC froze the account holding the remaining 12.9 percent in about an hour. That is an unusually complete containment for a crypto incident, and it is worth being precise about why it worked.

It worked because the asset never became bearer cash. Theft on a chain is a state change that the network agrees on, so a coordinated halt converts a completed theft into a pending one, and a centralised exchange converts an off-chain exit into a frozen account. Both moves need something that a bearer asset does not have: an authority that can act after the fact. Qubic has a way to halt its chain, and MEXC was able to freeze the receiving account.

The trade-off is on the record too. A chain that can be halted to protect funds can be halted for reasons its holders do not endorse, and the recovery decision now sits with a vote of the Computors. Qubic says a full technical report follows once a recovery plan is settled. Anyone treating this containment as a property of the asset class should read that report before generalising it.

Netics editorial diagram: the published numbers behind the Qubic drain, including the share of funds that stayed on the network and the share that reached an exchange.

What to check before you trust a key generator

The useful question for anyone who ships software that creates keys, seeds or recovery phrases concerns three things: which function produces the value, in which runtime it runs, and whether the answer stays unpredictable to someone who can read the code.

Four checks catch this class of bug before an incident report does:

Name the API in the code review. A seed that comes from Math.random(), a language's default shuffle, a timestamp, or a hash of the device name is predictable by construction, whatever entropy the surrounding system claims to have.
Check the source on every platform. The same call can behave differently across browser engines and OS versions, which is exactly what Qubic's three-environment list describes. Confirm that crypto.getRandomValues() is the source on each target runtime. A distribution test cannot prove that a generator is unpredictable: Math.random() can pass one while staying predictable to anyone who models its internal state.
Treat a generator fix as a rotation event. Patching the code protects future seeds. It says nothing about the keys already written down, exported or stored, and those are the ones an attacker can work on at leisure.
Write down the blast radius before the incident. Qubic could contain 87.1 percent because a halt was available and an exchange answered in an hour. Most systems do not, which makes early detection the only containment they own.

For teams in France and Morocco building products on top of key material, the same four checks map onto a procurement question that is easy to ask a supplier: which API generates our keys, on which runtimes, and how do we know the output is not reproducible. A vendor that cannot answer that in writing is asking you to take on the entropy risk without a description of it. We build and review exactly that kind of boundary in our security work, and the first output of the exercise is usually the answer to a question nobody had asked before.

Qubic's own wallet release notes (github.com/qubic/wallet/releases, captured 2026-10-19), which list version 4.26.4 dated 18 October 2026 and the fix line naming cryptographically secure seed generation: the change that protects future seeds and leaves every earlier one untouched.

Originally published on the Netics blog.

A Public Exploit Escapes Ubuntu Containers Before the Patch Ships

Taha Echakiri — Thu, 22 Oct 2026 19:00:03 +0000

TL;DR

On September 22, 2026, security firm DepthFirst published a working container-escape exploit for CVE-2026-80521, a heap use-after-free in the Linux kernel's AF_UNIX socket subsystem, demonstrated against Ubuntu 26.04 LTS.
The upstream kernel fix landed on August 6, 2026 — seven weeks earlier. As of September 24, Ubuntu's security tracker still lists the main kernel package on 26.04 as "Vulnerable, work in progress" and 24.04 as vulnerable, including AWS, Azure, and GCP kernel variants.
The vulnerable code is reachable from inside default Docker and Kubernetes containers: AF_UNIX sockets and SCM_RIGHTS descriptor passing are permitted by standard seccomp profiles, so the race bypasses namespace isolation, cgroup limits, and seccomp filtering in one chain.
No confirmed in-the-wild exploitation and no CISA KEV listing as of publication — but public exploit code plus an unpatched LTS distribution is exactly the combination that stops being academic.
The disclosure timeline is the story: kernelCTF slot on July 24, disclosure to security@kernel.org on August 5, upstream fix on August 6, public exploit on September 22 — with Ubuntu's kernel still listed as vulnerable on September 24.
5,976 unique Linux kernel CVEs were published between January and mid-September 2026, with August alone peaking at 1,650 — the AI-discovery economics that make public exploits before distribution patches the new normal.
DepthFirst's recommendation is structural: move untrusted and multi-tenant workloads to microVMs (Firecracker, Kata Containers) that give each workload its own kernel.

The seven-week gap nobody benefits from

The story of CVE-2026-80521 is not the bug; kernel use-after-frees are routine. The story is the timeline. DepthFirst won a Google kernelCTF slot with the zero-day on July 24, reported it to the kernel security team on August 5 — the team replied that Kyle Zeng of OpenAI had independently reported the same bug — and the fix shipped upstream on August 6. Then the exploit sat with a public timeline while the distributions caught up. DepthFirst published the write-up and exploit on September 22. As of September 24, Ubuntu's own tracker says of the main kernel package on Ubuntu 26.04 LTS: "Vulnerable, work in progress." Ubuntu 24.04 LTS is listed as vulnerable, and so are the AWS, Azure, and GCP kernel packages for both releases. A reproduction of the tracker's status is below.

Official Ubuntu security tracker page (ubuntu.com/security/CVE-2026-80521, captured 2026-09-29): CVSS 7.8 High, Ubuntu priority Medium, with the linux package on 26.04 listed as "Vulnerable, work in progress" and on 24.04 as "Vulnerable."

This is precisely the window that public exploit code makes dangerous. There is no confirmed evidence of in-the-wild exploitation yet, and the CVE is not in CISA's Known Exploited Vulnerabilities catalog — but the exploit is on GitHub, targeting the newest LTS, and the fix has been upstream since August 6. Every day the distribution backport lags is a day where a competent operator with no novel skill can run a published exploit against a production host. That is the difference between a theory and an exposure.

A race in the AF_UNIX garbage collector

The bug lives in the kernel's AF_UNIX socket subsystem, the foundation of local inter-process communication on Linux. Every local database connection, systemd socket, and Docker interaction touches it. AF_UNIX sockets let processes pass file descriptors to each other using SCM_RIGHTS messages, and the kernel garbage collector tracks those references to clean up circular dependencies. The flaw is a race inside that collector. When a process sends file descriptors, the kernel publishes the new graph edges before the socket buffer that carries them is queued: scm_stat_add() calls unix_add_edges() to make the edges visible to the collector, then the buffer is queued with __skb_queue_tail() a few instructions later. Because unix_add_edges() takes and releases the unix_gc_lock, the collector can interrupt the send path and see edges whose owning socket buffer is not yet on the queue. If the collector then decides a strongly connected component is dead, it can free part of it — through unix_del_edge() and unix_free_vertices() — without unlinking the vertex from the persistent scc_entry ring. A surviving socket in the same SCC still points into freed memory. The next collection pass walks the stale ring via unix_walk_scc_fast() and dereferences freed memory: a textbook use-after-free whose upstream fix is literally named "af_unix: Unlink scc_entry in unix_del_edge()." The fixed commit is 594d905 in net/unix/garbage.c.

Official kernel.org commit page (git.kernel.org, commit 594d905195024b228c962627ae5ae7c17bd582a4, captured 2026-09-29): the fix that unlinks scc_entry before the vertex is freed, addressing the AF_UNIX garbage-collector use-after-free.

Reachability is the second half of the story. AF_UNIX sockets and SCM_RIGHTS descriptor passing are allowed by default Docker and Kubernetes seccomp profiles — they are ordinary operations inside a standard container. The exploit needs no CAP_SYS_ADMIN, no privileged flag, no kernel module loading, no seccomp modification. It bypasses namespaces, cgroups, and seccomp in a single chain because the corrupted structure is shared kernel state. DepthFirst notes the same subsystem is relied on by nsjail, Firejail, and Bubblewrap, so OS-level sandboxes built on the kernel are exposed too, not just Docker and Kubernetes.

The AI-discovery angle changes the economics

DepthFirst did not find this bug by eyeballing code. It used dfs-large1, an in-house AI model trained for vulnerability detection, combined with a human-operated testing harness — and the company is explicit about what the model means for the threat model: "the barrier to escaping containers by attacking kernel has fallen so significantly that we must assume attackers can do so at will." The escalation figures in the write-up are stark: 5,976 unique Linux kernel CVEs published between January and mid-September 2026, with August alone peaking at 1,650 published vulnerabilities — over 27% of the year's total in a single month. Of the 36 CVEs publicly disclosed through Google's kernelCTF program, 13 are reachable from ordinary, unprivileged interfaces in subsystems exposed to containers by default.

Official DepthFirst research image (2026-09-22, "Containers Are No Longer a Security Boundary"): containers share a single monolithic kernel with the host, so a kernel bug reachable from inside a container is a container escape by definition.

That is the uncomfortable synthesis. Containers share the host kernel by design; the kernel has thousands of published holes per year, many reachable from default container syscalls; and AI-accelerated discovery is compressing the time between "bug exists" and "working exploit published." DepthFirst's own numbers frame the shift: 5,976 unique Linux kernel CVEs published between January and mid-September 2026, August alone peaking at 1,650. The traditional stance — patch the distribution kernel when a USN lands — still works, but it now operates on attacker timescales where public exploit code can precede the distribution fix by weeks. This is exactly the pattern our copy-fail analysis of container security warned about: the host kernel is the real boundary, and a kernel bug is a container-escape bug by definition.

The operational response: microVMs for untrusted workloads

With no distribution fix shipped and no workaround published by either DepthFirst or Canonical, the only structural mitigation is to stop sharing the kernel with workloads you do not trust. DepthFirst's recommendation is direct: migrate untrusted and multi-tenant workloads to microVMs such as Firecracker (used by AWS Lambda and Fargate) or Kata Containers. Each microVM gets its own lightweight guest kernel, so even a successful kernel exploit compromises only that ephemeral instance — the host and neighboring tenants stay intact. It is the only response that addresses the architecture rather than the specific CVE.

For everything that must stay in containers, the checklist is short and uncomfortable. Check which Ubuntu kernel flavor your nodes actually run — the stock 22.04 kernel is not affected, but the 6.8-based HWE and the cloud kernels for 22.04 are, and 24.04/26.04 mainline variants are vulnerable. Apply the upstream patch directly if your distribution lags and you can test a rebuild. Watch Ubuntu security notices for the kernel USN and treat the reboot slot as urgent, not scheduled. And if the workload is untrusted — third-party code, customer code, anything multi-tenant — ask whether it needs a shared kernel at all.

Netics editorial diagram: the timeline — July 24 kernelCTF slot, August 5 kernel-team disclosure (with OpenAI's Kyle Zeng reporting independently), August 6 upstream fix, September 22 public research and exploit, and an Ubuntu kernel still listed as vulnerable at the end of the tracker's September update.

The era where containers were assumed to be a robust isolation boundary is over; that is DepthFirst's title, and the timeline of this CVE is its evidence. Infrastructure that assumes the kernel can be breached has to stop putting the kernel in reach. More infrastructure and container-security analysis is on neticslabs.com.

Originally published on the Netics blog.

DeepSeek Just Published the Blueprint for Training Agents at Scale

Taha Echakiri — Thu, 22 Oct 2026 14:00:39 +0000

TL;DR

The headline claims are production measurements: about three million sandboxes a day, over 380,000 concurrent, more than 5,000 creations per second, from a scale unit of about 160 nodes.
Four backends behind one libdsec SDK — FnCall, container, microVM, full VM — with different isolation and startup cost.
Most agent sandboxes are nearly idle — about 90% draw no more than 5% of requested CPU — while memory stays pinned and images see low fanout.
Density comes from EROFS layers loaded on demand from 3FS, virtio-pmem DAX for read-only layers, and DAMON balloon free-page reporting.
RL co-design keeps rollouts alive across GPU preemption: pause/resume, agent-built environments, access controls hardened against agents that try to cheat.
The honest costs: heavier VM backends, persistent workspaces, and a residual latency toll — treat the sandbox tier as a platform decision, not glue code.

The 160-node sandbox factory behind the report

On September 19, 2026, DeepSeek published its DSec technical report on arXiv, documenting the production platform that has carried agentic training and evaluation from DeepSeek V3.2 through V4.1. The abstract: supporting such workloads "requires an elastic execution platform rather than a single sandbox runtime".

Our stance is stronger: training agents at scale is an infrastructure problem first, and DSec is the most complete public blueprint yet — elastic backends, on-demand image delivery, high-density memory management, RL co-design.

Netics editorial visual: DSec production scale — about 3 million sandboxes a day, 380,000 concurrent, over 5,000 creations per second, per the DeepSeek report.

Three million short-lived sandboxes a day

One scale unit spans roughly 160 nodes and serves about three million sandboxes a day — over 380,000 concurrent, over 5,000 creations per second — with single jobs requesting up to 32,000 instances in a burst. Median lifetimes: 17.4 minutes for containers, 15.5 for microVMs, p99 over three hours. Per-node peaks: 1,048 containers and 524 microVMs; stable operation runs 3,200 containers or 800 microVMs per node.

Four isolation backends behind one libdsec client

DSec exposes four backends through one Python SDK, libdsec: FnCall for short, stateless tasks in precreated reusable containers; containers for the bulk of engineering workloads; Firecracker microVMs where isolation matters more; full VMs for complete operating systems. Each container or VM sandbox runs a per-sandbox proxy (aether) with chronus instances for commands and filesystem access; FnCall skips the proxy. Network policy is per sandbox too: the documented example allows PyPI but denies NPM.

Official DeepSeek DSec report, Figure 1: DSec architecture — per-sandbox proxies (aether) mediate between container/VM sandboxes and the platform; FnCall follows a separate path. (arXiv:2609.22978, 2026-09-19)

Pinned memory, idle CPUs, and images nobody reuses

The naive just-run-a-container approach dies on this profile. While an agent waits for the model's next action, a sandbox's "CPU usage is sparse and naturally suitable for overcommit" — about 90% of sandboxes use no more than 5% of requested CPU. Yet sandboxes are stateful: memory, page caches, and writable state stay pinned after the CPU goes idle, so resident cost caps capacity. Images add a second constraint — containers show a median fanout of three, microVMs a median fanout of one — and eager pulling concentrates load on the distribution path: the report's ablation shows eager pulling stretches completion time by 1.7× while on-demand loading cuts disk writes by 57%.

EROFS layers, on-demand reads, and a 21.2% memory reclaim

Environments compose into independently versioned read-only EROFS layers, served block-by-block on demand instead of unpacked from tarballs. Image data lives in 3FS, fetched only when accessed and in bulk, while writes stay on local disk; a de-duplicated EROFS set totaling 30 TB covers the files accessed by 70% of container tasks. Memory density gets two mechanisms: virtio-pmem with DAX removes page-cache duplication for read-only layers, and DAMON with virtio-balloon free-page reporting, which the evaluation says "reduces memory consumption by 21.2% without significant CPU overhead". For CPU, sandboxes are latency-sensitive (LS) or best-effort (BE); BE sits under SCHED_IDLE with core scheduling, cutting SMT latency inflation from 45.2% to 17.3%.

Netics editorial visual: the four DSec sandbox backends — FnCall, container, microVM, full-VM — behind the libdsec SDK, with per-sandbox network rules.

Rollouts that survive GPU preemption, agents that try to cheat

DSec "decouples stateful rollout execution from preemptible GPU training". Earlier generations ran the agent loop inside the preemptible GPU pod and recovered via command-log replay; since V4.1, rollout execution lives on DSec — a worker container plus an agent sandbox, both outside the GPU pool — as the single source of truth. On preemption, the framework pauses sandboxes — containers freeze process trees and swap memory; microVMs snapshot, terminate, restore — and rollouts resume. Environments follow the heading "Build environments of Agents, by Agents, for Agents": agents checkpoint sessions via pack_diff, turning incremental snapshots into reusable environments with no separate image pipeline.

The report is equally blunt about untrusted agents: they forge requests to chronus sockets and overwrite /bin/bash; after file and socket access controls landed, "an agent attempted to bypass them using XFS_IOC_SWAPEXT" — an ioctl swapping the data-extent mappings of two files — and the attempt corrupted XFS metadata and forced a filesystem shutdown. Access control is a live security problem, the principle behind our zero-trust boundaries outside the model piece.

Official DSec report, Figure 9: overview of the core system mechanisms — composable layers, high-density resource management, image distribution, and RL co-design (arXiv:2609.22978).

The honest costs of a sandbox factory

The inconvenient parts are in the source, so we state them. Full-VM and microVM backends are heavier to operate, and microVM workspaces persist as task-specific disk images — 53,590 of them in one production week, on top of just two shared base images.

QoS is not free: core scheduling limits SMT-induced latency inflation to 17.3% at 50% best-effort load, yet the report concedes residual degradation from turbo reductions plus "memory bandwidth and shared last-level cache (LLC) contention that core scheduling does not address". The 21.2% reclaim is a measured win, not a zero: resident costs remain, and every mechanism adds surface — placement, admission control, proxies, snapshot plumbing.

Treat the sandbox tier as a platform decision

Four decisions follow for any team running agentic training at real scale: pick an isolation tier per workload, deliver images on demand from shared storage, set an explicit overcommit policy — measurement, reclamation, QoS classes — and co-design the training loop with the sandbox layer so preemption stops destroying rollout state.

Netics editorial visual: what the sandbox-factory approach buys — on-demand image layers, about 21.2% of memory reclaimed by DAMON, RL co-design — and what it costs in operating complexity.

Run the exercise on your own fleet: list the environments, measure the fanout, count the minutes sandboxes actually burn CPU. If your stack still builds a fresh environment per task and pulls the full image first, DSec shows the ceiling. The harness half is being productized fast — our breakdown of OpenAI's Agents API covers it — and the execution tier is now public knowledge too. More analysis lands on neticslabs.com.

Originally published on the Netics blog.