DEV Community: Indra Gusti Prasetya

Fix conflicting SELinux labels of volume in K8s 1.37

Indra Gusti Prasetya — Fri, 28 Aug 2026 04:42:57 +0000

Kubernetes 1.37 shipped on 26 August 2026, and with it SELinuxMount went stable and on by default. A qualifying PersistentVolume is now handed to the container runtime with -o context=<label> instead of being walked inode by inode. On a volume with four million files, that converts minutes of pod startup into milliseconds, which is the reason KEP-1710 exists in the first place. It also means a mounted volume carries exactly one SELinux label for as long as it stays mounted, and two pods with different labels can no longer share it on the same node.

The second pod sits in ContainerCreating. Kubelet emits an event naming the conflict, phrased in KEP-1710 as "volume XYZ is already mounted with a different SELinux label", and described in the docs as conflicting SELinux labels of a volume, with the other pod's name attached when both live in the same namespace.

Which pods actually take the new mount path?

Four conditions, all of them, per the security-context task docs: the pod uses a PersistentVolumeClaim; spec.securityContext.seLinuxChangePolicy is nil or MountOption; the containers using that PVC set seLinuxOptions; and the PV is an in-tree iscsi, rbd, or fc volume, or a CSI volume whose CSIDriver sets spec.seLinuxMount: true (a field that has been available since 1.25). Fail any gate and you get 1.36 behaviour.

The uncomfortable consequence is who qualifies. A pod that never sets seLinuxOptions is untouched. The pods that pass all four gates are the ones that deliberately pin a label: sidecar-injected workloads, OpenShift-style deployments, anything setting seLinuxOptions.level so two workloads can share a volume on purpose. The teams that did the SELinux work are the ones this lands on, which puts it alongside the mirror pod secret reference break and the DRA GPU double-allocation hazard in the 1.37 category of "correct configuration is the entry condition for the failure".

flowchart TD
  A["Pod with a PVC starts on a 1.37 node"] --> B{"seLinuxChangePolicy = Recursive?"}
  B -->|Yes| R["Runtime relabels every inode (1.36 behaviour)"]
  B -->|No| C{"Containers set seLinuxOptions?"}
  C -->|No| R
  C -->|Yes| D{"CSIDriver seLinuxMount: true?"}
  D -->|No| R
  D -->|Yes| E["kubelet mounts with -o context=label"]
  E --> F{"Volume already mounted on this node under another label?"}
  F -->|Yes| G["Pod stuck in ContainerCreating\nconflicting SELinux labels of volume"]
  F -->|No| H["Pod runs; chcon on that volume returns EOPNOTSUPP"]

One command scopes the cluster:

kubectl get csidriver -o custom-columns=NAME:.metadata.name,SELINUXMOUNT:.spec.seLinuxMount

AWS EBS CSI and Ceph RBD both advertise seLinuxMount: true, so most SELinux-enforcing fleets on RHEL, CoreOS, Bottlerocket, or Flatcar are in scope the moment the nodes reach 1.37. The CSI driver object reference documents the field if you need to confirm what a third-party driver claims.

Four post-upgrade symptoms that get blamed on the same thing

Upgrade threads currently blend these together. They have different causes and different discriminating checks.

Symptom after upgrading to 1.37	Cause	Check that proves it
Pod stuck in `ContainerCreating`, event names two SELinux labels	Two labels, one `-o context=` mount, same node	`kubectl describe pod`; node metric `volume_manager_selinux_volume_context_mismatch_errors_total`
Nothing changed, startup still slow on large volumes	A gate failed, usually `seLinuxMount: false` on the driver or containers with no `seLinuxOptions`	`findmnt -o TARGET,OPTIONS` on the node; no `context=` in the options
App runs, then `chcon` / `restorecon` / `setfattr` fails with "Operation not supported"	Kernel forbids per-file relabelling on mountpoint-labelled filesystems	Reproduce with `chcon -t svirt_sandbox_file_t <file>` inside the container
Second pod on a shared NFS export silently gets the first pod's label	Superblock reuse; KEP-1710 notes NFS needs `nosharecache`, CIFS `nosharesock`	Compare `context=` in `findmnt` against the pod's own `seLinuxOptions.level`

Row three deserves a look at the source rather than the release notes. SELinux disables setxattr on security.selinux for filesystems labelled at the mountpoint, in selinux_inode_setxattr() in security/selinux/hooks.c, added by the commit "selinux: disable setxattr on mountpoint labeled filesystems". Red Hat's SELinux documentation states the user-visible result directly: chcon on a filesystem mounted with a context option returns "Operation not supported". Any application that manages its own file labels on a PVC stops working on 1.37, and Kubernetes reports nothing at all, because from kubelet's point of view the pod started fine.

Why does the recommended pre-flight check come up clean on EKS?

The docs and the SIG-Storage announcement of 22 April 2026 both point at the SELinuxWarningController. It watches running pods, emits an event on both sides of a conflict, and raises selinux_warning_controller_selinux_volume_conflict. The docs also record that it ships disabled and is turned on with --controllers=*,selinux-warning-controller on kube-controller-manager, with the SELinuxChangePolicy gate (GA since 1.36).

That is a control-plane flag. EKS, GKE, and AKS do not expose it. So the advice showing up in upgrade checklists, "check for conflict events before you upgrade", returns zero on managed clusters because no controller was ever counting. The reachable signal is per-node: scrape kubelet /metrics for volume_manager_selinux_volume_context_mismatch_warnings_total, which KEP-1710 describes as counting pods that would fail once SELinuxMount is enabled. On a 1.36 fleet that counter is the honest forecast, and even then a zero only tells you the conflicting pair never happened to co-locate during the window you were scraping.

The failure is intermittent, because it depends on scheduling

Both pods have to land on the same node for the conflict to fire. A ReadWriteMany PVC shared between a privileged log-shipper and an unprivileged app can pass a staging upgrade for weeks, then break the first time a descheduler run, a node drain, or a bin-packing change puts them together. Absence of failures on upgrade day is weak evidence. This is the same reasoning that makes cluster-wide guarantees like default-deny egress worth more than per-workload assumptions: anything that only holds under a particular placement will eventually meet a different placement.

The blanket Recursive opt-out gives back the thing you upgraded for

The circulating fix is seLinuxChangePolicy: Recursive, pushed cluster-wide through MutatingAdmissionPolicy, Kyverno, or Gatekeeper. The docs suggest roughly that, and I would not apply it fleet-wide. Setting it everywhere returns every qualifying volume to full-tree relabelling, which is the startup cost KEP-1710 was written to remove, and on a large volume that is the difference between a pod starting in milliseconds and a pod starting in minutes. Scope it to the namespaces that genuinely share a volume across two labels, keep the mount path for everything else, and accept that you now have a per-namespace policy to maintain.

There is a second-order problem with the opt-out that I have seen bite in a different guise: the field can be silently dropped. An API server older than 1.36 prunes seLinuxChangePolicy from the pod spec, so the manifest in Git says Recursive and the running pod says nothing. Schema mismatches between what you submit and what the API server retains are a recurring class of Kubernetes bug, the same shape as "unrecognized format int32". Confirm against the live object, not the file.

What to check before you promote 1.37 to production

List the exposed drivers. Run the kubectl get csidriver command above. Any driver with SELINUXMOUNT: true puts every PVC it serves in scope.
Find the conflicting pairs. For each of those PVCs, list the pods that mount it and compare seLinuxOptions.level across them. Differing levels on one PVC is the blast radius, and it is usually a short list.
Scrape the forecast metric on every node while still on 1.36: volume_manager_selinux_volume_context_mismatch_warnings_total from kubelet /metrics. Treat zero as unproven until you have confirmed from step 2 that a conflicting pair exists and has been co-located.
Enable the SELinuxWarningController if you run your own control plane: --controllers=*,selinux-warning-controller, with the SELinuxChangePolicy gate. On EKS, GKE, and AKS, write in the upgrade plan that this check is unavailable and that the node metric is the substitute, so nobody later reads a clean event log as a pass.
Set seLinuxChangePolicy: Recursive on the pairs from step 2 only, then verify it persisted: kubectl get pod -o jsonpath='{.spec.securityContext.seLinuxChangePolicy}'.
Grep your images and charts for chcon, restorecon, and setfattr run against a PVC path. Those calls will start returning EOPNOTSUPP on qualifying volumes, with no Kubernetes event to explain it. Reproduce with chcon -t svirt_sandbox_file_t <file> inside the container before the upgrade, not after the pager goes off.

Steps 1 and 2 take an afternoon on a mature cluster. Step 6 is the one teams skip, and it is the one that produces a silent application failure weeks later with no trace in kubectl describe.

FAQ

Does SELinuxMount affect every pod with a PVC in Kubernetes 1.37?
No. All four gates must pass: PVC-backed volume, seLinuxChangePolicy nil or MountOption, seLinuxOptions set on the containers using that PVC, and either an in-tree iscsi/rbd/fc volume or a CSI volume whose CSIDriver sets spec.seLinuxMount: true. Otherwise the runtime relabels recursively as before.

Why does chcon fail with "Operation not supported" on a PVC after upgrading?
SELinux refuses setxattr on security.selinux for filesystems labelled at the mountpoint, in selinux_inode_setxattr() in security/selinux/hooks.c. Once kubelet mounts with -o context=, per-file relabelling inside that volume is denied by the kernel.

How do I opt a single workload out?
Set spec.securityContext.seLinuxChangePolicy: Recursive on the pod, then confirm with kubectl get pod -o jsonpath='{.spec.securityContext.seLinuxChangePolicy}'. An API server older than 1.36 can prune the value without error.

Can I use the SELinuxWarningController on a managed cluster?
No. It needs a kube-controller-manager flag that EKS, GKE, and AKS do not expose. Use volume_manager_selinux_volume_context_mismatch_warnings_total from kubelet on each node.

Do NFS and CIFS volumes get per-pod contexts?
Not by default. KEP-1710 notes NFS needs nosharecache and CIFS needs nosharesock, otherwise superblock reuse hands the second pod the label from the first mount.

Originally published at indragustiprasetya.com

Fix ESLint Cannot read properties of undefined (Intrinsic)

Indra Gusti Prasetya — Fri, 28 Aug 2026 02:10:20 +0000

Two teams hit the same wall this month with the same tool and needed opposite fixes. One upgraded to TypeScript 7 and forced past a peer range that had told them no. The other never touched TypeScript: their package.json still pins typescript: ^6.0.3, and their package manager wrote TypeScript 7 into the tree anyway.

Short version: ESLint throwing Cannot read properties of undefined (reading 'Cjs') or (reading 'Intrinsic') means typescript-eslint loaded TypeScript 7, whose Go-native rewrite dropped the JS enum exports it reads. Run pnpm why typescript to see which of three trees you have. If the path runs through @typescript-eslint/types, pnpm's bundled compatibility database put it there, and editing your own package.json will not move it.

Two error strings, one missing enum

The crashes look like this:

TypeError: Cannot read properties of undefined (reading 'Cjs')
    at .../@typescript-eslint/typescript-estree/dist/create-program/shared.js:59:18

TypeError: Cannot read properties of undefined (reading 'Intrinsic')

Both are the same read against a different missing export. shared.js:59 reaches for .Cjs on a TypeScript enum object that came back undefined, and the 'Intrinsic' variant does the same thing one enum over. TypeScript 7.0 went GA on 8 July 2026 as the Go-native compiler and shipped with no stable programmatic API, so the JS module every TypeScript major had exposed for a decade is no longer there in the shape the linter expects. This is the same class of failure as the npm arborist edgesOut crash: a null-property read that is really a resolution problem three layers up.

The threads on this run the two cases together, which is why the popular fix works for half the people who try it.

Where TypeScript 7 came from if you never installed it

pnpm ships a compatibility database: a table of packageExtensions patches, maintained partly with Yarn as @yarnpkg/extensions plus pnpm's own curated entries, that rewrites third-party package.json files during resolution. One entry gave @typescript-eslint/types a dependency on typescript.

It started reasonably. typescript-eslint issue #3622, opened in 2021, correctly reported that @typescript-eslint/types reaches for a typescript module it never declares. The patch declared it, unpinned. For five years that resolved to whatever TypeScript major was current and nothing broke, because every major kept the same JS module surface. Then one did not.

The pnpm 12.0.0 release notes, published 27 August, state the outcome directly: the removed entries "named packages that are only imported for their types, so installing them was at best unnecessary and at worst broke the dependent: @typescript-eslint/types gained a typescript dependency resolved to the newest release, which put TypeScript 7 under older @typescript-eslint versions and made ESLint fail with 'Cannot read properties of undefined (reading Intrinsic)'."

flowchart TD
  A["your package.json\ntypescript: ^6.0.3"] --> B["pnpm resolution"]
  B --> C["compatibility database\npackageExtensions"]
  C -->|"injects typescript dependency"| D["@typescript-eslint/types"]
  D --> E["typescript@7.x\nGo-native compiler"]
  E --> F["require('typescript')\ninside typescript-estree"]
  F --> G{"enum exports present?"}
  G -->|"no"| H["TypeError:\nreading 'Intrinsic'"]
  G -->|"yes"| I["lint runs"]

Which of the three trees do you have

Run pnpm why typescript (or npm ls typescript) before you change a single line, and read the path.

What `pnpm why` shows	Cause	Fix that works
Path is your root `devDependencies`	You upgraded and overrode the peer range	Roll back to 6.x, or run TS 7 as a second non-blocking checker
Path is `@typescript-eslint/types > typescript`	Compatibility database injected the edge	pnpm 12.0.0, `overrides`, or `ignoreCompatibilityDb=true`
Two entries, TS 6 and TS 7 at different depths	Hoisting resolution, common under `node-linker=hoisted`	Dedupe, or pin via `overrides` so both depths agree

For the first row: typescript-eslint@8.63.0 declares a peer range of >=4.8.4 <6.1.0. Getting past that took --legacy-peer-deps, --force, or strictPeerDependencies=false. The range was accurate and the override turned a correct install-time refusal into a runtime TypeError several minutes into CI, which is the same trade the npm 12 install-script lockdown taught people to stop making. A flag that silences the message rarely removes the condition.

Then confirm which compiler the tool actually loaded, rather than the one you believe you installed:

node -p "const ts=require('typescript'); [ts.version, typeof ts.Extension, typeof ts.ScriptKind].join(' | ')"

A 7.x version string with undefined for both enum types is the signature. Those objects are exactly what typescript-estree reads.

The pin that does nothing

The top-voted advice on every one of these threads is "pin typescript in your package.json" or "downgrade." On the injected path, the edge was written into a transitive package's manifest during resolution, so your root pin is not in the conversation. Neither is your .pnpmfile.cjs: the compatibility database is applied after it, so your own transformations cannot see or override the patch.

Three levers actually reach it, in the order I would try them:

Upgrade to pnpm 12.0.0. It removes the statically-analysed entries while keeping the curated @yarnpkg/extensions ones. Read the pnpm 12 upgrade traps first, because the Rust CLI rewrite lands several unrelated breaks in the same bump.
Add an overrides entry pinning typescript to your 6.x line. This wins over the injected version and works on any pnpm from 7 onward.
Set ignoreCompatibilityDb=true in pnpm-workspace.yaml, available since pnpm 7.9.0. It works, and it disables every fixup in the table, including ones other packages in your tree quietly depend on. Use it to unblock a build today and upgrade this week.

A patch table with no Git diff

The interesting part of this failure is the supply-chain shape, and it deserves more attention than the workaround. A package manager added an edge to your dependency graph from a table compiled into its own binary. The edge does land in pnpm-lock.yaml, so it is technically visible, but it appears on a pnpm version bump with no package.json change, which is the diff every reviewer scrolls past.

Rush's documentation has flagged the structure for years, calling the fixups "hidden magic bundled into the PNPM binary with no Git diff visibility", coupled to whichever @yarnpkg/extensions snapshot that particular pnpm build happens to carry. Your dependency graph is a function of your lockfile and your package manager build. If you generate an SBOM from the installed tree, it will faithfully record TypeScript 7 as a real component of your project, sourced from nothing you ever wrote down.

Pin packageManager in package.json and treat a bump to it as a dependency change in review.

What actually links against tsgo today

There is a working path to type-aware linting on TypeScript 7 right now. oxc's tsgolint went stable on 22 July 2026, tracks TypeScript v7.0.2, and implements 59 of typescript-eslint's 61 type-aware rules. The oxc announcement reports 12-18x faster runs than ESLint with typescript-eslint across vscode, TypeScript, typeorm, and vuejs/core. It gets there by linking typescript-go directly instead of importing the JS module, which is why the API gap does not touch it.

The cost is two missing rules and a different config surface. If lint wall-clock is your CI bottleneck, that is a cheap trade. If your rule set is heavily customised, budget real time for the migration.

What 7.1 fixes on 10 November, and what it does not

typescript-eslint closed the TypeScript 7 support request (#12518) as not planned on GA day. The TypeScript 7.1 iteration plan lists "Stabilize API" covering Content Mapper, Emit, and Language Service, against a 7.1 stable target of 2026-11-10, roughly ten weeks out.

That date unblocks the version range. It does not finish the job. Per typescript-eslint #10940, tsgo is consumed over native or WASM bindings, ESLint core has no async parser support, and rules expect JS-land AST nodes. A stable API gives the maintainers something to build against; ESLint's synchronous rule model still has to meet a Go type checker somewhere in the middle. Plan on 7.x type-aware linting through the official parser arriving well after November.

The blast radius extends past ESLint for the same reason. ts-jest, ts-morph, API Extractor, and the template checkers behind Volar, Svelte, and Astro all consume that missing programmatic API.

Do this before your next CI run

pnpm why typescript. Read the path and match it against the table above. Everything else is downstream of knowing which tree you have.
If the path runs through @typescript-eslint/types, upgrade to pnpm 12.0.0 today. If a Rust-CLI break blocks that upgrade, add an overrides entry for typescript at your 6.x version and revisit the upgrade this sprint.
Grep your CI configs and Dockerfiles for --legacy-peer-deps, --force, and strictPeerDependencies=false, and delete every hit that touches a TypeScript install. Each one converts an accurate refusal into a stack trace ten minutes later.
Pin packageManager in package.json if it is not already, and add "pnpm version bump" to whatever your team treats as a dependency-review trigger.
If you upgraded on purpose, keep 6.x as the source of truth for emit and tooling, and run TypeScript 7 as a second, non-blocking tsc job until 7.1 lands on 2026-11-10.
If lint takes more than a couple of minutes in CI, benchmark tsgolint against your rule set this week and decide the two-rule gap knowingly rather than in an incident.

What I read for this piece: the pnpm 12.0.0 release notes, typescript-eslint issues #3622, #10940 and #12518, the TypeScript 7.1 iteration plan, the oxc type-aware linting announcement, and Rush's compatibility database page.

FAQ

Does typescript-eslint support TypeScript 7?
No. The support request, #12518, was closed as not planned on GA day, because TypeScript 7.0 shipped without a stable programmatic API. The 7.1 iteration plan targets API stabilization for 2026-11-10.

How do I stop pnpm from installing TypeScript 7 when my package.json pins 6.x?
Upgrade to pnpm 12.0.0, which removed the compatibility-database entry that gave @typescript-eslint/types an unpinned typescript dependency. Failing that, add an overrides entry pinning typescript to your 6.x line.

Is ignoreCompatibilityDb safe to turn on?
It resolves this specific break and disables every other bundled packageExtensions fixup at the same time, including ones unrelated packages rely on. Use it as a temporary unblock, then upgrade.

Can I lint with type information on TypeScript 7 today?
Yes, through oxc's tsgolint, stable since 22 July 2026. It links typescript-go directly, tracks v7.0.2, and covers 59 of the 61 type-aware rules.

Originally published at indragustiprasetya.com

pnpm 12 Upgrade: Fix 12 Rust CLI Traps Before CI Breaks

Indra Gusti Prasetya — Thu, 27 Aug 2026 02:09:16 +0000

pnpm 12.0.0 was published on 2026-08-26 with the Rust CLI as the real CLI. The happy path is uneventful: same commands, same flags, same pnpm-lock.yaml format. The traps live in the edges, and every one of them shipped less than eight weeks ago, which means any assistant answering from training data will describe pnpm 11 behaviour with total confidence. These are the twelve things I check before pnpm 12 goes near a CI runner.

Short version: pin packageManager before you upgrade anything. pnpm 12 finally validates pnpm-workspace.yaml instead of dropping unknown keys in silence, but the check only fails hard when the project pins a pnpm version the running pnpm satisfies. With no pin, a typo in minimumReleaseAge stays a warning and your dependency cooldown stays off.

flowchart TD
  A["Unknown key in pnpm-workspace.yaml"] --> B{"Project pins a pnpm version?"}
  B -->|"yes, pnpm 12 satisfies it"| C["ERR_PNPM_UNRECOGNIZED_WORKSPACE_SETTINGS\ncommand fails"]
  B -->|"no pin"| D["Warning only\ninstall continues, key still ignored"]

What actually breaks when you upgrade to pnpm 12?

Pin the pnpm version before you touch anything else. The v12.0.0 release notes give the new error as ERR_PNPM_UNRECOGNIZED_WORKSPACE_SETTINGS, and the RC 9 notes spell out the condition: the command fails when the project pins a pnpm version that the running pnpm satisfies. Everywhere else, the same unknown key prints a warning, the install continues, and the setting is still dropped. Put the pin in package.json (or use devEngines.packageManager) so the upgrade runs against a check with teeth:

   { "packageManager": "pnpm@12.0.0" }

Grep for a misspelled minimumReleaseAge first. This is the one that made me write the list. Under pnpm 11 a mistyped key in pnpm-workspace.yaml disappeared without a word, and minimumReleaseAge is the key that holds your supply-chain cooldown: the delay that stops an install from pulling a package published an hour ago. A typo there means every install has been taking the newest thing on the registry while the config file said otherwise, which is exactly the exposure window that npm addressed with its install-script allowlist. pnpm 12 suggests the closest real setting name when a key looks like a typo, so let it audit for you:

   pnpm install --frozen-lockfile 2>&1 | grep -iE 'unrecognized|did you mean'

Run that on every repo before you bump a single dependency.

Move the SSH rewrite out of the repo and into git config. Per pnpm's What's different in pnpm 12 post (2026-08-10), GitHub, GitLab and Bitbucket specifiers are identities now: kevva/is-positive, github:…, git+https://… and git+ssh://… all resolve to the same dependency, and pnpm never records an SSH URL for those hosts. Private repos that relied on the SSH form living in the lockfile fail to fetch on a runner with a deploy key. Push the rewrite down to git, where it applies to every fetch:

   git config --global url."ssh://git@github.com/".insteadOf "https://github.com/"

Worth remembering what a git dependency is while you do this: upstream source you build locally, the same trust position that made the arrayref build.rs compromise work.

globalShims changes what node means on a build agent. pnpm 12 adds globalShims, defaulting to { node: true, deno: true, bun: true }, so a globally installed node honours the version the current project pins. That is excellent on a laptop and unwelcome on an agent that was built around one fixed toolchain and runs jobs from six repos. I set globalShims: false in pnpm-workspace.yaml on runners and keep version selection in the CI image, because a global binary whose meaning depends on the working directory turns "which node ran this build" into archaeology.
pnpm add -g yarn now installs current Yarn. The same post confirms pnpm 12 installs the actual package-manager tools rather than npm wrappers: pnpm add -g yarn lands on the current Yarn line, and pnpm add -g node installs a real Node.js release instead of a wrapper that downloads one at first run. Any bootstrap script that has been quietly shipping Yarn 1 for years now ships Yarn 4. Read your provisioning scripts before the runners do.
--resolution-only is gone, and the failure carries no pnpm error code. pnpm peers check replaces it. What the old flag gets you is a clap argument-parser message:

   error: unexpected argument '--resolution-only' found

Nothing in that line says ERR_PNPM_, which is why it gets filed as a shell quoting problem. Treat any bare error: unexpected argument from pnpm as "the Rust CLI dropped this flag" and go find the replacement command. It needs the same reading discipline as the npm approve-scripts and --allow-scripts split: the flag name that used to work tells you nothing about the one that works now.

Audit .pnpmfile.cjs for filterLog. The v12.0.0 notes deprecate the pnpmfile filterLog hook: the Rust CLI ignores it and emits a warning. Teams used it to scrub tokens and cut noise out of install logs, and that scrubbing stops the moment you upgrade, with a single warning line as the only marker. Check the file, then check what your CI log retention has been capturing since the upgrade landed.
Reject .. in workspace globs before pnpm does. Issue #13880, opened 2026-08-12, reports parent-relative workspace packages breaking on 12: pnpm omits them from discovery, then refuses the lockfile importer with Refusing to install importer with unsafe path key '../shared'. The message goes on to require POSIX paths relative to the workspace root, rejecting absolute paths, drive prefixes and .. components. One grep tells you whether you are exposed:

   grep -n '\.\.' pnpm-workspace.yaml

Dry-run engineStrict against your optional subtrees. Under engineStrict, pnpm 12 fails when an incompatible package is reached through a regular dependencies edge, even where that edge hangs below an optionalDependencies subtree. Packages tolerated for years because an optional parent covered for them now block the install outright. Flip engineStrict on locally and run a full install before you change the version in CI.
Land the lockfile churn as its own commit. pnpm 12 breaks dependency cycles in a canonical order, which buys deterministic, smaller lockfiles and 2 to 3 times faster peer resolution, and it re-keys the affected peer variants on the first resolution pass after the upgrade. Run pnpm install, commit pnpm-lock.yaml alone, and keep that diff away from any dependency bump. A reviewer can only spot a real change when the mechanical churn shipped separately.
Check readlink on the deploy output, because the build stays green either way. Issue #13618 documents a regression introduced in 11.19.0 and still labelled state: needs design: pnpm deploy --legacy leaves a workspace dependency that carries peerDependencies as a symlink escaping the deploy directory. CI passes, because the target still exists on the runner, and the container dies at startup with ERR_MODULE_NOT_FOUND. One line in the pipeline catches it:
```
readlink deploy/node_modules/shared | grep -q '^\.\./\.\./' && exit 1
```
Workspace deps without peers still pack correctly, which is why the small reproduction someone builds to check this comes back clean.
Drop the overrides you added for the ESLint Intrinsic crash. RC 10 (2026-08-24) removed statically-detected entries from the built-in compatibility database. Those entries had given @typescript-eslint/types a typescript dependency that resolved to the newest release, putting TypeScript 7 under older @typescript-eslint versions and making ESLint fail with Cannot read properties of undefined (reading 'Intrinsic'). The database keeps its @yarnpkg/extensions and pnpm-curated entries. Any pin you added to work around that crash is dead weight in the manifest, and dead pins outlive the people who remember why they exist.

Wrap-up

Pin the pnpm version before you upgrade, then read the warnings the pinned run produces. pnpm 12 has genuinely good diagnostics: typo suggestions, an unrecognized-settings error, a strict importer check. Almost all of them degrade to a warning, or to nothing at all, when the project gives them no pin to check against. That is the same failure mode as an npm audit that returns 410 and passes anyway: the tool tells you the truth only once you have configured it to be able to fail.

FAQ

Does pnpm 12 change the lockfile format? No. The format is unchanged from pnpm 11. The content changes, because cycles are now broken in a canonical order and the affected peer variants get re-keyed on the first resolution pass, so expect one large diff and commit it alone.

Why did my unknown pnpm-workspace.yaml key only warn? The project has no pnpm version pin. ERR_PNPM_UNRECOGNIZED_WORKSPACE_SETTINGS fails the command only when the project pins a version the running pnpm satisfies; otherwise you get a warning and the key is still ignored.

What replaced pnpm install --resolution-only? pnpm peers check. The removed flag surfaces as error: unexpected argument '--resolution-only' found, straight from the argument parser, with no ERR_PNPM_ code attached.

Is pnpm deploy safe on 12? pnpm deploy --legacy still carries issue #13618, open since 11.19.0 and labelled needs design. Add the readlink check from tip 11 to the pipeline rather than waiting for the fix.

What I read: the v12.0.0 release notes and RC 10 notes on GitHub, pnpm's own What's different in pnpm 12, and issues #13618 and #13880.

Originally published at indragustiprasetya.com

Stop Trusting C2PA Content Credentials on Android

Indra Gusti Prasetya — Wed, 26 Aug 2026 02:09:25 +0000

On 25 August 2026, David Buchanan published an AI-generated image carrying a valid C2PA Content Credential that says a Google Pixel camera captured it, along with a YouTube clip signed the same way. The official C2PA verification tool accepts both. The signing key was never extracted from the phone, and that detail is the reason this is hard to fix.

TL;DR: A Content Credential proves which key signed some bytes. It does not prove a sensor saw anything. On a Pixel rooted by a software exploit, StrongBox will sign attacker-supplied content with the real camera key, and the attestation certificate still reports a locked bootloader. Because Pixel issues one certificate per image, there is nothing to revoke. Score verified credentials as weak corroboration with a device-class caveat, and score a missing credential as nothing at all.

What does a valid Content Credential actually prove?

The C2PA binding runs from the manifest to the asset bytes, and from the signature to a certificate that chains to a trust list. At no point does anything in that chain observe the image sensor. The guarantee on offer is "a key on the trust list signed these bytes," and everything else people read into a green check is inference about how well that key is guarded.

On Pixel, the guard is real hardware. Google's September 2025 announcement states that Pixel Camera's "C2PA claim signing keys are generated and stored using Android StrongBox in the Titan M2 security chip," and that Pixel Camera reached Assurance Level 2, "the highest security rating currently defined by the C2PA Conformance Program," which Google notes is "currently only possible on the Android platform."

Buchanan's result leaves that hardware claim intact. Root on the device can call the KeyStore API while impersonating the camera app, hand StrongBox arbitrary bytes, and get a signature over them. His tool, Keystork, does exactly that. He states every C2PA camera app relying on Android_KeyAttestation or Google_PlayIntegrity is affected, which covers the Android field as it exists today.

The attestation certificate that ships alongside is the part that fools operators. It reports genuine hardware, a locked bootloader, unmodified AVB keys, and Play Protect Certified boot. Every one of those statements is still true on a phone rooted through a kernel bug. Attestation measures the boot chain. Root acquired after boot is invisible to it. Anyone who has argued about what a JWS on an agent card really certifies has met this gap before: the signature is honest about origin and silent about everything the operator cares about.

How expensive is the root that makes this work?

Cheap enough to stop treating as a nation-state problem. Buchanan attributes in-the-wild one-click root on fully patched Pixels to CVE-2026-43499, a Linux kernel rtmutex use-after-free (CWE-416) published to NVD on 21 May 2026 with a CVSS base score of 7.8, fixed in 6.1.175, 6.6.140, 6.12.86, 6.18.27 and 7.0.4. He reports testing on Pixel 8a and 9a at August patch levels. He also describes a hardware fault-injection path, flipping DRAM bits into page table entries, which no kernel patch closes.

Google's response was to close the report "Won't fix (infeasible)" and pay a $7,500 out-of-scope bounty. Read that as a statement about the threat model rather than a shrug: Android key attestation was designed to prove device integrity at boot for DRM and payments, and it does that job. C2PA asked it to prove something it was never built to prove, and Google is declining to redefine it.

flowchart TD
  A["Sensor capture"] --> B["Camera app builds\nC2PA claim"]
  B --> C["StrongBox / Titan M2\nsigns the claim"]
  C --> D["Attestation cert:\nlocked bootloader,\nAVB keys, patch level"]
  D --> E{"Validator:\nchains to trust list?"}
  E -->|yes| F["Verified as\ncamera-captured"]
  G["Root via CVE-2026-43499\nor DRAM fault injection"] --> H["Attacker requests\na signature over\narbitrary bytes"]
  H --> C

Four different states, one identical verifier output

Most of the confusion in the last week comes from teams collapsing distinct situations into a single boolean.

What the verifier shows	What actually happened	The check that separates it
Valid, trusted signer	Genuine capture	Nothing available today distinguishes this from row 2
Valid, trusted signer	Forgery on a software-rooted device	Device class and attestation method in the manifest; treat Android AL2 as weak evidence
Valid, trusted signer	Certificate known-compromised and revoked	A manual revocation check; conforming validators skip it
No credential	Screenshot, CDN resize, re-encode, or a deliberate strip	Absence carries no signal in either direction

Row four matters more than it looks. Stripping happens by accident on almost every resize and screenshot, and purpose-built C2PA strippers have been easy to find on GitHub this month. An intake policy that raises a risk score for missing provenance will spend its life flagging holiday photos that went through an image CDN.

Why revocation cannot save the Pixel design

Google describes a "One-and-Done" strategy where each key and certificate is used to sign exactly one image. That is excellent hygiene against key reuse and it removes the object you would revoke. There is no device certificate. Each forgery arrives with a fresh certificate, freshly attested. The remaining lever is refusing to provision new certificates to a compromised device, which requires detecting software root, which is precisely the thing attestation cannot see.

Nikon had the opposite design and used it. After Adam Horshack demonstrated the Z6 III's multiple-exposure mode signing an image the camera never captured, Nikon suspended its Authenticity Service and, per DPReview and Nikon Rumors, revoked every certificate issued since the 21 September 2025 launch. That invalidated legitimate photographs along with the forgeries. Those are the two shapes available in 2026: revoke nothing, or revoke everyone.

The academic picture is worse. The first formal-methods analysis of C2PA, by Golaszewski, Krawetz, Sherman and colleagues (IACR ePrint 2026/804, revised 2 June 2026), reports that "inadequate certificate revocation policies cause conforming validators to accept manifests signed with known-compromised certificates, violating all security goals." The same paper finds that generators and validators disagree on the trusted timestamp, that conforming validators return inconsistent results on the same asset, that the exclusion range permits undetectable alterations, and that "the conformance program certifies products without technical review or defined requirements." The authors note some fixes landed in Pixel 10 Pro and C2PA v2.3, so the specification is moving. The deployed fleet is not.

The timestamp is right for builds and wrong for cameras

The C2PA 2.2 specification (2025-05-01) directs generators to obtain an RFC 3161 countersignature so a manifest keeps verifying after its certificate expires or is revoked. For build provenance that is the correct default, and it is why a two-year-old release artifact still validates. Anyone who has fought through a Cosign v3 rollout and the trust-chain breakage that came with it knows how much that durability is worth.

Capture provenance wants the opposite behaviour. When a camera is known compromised, its output should stop verifying immediately, including images signed last year. C2PA inherited the build-provenance default and applied it to real-world claims, and the countersignature now works to keep forgeries alive. Decide which of those two claims your own signatures are making before you copy the pattern into anything else you sign.

Your verification gate is probably trusting nothing

c2patool -d asset.jpg prints the detailed manifest and certificate chain. Trust anchors are supplied through C2PATOOL_TRUST_ANCHORS, C2PATOOL_ALLOWED_LIST and C2PATOOL_TRUST_CONFIG, per the c2pa-rs CLI usage docs, and contentcredentials.org publishes anchors.pem and allowed.sha256.txt.

The failure mode to hunt for is the one that reports success while doing nothing. In a settings file, trust.trust_anchors takes PEM text rather than a path. Hand it a filename and the tool configures no anchors at all, then happily reports valid on assets that chain to nowhere. There is also a dead-list trap: the C2PA Interim Trust List was frozen as of 1 January 2026, leaving the Conformance Programme as the only issuance path, so a gate still pinning the ITL is validating against a list that stopped growing eight months ago.

What to change in your intake pipeline this week

Rewrite any rule that reads verified == true → accept. For the Android fleet, that rule is now wrong. Downgrade a passing Content Credential to weak corroborating evidence carrying a device-class caveat, and require a second, independent signal before an image drives a payout, a publication, or a KYC decision.
Delete any risk score attached to a missing manifest. Absence proves a resize happened, nothing more.
Prove your gate pins an anchor. Put a known-untrusted signed asset in CI and require the pipeline to fail on it. If it passes, your trust.trust_anchors is holding a path instead of PEM text, and you have been rubber-stamping everything.
Re-pull anchors.pem and allowed.sha256.txt from contentcredentials.org and confirm nothing in your config still references the Interim Trust List frozen on 1 January 2026.
Record the signer identity and attestation method (Android_KeyAttestation, Google_PlayIntegrity, or a hardware vendor's own) from every manifest at intake, alongside the asset hash. When the next device class breaks, that field is how you find the affected corpus. The same discipline that makes a stored SBOM plus VEX useful the day a CVE lands applies here, and neither can be reconstructed after the fact.
Flag Pixel 8a and 9a class devices at August 2026 patch levels specifically, since those are the models Buchanan reports testing, and keep the flag until you can point at a device generation where the ePrint 2026/804 fixes shipped.

What I read for this: Buchanan's write-up for the mechanism, ePrint 2026/804 for the formal analysis of revocation and validator disagreement, Google's Pixel C2PA announcement for the StrongBox and Assurance Level 2 claims, NVD's entry for CVE-2026-43499 for the kernel bug and fixed versions, and the C2PA 2.2 specification for the RFC 3161 countersignature requirement.

Originally published at indragustiprasetya.com

Stop Kubernetes DRA Giving One GPU to a VM and a Pod

Indra Gusti Prasetya — Tue, 25 Aug 2026 03:49:07 +0000

NVIDIA GPU Operator 26.7.0 shipped on 21 August with the donated DRA driver as a managed component, and buried in the release notes is a sentence that should stop a platform team mid-rollout: with passthrough enabled, one physical GPU is published to the scheduler twice, and kube-scheduler can grant both claims.

Short version: turn on DRA VFIO passthrough and every eligible GPU is advertised as a container device under gpu.nvidia.com and as a passthrough device under vfio.gpu.nvidia.com. The driver only notices the overlap during Prepare, which runs after the scheduler has committed, so a container claim and a VM claim for the same card can both succeed. NVIDIA suggests submitting workloads sequentially. Dedicate the nodes instead.

What the 26.7.0 release notes actually admit

The wording in the GPU Operator 26.7.0 release notes is unusually direct: "With DRA passthrough enabled, the driver initially advertises each eligible physical GPU as both a container GPU and a VFIO device. If a container GPU claim and a VFIO passthrough claim for the same physical GPU are allocated before either device is prepared, the scheduler can allocate both claims."

That paragraph is the complete public documentation of the problem. It describes the symptom and stops. The mechanism underneath it lives in a scheduling KEP that changed shape between Kubernetes 1.35 and 1.36, which is why an assistant asked about this today will answer from the device-plugin era, when a GPU had exactly one owner because there was exactly one nvidia.com/gpu integer to decrement.

Context for how fast this spreads: NVIDIA donated the GPU DRA driver to the Kubernetes community at KubeCon Europe 2026 in Amsterdam, and it now lives at kubernetes-sigs/dra-driver-nvidia-gpu. Release 26.7.0 promotes it to a first-class operand next to driver 595.91.07 and container toolkit v1.20.0. Teams that were waiting for DRA to feel supported are about to enable it. This is the same pattern as the mirror pod secret rejection in 1.37: a node-level component is the only thing that knows the truth, and it learns it too late to say no.

Why kube-scheduler cannot catch this

DRA splits a device lifecycle into three stages. The kubelet plugin publishes ResourceSlices describing what exists on the node. kube-scheduler allocates claims against those slices, cluster-wide, with no node-local knowledge beyond what was published. Then NodePrepareResources runs on the node and produces the CDI spec that actually wires the device into the workload.

Physical exclusivity is only knowable at stage three. The binding decision happened at stage two.

flowchart TD
  A["Physical GPU 0\nnode gpu-a1"] --> B["ResourceSlice\ndriver gpu.nvidia.com"]
  A --> C["ResourceSlice\ndriver vfio.gpu.nvidia.com"]
  B --> D{"kube-scheduler\nallocates from slices"}
  C --> D
  D -->|"container claim"| E["Prepare\nCDI spec for /dev/nvidia0"]
  D -->|"VFIO claim"| F["Prepare\nbind vfio-pci, /dev/vfio/devices/vfio0"]
  E --> G["One GPU, two owners"]
  F --> G

Two claims, two Prepare calls, one card. The container path opens /dev/nvidia0. The VFIO path unbinds the nvidia kernel driver from the PCI function and binds vfio-pci. If a training job already holds the character device open, that unbind either returns EBUSY or tears the device out from under a running process.

The asymmetry is the part I would plan around. The container job dies loudly and someone files a ticket. The VM boots against a card whose teardown path never completed. Whether framebuffer contents survive that exact sequence is a test I would run on a spare node before trusting a tenancy boundary to it, and I have found no published result either way. If you sell GPU capacity to more than one tenant, that is now a boundary you own, in the same family of questions as running workloads under user namespaces in 1.36 and default-deny egress between tenants.

Seeing the VFIO DeviceClass is not evidence you are exposed

Here is where I expect the panic to be wrong. The PassthroughSupport feature gate is Alpha and off by default in the DRA driver, and that single fact is what keeps this from being a fleet-wide incident this week.

NVIDIA's DRA install documentation notes that the passthrough capability "is always enabled, although VFIO passthrough is unusable without the Alpha PassthroughSupport feature gate." So vfio.gpu.nvidia.com shows up in kubectl get deviceclasses on clusters that have no passthrough at all. Someone on your team will screenshot that list and call it a finding. Check the gate on the kubelet plugin before you page anyone.

Your 2023 pod manifest may already be a DRA claim

The second-order effect is nastier than the headline, and it has nothing to do with VMs.

DRAExtendedResource (KEP-5004) is enabled by default from Kubernetes v1.36.0. It makes the scheduler translate a plain resources.limits: nvidia.com/gpu: 1 into an auto-created ResourceClaim against whichever DeviceClass carries spec.extendedResourceName: nvidia.com/gpu. A manifest written three years ago and never touched since is now flowing through the DRA allocation path. Nobody opted in, and nothing in the pod spec changed.

That translation has already produced a leak of its own. kubernetes/kubernetes#134311, filed against v1.34.0 with DRAExtendedResource: true, reported scheduler-created claims still sitting in allocated,reserved after their pods had completed. It is closed and fixed, and it marks where the sharp edges are on this path: claims whose lifecycle is owned by the scheduler rather than by you.

Kubernetes already has the primitive that would close this

KEP-4815 Partitionable Devices exists for exactly this shape of problem. It adds sharedCounters and consumesCounters to ResourceSlice so a driver can model overlapping views of one piece of hardware: allocate one device, and the counters it consumes stop being available to its siblings. That is how MIG profiles express that two partitions want the same memory slice.

Per the KEP README, DRAPartitionableDevices graduates to beta in 1.36, and a backwards-incompatible revision between 1.34 and 1.35 now requires sharedCounters to live in a separate ResourceSlice from the devices consuming them, in the same pool with matching generation. A driver migrating across that break has a defensible reason to ship the overlap unmodeled and write a known-issue paragraph. It still leaves operators holding the risk.

Find double-advertised GPUs in your own cluster

The condition is invisible in kubectl describe node. Node capacity looks normal because neither driver is reporting an integer count any more. What you are hunting is one GPU UUID appearing under two driver names:

kubectl get resourceslices -o json | jq -r '
  .items[] | .spec.driver as $d | .spec.nodeName as $n |
  .spec.devices[]? | [$n, $d, .name, (.attributes.uuid.string // "-")] | @tsv' \
  | sort -k4 | uniq -f3 -D

Any UUID on two rows with different drivers is double-advertised. Then confirm the gate rather than inferring it from the DeviceClass list:

kubectl get deviceclass gpu.nvidia.com -o jsonpath='{.spec.extendedResourceName}{"\n"}'
kubectl -n gpu-operator get pods -l app=nvidia-dra-driver-gpu-kubelet-plugin \
  -o jsonpath='{.items[0].spec.containers[0].args}' | tr ',' '\n' | grep -i feature

The first command tells you whether legacy nvidia.com/gpu manifests are being translated into claims. The second tells you whether passthrough is genuinely live.

Before you turn on PassthroughSupport

Split the node pool first, at the provisioner level. Label passthrough nodes and taint them (nvidia.com/gpu.workload=vfio:NoSchedule), then make the container DeviceClass selector exclude them. Encode it in the Karpenter NodePool or node-group config so it survives autoscaling. Typed GPU pools strand capacity, and on a card this expensive that hurts. Take the strand until DRAPartitionableDevices closes the gap at the scheduler.
Reject sequential submission as a control. NVIDIA's first workaround asks you to hand-serialize a system built to avoid serializing. It survives a demo. It does not survive one Argo burst or one Karpenter node coming online with two pending pods.
Run the ResourceSlice UUID query on every cluster already on 26.7.0. A collision means a live allocation race, today, with no alert attached to it.
Audit leaked claims if you are on Kubernetes 1.36 or later. List ResourceClaims whose ownerReferences point at a Pod in Succeeded or Failed while status is still allocated. Those are GPUs you are billed for and cannot schedule, per the failure mode in issue #134311.
Pin the stack explicitly. GPU Operator 26.7.0, driver 595.91.07, container toolkit v1.20.0, on Kubernetes v1.34.2 or later, which the DRA driver requires for the resource.k8s.io DeviceClass API. Version drift here changes which of the above applies.
Set a watch on DRAPartitionableDevices reaching beta in your distro. Once the NVIDIA driver models the container and VFIO views as counter-consuming siblings in one pool, the scheduler enforces exclusivity and the typed node pools become an optimization rather than a safety mechanism.

FAQ

Does GPU Operator 26.7.0 double-allocate GPUs by default?
No. PassthroughSupport is Alpha and off by default, so the VFIO view of the card is unusable until you enable the gate. The DeviceClass is registered either way.

How do I tell if a GPU is advertised twice?
Group ResourceSlice devices by their uuid attribute across driver names. A UUID under both gpu.nvidia.com and vfio.gpu.nvidia.com on one node is double-advertised.

Why can kube-scheduler not prevent this?
It allocates at stage two of publish, allocate, prepare. The driver reconciles the two views during NodePrepareResources at stage three, after the binding decision is already made.

Does an old manifest using nvidia.com/gpu limits go through DRA now?
On Kubernetes 1.36 and later it can. DRAExtendedResource is on by default from v1.36.0 and auto-creates a ResourceClaim against the DeviceClass carrying spec.extendedResourceName: nvidia.com/gpu.

What breaks first if both claims land?
The container job, when the nvidia driver unbind hits an open /dev/nvidia0 and returns EBUSY or pulls the device away mid-run. The VM starts against a card whose teardown never finished cleanly.

Originally published at indragustiprasetya.com

Fix "a mirror pod may not reference secrets" in K8s 1.37

Indra Gusti Prasetya — Tue, 25 Aug 2026 02:09:53 +0000

Kubernetes v1.37 goes GA tomorrow, Wednesday 26 August 2026, per the release team's schedule. Under the ipvs deprecation and the containerd 2.0 floor sits a two-line removal that closes a four-year-old reporting hole: the PreventStaticPodAPIReferences feature gate is gone, merged as PR #140226 on 6 July 2026. Short version: a static pod that references a Secret, ConfigMap, ServiceAccount, or PVC is now refused by the kubelet with no opt-out, and any node still pinning that gate to false will fail to start on 1.37. Two greps on one node per pool tell you whether you are affected.

What actually happened when a static pod referenced a Secret?

The rule itself is documented and boring. The Static Pods docs state that "the spec of a static Pod cannot refer to other API objects, such as ServiceAccount, ConfigMap, or Secret." What almost nobody has looked at is the behavior when you ignored it, because the failure was split across two subsystems that never compared notes. The container started. The Pod object did not exist. Both were true at once, and only one of them was visible from kubectl.

Mike Spreitzer filed issue #103587 on 8 July 2021 with the log lines that prove it. The kubelet writes "Creating a mirror pod for static pod" pod="default/test5-init1", then immediately "Failed creating a mirror pod for" err="pods \"test5-init1\" is forbidden: a mirror pod may not reference service accounts". The workload keeps serving traffic the entire time. That asymmetry survived until v1.34 put a fix behind a gate, and v1.37 welds the gate shut.

flowchart TD
  A["/etc/kubernetes/manifests/*.yaml"] --> B["kubelet static pod source"]
  B --> C["container starts via CRI"]
  B --> D["kubelet POSTs a mirror pod"]
  D --> E{"spec references an API object?"}
  E -->|no| F["Pod visible to kubectl,\npolicy engines, audits"]
  E -->|yes| G["NodeRestriction returns 403"]
  G --> H["No Pod object exists.\nContainer keeps running."]

The 403 comes from the NodeRestriction admission plugin. In current master, plugin/pkg/admission/noderestriction/admission.go formats the message generically as "node %q can not create pods that reference %s", alongside "error checking mirror pod for API references: %w". That %s is filled by podutil.HasAPIObjectReference() in pkg/api/pod/util.go, which walks the spec and returns the plural resource name it found: serviceaccounts, secrets, configmaps, resourceclaims, persistentvolumeclaims, plus qualified variants like secrets (via azureFile volumes), csidrivers (via CSI volumes), and persistentvolumeclaims (via ephemeral volumes).

Two things follow for operators. The wording changed between releases, so older nodes print the 2021-era phrasing (a mirror pod may not reference service accounts) and newer ones print the generic form. Grep for mirror pod and read the suffix to learn which resource tripped it. And that helper returns an error on any spec field it does not recognise, so the check fails closed as new volume types land. That is the design choice that made removing the gate safe three releases after it shipped.

Why an invisible pod is worse than a failed one

A pod that runs without a Pod object is missing from the entire governance stack. Kyverno and Gatekeeper are admission webhooks on the API server; static pods bypass admission by construction, and the mirror pod is the one API artifact that would have exposed them. CIS benchmark scanners, image inventories built from kubectl get pods -A -o json, runtime-to-API reconciliation in a SIEM, cost allocation, and every "what is running in my cluster" dashboard read the same source of truth, and that source of truth has a hole in it.

I care about this more than the raw severity suggests because of what it does to inventory. If you build an SBOM per running image from an API-server listing, the container you most want to inspect (a hand-edited control-plane component pulling from a private registry) is precisely the one missing from the input list. Add an imagePullSecrets block to a static pod manifest and the container gets harder to see while staying just as easy to run. That is a strange property for a security boundary, and it is why this landed as a removal rather than a docs clarification.

The population at risk is narrow. Vanilla kubeadm control-plane manifests use hostPath volumes only, so a stock cluster upgrades clean. The clusters that break are the ones where someone edited /etc/kubernetes/manifests by hand: private-registry pulls that added imagePullSecrets to kube-apiserver.yaml, air-gapped installs, and vendor node agents shipped as static pods with a projected serviceAccountToken volume for telemetry. If you have already moved workloads to bound ServiceAccount tokens, that projected volume is exactly the pattern you were encouraged to adopt, which is how well-meaning teams ended up here.

Three causes of "it runs but kubectl cannot see it"

The symptom is one line in a ticket. It has at least three unrelated causes, and the GitHub threads conflate all of them.

An API reference in the manifest. The kubelet log shows a create attempt followed by a 403 naming a resource. journalctl -u kubelet | grep -i "mirror pod" is the whole diagnosis.
A stale mirror pod after a hash mismatch. The Pod object exists but describes an older manifest. Compare kubectl get pod <name>-<node> -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.hash}' against the value after a fresh kubelet restart. The kubelet deletes and recreates on divergence.
The kubelet cannot reach the API server, or the Node object is missing. Then no mirror pods exist for that node at all, while every static pod still runs. Count them:

kubectl get pods -A -o json | jq '[.items[]
  | select(.metadata.annotations["kubernetes.io/config.mirror"])
  | select(.spec.nodeName=="'$NODE'")] | length'

Zero means cause 3. A partial count means cause 1.

The check that separates all three without guessing is a diff between runtime truth and API truth, run on the node:

crictl pods -o json | jq -r '.items[].metadata.name' | sort > /tmp/runtime.txt
kubectl get pods -A --field-selector spec.nodeName=$NODE \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort > /tmp/api.txt
comm -23 /tmp/runtime.txt /tmp/api.txt

Anything in the first list and absent from the second is your invisible set. It takes five seconds per node class, and it is the only inventory that does not inherit the bug. Run it on 1.33 and older too, where nothing else will tell you.

The kubelet config line that stops a node from registering

The failure that actually pages you at 03:00 is smaller than the security story. A node whose KubeletConfiguration still carries PreventStaticPodAPIReferences: false, pinned during a v1.34 or v1.35 upgrade to defer this work, meets a removed gate on v1.37. The kubelet exits at startup with failed to set feature gates from initial flags-based config: unrecognized feature gate, and the node never registers. k3s users saw the identical shape when KubeletCredentialProviders was removed in 1.28.

This is the same class of upgrade break as the unrecognized format int32 surprise in 1.34: a change that is trivially fixable if you find it in a canary node, and miserable if you find it after draining half the control plane. Do the node-side prep first, the way you would for enabling user namespaces with hostUsers false, and keep the config edit in a separate change from the version bump.

The fix in circulation is the wrong artifact

The workaround people reach for is to keep imagePullSecrets in the manifest and disable the gate. That credential never worked for a static pod in the first place. Registry auth for control-plane images comes from the runtime: /etc/containerd/certs.d/<registry>/hosts.toml, the registry.configs.<host>.auth section of config.toml, or a kubelet credential provider configured with --image-credential-provider-config. Clusters that "proved" imagePullSecrets worked were pulling from a warm image cache or an already-authenticated containerd, and the proof evaporates on a rebuilt node.

Here is my opinion, and it is arguable: I would rather see node-level config and credentials in hostPath files with 0600 ownership, owned by whatever config management already owns the node, than reachable through the API server. Yes, that means the value is not rotated by a Secret controller and not visible in kubectl describe. A control-plane component that depends on the API server to fetch the credential it needs to start is a bootstrap loop waiting for a bad day, and the same reasoning applies to any node agent you expect to keep working while the API server is down.

What to run before you upgrade to 1.37

In this order, on one node per pool, this week:

Grep the kubelet config for the dead gate. grep -n PreventStaticPodAPIReferences /var/lib/kubelet/config.yaml. A hit means that node fails to start on 1.37. Delete the line, restart the kubelet on the current version, confirm the node stays Ready, then upgrade. This is the only step that causes an outage if skipped.
Grep the manifests for references. Read staticPodPath from the config first, because it is not always /etc/kubernetes/manifests:

SPP=$(awk '/staticPodPath/{print $2}' /var/lib/kubelet/config.yaml)
grep -REl 'secretRef|configMapRef|imagePullSecrets|serviceAccountName|persistentVolumeClaim|projected' "$SPP"

Every file listed is a node that will lose its static pod on 1.37.

Run the invisible-pod diff from the section above on each node class and keep the output. If it is non-empty on a pre-1.34 node, you have been running unreported containers, and your compliance evidence has been under-reporting since 8 July 2021 when the behavior was first written down.
Move registry credentials into containerd (hosts.toml or a credential provider) and node config into hostPath files, then delete the offending manifest lines. Verify with a cold pull: crictl rmi <image> && crictl pull <image> on a node with no cached layer. A pull that succeeds only because the layer was already local is the failure mode that hides here.
Add step 3 to your node readiness check permanently. Treat a non-empty diff as a failed check, the same way you would treat a default-deny egress policy that never got applied.

What I read while writing this: the v1.37 sneak peek for the release framing, PR #140226 for the removal itself, issue #103587 for the 2021 logs, and the Static Pods documentation for the rule that was always there.

Originally published at indragustiprasetya.com

Fix Terraform Deposed Object Panic in readDiff on 1.15

Indra Gusti Prasetya — Mon, 24 Aug 2026 02:13:28 +0000

On Terraform 1.15.x, terraform apply can end in a Go stack trace rather than a diagnostic: panic: runtime error: invalid memory address or nil pointer dereference, with readDiff in the frame. It is not your config. It is a dropped map key, and the object that key points at is a live cloud resource that almost none of your tooling can see.

Short version: a failed create_before_destroy leaves an old object in state under a deposed key. If that resource also declares a precondition or postcondition, Terraform 1.15.x panics in readDiff while planning it (hashicorp/terraform#38586). The fix ships in 1.16.0-rc2, released 19 August 2026 and still not GA as of 24 August 2026. Sweep raw state for deposed keys before you upgrade, because 1.16 removes the crash and leaves the orphan.

What actually triggers the readDiff panic

Issue #38586 names the mechanism plainly. The DiffTransformer builds an update node for the deposed object because conditions exist on the resource. It does not propagate the deposed key onto that node. readDiff then asks the plan for a change to the current object, gets nil back, and dereferences it. The reporter filed against Terraform v1.15.2 on darwin_arm64 with AWS provider v6.42.0 and traced the crash to internal/terraform/node_resource_abstract_instance.go:178.

That line matters more than the stack trace does, because it tells you rewriting the precondition expression will not help. The condition's content is irrelevant. Its existence is the trigger.

The fix landed in v1.16.0-alpha20260715 on 15 July 2026 and is carried into v1.16.0-rc2. Every stable Terraform running in production today is on the panicking code path. If a model-generated answer told you this was fixed, check which version it meant.

Terraform's diagnostics are normally better than a raw panic. Even the genuinely annoying cases, like the Terraform "The deprecation originates from" warnings that point at a module which does not own the deprecation, still hand you a message. Here you get a nil dereference in a graph walker, which is the same unhelpful shape as npm's "Cannot read properties of null (edgesOut)" crash: the tool loses a reference mid-walk and takes the process with it. Worse, in CI it usually happens while holding a state lock, under whatever GitHub Actions OIDC role runs your applies, so the next job queues behind a lease nobody released.

How do you confirm a deposed object exists?

One command, and it deliberately bypasses the CLI's own view of state:

terraform state pull > /tmp/state.json
jq -r '.resources[] as $r | $r.instances[]
       | select(.deposed != null)
       | "\($r.type).\($r.name) deposed=\(.deposed) id=\(.attributes.id // "unknown")"' /tmp/state.json

If your state version serializes it differently, grep -n '"deposed"' /tmp/state.json settles the question. Zero hits means the deposed theory is dead and your panic is something else. One or more hits, on a resource that also carries a precondition or postcondition, is #38586.

For the plan side there is a documented path:

terraform show -json tfplan | jq '.resource_changes[] | select(.deposed != null)'

What I actually read. Issue #38586 for the DiffTransformer and readDiff mechanism and the exact source line, and issue #29713 for the missing deposed state variant, where core maintainer jbardin confirms the deposed entry was readable during walkPlanDestroy and gone by walkDestroy. The visibility rules come from HashiCorp's own JSON output format reference.

Three error strings, one dropped deposed key

Operators treat these as one bug. They need different responses.

Error string	Issue	Where it fails	What to do
`panic: ... nil pointer dereference` with `readDiff` in frame	#38586	Plan graph builds an update node for the deposed object, key not propagated	Drop the condition block for one apply, or move to 1.16.0-rc2
`Error: missing deposed state for aws_api_gateway_rest_api.this (4ece7f68)`	#29713	Entry readable in `walkPlanDestroy`, absent in `walkDestroy`	Re-plan; the deposed entry is real, the destroy walk lost it
`panic: MaybeRestoreResourceInstanceDeposed called without DeposedKey`	#22974	Older restore path, reported 2019	Historical, same defect class

Seven years of the same failure: the deposed key does not travel with the object it identifies. That is worth knowing before you attribute the crash to your module, your provider, or your backend.

Why terraform state list will never show it

The plan representation and the state representation disagree, and the disagreement is documented. HashiCorp's JSON format reference calls the deposed key "an opaque key representing the specific deposed object" and notes that address plus deposed key together uniquely identify a change. The values representation, which is what terraform show -json gives you for state, says the opposite in plain words: "Only the 'current' object for each resource instance is described. 'Deposed' objects are not reflected in this structure at all."

So your inventory tool, drift detector, cost report and tag-compliance scan all read a document that structurally cannot mention the resource you are paying for.

flowchart TD
  A["Config change forces replacement\nlifecycle create_before_destroy"] --> B["New object created\nappears in state list"]
  B --> C["Old object moved aside\nunder deposed key 4ece7f68"]
  C --> D{"Destroy of old object succeeds?"}
  D -->|"yes"| E["Deposed entry removed\nstate list and show -json agree"]
  D -->|"no"| F["Deposed entry persists\ncloud resource still running and billed"]
  F --> G["Invisible: terraform state list\nand show -json values representation"]
  F --> H["Visible: plan JSON deposed field\nand terraform state pull raw JSON"]
  F --> I["On 1.15.x with a condition block\napply panics in readDiff"]

There is also no address you can type. The resource addressing reference documents module paths, type and name, numeric count indices and string for_each keys. The word "deposed" does not appear on that page, and the terraform state rm documentation never mentions it either. Every "just state rm the deposed one" answer in circulation describes a grammar that does not exist.

Policy as code inherits the blind spot in a weaker form. The Sentinel tfstate/v2 import exposes a deposed_key attribute on a resource, described as matching the deposed field in resource_changes. It is a flag, not a collection. There is no deposed object whose attributes a policy can evaluate, so a rule like "every instance must carry an owner tag" cannot see the untagged orphan at all. If you are already tracking cloud resources that no identity or team owns, this is a category your governance stack is silently missing.

I have watched a team spend a quarter arguing about an unexplained line item in an AWS bill that turned out to be a deposed NAT gateway from a botched replacement. Their scanner was correct. Their state was correct. The interchange format between them omitted the object.

The documented recovery deletes your only evidence

HashiCorp's support article on create_before_destroy and deposed objects walks operators through removing all instances of the resource from state, importing the wanted object back, then deleting the unwanted one by hand in the cloud console. Its old URL now 302s into IBM's support portal, which tells you how much attention this corner gets.

Step one erases the deposed object's attributes, and that is where the real resource ID was written down. Step three then depends on a human remembering an ID that no longer exists anywhere in the repo, the state, or the plan. If you follow that procedure without recording the ID first, you have converted a tracked orphan into an untracked one.

Related behavior worth naming while you are in here: issue #31635 reports that with create_before_destroy, a failure in a later resource still leaves Terraform destroying the deposed object. Replacement ordering and failure handling around this feature are less deterministic than the lifecycle documentation implies. Deposed leftovers are common, not exotic.

Upgrading to 1.16 makes this quieter, not cleaner

The panic was a loud, unignorable signal that a deposed object existed. On 1.16 the same state applies cleanly and the object is still there, still running, still outside configuration. My read is that most teams will upgrade, watch the error disappear, and never learn the orphan was there. Given the choice I would keep the crash, which is why the jq check belongs in CI before the upgrade rather than after it.

What to run before you move CI to 1.16

Ordered by what bites first.

Sweep every state you own now, while the panic is still there to get your attention. Run the terraform state pull plus jq check above against each workspace. Any hit is a running resource with no configuration governing it.
Record attributes.id, region and ARN in a ticket before any state surgery. Not in terminal scrollback. terraform state rm removes current and deposed instances together, and that command is the moment the ID stops existing.
Prefer letting Terraform destroy it over removing it from state. If the remote object still exists, keep the resource in configuration and let a normal apply plan the destroy of the deposed object. State surgery is the fallback, not the first move.
If that apply panics in readDiff on 1.15.x, comment out the precondition or postcondition block on that one resource, apply, then restore it. This follows from the mechanism in #38586: no conditions means no update node for the deposed object. You are disabling a safety check to clear a leftover, so test against a copy of state first and put the block back in the same pull request.
Hold CI on 1.15.x until 1.16.0 goes GA, then upgrade with the deposed sweep already wired in as a post-apply step. Alerting threshold: a deposed key that survives two consecutive applies is an orphan, not a transient.
Point drift and cost tooling at plan JSON resource_changes[].deposed or at raw state, never at the values representation of terraform show -json, which is documented to omit these objects.

FAQ

What is a deposed object in Terraform? When a resource with create_before_destroy is replaced, Terraform creates the new object first and moves the old one aside under a deposed key, an 8-character hex string such as 4ece7f68. If the destroy of the old object then fails, that entry stays in state and the real cloud resource keeps running.

Why does terraform state list not show deposed objects? No address grammar exists for them, and the values representation of terraform show -json is documented to describe only the current object per instance. Their absence is by design.

Is it safe to run terraform state rm on a deposed resource? There is no syntax to remove only the deposed object, so state rm drops current and deposed together and takes the real resource ID with it. Record the ID first and prefer a planned destroy.

Does Terraform 1.16 remove deposed objects automatically? No. The release candidates fix the nil pointer dereference so the apply completes. The orphan survives the upgrade.

Originally published at indragustiprasetya.com

Fix Terraform "The deprecation originates from" Warnings

Indra Gusti Prasetya — Mon, 24 Aug 2026 02:09:04 +0000

TL;DR: Terraform 1.15 reports a deprecated attribute you never referenced because deprecation rides inside the value as a cty mark, and exporting a whole resource object carries the mark with it. Three different situations print the identical warning. Run terraform validate -json, compare warning_count to the console, then grep each originates from attribute against your own HCL: hits are real migration work, misses are propagation. Terraform 1.16 fixes the child module case only.

Nobody wrote website_domain. Terraform 1.15.8 printed it anyway, attached to an output block that exports an S3 bucket:

Warning: Deprecated value used

  on bucket.tf line 11, in output "bucket":
  11:   value       = aws_s3_bucket.example

  The deprecation originates from aws_s3_bucket.example.website_domain

website_domain is deprecated. [more deprecation instructions here]

(and 13 more similar warnings elsewhere)

That is hashicorp/terraform issue #38982, and the last line matters as much as the text above it. Issue #38554, filed 2026-05-11 against 1.15.2, counted roughly 30 of these coming out of one S3 module, and the reporter's summary does the diagnosis for you: "no value is passed to the variable, it is never referenced, so no warning should be produced." One warning summary with several unrelated triggers is the same trap as npm's null edgesOut crash. You cannot fix it until you know which trigger you have.

Why Terraform blames an attribute you never wrote

Terraform 1.15 shipped variable and output deprecation in PR #38001, and it is implemented with marks rather than usage analysis. internal/lang/marks defines a DeprecationMark{Message, OriginDescription}, and deprecated provider schema attributes get the same treatment. A mark lives inside the value. Export the resource object and the mark on website_domain goes with it, so when the output node walks the value looking for marks, it finds one and reports the export as a use.

The The deprecation originates from line is literally the mark's OriginDescription. That is the reason it names an attribute that appears nowhere in your configuration, and the reason no amount of editing your own HCL clears it.

The blast radius is a module you did not write. The terraform-aws-eventbridge module tripped this at version 4.3.0 on Terraform 1.15.4 with AWS provider 6.46.0 from a single line, output "eventbridge_rules" { value = aws_cloudwatch_event_rule.this }, as filed in module issue #199. Any module that exports whole resource objects, which is most of them, becomes a warning generator the moment a provider deprecates one nested attribute. And a deprecation is a removal clock, the same kind of clock that Ingress-NGINX retirement put on everyone's ingress layer, so the noise is worth reading rather than muting.

What the 1.16 changelog line actually covers

Terraform 1.16.0-rc2 landed on 2026-08-19 with a changelog entry that reads like the end of this: "Child module outputs with unreferenced deprecated nested attributes no longer return deprecation warnings" (#38778).

The merged diff is eleven lines in internal/terraform/node_output.go (merged 2026-06-25, approved by jbardin), and it branches on IsRoot():

if n.ModulePath().IsRoot() {
    val, deprecationDiags = ctx.Deprecations().ValidateExpressionDeepAndUnmark(val, n.ModulePath(), n.Config.Expr)
} else {
    // If the output is in a child module, only check for deprecations
    // at the "top level". This avoids deprecation warnings when
    // outputting an entire resource with a nested deprecated attribute.
    // (References to said attribute should still incur a warning)
    val, deprecationDiags = ctx.Deprecations().ValidateAndUnmark(val, n.ModulePath(), n.Config.Expr.Range().Ptr())
}

The doc comments in internal/deprecation/deprecation.go name the difference plainly: ValidateExpressionDeepAndUnmark uses GetDeprecationMarksDeep and traverses the value, ValidateAndUnmark uses GetDeprecationMarks and handles top-level marks only. Root outputs keep the deep walk.

The repro in #38982 is a root module output (on bucket.tf line 11, in output "bucket"), and my read of the diff is that this case still warns on 1.16 even though the issue is closed. I would spend ten minutes reproducing your own case on rc2 before you schedule an upgrade around that changelog line.

flowchart TD
  A["aws_s3_bucket.example\nprovider marks website_domain deprecated"] --> B{"where is the\noutput block?"}
  B -->|"child module"| C["ValidateAndUnmark\ntop-level marks only\nno warning in 1.16"]
  B -->|"root module"| D["ValidateExpressionDeepAndUnmark\nwalks nested marks\nwarning still fires"]
  C --> E["nested marks survive into the caller"]
  E --> F["caller re-exports the object\nfrom a root output"]
  F --> D

Worth noting that the 1.15 line moved the opposite way in the same week. Terraform 1.15.9, released 2026-08-19 alongside rc2, carries "Child module validation has been fixed and will now raise errors or warning diagnostics for invalid blocks." Patch forward on 1.15 and you get more child module diagnostics; move to 1.16 and you get fewer of this specific kind. 1.15.6 had already fixed a terraform console panic when evaluating expressions with deprecated values. Four minor releases in, the marks implementation is still settling.

Three causes hide behind one warning summary

The issue threads run these together, which is why the advice inside them contradicts itself.

Cause	What the "originates from" path points at	Fixed in 1.16.0	The right move
You actually reference the deprecated attribute	An attribute you can grep in your own HCL	No, and correctly so	Migrate the attribute
A child module output exports a whole resource or data source object	An attribute of a resource inside the module	Yes (#38778)	Upgrade Terraform
A root module output or local exports a whole object	An attribute of a resource in your root config	Not per the diff	Export named attributes, or accept the noise

The two commands that separate them

Run terraform validate -json and compare its warning_count against what the console showed you. The human renderer consolidates same-summary diagnostics into (and N more similar warnings elsewhere), a behavior still tracked as open issue #32104, so the collapsed remainder is exactly where a genuine deprecation hides behind a dozen propagation artifacts. If the console printed 4 and the JSON says 30, you have been reading 13 percent of your deprecation surface.

Then pull the attribute name out of each originates from path and grep your own HCL for it:

terraform validate -json \
  | jq -r '.diagnostics[] | select(.summary=="Deprecated value used") | .detail' \
  | grep -o 'originates from [^ ]*' | sort -u

Zero grep hits in your configuration means propagation. Hits mean migration work. That split is the only thing that tells you whether a Terraform upgrade helps you at all, and it takes about a minute per workspace inside the same CI job that already runs plan against AWS.

ignore_nested_deprecations deletes the signal, it does not hide it

The workaround circulating in the module threads is ignore_nested_deprecations = true on the module call. The syntax page gives it one sentence: "If the ignore_nested_deprecations argument is set to true, Terraform does not show deprecation warnings in the module call or nested modules." No mention of scope, and nothing about what happens to the underlying data.

The source is more specific than the docs. Suppression runs through IsModuleCallDeprecationSuppressed, checked against a suppressedModules list, and both validation paths remove the marks from the value whether or not suppression is on. This is not a display filter. It strips the deprecation information for that module and everything under it, real deprecations included. Set it to quiet fourteen false positives and you have also disabled the fifteenth, which was the one you needed.

These warnings are your AWS provider v7 backlog

Ignoring them is tempting because they do not change an exit code. Plan stays green, CI stays green.

The AWS provider is currently using deprecations as its v7 removal queue. Provider issue #42468, milestone v7.0.0, is open to delete name from the aws_region data source. Issue #44803 tracks removing deprecated status attributes from the organizations resources. Each of those turns into a hard plan error on the day you bump the major version, and the warnings you are scrolling past are the inventory of that work. It is the same warning-window economics as the npm allowScripts notices before npm 12: free to fix while it is a warning, a broken pipeline once the major lands. Suppress the list at the module boundary in August and you rediscover it as an upgrade-day outage, which is how a Kubernetes minor bump turns into an "unrecognized format int32" scramble.

What to run before your next provider bump

Capture the real count. terraform validate -json | jq '.warning_count' in every workspace, and record the delta against the console number. Anything above roughly 10 means the console has been collapsing most of it (#32104).
Classify with grep, not by reading. Extract each originates from attribute and search your own HCL. Hits go on the migration list; misses are mark propagation and no config edit will clear them.
Fix the ones you own at the source. In your own modules, export named attributes instead of whole resource objects wherever the consumer contract allows. That kills the noise on every Terraform version, including the root module case 1.16 does not cover.
Leave ignore_nested_deprecations alone unless the module is vendored and unchangeable. If you must set it, put the date you will revisit it in a comment next to the argument, because it blinds the entire subtree.
Turn the surviving genuine warnings into a v7 ticket now. Provider issues #42468 and #44803 are open against the v7.0.0 milestone; every attribute on that list is a plan error waiting for your major bump.
If a 1.16 upgrade is on the plan specifically to clear this, reproduce your case on 1.16.0-rc2 first. The changelog says child module outputs, and the eleven-line diff means it: https://github.com/hashicorp/terraform/pull/38778

Originally published at indragustiprasetya.com

npm approve-scripts vs --allow-scripts: pick the right flag

Indra Gusti Prasetya — Sun, 23 Aug 2026 08:36:31 +0000

npm 12 blocks dependency install scripts by default and gives you four different ways to let one run again. Two of those ways reject each other outright, so what follows is the decision rule: which command inside a project, which flag for a global install, what actually lands in git, and the CI gate that catches what the review command quietly hides from you.

TL;DR: Inside a project, never pass --allow-scripts on the command line. Run npm approve-scripts --allow-scripts-pending to review, then npm approve-scripts <pkg> (or --all) to write the allowScripts field into package.json, and commit package.json. For a global install (-g) or npx there is no package.json to write to, so you must use the install-time --allow-scripts=<pkg> flag, the same flag that is forbidden inside a project. Gate CI with npm ci --strict-allow-scripts.

The confusion is worth naming precisely, because the two failures look similar and have opposite fixes. npm approve-scripts exits EGLOBAL when you run it for a global install (npm/cli #9463). --allow-scripts is refused inside a project install, with an error telling you to edit package.json instead. Same person, two terminals, two contradictory messages. If you have already hit the second one, the walkthrough lives in Fix npm --allow-scripts not allowed in project installs; the reason the field belongs in package.json rather than where most people reach for it is covered in npm allowScripts: the package.json schema, not .npmrc.

Which flag, in which scope?

You want to	Command	Scope	Mutates?	What it touches
See what needs approval (read-only)	`npm approve-scripts --allow-scripts-pending` or `npm install-scripts ls`	project	no	nothing
Approve one package	`npm approve-scripts <pkg>` / `npm install-scripts approve <pkg>`	project	yes	`allowScripts` in `package.json`
Approve everything pending	`npm approve-scripts --all`	project	yes	`allowScripts` in `package.json`
Permanently deny a package	`npm deny-scripts <pkg>` / `npm install-scripts deny <pkg>`	project	yes	`allowScripts` (name-only `false`)
Allow scripts for a global or npx install	`npm i -g --allow-scripts=<pkg>`	global	no	nothing persisted (per invocation)
Persist a global allowance	`npm config set allow-scripts=<pkg> --location=user`	global	yes	your user `.npmrc`
Enforce in CI	`npm ci --strict-allow-scripts`	project	no	fails the build on gaps

The model to keep in your head: approve-scripts writes a committed allowlist into package.json, while --allow-scripts is a one-off install-time override for contexts that have no package.json to commit. Each one is rejected in the other's territory.

flowchart TD
    A["Install script was skipped"] --> B{"Global install or npx?"}
    B -->|"global"| C["npm i -g --allow-scripts=pkg"]
    B -->|"project"| D["npm approve-scripts --allow-scripts-pending\n(review only)"]
    D --> E["npm approve-scripts pkg\nor --all"]
    E --> F["Commit package.json"]
    F --> G["CI: npm ci --strict-allow-scripts"]
    C --> H["npm config set allow-scripts=pkg --location=user\nto persist per machine"]

Prerequisites

npm 12.x (npm -v). The npm install-scripts command namespace and the strict-allow-scripts config are npm 12 features, shipped with the breaking changes the npm team published on June 9, 2026 (GitHub Changelog). On 11.16.x you get warnings only.
Node.js 22+ (bundled with npm 12).
A repo with a package.json and a package-lock.json for the project workflow, plus global install access for the -g case.

Step-by-step

1. Inventory what would run, without writing anything

npm approve-scripts --allow-scripts-pending

This lists every dependency whose install scripts are not yet covered by allowScripts and changes nothing on disk. npm install-scripts ls is the equivalent under the namespaced command. Run it first, every time. It is the audit output you paste into the PR that adds the allowlist, so a reviewer can see what you decided to trust and what you left out.

2. Approve only the packages you actually trust

# one at a time, the safe default
npm approve-scripts esbuild

# or approve everything pending in one shot
npm approve-scripts --all

Either form writes entries into the allowScripts object in package.json. Approvals are version-pinned by default (--allow-scripts-pin is true), producing "esbuild@0.25.0": true. Setting --allow-scripts-pin=false gives you name-only entries that survive version bumps, which is more convenient and strictly weaker: you lose the property that a new version drops back into "pending" and gets re-reviewed. I keep pinning on and treat the churn as the feature, because a postinstall added in a patch release is exactly the event an allowlist exists to catch. That is the same fast-publish window that dependency cooldowns close from the release-timing side.

--all is fine on the first migration of a repo you already know. It is a bad habit after that, since it approves whatever happens to be pending rather than whatever you read.

3. Deny what should never run, and know the asymmetry

npm deny-scripts core-js

A denial is always written name-only as "core-js": false, whatever the pin setting says. That asymmetry is deliberate and useful: --all approves everything pending, but a false entry is sticky and is not overwritten by a later --all. Deny is the permanent "no" that a blanket approve cannot undo by accident.

4. Commit the one artifact this produces

The only thing that gets committed is the allowScripts field in package.json. Here is a real one, from the eslint project's npm 12 migration (eslint/eslint#21092):

{
  "allowScripts": {
    "core-js": false,
    "cypress": false,
    "re2": true,
    "yorkie": true
  }
}

Commit package.json. Do not adopt hand-editing as a workflow, let the commands maintain the field, but do review the diff the way you would review an IAM policy change: each true is a package you have granted arbitrary code execution on every developer laptop and every CI runner. The error message from a mis-run install names the two legal homes, allowScripts in package.json or .npmrc. Keep the project allowlist in package.json where it is committed, PR-reviewed, and shared, and reserve .npmrc for machine-local or global allowances that should not travel with the repo.

5. Gate CI so an unapproved script fails the build

npm ci --strict-allow-scripts

In a normal install npm skips unapproved scripts, prints a summary, and exits 0. A green local npm install therefore proves nothing about whether your allowlist is complete. --strict-allow-scripts (or npm config set strict-allow-scripts true) turns a skipped script into a hard failure, so CI catches the dependency that started shipping a postinstall between two commits instead of letting a native module go missing until it explodes inside a request at runtime.

Why does npm approve-scripts fail on a global install?

Install a global CLI on npm 12 and you get a warning that points you at the wrong command:

npm install -g some-cli
# npm warn allow-scripts Run `npm approve-scripts --allow-scripts-pending` to
#   review, or `npm approve-scripts <pkg>` to allow.

Follow that advice and you hit npm/cli #9463:

npm approve-scripts does not work for global installs

It exits with code EGLOBAL. The reason is structural rather than a bug in your invocation: approve-scripts writes to a project package.json, and a global install does not have one. The suggested command cannot work in that context, so treat the warning text as wrong there. The working fix is the install-time flag, optionally persisted to your user .npmrc:

npm install -g --allow-scripts=some-cli some-cli

# persist it for future global installs of that package
npm config set allow-scripts=some-cli --location=user

The mirror image is the project-scoped rejection of that same flag:

npm error --allow-scripts is not allowed in project-scoped installs.
Add the entries to the "allowScripts" field in package.json, or to .npmrc, instead.

One rule resolves both messages: project goes through approve-scripts and package.json, global and npx go through --allow-scripts at install time.

Verify it works

# 1. the allowlist is populated
node -p "Object.keys(require('./package.json').allowScripts || {}).length"

# 2. strict CI passes with nothing pending
npm ci --strict-allow-scripts && echo "OK: all scripts covered"

The first prints a non-zero count. The second exits 0 with no install scripts not covered line. If the second command fails while --allow-scripts-pending showed you nothing, that is not a mistake on your part, it is the pitfall below.

Common pitfalls

The review command hides optional deps that the strict check rejects. This is the sharpest trap in the feature, filed as npm/cli #9562. On Linux, npm approve-scripts --allow-scripts-pending does not list fsevents, because it is an optional dependency marked os: ["darwin"] and your platform skips scanning it. But npm ci --strict-allow-scripts validates the lockfile, sees fsevents in there, and fails:

npm error --strict-allow-scripts: 1 package(s) have install scripts not covered by allowScripts: fsevents@2.3.3

Your review said "all clear" and CI said "rejected" for the same dependency tree. Approve the optional package explicitly even though the inventory never surfaced it (npm approve-scripts fsevents@2.3.3), or run the inventory on the OS where that optional dependency actually installs. When the two disagree, the lockfile-based strict check is the source of truth and the platform-filtered review is the false negative.

Version pins go stale after upgrades. With pinning on, bumping esbuild@0.25.0 to 0.26.0 moves it back to pending, which is intended. Clear dead entries with npm install-scripts prune, and preview with --dry-run first.

A passing npm install is not proof of coverage. Non-strict installs skip and warn while still exiting 0. Only --strict-allow-scripts makes coverage a gate, so wire it into the pipeline rather than into your own shell aliases.

Wrap-up

The decision rule is short: approve-scripts plus package.json for projects, --allow-scripts for global and npx, deny-scripts for the permanent no, --strict-allow-scripts for CI. In that order, and with the pin left on.

Two follow-ups worth doing this week. Run npm ci --strict-allow-scripts in a fresh clone on the same OS as your CI runner, so platform-specific optional dependencies surface before they fail a real pipeline. Then check that a package you approved is one you would still approve at its current version, because install scripts are where a compromised release actually executes. For the full npm 12 migration timeline and the 11.16 warning window, see the npm v12 breaking-changes walkthrough; for the same problem in another ecosystem, cargo build.rs runs any code shows how it plays out in Rust.

FAQ

Do I commit package.json after npm approve-scripts? Yes. The allowScripts field lives in package.json, and committing it is the point: the allowlist gets reviewed in a PR and shared by the team. Let the commands maintain the field instead of hand-editing it, but read the diff like any other security change.

What is the difference between --allow-scripts-pending and approving a package? --allow-scripts-pending is read-only. It lists the packages whose install scripts are not covered by allowScripts and writes nothing. npm approve-scripts <pkg> mutates package.json to allow that package.

Why does npm approve-scripts fail on a global install? Because it writes to a project package.json and a global install (-g or npx) has none, so it exits EGLOBAL with "does not work for global installs" (npm/cli #9463). Use npm i -g --allow-scripts=<pkg> instead.

Why is --allow-scripts rejected inside my project? npm forbids the install-time flag in project scope and points you at the allowScripts field in package.json or at .npmrc. Inside a repo use npm approve-scripts; the CLI flag is for global and npx installs only.

My local install passed but CI failed on fsevents. Why? --allow-scripts-pending skips optional and off-platform dependencies, while npm ci --strict-allow-scripts validates the lockfile and sees them anyway (npm/cli #9562). Approve the optional package explicitly, or run the inventory on the same OS as your CI runner.

Sources

Originally published at indragustiprasetya.com

npm allowScripts: the package.json Schema, Not .npmrc

Indra Gusti Prasetya — Sun, 23 Aug 2026 08:32:21 +0000

By the end of this you will have a correct allowScripts block in package.json, a CI install that fails loudly when a dependency ships an unreviewed install script, and zero allow-scripts entries left in .npmrc where they cause a confusing error instead of a policy.

Short version: allowScripts is a plain object in package.json mapping a dependency to a boolean. true lets its install script run, false blocks it. Keys are either pinned ("pkg@1.2.3": true) or name-only ("pkg": false). It is a package.json field. .npmrc only holds the toggles around it (ignore-scripts, strict-allow-scripts, allow-scripts-pin) plus the allow-scripts list, which works for global and npx installs and is rejected in a project install.

The field shipped in npm 11.16 (commit a10c7ca, "Phase 1 of allowScripts opt-in install-script policy") and becomes enforcing in npm 12, which is late enough that most generated answers conflate the two config surfaces. npm's own error text conflates them too, so the confusion is earned. If you want the migration walkthrough rather than the semantics, read fix the "install scripts not covered" warning before npm 12; if you already hit the flag error, Fix npm --allow-scripts not allowed in project installs covers that one case. This page is the reference both of those point back to.

Prerequisites

npm 11.16.0 or newer to see the pending-package warning, npm 12.x to see the default block. Check with npm --version.
A project with a package.json. The field is meaningless without one, and approve-scripts / deny-scripts return EGLOBAL outside a project.
Knowing which scripts are in scope: preinstall, install, postinstall, and prepare for non-registry (git or local) sources. Those are the only lifecycle scripts allowScripts governs. Your own root-project scripts always run.

Step-by-step

1. Find out which npm you actually have

npm --version

On 11.16 through 11.x a pending package produces a warning and nothing else. On 12.x the same package gets its script skipped by default. The behavior change is the reason a lockfile that installed fine last quarter can quietly stop running sharp's build step after a runner image bump.

2. List what is pending, without writing anything

# npm 12
npm install-scripts

# npm 11
npm approve-scripts --allow-scripts-pending

--allow-scripts-pending is read-only. It prints every package whose install scripts are not yet covered and touches nothing on disk, which is what you want on the first pass so you can review each package before approving it. Do the review here, not later: this list is the only point in the process where you look at what code you are agreeing to execute.

3. Write the field, and pick the key shape deliberately

allowScripts is Record<PackageIdentifier, boolean>. That is the entire schema.

{
  "name": "my-app",
  "allowScripts": {
    "esbuild@0.24.2": true,
    "@swc/core@1.7.0": true,
    "sharp": true,
    "some-typosquat": false
  }
}

Two key shapes exist, and choosing between them is the only real decision the field asks of you:

Key shape	Example	Meaning
Pinned	`"esbuild@0.24.2": true`	Approval is narrowed to that exact version. `0.24.3` is not covered and returns to pending.
Name-only	`"sharp": true`	Applies to any version of the package.

Per the npm-install-scripts docs, npm writes pinned entries by default, because "By default it writes pinned entries (pkg@1.2.3), which keep their approval narrowed to the specific version you reviewed." The allow-scripts-pin config (default true) controls that; set it to false if you want generated entries to be name-only.

My rule: allow pinned, deny name-only. A true should expire when the code changes, and a false should not. Writing denials as "some-typosquat": false means the block survives every version bump, so you never re-review a package you already decided should never execute.

4. Rule out ignore-scripts before you debug anything else

npm config get ignore-scripts

If that returns true, allowScripts is inert. The docs are explicit that ignore-scripts "takes precedence and no scripts run, the allowlist does not override it." An .npmrc at any layer (project, user, global) can set it. Pick one model and stay in it: global-off via ignore-scripts, or granular via allowScripts. Running both means the allowlist is decoration.

Here is the resolution order for a dependency that declares an install script:

flowchart TD
    A["Dependency declares an install script"] --> B{"ignore-scripts = true?"}
    B -->|yes| C["No scripts run and allowScripts is ignored"]
    B -->|no| D{"Entry in allowScripts?"}
    D -->|denied| E["Script blocked"]
    D -->|covered| F["Script runs"]
    D -->|pending| G{"strict-allow-scripts set?"}
    G -->|yes| H["Install fails with ESTRICTALLOWSCRIPTS"]
    G -->|no| I["Script skipped, WARN, install succeeds"]

Three states matter for anything not blanket-disabled by ignore-scripts. Covered (true) runs normally. Denied (false) is blocked, and per the docs "Existing false entries always win; approve will not silently re-allow a package you previously denied," which holds even under approve-scripts --all. Pending (no entry) gets its script skipped with npm warn allow-scripts N packages have install scripts not yet covered by allowScripts, and the install still exits 0. The community migration discussion frames it the same way: "an install script you haven't approved gets skipped, you get a warning, and the install still succeeds."

That exit 0 is the part worth reacting to. A skipped postinstall on a native module does not fail the install; it fails at runtime, in whatever service imports the module, with a missing binary. The warning scrolls past in CI logs and the build goes green.

5. Turn pending into a hard failure in CI

# .npmrc
strict-allow-scripts=true

A pending package now fails the install with ESTRICTALLOWSCRIPTS instead of warning (npm/cli #9562). Set this in CI today, on npm 11, and you get npm 12's enforcement on your schedule rather than on your base image's. It pairs well with dependency cooldowns: a new version arrives, it lands as pending, and the pipeline stops until a human looks at it.

6. Keep the allowlist out of .npmrc

.npmrc can carry ignore-scripts, strict-allow-scripts, allow-scripts-pin, and allow-scripts=<comma,separated,names>. That last one only applies to global and npx contexts, where there is no package.json to hold a field:

# valid: global install, allow named packages' scripts
npm install -g --allow-scripts=canvas,sharp some-cli

# or persist it for global/npx use
npm config set allow-scripts=canvas,sharp --location=user

Pass --allow-scripts to a project install, ci, update, or rebuild and npm rejects it deliberately: it is "blocked in project installs on purpose (it's for -g and npx)."

The trap, filed as npm/cli #9783, is that a user or global .npmrc carrying allow-scripts=... can leak into a project install you never flagged. Preparing a git dependency spawns an inner npm install and forwards config as npm_config_* environment variables. The inner install reads allow-scripts off its env layer, classifies it as a command-line or env policy, and fails with EALLOWSCRIPTS. The message then tells you to move the entry into .npmrc, where it already is. Remove allow-scripts from your user and global .npmrc, and express project intent through package.json.allowScripts only.

7. Commit the field

node -p "require('./package.json').allowScripts"
git add package.json && git commit -m "chore: pin allowScripts allowlist"

allowScripts is a reviewed-code policy, so it belongs in version control with the same weight as the lockfile. Uncommitted, every teammate and every CI job re-hits the warning and someone eventually approves in a hurry.

Verify it works

npm ci --strict-allow-scripts

Expected on success: exit 0 with no allow-scripts warning, meaning every script-carrying dependency is covered. Expected on failure: a non-zero exit and ESTRICTALLOWSCRIPTS naming the offending package. Run it once with a deliberately bumped dependency version to confirm the pinned key stops covering it, since that is the behavior most likely to surprise you later.

Common pitfalls

A version bump drops coverage. "esbuild@0.24.2": true does nothing for 0.24.3. The package returns to pending, which is the design, not a bug. Reserve name-only true keys for packages you trust across every future version.
ignore-scripts=true masks a working allowlist. Scripts not running despite a complete allowScripts? Check npm config get ignore-scripts first, including the user and global layers.
--allow-scripts is not the granular tool. It is for -g and npx. Copying it out of a global-install tutorial into a project command produces EALLOWSCRIPTS, and #9783 means you can hit that error without typing the flag at all.
approve-scripts and deny-scripts fail outside a project with EGLOBAL, because there is no package.json to write. Use the allow-scripts config for global and npx work.
A green install is not a working install. On npm 12 a pending native module installs cleanly and breaks at import time. If you are already chasing odd install-tree behavior, rule out lockfile damage separately, as in Fix npm "Cannot read properties of null (edgesOut)".

FAQ

Does allowScripts go in package.json or .npmrc? The per-package allowlist lives only in package.json. .npmrc carries the toggles (ignore-scripts, strict-allow-scripts, allow-scripts-pin) plus the allow-scripts list for global and npx installs.

What is the difference between "pkg": true and "pkg@1.2.3": true? Name-only allows any version. Pinned allows only the version you reviewed, and a new version resets to pending. npm writes pinned entries by default.

Does allowScripts override ignore-scripts? No. With ignore-scripts=true, no scripts run and allowScripts is ignored entirely.

What happens to a package that is not listed? It is pending: skipped with a warning on npm 12, install still exits 0. With strict-allow-scripts set, the install fails with ESTRICTALLOWSCRIPTS.

Do false entries ever get re-enabled automatically? No. An existing false wins, and approve-scripts --all will not undo it.

Wrap-up

The model in one line: allowScripts is Record<PackageIdentifier, boolean> in package.json, pinned by default, false always wins, and ignore-scripts short-circuits all of it. The per-package allowlist never belongs in .npmrc; that file holds the toggles and the global-only allow-scripts list.

Do these in order. Run npm install-scripts (or approve-scripts --allow-scripts-pending on 11.x) and read the list before approving anything. Write approvals pinned and denials name-only. Set strict-allow-scripts=true in CI so an unreviewed script fails the build now instead of after your runner picks up npm 12. Grep your user and global .npmrc for allow-scripts and delete it. Commit package.json. The reason to bother is the same reason build.rs in Cargo deserves an allowlist it does not have: an install script executes with your developer's credentials and your runner's network access, before any code you wrote has run.

Sources

Originally published at indragustiprasetya.com

arrayref Attack Shows cargo build.rs Runs Any Code

Indra Gusti Prasetya — Fri, 21 Aug 2026 03:35:17 +0000

On August 20, 2026, someone published arrayref@0.3.10 to crates.io at 07:15 UTC. It was gone by 08:41, alive for 86 minutes. During that window nobody had to call the crate. Resolving it into your dependency tree and running cargo build was enough to download and execute a remote binary with your user privileges.

Short version: the malicious code shipped inside a build script (build.rs) carried by proc-macro1, a one-character typosquat of proc-macro2 that arrayref@0.3.10 added as a dependency. Cargo runs build.rs automatically at compile time, so cargo build or cargo update triggered it with no function call required. Rust has no cargo --ignore-scripts equivalent, so the fix is not a flag. It is committing Cargo.lock, building with --locked, and refusing brand-new versions with a cooldown.

arrayref is a nine-line convenience crate with roughly 245 million downloads, sitting under winit, egui, iced, blake3, and a swath of Solana and Ethereum tooling, per The Hacker News and the Rust Security Response Team. The poisoned release added one new dependency: proc-macro1, published from dtolney, a near-miss of the real maintainer account dtolnay. Two sibling crates from the same owner, internment@0.8.7 and append-only-vec@0.1.9, were poisoned the same morning and pulled within 90 and 107 minutes.

What actually executed during cargo build?

The payload ran at build time, not run time. Per StepSecurity's teardown and the Rust Project advisory, the malicious logic lived in proc-macro1's build.rs. Nothing from arrayref or its siblings had to be invoked. Resolving the tree and compiling it was the trigger.

The script was written to slip past a casual look. It reconstructed its C2 host (23.254.165.112, a Hostwinds VPS) from base64 URL fragments, used an AcceptAll TLS verifier so it could talk to a bare IP with no valid certificate, wrote /tmp/rust-setup on Unix (a .ps1 plus .vbs pair driven by wscript.exe on Windows), and called std::mem::forget(child) so the spawned process outlived the build. On one Linux victim, stage two dropped MonoService and MonoXpc binaries under ~/.config/AzureKits and registered a systemd service for persistence.

flowchart TD
  A["cargo build or cargo update"] --> B["Resolve dependency tree"]
  B --> C["arrayref 0.3.10 pulls proc-macro1"]
  C --> D{"proc-macro1 ships a build.rs?"}
  D -->|yes| E["Cargo runs build.rs at compile time"]
  E --> F["Rebuild C2 host from base64 fragments"]
  F --> G["Fetch binary over TLS with AcceptAll verifier"]
  G --> H["Spawn /tmp/rust-setup, std mem forget child"]
  H --> I["Stage two: AzureKits binaries plus systemd persistence"]

The trust boundary that matters here is not "code I wrote" versus "libraries I use." It is "code that runs when I compile." In Rust those are different sets, and build.rs is the second one.

Why "I never called that crate" is the wrong model

Every JavaScript developer reading this already survived the install-script era. npm shipped --ignore-scripts, then npm 12 turned install scripts off by default behind an allowlist you commit to package.json, and pnpm added its own onlyBuiltDependencies gate. If you have wrestled with the npm --allow-scripts not allowed in project installs error or the npm 12 allowScripts warning, you already know the muscle: name the packages allowed to run code at install, deny the rest.

Rust's build.rs is the structural twin of an npm install script, with one difference. Rust never shipped the seatbelt. There is no flag today that says "compile this project, but do not execute my dependencies' build scripts." So the assumption a JavaScript team now gets for free is unavailable to a Rust team in August 2026, and this incident is the first at-scale demonstration of what that gap costs.

The second-order effect is where teams get burned. Because the payload sits in a build script, static reasoning about your own code (grepping for use arrayref, checking which functions you call) tells you nothing. A dependency you pulled purely as a transitive requirement of winit or blake3, and never reference directly, ran code on your build box. Audit the set of crates in your resolved graph that carry a build.rs, not the set you import.

Is `--offline` the fix? No, and here is why

The reflex I saw within hours of the disclosure was "I'll just build --offline." That is false comfort. Offline builds only protect you if the malicious .crate is not already sitting in ~/.cargo/registry/cache. Once a poisoned cargo update has run, it is there, and an offline build will happily compile and execute it.

The controls that would have shrunk this to zero are duller and older. Commit Cargo.lock and build with --locked so CI never silently resolves a brand-new version behind your back. Then add a cooldown that refuses versions younger than a few days. An 86-minute exposure window does not survive a 3-day cooldown, because no build in that window ever sees the poisoned version as installable. This is the same argument the dependency cooldown work makes on the npm side, and it ports directly to Cargo.

One more trap: yanking the version upstream does not clean your machine. The Rust advisory ships a find command against ~/.cargo/registry/cache for exactly this reason. Cached .crate files survive an upstream yank. Any host that ran cargo build or cargo update in the 07:11 to 09:25 UTC window on August 20 should be treated as compromised, with its reachable credentials rotated.

How a one-character typosquat survived a glance

The attacker staged the name to look legitimate. They first published a clean proc-macro1@1.0.106, a bit-for-bit copy of proc-macro2 source with no payload, to build innocent history. Then they weaponized 1.0.107 with the malicious build.rs and three build-only dependencies (base64, rustls, ureq), per StepSecurity. A developer who clicked through to vet the crate would have found a real, building, plausible package with prior releases. Provenance-by-eyeballing loses against this.

cargo vet and cargo-deny are the durable answer, and it is worth being honest that neither would have flagged 1.0.107 the instant it landed. What cargo vet gives you is a "who audited this, and this exact version" model, which is the thing typosquat-by-history is designed to defeat. That is a different job from a CVE scanner. If you want the distinction between scan-what-exists and attest-what-you-trust laid out for the container world, the same split shows up in Trivy versus Grype.

Who ran this, and why the target set fits

Wiz reports significant overlap between this campaign's infrastructure and tradecraft and prior DPRK-linked operations. That reading is consistent with the stage-two persistence, which focused on credentials and crypto tooling, and with the choice of crates sitting under Solana and Ethereum ecosystems. The counterpoint worth stating: attribution from infrastructure reuse is a probabilistic claim, not proof, and the defensive posture does not change based on who did it. Whether the actor is a nation-state or a bored opportunist, the control gap (build scripts run arbitrary code, and Cargo cannot stop them) is identical.

The broader signal is the cadence. Published-then-yanked-in-under-two-hours is now a repeatable pattern across npm, PyPI, and Cargo. Speed is the attacker's advantage, and every defense that depends on a human noticing in time is already beaten.

What to do before your next CI build

Order these by what bites first.

Commit Cargo.lock and add --locked to every CI build command today. For applications there is no excuse: cargo build --locked fails the build rather than silently resolving 0.3.10. This one line closes the exact vector that pulled the poisoned version in during the 86-minute window.
Set a cooldown of 3 to 7 days on new crate versions via cargo-deny policy or a filtering mirror. Every published-then-yanked-in-90-minutes attack dies against a 3-day floor. Pick 3 days as a minimum; go to 7 if your release pace allows.
Scan your cache for the known-bad versions now. Run the advisory's find against ~/.cargo/registry/cache for arrayref@0.3.10, internment@0.8.7, proc-macro1 (any version), and append-only-vec@0.1.9. If any host ran cargo build or cargo update between 07:11 and 09:25 UTC on Aug 20, rotate every credential reachable from that host, including cloud tokens and signing keys. Check ~/.config/AzureKits and for a rogue systemd service.
Baseline build-time egress, not just runtime. StepSecurity's Harden-Runner caught this because a build phoning 23.254.165.112:9089 had never appeared in the workflow's network baseline. If your CI cannot see what cargo build talks to, wire up egress monitoring; the same default-deny egress logic that contains a pod contains a build runner.
Adopt cargo vet for your critical dependency set and pin trusted publishers. It will not catch a fresh typosquat instantly, but it makes version-level provenance a gate instead of a hope.
Track cargo #13681 (build-script allowlist) and #5720 (sandbox). Until one ships, treat every cargo build on a credentialed machine as arbitrary code execution by your entire transitive tree, because that is what it is.

FAQ

Does building with --offline protect me? Only if the poisoned .crate never reached your cache. After a cargo update pulled it, the file is in ~/.cargo/registry/cache and offline builds still run its build.rs. Use --locked against a committed lockfile.

I never called arrayref. Am I safe? No. The payload was in a build.rs, which Cargo runs for any crate in the resolved graph regardless of whether you import it.

Is there a cargo --ignore-scripts? No. As of August 2026 no flag skips dependency build scripts. Proposals cargo #13681 and #5720 are open and unmerged.

How do I know if I was hit? Run the advisory's find over ~/.cargo/registry/cache, and treat any host that compiled between 07:11 and 09:25 UTC on Aug 20 as compromised. Rotate its credentials.

Sources

Originally published at indragustiprasetya.com

Fix "unrecognized format int32" in Kubernetes 1.34

Indra Gusti Prasetya — Tue, 18 Aug 2026 04:43:14 +0000

After upgrading a cluster to Kubernetes 1.34.0, kubectl apply and helm install start printing Warning: unrecognized format "int32" (and int64, float, double) against CRDs that were valid the day before. Your CRDs are fine. This guide shows how to prove the warning is a kube-apiserver regression, tell it apart from a real format typo, and either patch the control plane or filter the noise cleanly until you can.

Overview

Short version: the warning is a server-side regression in kube-apiserver 1.34.0 that wrongly applied a new unknown-format check to integer and number properties. It was fixed in PR #133896, cherry-picked to the 1.34 patch line on 2025-09-05, and it ships clean in 1.35.0. The fix is a control-plane upgrade, not a kubectl or CRD change.

This is for platform and cluster operators who suddenly see a wall of format warnings from kube-prometheus-stack, cert-manager, Cilium, or their own operators, and have to decide whether to act. The symptom is all over the issue trackers, and almost every thread misattributes the cause: people get told to edit their CRD schemas or downgrade kubectl. Both are wrong. The warning is emitted by the API server, not the client, and the schemas were correct all along. Below is the one command that proves where the warning originates, so you can separate this false positive from a genuine format typo. If you are mid-rollout, file it next to the other 1.34/1.35 surprises like user namespaces going GA, and treat it the same way: it is a new server version changing behavior underneath stable tooling, the same shape of break as Cosign v3 breaking image verification in Harbor.

Prerequisites

A cluster running Kubernetes 1.34.0 (control plane). Check with kubectl version.
kubectl v1.28+ with server-side field validation (the default since 1.27).
Cluster-admin or the ability to apply a CRD in a scratch namespace.
If you run EKS, GKE, or AKS, note your provider's control-plane patch version. You likely cannot bump it yourself, which changes your options (see pitfalls).

Step-by-step

1. Reproduce the warning with a minimal CRD

Apply a CRD that declares an integer property with format: int32, the exact shape kube-prometheus-stack and cert-manager ship:

# int32-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  scope: Namespaced
  names:
    plural: widgets
    singular: widget
    kind: Widget
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                  format: int32

kubectl apply -f int32-crd.yaml

On 1.34.0 you get:

Warning: unrecognized format "int32"
customresourcedefinition.apiextensions.k8s.io/widgets.example.com created

The resource is still created. The warning is non-blocking, but in a real chart it repeats dozens of times and buries anything that actually matters.

2. Confirm the warning is server-side, not from kubectl

This is the step everyone skips. Add -v=8 and watch for a Warning response header from the API server:

kubectl apply -f int32-crd.yaml -v=8 2>&1 | grep -i warning

You'll see the warning arrive as an HTTP response header:

Response Headers:
    Warning: 299 - "unrecognized format \"int32\""

A 299 warning header is generated by kube-apiserver, not printed locally by kubectl. That single line tells you upgrading or downgrading the CLI will do nothing, because the behavior lives in the control plane. This matches the 1.35 release note, which files the fix under "Kube-apiserver."

3. Confirm it's a false positive, not a real typo

The check itself is legitimate: it exists to catch format typos like format: itn32. The bug is that 1.34.0 applied it to non-string types. Per the CRD documentation, int32, int64, float, and double are documented, supported formats, so a warning on any of those four attached to a type: integer or type: number field is the regression, not a mistake. Issue #133880 is the upstream confirmation.

A rule of thumb you can apply while scanning output:

Warning on int32/int64/float/double under type: integer/number is a false positive, ignore it.
Warning on a type: string field with a genuinely unknown format (format: emial) is real, fix your schema.

4. Apply the real fix: patch the control plane

The fix (PR #133896) restricts the unknown-format warning to type: string properties, so integer and number formats stop warning. It merged to master under the v1.35 milestone and was cherry-picked to release-1.34 (PR #133901, merged 2025-09-05).

On a self-managed cluster, upgrade the control plane to a 1.34 patch that includes the cherry-pick, or to 1.35.0+:

# self-managed / kubeadm example, upgrade the control-plane node
kubeadm upgrade plan
kubeadm upgrade apply v1.35.0

Verify the served version moved:

kubectl version -o json | jq -r '.serverVersion.gitVersion'

5. If you can't upgrade yet, filter, don't rewrite

On a managed cluster you are stuck on the provider's control-plane version until they roll the patch. Do not edit vendored CRDs to drop format: int32; you will silently weaken validation and create drift you have to unwind later. Filter the noise at the point of apply so real warnings still surface:

# keep every line EXCEPT the four known false positives
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  2> >(grep -vE 'unrecognized format "(int32|int64|float|double)"' >&2)

This suppresses only the four documented-format false positives and lets a genuine type: string typo through untouched.

Verify it works

Re-apply the same CRD after the control plane is patched:

kubectl apply -f int32-crd.yaml -v=8 2>&1 | grep -i 'unrecognized format' || echo "clean"

Expected output on a fixed control plane:

clean

No 299 header, no terminal warning. If you still see it, your serverVersion from step 4 has not actually moved, and the client version is irrelevant here.

Where the warning comes from

flowchart TD
    A["kubectl apply -f crd.yaml"] --> B["kube-apiserver\nvalidates CRD OpenAPI schema"]
    B --> C{"format on a\nstring property?"}
    C -->|"string, unknown format\n(real typo)"| D["Warning: 299\nunrecognized format"]
    C -->|"integer/number in 1.34.0\n(the regression)"| D
    C -->|"integer/number on\npatched apiserver"| E["no warning"]
    D --> F["kubectl prints the\nserver Warning header"]

Common pitfalls

"Just update kubectl." The warning is a server response header (step 2). A newer or older CLI changes nothing. This is the single most common wrong answer in the wild.
Editing the operator's CRDs to remove format: int32. You are patching a false positive by degrading real schema validation, and your change gets clobbered on the next chart upgrade. Filter at apply time instead (step 5).
Assuming every warning in the batch is the same bug. On 1.34, kube-prometheus-stack installs also emit Warning: spec.SessionAffinity is ignored for headless services (issue #6108). That one is unrelated to formats: it flags a sessionAffinity set on a clusterIP: None Service and is a real, if harmless, config smell. Only the four numeric-format warnings are the regression.
CI gates that fail on any stderr. If your pipeline treats warning output as failure, a control-plane bump you don't control (managed cluster auto-patch) can flip a job red or green with no change on your side. Pin the filter from step 5 into the pipeline until you are on a fixed control plane, then remove it.
Chasing it as a Cilium or cert-manager bug. Both projects got issues filed against them (Cilium #41826) for warnings that were never theirs to fix. If the format is one of the four numeric ones, close it and point at the apiserver.

FAQ

Is Warning: unrecognized format "int32" breaking my CRD install?
No. It's a non-blocking warning; the CRD is still created and validated normally. The only cost is noise.

Which Kubernetes version fixes it?
kube-apiserver 1.35.0 ships the fix, and it was cherry-picked to the 1.34 patch line (PR #133896 / #133901, merged 2025-09-05). Upgrade the control plane, not kubectl.

Do I need to change my CRD YAML?
No. int32, int64, float, and double are documented, supported formats. Removing them only weakens validation.

I'm on EKS/GKE/AKS and can't pick the patch, what do I do?
Filter the four false-positive strings at apply time (step 5) and wait for your provider to roll a fixed 1.34 patch. Confirm with kubectl version -o json.

How do I tell this apart from a real format typo?
Real typos warn on type: string fields with an unknown format. The regression warns on type: integer/number fields using one of the four numeric formats. Check the field's type.

Wrap-up

Warning: unrecognized format "int32" on Kubernetes 1.34 is a kube-apiserver false positive. Prove it's server-side with -v=8, separate it from a genuine format typo and from the unrelated SessionAffinity warning, then either patch the control plane to a fixed 1.34 point release (or 1.35.0+) or filter the four numeric formats until you can. Fold it into your 1.34/1.35 upgrade checklist alongside other version-pinned gotchas, including the Ingress-NGINX to Gateway API migration, where the same trap bites: the docs still describe the pre-change behavior. The discipline that saves you on all of them is the same, check the served version, not the client.

Sources

Kubernetes issue #133880, CRD validation warning on valid formats: https://github.com/kubernetes/kubernetes/issues/133880
Kubernetes PR #133896, restrict unknown-format warning to type=string: https://github.com/kubernetes/kubernetes/pull/133896
prometheus-community/helm-charts issue #6108 to 1.34 warnings in kube-prometheus-stack: https://github.com/prometheus-community/helm-charts/issues/6108
cilium/cilium issue #41826, unrecognized format int32/int64 on helm install: https://github.com/cilium/cilium/issues/41826
Kubernetes v1.34 release announcement (2025-08-27): https://kubernetes.io/blog/2025/08/27/kubernetes-v1-34-release/

Originally published at indragustiprasetya.com