DEV Community: Indra Gusti Prasetya

pnpm readPackage Edits Leak to Deps You Never Matched

Indra Gusti Prasetya — Fri, 18 Sep 2026 02:33:45 +0000

Until pnpm 11.27.0, the resolver kept one manifest object per resolved name@version in its metadata cache and passed that object straight into your readPackage hook. The documented pnpmfile example tells you to edit it in place and return it. Following the documentation wrote into the resolver's cache, and the next dependent that resolved to the same version inherited your edit whether your condition matched it or not.

The short version: pnpm PR #14014, merged 9 September 2026 and shipped in 11.27.0, adds a shallow copy of the manifest before the hook sees it. If you run a .pnpmfile.cjs or .pnpmfile.mjs that assigns to pkg.*, upgrade, then rewrite the hook to return a new object instead of mutating the argument. The copy covers dependency and peer fields only, so edits to pkg.scripts, pkg.bin or pkg.engines are still writes into an object you do not own.

flowchart TD
  R["resolver metadata cache\none manifest object per name@version"] --> M["manifest for baz@1.0.0"]
  M -->|"passed by reference"| H1["readPackage runs for dependent A"]
  H1 -->|"pkg.dependencies.qux = 2.0.0"| M
  M -->|"same object, already edited"| H2["readPackage runs for dependent B"]
  H2 --> L["pnpm-lock.yaml records A's edit under B"]

One object, many dependents

The aliasing lives in resolveDependencies.ts. When several dependents resolve to the same package version, pnpm hands each of them the identical manifest object from its metadata cache. Your hook mutation is never scoped to the dependent being processed; it lands in cache and stays there for the rest of the install.

The fix is a helper named copyResolvedManifest, added in pnpm11/installing/deps-resolver/src/resolveDependencies.ts and applied before the manifest reaches readPackageHook. That path matters. The change landed in the v11 tree of the monorepo, so anyone who has already moved to the 12.x Rust CLI should confirm the equivalent rather than assume it carried over. The pnpm 12 migration has enough surface of its own that assumption is expensive, as the twelve traps worth checking before a pnpm 12 CI upgrade covers. The reproduction at the end of this piece answers the question on any version without reading source.

This sits underneath a practice most platform teams now depend on. readPackage and packageExtensions are how you pin a transitive dependency away from a bad release, repair a missing peer, or force a resolution upstream will not ship. Those are supply-chain controls written as JavaScript, and they were running against shared mutable state.

What the 11.27.0 release note leaves out

The release note describes the bug as a hook that "no longer changes what a later install in the same command resolves." The changeset attached to the PR is broader and more accurate: "Fixed manifest edits leaking between dependencies that resolve to the same package version."

Take the second framing seriously, because it removes the precondition the first one implies. You do not need two installs. One install with two dependents on the same resolved version is enough, which is the ordinary shape of a monorepo: dozens of dependents collapsing onto one hoisted name@version, one object in the cache. A team reading only the release note will conclude they are unaffected because they run a single pnpm install in CI.

Two symptoms that read as flaky CI

The loud symptom is a lockfile that stopped being a pure function of package.json, the registry and the pnpmfile. Resolution order became an input. Two runners that walk the graph in a different order can write different pnpm-lock.yaml files, which shows up downstream as a --frozen-lockfile failure nobody can reproduce locally, gets labelled flake, and gets a retry. It joins a small family of pnpm failures that look environmental and are not, alongside the four causes behind ERR_PNPM_PACKAGE_MANAGER_REMOVE_MODULES_DIR.

The quiet symptom is worse, and it is the one I would go looking for first. A conditional hook that inspects pkg.dependencies before deciding whether to act can read a value some other dependent already mutated, take the wrong branch, and skip the package the rule exists for. Over-application is visible in a lockfile diff. Under-application produces a control that silently no-ops on the single dependency it was written to constrain, and the review that approved that control does not run again.

There is a third artifact riding the same object. The changeset names it: a deprecated notice read from the lockfile was written into the cached manifest and reused for the next dependent, producing a deprecation warning attributed to a package that is not deprecated in that position. Chasing that one through a registry and a lockfile burns an afternoon.

Why is the copy deliberately shallow?

The PR is explicit that the copy is "deliberately shallow and limited to the fields those writes reach": dependency and peer fields, with a follow-up commit extending it to copy each peerDependenciesMeta entry. The justification is cost. The PR cites roughly 0.1µs for the shallow copy against roughly 37µs per manifest for a deep clone, which on a large tree is the difference between free and noticeable.

I think that tradeoff is correct, and it also defines your remaining exposure precisely. A hook that writes to pkg.scripts, pkg.bin, pkg.engines, or anything nested below a field outside the copied set is still mutating a shared object on 11.27.0. The version number is not the condition for safety. The absence of pkg.* assignments in your pnpmfile is.

The docs still teach the unsafe pattern

The example on pnpm.io/pnpmfile mutates in place (pkg.dependencies.baz = '1.2.3') and returns pkg. The page warns that mutations "will affect what gets resolved in the lockfile" and that you may need to delete pnpm-lock.yaml. It says nothing about aliasing. Anything trained on that page, a coding model included, will hand you the same shape, which is how the pattern spread in the first place.

The same page already documents a limit that a lot of teams have misread. Deleting pkg.scripts in readPackage does not stop pnpm building the package, because pnpm reads the real package.json out of the archive at build time. Install-script suppression belongs in pnpm's trust and build-approval settings. If your threat model for postinstall execution is a delete pkg.scripts line in a pnpmfile, that control has never fired, and the npm side of the same argument is worth reading in what breaks after npm 12 CI goes green.

Same object problem, three surfaces

Resolution is where this was found, not where it ends. Open PR #14932 reports pnpm update and pnpm audit --fix=update writing hook-injected dependencies into the on-disk package.json, because the same edits were applied both to the effective manifest used for resolution and to the original manifest written back to disk. A hook meant to be a resolution-time overlay quietly becomes a commit.

Open issue #15025 shows the same family of defect at the fetch layer: resolution keys a cache entry by tarball URL while install derives name@version, so a custom fetcher runs twice for the same package. Three different surfaces, one root pattern of identity confusion between a cached artifact and a per-dependent view of it. That pattern tends to come in clusters, which is why the audit below is worth running even if your lockfile looks clean today.

What to change in your pnpmfile this week

Upgrade to 11.27.0 or later, then stop mutating regardless. The hook contract is the return value, not the argument. Returning pkg untouched is fine because no write happens. Everything else gets a spread:

export const hooks = {
  readPackage (pkg) {
    if (!pkg.dependencies?.baz) return pkg
    return { ...pkg, dependencies: { ...pkg.dependencies, baz: '1.2.3' } }
  }
}

Find the writes you already have. Name them rather than eyeballing the file:

grep -nE 'pkg\.[A-Za-z]+(\[[^]]*\])?\s*=|delete pkg\.' .pnpmfile.cjs .pnpmfile.mjs 2>/dev/null

Any hit outside dependency and peer fields is still unprotected on 11.27.0.

Separate an aliasing leak from an ordinary resolution change with a two-run diff:

cp pnpm-lock.yaml /tmp/lock.base
pnpm install --lockfile-only --ignore-pnpmfile && cp pnpm-lock.yaml /tmp/lock.nohook
git checkout pnpm-lock.yaml
pnpm install --lockfile-only && diff -u /tmp/lock.base pnpm-lock.yaml

A diff that appears only with the hook enabled, on a package your condition does not match, is the leak. A diff that survives --ignore-pnpmfile is a normal resolution change and needs no pnpmfile work. For the deprecation ghost, re-run against a throwaway store with pnpm install --store-dir "$(mktemp -d)" and see whether the warning survives a cache holding no lockfile-sourced notice.

Gate it in CI instead of in review. pnpm install --frozen-lockfile followed by git diff --exit-code pnpm-lock.yaml converts order-dependent resolution into a failing job with a name, rather than an intermittent install error three steps later. Check git diff --exit-code package.json in the same step while #14932 is open.
Retire the extra job on a condition, not a date. Drop it when the grep in step 2 returns nothing, meaning your pnpmfile performs no assignment to pkg.* at all. As of 18 September 2026, upgrading alone does not get you there.

FAQ

Is returning pkg unchanged from readPackage safe?
Yes. The hazard is the write, not the return. A hook that inspects the manifest and returns the same object without assigning to it never touches the resolver's cached copy, on any version.

Does pnpm 11.27.0 cover edits to pkg.scripts?
No. PR #14014 states the copy is deliberately shallow and limited to the fields the known writes reach, meaning dependency and peer fields plus each peerDependenciesMeta entry. Edits to scripts, bin, engines or anything nested under an uncopied field still mutate a shared object.

Why does my pnpm lockfile differ between CI runners?
If you run a pnpmfile that mutates manifests on pnpm older than 11.27.0, resolution order is an input to the result, so two runners can legitimately produce different lockfiles from identical inputs. Use the --ignore-pnpmfile comparison above to confirm before blaming the runner image.

Can readPackage block a package's install scripts?
No. pnpm reads the real package.json from the package archive at build time, so deleting pkg.scripts in the hook changes resolution metadata and nothing about what runs. Use pnpm's trust and build-approval settings for that, per pnpm.io/pnpmfile.

Originally published at indragustiprasetya.com

Fix Terraform 1.16.3 binary/octet-stream: the zip is fine

Indra Gusti Prasetya — Fri, 18 Sep 2026 02:12:16 +0000

Terraform 1.16.3 shipped on 16 September 2026, and within hours every CI job that fetches the CLI through HashiCorp's own Go installer started refusing it. This takes you from the error string to a running plan in about ten minutes, and leaves you with a one-line test that tells you when the workaround can come out.

TL;DR: The archive is not corrupt. releases.hashicorp.com serves terraform_1.16.3_*.zip with content-type: binary/octet-stream, and hc-install only accepts application/zip or application/x-zip-compressed, so it aborts before it unpacks anything. curl, tfenv and tfswitch never read that header, so the same URL works by hand. Pin != 1.16.3 or hand your consumer a binary it already has, then watch the header for the server-side fix.

The failure looks like this, from Atlantis, from provider acceptance tests, from anything embedding hashicorp/hc-install:

error downloading terraform version 1.16.3: unexpected content-type: binary/octet-stream (expected any of ["application/x-zip-compressed" "application/zip"])

Three different problems print almost that same sentence, and the open threads run them together: a release-side metadata regression from last week, a 2022-era check at a completely different stage of the install, and a corporate proxy rewriting the header in transit. hashicorp/terraform#39235 and hashicorp/hc-install#407 were both filed on 16 September 2026 and, at the time of writing, neither carries a maintainer fix. Any model answering from training data older than that will call it a corrupt download or a bad mirror and send you to clear caches. One warning summary with several unrelated triggers is the same diagnostic trap as Terraform 1.15's deprecation-mark warnings: you cannot pick a fix until you know which trigger you have.

Prerequisites

curl 7.x or later with outbound access to releases.hashicorp.com, plus sha256sum and unzip.
One affected consumer to reproduce against: Atlantis v0.44.0 or similar, a provider repo whose acceptance tests use terraform-plugin-testing, or a Go program importing hc-install.
A known-good binary to fall back to. 1.16.2 is unaffected, and a patch-level rollback is cheap here, unlike the version-specific state behaviour behind the deposed-object panic in readDiff.
Shell access on the host where the download actually runs. Reproducing on a laptop will not reproduce the proxy variant, which is the point of step 2.

What the three threads actually say: #39235 is the Terraform-side report with the confirmed-good checksum and the != 1.16.3 workaround; hc-install#407 argues the library should accept the generic type because checksum and signature verification happen after the download anyway; runatlantis/atlantis#6906 is the same break surfacing as failed plans on pull requests.

Step-by-step

1. Read which stage of the install failed

hc-install validates content types in two separate places and the messages differ by one substring. A message naming application/vnd+hashicorp.releases-api.v0+json comes from the releases-index stage in releasesjson/releases.go, which is the 2022 problem from hc-install issues #56 and #59, relaxed by PR #57. A message naming binary/octet-stream comes from the archive stage in internal/releasesjson/downloader.go, inside DownloadAndUnpack(), which compares the response header against a two-entry list:

var zipMimeTypes = []string{
    "application/x-zip-compressed", // Windows
    "application/zip",              // Unix
}

Only the second check is live today. Sorting this out first keeps you from applying a four-year-old library upgrade to a two-day-old server-side break.

2. Ask the origin what it is serving, and compare two versions

curl -sSIL https://releases.hashicorp.com/terraform/1.16.3/terraform_1.16.3_linux_amd64.zip | grep -i '^content-type'
curl -sSIL https://releases.hashicorp.com/terraform/1.16.2/terraform_1.16.2_linux_amd64.zip | grep -i '^content-type'

Expect content-type: binary/octet-stream on the first line and content-type: application/zip on the second. The 1.16.3 object did not get the per-object metadata that every previous release got. Run this from the failing host rather than your workstation: a header rewritten by an intercepting proxy or an Artifactory remote repository only shows up on the path CI takes. If 1.16.3 already returns application/zip for you, HashiCorp has re-tagged the object and whatever you are still seeing is cached or proxy-side.

3. Prove the bytes are good before you touch anything else

curl -sSLO https://releases.hashicorp.com/terraform/1.16.3/terraform_1.16.3_linux_amd64.zip
curl -sSL https://releases.hashicorp.com/terraform/1.16.3/terraform_1.16.3_SHA256SUMS \
  | grep linux_amd64 | sha256sum -c -

Expected output: terraform_1.16.3_linux_amd64.zip: OK. The reporter on #39235 did the same check against terraform_1.16.3_darwin_arm64.zip and got a valid zip with a matching SHA256. Two things follow. You can stop clearing caches, and you now have the argument that makes a correct fix stick: hc-install verifies checksum and signature after the download, so the header it rejected was never the integrity control. Trusting an artifact by its checksum instead of by transport metadata is the same discipline that makes SHA-pinned GitHub Actions worth the friction.

4. Apply the fix that matches your consumer

Atlantis resolves a Terraform version from PATH and its own binary directory before it reaches out to releases.hashicorp.com, and per the server configuration reference --tf-download=false stops the outbound attempt entirely:

atlantis server --tf-download=false --default-tf-version=v1.16.2

Provider acceptance tests take the binary from an environment variable instead. HashiCorp's acceptance testing docs describe TF_ACC_TERRAFORM_PATH as the path to an existing binary; leaving both it and TF_ACC_TERRAFORM_VERSION unset is what makes the harness install the latest CLI into a temp directory on every run:

export TF_ACC_TERRAFORM_PATH="$(command -v terraform)"
TF_ACC=1 go test ./internal/provider/... -v

One trap is documented in that same page and costs people an afternoon: if TF_ACC_TERRAFORM_PATH points at something missing or non-executable and TF_ACC_TERRAFORM_VERSION is set, the harness installs anyway and the error comes straight back. Set the path, then confirm the binary is there.

For a Go program of your own, change the install source so it locates a binary rather than fetching one:

execPath, err := install.NewInstaller().Ensure(ctx, []src.Source{
    &fs.AnyVersion{Product: &product.Terraform}, // was: &releases.LatestVersion{Product: product.Terraform}
})

Ensure() walks the slice and falls through to the next source, so leaving the releases entry in the list puts the download straight back on the critical path.

5. Pin around the release where a constraint drives the download

terraform {
  required_version = ">= 1.14.0, != 1.16.3"
}

This is the workaround posted on #39235 and it resolves to 1.16.2. Be clear with your team about what it does not cover: it constrains version selection, so it helps tooling that reads the constraint and does nothing for a pipeline with 1.16.3 hardcoded in a Dockerfile, a setup-terraform step, or a build matrix.

6. Keep the decision tree somewhere the on-call can find it

flowchart TD
  A["error downloading terraform version 1.16.3"] --> B{"Which content-type\nis named?"}
  B -->|releases-api v0 json| C["Index stage\nUpgrade hc-install past PR #57"]
  B -->|binary/octet-stream| D{"curl -sSIL the same URL\nfrom the failing host"}
  D -->|binary/octet-stream| E["Release-side metadata\nPre-install or pin != 1.16.3"]
  D -->|application/zip| F["A proxy rewrote the header\nFix the artifact proxy"]

Verify it works

Re-run whatever failed, and check the binary rather than the exit code:

terraform version
# Terraform v1.16.2
# on linux_amd64

For Atlantis, a re-planned pull request is the proof. For acceptance tests, TF_LOG=info go test ... shows which path the harness chose, which is how you catch the TF_ACC_TERRAFORM_PATH trap from step 4.

Then leave the durable check behind, because the real fix lands server-side and nobody will announce it to you. The header test from step 2 is your tripwire:

curl -sSIL https://releases.hashicorp.com/terraform/1.16.3/terraform_1.16.3_linux_amd64.zip \
  | grep -qi 'content-type: application/zip' && echo "re-tagged, drop the pin"

Put that in the same cron or CI job that already checks for stale pins, with a link to #39235 in the comment.

Common pitfalls

Clearing caches and re-pulling. Step 3 already proved the bytes are correct. Nothing local is wrong, and every minute spent on Docker layers or ~/.terraform.d is wasted.
Blaming tfenv, mise, or the laptop. Those downloaders do not inspect the content type, so a clean local install is perfectly consistent with a hard CI outage. It is not a signal either way.
Forking hc-install to skip the MIME check and disabling VerifyChecksum in the same commit. Relaxing the allowlist is defensible precisely because signature and checksum verification run after the download. Turning off the checksum removes the control that made the argument work.
Expecting a provider mirror to cover it. filesystem_mirror and network_mirror apply to providers. The CLI binary is fetched by a different mechanism and mirror configuration has no effect on it.
A warm GitHub Actions tool cache hiding the break until a cold runner picks up a job, which turns a deterministic failure into an intermittent one. That class of late-surfacing breakage is worth reading up on separately in what breaks after CI goes green.
Over-extending the downgrade reasoning. 1.16.3 to 1.16.2 is patch-level and state stays readable. Do not reuse that logic for a minor downgrade after a state format upgrade.

FAQ

Is the Terraform 1.16.3 download corrupt?
No. The reporter on hashicorp/terraform#39235 verified the downloaded zip against the published SHA256SUMS and it matched. The rejection happens on the HTTP response header, before any unpacking or verification.

Why does curl or tfenv work while Atlantis fails on the same URL?
Only hc-install compares the response content-type against an allowlist. Generic downloaders write the bytes to disk regardless of what the header says.

Will upgrading hc-install fix it?
Only if your error names application/vnd+hashicorp.releases-api.v0+json, which is the index-stage check relaxed back in 2022 by PR #57. The binary/octet-stream message comes from a separate check in DownloadAndUnpack() that, as of hc-install#407, has not been changed.

How do I know when to remove the pin?
Run the curl -sSIL tripwire from the Verify section. When 1.16.3 answers with application/zip, the object has been re-tagged and the constraint can go.

Wrap-up

You end with a plan that runs, a one-line test that separates the release-side regression from a proxy rewriting headers, and something that tells you when the workaround expires. If you are doing this across more than a handful of pipelines, the lasting fix is to move CLI acquisition behind an internal artifact store with pinned checksums, so a vendor's object metadata never sits on the critical path to a plan again.

Originally published at indragustiprasetya.com

Fix ERR_PNPM_PACKAGE_MANAGER_REMOVE_MODULES_DIR: 4 causes

Indra Gusti Prasetya — Fri, 18 Sep 2026 02:07:04 +0000

On Windows, pnpm 12.3.4 through 12.4.1 kills pnpm install with ERR_PNPM_PACKAGE_MANAGER_REMOVE_MODULES_DIR Failed to remove modules directory contents: Access is denied. (os error 5) while clearing node_modules. By the end of this you will know which of four causes produced that string on your machine, the upgrade that fixes the common one, and the unlink sequence for a host you cannot upgrade today.

Overview

The short version: Windows reports a junction as a symlink rather than a directory, so pnpm's Rust purge path took the file branch and called DeleteFileW, which refuses directory links and returns ERROR_ACCESS_DENIED. Upgrade to pnpm 12.4.2, released 15 September 2026. Everything after step 4 is for readers stuck on an older binary, or readers who upgraded and still see the string, because three other conditions print it.

Two details make this worth writing down rather than guessing at. The break and the fix both landed inside a ten-day window in September 2026, and the reason a junction escapes the directory branch is visible only in the merged patch, not in any published doc. If you have been working through the pnpm 12 upgrade traps, treat this as one more Rust-CLI regression in that family. The shape also rhymes with uv refusing to install into a symlinked directory: a package manager rewritten in Rust, a link type the old implementation tolerated, a new one that does not.

flowchart TD
  A["pnpm install decides to purge node_modules"] --> B{"Which string printed?"}
  B -->|"ABORTED_REMOVE_MODULES_DIR_NO_TTY"| C["No terminal to confirm the purge\nSet CI=true or confirmModulesPurge=false"]
  B -->|"REMOVE_MODULES_DIR, os error 5"| D{"Windows host?"}
  D -->|"No"| E["Real permission denial\nRead the path named in the error"]
  D -->|"Yes"| F{"pnpm 12.3.4 to 12.4.1?"}
  F -->|"Yes"| G["Junction routed to DeleteFileW\nUpgrade to 12.4.2"]
  F -->|"No"| H["Entry held open by AV, indexer or editor\n12.4.2 retries, then reports the path"]
  A --> I{"Why a purge at all?"}
  I -->|"nodeLinker or virtualStoreDir moved"| J["Diff node_modules/.modules.yaml"]
  I -->|"node_modules is itself a symlink"| K["Issue 9973, still open"]

Prerequisites

Windows 10 or 11, or a windows-latest GitHub Actions runner. Linux and macOS never hit the junction path, since remove_file unlinks a symlink there without complaint.
pnpm 12.x, checked with pnpm --version. Affected releases run from v12.3.4 (4 September 2026) to v12.4.1 (10 September 2026). The reporter in pnpm issue #14790 confirmed 12.3.4 and 12.4.0.
Node.js 20 or newer, and a shell from which you can run PowerShell and cmd.exe.
Write access to package.json for the packageManager pin, and to your continuous integration (CI) workflow file.

One assumption I am making: the purge itself is legitimate. pnpm clears node_modules when the settings that shaped the existing tree no longer match the settings for the install about to run, and step 2 is how you confirm that rather than assume it.

Step-by-step

1. Read the string before you touch anything

pnpm install 2>&1 | tail -20

Two error codes carry the words "modules dir" and they want opposite fixes. ERR_PNPM_PACKAGE_MANAGER_REMOVE_MODULES_DIR means pnpm tried to purge and the filesystem refused. ERR_PNPM_ABORTED_REMOVE_MODULES_DIR_NO_TTY means pnpm never tried, because there was no terminal attached to confirm on, which is the case described in issue #11562. Step 6 covers that second one; the rest of this page is about the first.

2. Find out why pnpm wants to purge

cat node_modules/.modules.yaml | grep -E "nodeLinker|virtualStoreDir|hoistPattern|publicHoistPattern"

pnpm writes the settings your current tree was built with into node_modules/.modules.yaml and compares them against the settings for this run. A changed nodeLinker is the usual trigger, and it is the exact reproduction in issue #14790: add a pnpm-workspace.yaml containing nodeLinker: hoisted after a default install. The default is isolated, per the node-modules settings reference. A drifting virtualStoreDir is the other one. Issue #12307 shows .modules.yaml written twice with conflicting values when enableGlobalVirtualStore: true and the install runs from inside a workspace package, which re-purges on every install even though nothing changed.

3. List the directory links pnpm is choking on

Get-ChildItem -Force node_modules |
  Where-Object { $_.LinkType } |
  Select-Object Name, LinkType, Target

Every row with a LinkType of Junction or SymbolicLink is a candidate. From cmd.exe, dir /A:L node_modules gives the same picture, and fsutil reparsepoint query node_modules\is-odd confirms one specific entry is a reparse point. An isolated install creates one such link per direct dependency, so a plain pnpm install on a project with a single dependency is enough to reproduce the failure.

4. Upgrade to 12.4.2, then pin it

pnpm self-update
pnpm --version   # expect 12.4.2 or newer

PR #14798, merged 14 September 2026, replaces the std::fs::remove_file call in crates/package-manager/src/install/prepare_modules_state/purge.rs with pnpm_fs::remove_dirent, which routes directory links through remove_symlink_dir and retries transient locks. The v12.4.2 release notes state it plainly: pnpm install on Windows no longer fails with this code when clearing a node_modules that contains linked dependencies, "such as when changing nodeLinker."

Pin the version so a stale global binary on one laptop or one runner does not walk you back into it:

{
  "packageManager": "pnpm@12.4.2"
}

5. If you cannot upgrade today, unlink before you install

for /d %d in (node_modules\*) do rmdir "%d" 2>nul
rmdir /s /q node_modules

rmdir against a junction removes the link and leaves the target alone, which is the call the old code path skipped. Run it from cmd.exe, not from Windows PowerShell 5.1, for the reason in the pitfalls below. Then run pnpm install normally.

6. Make CI answer the purge question instead of aborting

- run: pnpm install --config.confirmModulesPurge=false
  env:
    CI: "true"

confirmModulesPurge appears in neither the settings reference nor the pnpm install CLI page; the error text is its documentation, and issue #6778 is where the --config. prefix form is confirmed. Setting CI=true has the same effect. Apply this only to the NO_TTY variant, since it does nothing at all for os error 5.

Verify it works

Reproduce the original trigger deliberately in a throwaway directory:

mkdir pnpm-purge-check && cd pnpm-purge-check
pnpm init
pnpm add is-odd@3.0.1
printf 'nodeLinker: hoisted\n' > pnpm-workspace.yaml
pnpm install

On 12.3.4 through 12.4.1 that last pnpm install exits non-zero with Failed to remove modules directory contents: Access is denied. (os error 5). On 12.4.2 it completes on the first attempt, and the word "first" is load-bearing: the broken versions often succeed on a second run, because the failed pass removed enough of the tree to change what the next pass encounters.

Then confirm the tree really flipped linker:

Select-String -Path node_modules\.modules.yaml -Pattern "nodeLinker"
Get-ChildItem -Force node_modules | Where-Object { $_.LinkType } | Measure-Object

A hoisted tree reports a nodeLinker: hoisted line and a link count at or near zero.

Common pitfalls

Filing it as flaky CI. The retry-succeeds pattern looks exactly like a race, and the usual response is a retry: 2 on the job. The failure is a deterministic type check, and the retry buys silence until the first pass fails somewhere less convenient, such as a release build.

Remove-Item -Recurse in Windows PowerShell 5.1. It has a long history of walking through a junction instead of deleting it, which turns a cleanup step into deleting the contents of your pnpm content-addressable store. Use cmd /c rmdir, or PowerShell 7.

Reaching for confirmModulesPurge=false first. It suppresses the confirmation prompt, and the removal still runs. On the Windows bug the purge proceeds and fails anyway; on a repo whose node_modules should never have been purged, it deletes the tree without asking.

Assuming 12.4.2 ends every occurrence. A genuinely locked entry produces the same code: an antivirus (AV) scan, the Windows Search indexer, a node.exe still running, an editor holding a file open. On 12.4.2 the message is worth reading, because PR #14605, merged 8 September 2026, changed the output from a bare OS error to Failed to remove C:\...\node_modules\is-odd from the modules directory, naming the entry. Open the path, close whatever holds it.

A symlinked node_modules on a deploy host. Capistrano-style layouts that point node_modules at a shared cache take a full purge on every deploy. Issue #9973 has been open since September 2025 with no maintainer answer, so treat a symlinked node_modules as unsupported until one arrives.

Wrap-up

You have the version boundary (12.3.4 to 12.4.1 broken, 12.4.2 fixed), the mechanism (junction reported as a symlink, routed to DeleteFileW, refused), a PowerShell one-liner that lists the offending links, and a separating test for each of the three causes an upgrade will not touch.

The follow-up worth doing this week: pin packageManager in every repo and put that pinned version in your Windows runner matrix, so the next Rust-CLI regression surfaces in one job instead of on one engineer's laptop at 5pm. The pnpm deploy symlink escape on 11.19 is the same family of bug one layer up. For the npm-side version of one crash line covering several broken trees, see the edgesOut crash and the TypeScript 7 ESLint break.

Originally published at indragustiprasetya.com

Fix uv "Cannot install into symlinked directory" in 0.12.14

Indra Gusti Prasetya — Wed, 16 Sep 2026 02:08:14 +0000

If a Docker build or CI job that passed yesterday now dies on The wheel is invalid: Cannot install into symlinked directory, nothing is wrong with your wheel, image, or virtual environment. uv 0.12.14 (released 2026-09-15) started refusing to install a wheel whose destination tree contains a directory symlink, and uv 0.12.15 shipped the same day to revert it. Bump the version, then spend ten minutes finding the dependency that genuinely writes through a symlink, because the check it tripped is coming back.

The change that breaks you is not in the 0.12.14 changelog. Those notes cover resumable downloads, MAX_PATH support on Windows, and a new exit-code scheme; the word "symlink" does not appear. The behaviour arrived with pull request #21569, "Reject symlinked wheel installation destinations", merged 2026-09-10 and unmentioned in the release. An assistant answering from the changelog will tell you your environment is broken, which is the same failure mode that makes version-gated breakage in other toolchains so expensive to diagnose, catalogued for the JavaScript side in the pnpm 12 upgrade traps.

Prerequisites

uv 0.12.13, 0.12.14, or 0.12.15, and write access to wherever your pipeline decides the uv version. Check with uv --version.
A Linux host or container to reproduce on. The Docker surface needs Docker plus a Debian-based official image (python:3.11-slim-bookworm, python:3.13-bookworm, python:3.12-slim-trixie and siblings all carry the trigger). -alpine images do not.
Knowledge of where the version is actually pinned: a COPY --from=ghcr.io/astral-sh/uv:<tag> line, an astral-sh/setup-uv step, a Renovate-managed tool version, or a base image with uv baked in. An unpinned tool fetched at build time is the same exposure class covered in the gaps SHA-pinning GitHub Actions leaves open.

What I read while writing this: the 0.12.15 release notes, the 0.12.14 release notes, the Docker reproduction in astral-sh/uv#21692, the --target . report in astral-sh/uv#21694, and the reverted PR itself.

Step-by-step

1. Read the path after the colon, not the wheel name

The wheel named in the first line is a red herring. It is whichever wheel in the install set happened to carry a .data/ payload first. The diagnosis is the path:

error: Failed to install: clevercsv-0.8.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (clevercsv==0.8.4)
  cause: The wheel is invalid: Cannot install into symlinked directory: /usr/local/man

  cause: The wheel is invalid: Cannot install into symlinked directory: .../packages/robocar/simulation/.venv/lib64

The first is a system install into a Debian python:* image, where /usr/local/man is a symlink to share/man. The second was reported on a GitHub Actions runner during uv sync --extra dev, where uv venv had created lib64 as a symlink to lib on 64-bit Linux. A third surface prints a completely different string, which is why people miss it.

flowchart TD
  A["error: The wheel is invalid"] --> B{"Which cause string?"}
  B -->|Cannot install into symlinked directory| C{"Path after the colon"}
  B -->|Wheel directory entry escapes its destination| D["--target=. or --target=./\nrelative root normalizes to an empty path"]
  C -->|/usr/local/man| E["uv pip install --system\ninto a Debian python:* image"]
  C -->|.venv/lib64| F["install into a venv on 64-bit Linux\nlib64 -> lib"]
  D --> G["Upgrade to uv 0.12.15"]
  E --> G
  F --> G
  G --> H["Then audit the wheel that\nactually crosses the symlink"]

2. Confirm the symlink exists so you stop suspecting the wheel

find /usr/local -maxdepth 1 -type l -printf '%p -> %l\n'
find .venv -maxdepth 1 -type l -printf '%p -> %l\n'

In python:3.11-slim-bookworm the first prints /usr/local/man -> share/man. In a uv-created environment on x86-64 the second prints .venv/lib64 -> lib. Both layouts are normal, both predate uv by years, and neither is something you introduced. Confirming this first stops the two hours people otherwise spend rebuilding wheels.

3. Reproduce it in three lines before changing anything

FROM python:3.11-slim-bookworm
COPY --from=ghcr.io/astral-sh/uv:0.12.14 /uv /bin/uv
RUN uv pip install --system clevercsv==0.8.4

docker build on that fails with the /usr/local/man cause. It is the reproduction filed as astral-sh/uv#21692 on 2026-09-15. clevercsv is incidental: it ships man pages in its .data/data/ tree, so substitute any wheel that does. Having the failure on demand is what lets you prove the bump worked rather than hoping.

4. Pin 0.12.15 where the version actually comes from

COPY --from=ghcr.io/astral-sh/uv:0.12.15 /uv /uvx /bin/

- uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
  with:
    version: "0.12.15"

The 0.12.15 notes are explicit: "This release fixes a regression in 0.12.14 that lead to rejecting valid installation commands such as using uv pip install --system in python:* docker images or when using uv pip install --target .", with the single bug-fix entry "Revert 'Reject symlinked wheel installation destinations'".

For a standalone-installer uv, uv self update is enough. One trap on the Actions side: setup-uv's version input searches your config files before falling back to the latest release, so if a committed uv version constraint is what resolved 0.12.14, editing the workflow alone will not move it. Change the constraint, not the step.

5. If you cannot bump today, patch the destination

Only worth doing for a hard-pinned 0.12.14 you cannot reach. For the Docker surface, replace the symlink with a real directory before installing:

RUN rm -f /usr/local/man && mkdir -p /usr/local/man

For the virtual-environment surface, pin the offending package away or commit the uv.lock that never selected it. Rewriting .venv/lib64 is riskier, because the interpreter's own layout expects that symlink to be there.

6. Find the wheel that really crosses the symlink

This is the step that survives the revert. Pull the wheel and list its data tree:

pip download --no-deps --dest /tmp/wheels clevercsv==0.8.4
unzip -l /tmp/wheels/clevercsv-*.whl | grep '\.data/'

Entries under .data/data/, .data/scripts/, or .data/platlib/ are the ones mapped onto scheme roots at install time, so they are the ones that can land inside a symlinked directory. Write down which of your dependencies have them. That list is your blast radius when the hardened check returns, and it is worth keeping next to whatever inventory you already maintain of packages that execute code at install time, the hazard that the arrayref attack showed for cargo build.rs.

Verify it works

uv --version
uv pip install --system --no-cache clevercsv==0.8.4

Expect uv 0.12.15 and a successful Installed 1 package line. Rebuild the step 3 Dockerfile with the 0.12.15 tag and it completes. Then check the --target surface separately:

uv pip install --python python3.14 --system --no-compile --no-cache --target=. annotated-types==0.8.0

On 0.12.14 that failed for every wheel with The wheel is invalid: Wheel directory entry escapes its destination: annotated_types-0.8.0.dist-info, because the relative installation root . normalizes to an empty path (astral-sh/uv#21694). Same pull request, different message, and no search engine currently connects the two strings. On 0.12.15 it installs.

Common pitfalls

Deleting and recreating the virtual environment does nothing. uv venv recreates lib64 -> lib, because that is the normal layout on 64-bit Linux rather than corruption. Advice that opens with "remove .venv" is aimed at the one thing you cannot remove.

Your exit codes changed in the same release. The 0.12.14 notes state that "Package-operation exit codes now reflect the underlying cause: expected failures return 1, while recognized operational and internal failures return 2." A wrapper script branching on $? -eq 1 to mean "resolution failed" will now take the wrong branch on an operational failure, silently. Grep your CI for literal exit-code comparisons around uv while you are already in there.

A cached Docker layer hides the fix. If the COPY --from=ghcr.io/astral-sh/uv:0.12.14 line sits above your dependency install and you only edited the tag, rebuild that stage with --no-cache. Layer ordering makes image debugging misleading in general, which I went into alongside the containerd store in docker save "no suitable export target found".

Alpine passing convinces people the bug is not real. python:*-alpine has no /usr/local/man symlink, so a matrix that only exercises Alpine stays green while every Debian job fails.

Treat the rejection as returning. PR #21569 was reverted for the relative-path and system-directory regressions, not because its reasoning was wrong: when an installation destination already contains a directory symlink, merging the wheel payload can write files outside the environment. Pinning 0.12.15 buys time, which is why step 6 is the actual work.

Wrap-up

You have the mapping from error path to surface, a three-line reproduction, a one-line bump to 0.12.15, per-surface workarounds for a pin you cannot move, and a list of dependencies whose .data/ trees make them candidates when the check comes back hardened.

The next thing worth doing is to stop discovering uv releases through red CI. Add a scheduled matrix row that runs your install against the newest uv with --no-cache on a Debian-based image. The window between the 0.12.14 regression and the 0.12.15 fix was hours, and the only teams who never noticed were the ones whose pinned version does not move on its own: https://github.com/astral-sh/uv/releases/tag/0.12.15

Originally published at indragustiprasetya.com

Fix "NVIDIA-SMI has failed" in a Kubernetes DRA Pod

Indra Gusti Prasetya — Tue, 15 Sep 2026 04:58:09 +0000

You end this with a GPU visible inside a container scheduled through Dynamic Resource Allocation (DRA), a one-line check that tells this failure apart from every "your driver is broken" answer on the internet, and a systemd unit that stops the node re-arming the bug at the next reboot.

Short version: if nvidia-smi works on the host but the pod prints NVIDIA-SMI has failed, grep the claim's Container Device Interface (CDI) spec under /var/run/cdi for /dev/nvidia. A count of zero means the kubelet plugin generated and cached a spec with the NVIDIA libraries and no device nodes. Run nvidia-modprobe -c 0 -u on the host, then delete the pod and claim and restart the plugin DaemonSet so the spec is regenerated.

Why nvidia-smi works on the host and dies in the pod

The failure looks like this inside the container, quoted from the report in kubernetes-sigs/dra-driver-nvidia-gpu issue #1380:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

Everything around it reports success. The ResourceClaim says allocated,reserved, the pod reaches Running, the kernel module is loaded, and nvidia-smi on the node lists the cards. Nothing crashed, and the GPU is not in the container.

That string has a well-known set of causes on a normal host: a failed DKMS rebuild, Secure Boot blocking the module, a userspace/kernel version mismatch, a card that fell off the PCIe bus. None of them apply here, because the host driver is fine. What broke is the CDI spec the kubelet plugin wrote for this specific claim: it carries the NVIDIA userspace libraries and no /dev/nvidia* device nodes, so the NVIDIA Management Library (NVML) inside the container finds the tooling on $PATH and no hardware to talk to. Same error text, different layer, and the split is the same shape as Xid 154 routing four different remediations through one log line: the symptom is a pointer, not a diagnosis. If the node also does GPU passthrough, settle the VFIO and DRA overlap before you start, because a card bound to vfio-pci fails earlier and differently.

flowchart TD
  A["Node boots, nvidia kernel module loads"] --> B["Pod scheduled, kubelet plugin runs NodePrepareResources"]
  B --> C["Plugin generates a CDI spec for the claim"]
  C --> D{"Do /dev/nvidia* exist yet?"}
  D -->|yes| E["Spec lists deviceNodes, container sees the GPU"]
  D -->|no| F["Spec has library mounts only, zero deviceNodes"]
  F --> G["Spec cached under /var/run/cdi plus checkpoint.json"]
  G --> H["Pod reaches Running, nvidia-smi exits 9"]

What you need before you start

Kubernetes 1.34 or newer, where core DRA is generally available and resource.k8s.io/v1 is on by default (Kubernetes v1.34: DRA graduates to GA). Issue #1380 was reproduced on v1.36.2+k3s1. If you are still landing on the 1.34 line, the unrecognized format int32 validation change came in the same wave.
dra-driver-nvidia-gpu v0.5.0, installed per the project install docs:

  helm install dra-driver-nvidia-gpu \
    oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
    --version 0.5.0 \
    --create-namespace \
    --namespace dra-driver-nvidia-gpu \
    --set gpuResourcesEnabledOverride=true

A root shell on the GPU node. Several steps read /var/run/cdi and /var/lib/kubelet, which the API server cannot show you.
An NVIDIA kernel driver on the host. The issue used 580.105.08 on RTX PRO 2000 and RTX PRO 4000 Blackwell cards; nothing about the mechanism is card-specific.
kubectl able to read resourceclaims and delete pods in the workload namespace.

One version check first. As of 15 September 2026, issue #1380 is open against v0.5.0 and carries the v0.5.1 milestone, which has not shipped. Look at the releases page before you do any of this: if v0.5.1 is out by the time you read it, upgrade first and use the recovery below only for nodes already stuck.

Read the claim's CDI spec, not the pod logs

1. Confirm the claim really was allocated. Rule out the boring case, a pod that never got a claim.

kubectl get resourceclaims -n "$NS"
kubectl get resourceclaim -n "$NS" "$CLAIM" \
  -o jsonpath='{.status.allocation.devices.results}' | jq .

Empty status or a Pending pod is a scheduling problem and stops here. allocated,reserved plus a Running pod is the signature you are chasing.

2. Ask the host whether the device files exist.

ls -l /dev/nvidia*

On a fresh headless node that has never run an NVIDIA userspace client, this returns No such file or directory, which is the core observation in issue #1380. The module is loaded and the character devices were never created. nvidia-modprobe exists for exactly this reason: creating the device files is normally a distro or udev job and is not guaranteed to have happened, which is why the tool is installed setuid root to do it on demand (nvidia-modprobe(1)).

3. Count the device nodes in the spec. This is the command that splits the two failure modes.

CLAIM_UID=$(kubectl get resourceclaim -n "$NS" "$CLAIM" -o jsonpath='{.metadata.uid}')
grep -c '/dev/nvidia' "/var/run/cdi/k8s.gpu.nvidia.com-claim_${CLAIM_UID}.yaml"

Name the variable something other than UID: bash marks UID readonly, and the assignment fails in an interactive root shell before you ever reach the grep. A healthy spec lists /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm and /dev/nvidia-uvm-tools under containerEdits.deviceNodes, the structure NVIDIA documents for CDI-generated specs (NVIDIA CDI support). A count of 0 while the library mounts are still present is your answer: the container received the userspace stack and no hardware, so nvidia-smi exits 9, the NVML code for driver-not-loaded.

Three ways a claim ends up with no device nodes

The node has never had them. Headless machine, no X server, no container that ran nvidia-smi, no udev rule doing the work at boot. The plugin prepares a claim against a /dev that has no NVIDIA entries in it.
The node rebooted. The device files are not persistent across boot on a headless host, so a node that worked yesterday can come back with the same empty /dev and re-arm the bug on the next pod.
Something created them afterwards. Once the plugin has prepared a claim, the spec is cached, so device files that appear later do not reach that claim. This is why the bug is a race and not a static misconfiguration, and why the timing looks random across a node pool.

Recover the stuck pod, in this order

4. Create the device nodes on the host.

nvidia-modprobe -c 0 -u
ls -l /dev/nvidia*

-c 0 creates the device file for minor number 0 and -u loads nvidia-uvm and creates its node. Running nvidia-smi -L has the same side effect, which is how "it started working after I SSH'd in to check" becomes such a misleading bug report on this failure.

5. Invalidate the cached spec. Step 4 on its own does not fix the running pod, because a prepared claim is tracked in two places the host has no idea about: the per-claim spec under /var/run/cdi, and a PreparedClaims entry in checkpoint.json under /var/lib/kubelet/plugins/gpu.nvidia.com/.

kubectl delete pod -n "$NS" "$POD"
kubectl delete resourceclaim -n "$NS" "$CLAIM"   # skip if it came from a ResourceClaimTemplate
kubectl rollout restart daemonset/dra-driver-nvidia-gpu-kubelet-plugin \
  -n dra-driver-nvidia-gpu
kubectl rollout status daemonset/dra-driver-nvidia-gpu-kubelet-plugin \
  -n dra-driver-nvidia-gpu

Delete the workload first so NodeUnprepareResources runs and removes the stale spec instead of orphaning it.

6. Recreate the workload and repeat step 3 against the new claim UID. The grep count should now be non-zero.

Make the node create its device nodes before kubelet starts

7. The durable fix is ordering, since the bug is a race. A oneshot unit that runs ahead of kubelet:

# /etc/systemd/system/nvidia-device-nodes.service
[Unit]
Description=Create NVIDIA character device nodes before kubelet starts
After=systemd-modules-load.service
Before=kubelet.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-modprobe -c 0 -u
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now nvidia-device-nodes.service

On multi-GPU nodes, repeat -c per minor number. Put this in the node image or the bootstrap script rather than a runbook, because a fix that depends on someone reading a runbook after a reboot is not a fix. I would not assume nvidia-persistenced covers this for you either: its own startup expects the device files to exist, so verify against the driver version you actually run before you drop the unit.

Verify the container sees the GPU

kubectl exec -n "$NS" "$POD" -- nvidia-smi -L

Expect one GPU 0: NVIDIA ... (UUID: GPU-...) line per allocated device and exit status 0. Then check the container's view against the claim:

kubectl get resourceclaim -n "$NS" "$CLAIM" \
  -o jsonpath='{.status.allocation.devices.results[*].device}'

The device count from the claim and the line count from nvidia-smi -L have to agree. On the host, nvidia-ctk cdi list should enumerate the nvidia.com/gpu=* devices, confirming the toolkit itself sees hardware.

Where this goes wrong

Running nvidia-smi on the host and declaring victory. It creates the device files and changes nothing for an already-prepared claim, because the cached spec is not regenerated. The pod keeps failing and the operator concludes the driver is flaky.
Working the generic fix list. DKMS rebuilds, nvidia-uvm reloads, Secure Boot checks and driver reinstalls all target a broken host driver. If nvidia-smi -L works on the node, every one of them is wasted time, and grep -c '/dev/nvidia' on the claim spec is what tells you so in one second.
Misfiling a UVM-only gap. If /dev/nvidiactl and /dev/nvidia0 are present but /dev/nvidia-uvm is missing, nvidia-smi succeeds and CUDA initialization fails instead. Different symptom, different fix, and the -u flag in step 4 is what covers it.
Blaming the kubelet. The plugin ships as a DaemonSet, so node-level state surfaces as a workload failure two layers away from its cause, the same indirection as the kubelet's cgroup v1 check misreading a v2 host.

FAQ

Why does nvidia-smi work on the host but not in the pod? The host has a working driver and the container has a CDI spec with no device nodes in it. The binary and NVML are mounted in, the hardware is not.

Does restarting the pod fix it? Only if the claim is deleted too. A restart against the same prepared claim reuses the cached spec under /var/run/cdi and fails identically.

Where is the CDI spec for a DRA claim? /var/run/cdi/k8s.gpu.nvidia.com-claim_<claim-uid>.yaml, with a matching PreparedClaims entry in checkpoint.json under /var/lib/kubelet/plugins/gpu.nvidia.com/.

Is this fixed in a released version? Not as of 15 September 2026. Issue #1380 is open against v0.5.0 with the v0.5.1 milestone attached, and v0.5.1 has not shipped. Check the releases page before you assume you still need the workaround.

One grep tells you which failure you have

The whole diagnosis is grep -c '/dev/nvidia' against the claim's CDI spec. Zero means the spec is device-less and the host driver is a red herring. Non-zero and a still-broken pod means you have an actual driver problem and the generic advice applies after all. Everything above is what you do on either side of that number.

Originally published at indragustiprasetya.com

MCP Tool Permissions: 10 Tips to Prove Agent Write Access

Indra Gusti Prasetya — Mon, 14 Sep 2026 02:06:15 +0000

Short version: in the 2026-07-28 MCP revision a server's tool list may legally vary by the token on the request, which turns tools/list into an authorization surface instead of a static manifest. readOnlyHint remains a self-declared annotation with no enforcement behind it, so your risk tiers belong in an allowlist you version. These ten tips are for the person who already has MCP servers in production, an agent calling them, and nothing to hand an auditor who asks what the agent can write to.

Three mechanisms in that revision carry authorization consequences and nothing connects them in the vendor docs: token-varying tool lists, cacheScope on those lists, and x-mcp-header, which mirrors tool parameters into HTTP headers. A model trained before July will describe the old shape, where the tool list is a fixed property of the server. Getting a correctly audienced token in the first place is a separate job covered in fixing MCP OAuth 2.1 before the July 28 rewrite; if you are tracking this against a control framework, the renumbering in the OWASP LLM Top 10 2026 crosswalk moved the IDs you probably cite. What follows is the layer underneath both: what an authenticated agent may actually invoke.

flowchart LR
  A["Agent"] --> G{"Gateway\nallowlist + scope check"}
  G -->|"read scope"| R["Read-tier tools"]
  G -->|"403 insufficient_scope\nstep-up required"| W["Write-tier tools"]
  G -->|"unlisted: deny + log"| D["Default deny"]
  R --> S["MCP server"]
  W --> S

The tips

Treat readOnlyHint as a claim, never as a control. The spec's tools page carries a warning that clients "MUST consider tool annotations to be untrusted unless they come from trusted servers," and the MCP project's March 2026 post on annotations puts it plainly: an untrusted server can lie, and annotations "aren't enforcement." A policy engine that auto-approves on readOnlyHint: true has delegated risk tiering to the party being tiered. Hold the tier in your own allowlist and let the server's annotation do one job, raising an alert when it drifts from what you recorded.
Hash the tool surface and alert on the diff. Servers may change their tool set at any time and announce it with notifications/tools/list_changed, so the manifest you reviewed in March is not the one answering calls today. Baseline the name, description, inputSchema and annotations together, because a rug pull can move a tool from read to write without touching the name.

   curl -sX POST "$MCP_URL" -H "Authorization: Bearer $TOKEN" \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' \
   | jq -S '[.result.tools[] | {name, description, annotations, inputSchema}]' | sha256sum

Filter the tool list by the token rather than by client config. This is the sanctioned mechanism and it is new. The spec states the tool set "MUST NOT vary per-connection or as a side effect of other requests," then adds that it "MAY vary by the authorization presented on the request, for example returning only the tools the caller's granted scopes permit, since credentials are per-request input, not connection state." A read-scoped agent should never see a write tool in its context window at all. Prove it by listing under two tokens and diffing the name arrays.
Set cacheScope: "private" the day that filtering goes live. The caching page's Security Considerations describe the trap directly: a tools/list result marked "public" "may be cached by a client and may be shared outside of the initial request's authorization context (i.e. different access tokens can leverage the same cache)." A scope-filtered list served as public through a shared gateway hands the privileged tool set to every caller behind it. The same section warns that servers "MUST NOT rely on cacheScope alone to prevent unauthorized access," so treat it as a leak stopper sitting on top of a real check.

   ... | jq '.result | {cacheScope, ttlMs, count: (.tools|length)}'

Key the allowlist on the canonical resource URI plus the tool name. Tool names are unique only within a server. The spec notes that aggregating proxies "MAY encounter naming collisions (for example, two servers each exposing a search tool)," and that serverInfo.name "is not guaranteed to be unique across servers and SHOULD NOT be relied upon for disambiguation." An entry reading search: allow is ambiguous by construction. Use the RFC 8707 canonical resource value the token was issued for, the same string you put in the resource parameter.

   - resource: "https://mcp.example.com/mcp"   # the RFC 8707 resource value
     tool: "search"
     tier: read

Audit x-mcp-header on every tool schema you accept. A parameter may carry x-mcp-header, which mirrors its value into an Mcp-Param-{name} HTTP header so load balancers and WAFs can route on it without parsing the body. The spec tells server developers they "SHOULD NOT mark sensitive parameters (passwords, API keys, tokens, PII) with x-mcp-header, as header values are visible to network intermediaries." It cuts both ways, and the second edge is useful: a gateway can deny on Mcp-Param-Region with zero JSON parsing, which is the cheapest enforcement point you will find in this protocol.

   ... | jq '.result.tools[] | select([.inputSchema.properties[]?["x-mcp-header"]] | length > 0) | .name'

Make writes earn a step-up and match on the exact 403 shape. Issue the agent read scopes at session start and let the server challenge for anything more. The authorization spec fixes the response format: HTTP 403 Forbidden with WWW-Authenticate: Bearer error="insufficient_scope", scope="files:write", and it tells servers to emit every scope the operation needs in one challenge instead of trickling them out call by call. Assert on that header text in CI, since a gateway that returns 401 or a bare 403 will send your client into a retry loop.
Bound the token lifetime, because granted scope only grows. The step-up flow requires clients to compute "the union of the client's previously requested scope set and the scopes from the current challenge." Across a long-running session the agent accumulates permissions and sheds none of them, so expiry is the only thing that resets the union. That makes TTL an authorization control, and the reasoning is the same one behind killing static ServiceAccount tokens for bound ones in Kubernetes.
Re-authorize stateful handles on every single call. The sharpest line in the whole revision is the non-normative guidance on stateful tools: "a handle is a name, not a capability. The server should validate the caller's authorization against the handle on every call." For unauthenticated servers, "where the handle is necessarily a bearer token," the spec asks for UUIDv4-grade entropy and a bounded lifetime. Test it in ten minutes by creating a handle with token A, then replaying that handle with token B and confirming you get a denial rather than a result.
Decide your policy on input_required before an agent meets one. A tools/call can return resultType: "input_required" along with an elicitation/create request; the spec's own example asks the user for a GitHub username. That is a server-controlled prompt arriving mid-call inside an automated loop, which is a prompt injection delivery path with a protocol blessing. Route elicitation to a human or fail the call, and never auto-fill it from a secret store. MRTR results also "MUST NOT be cached," so these calls sidestep tip 4 entirely.

Wrap-up

The habit that carries the rest: keep the risk tier in an artifact you version, and let the server's annotations only ever raise an alert. The scope-filtered list, the private cache scope, the step-up 403 and the bounded token all follow from refusing to let a tool grade its own homework, which is the same default-deny posture that keeps Kubernetes admission control from failing open.

Turning this into evidence takes an afternoon. Write a CI job that lists tools under a read-scoped token, asserts zero write-tier names come back, then calls one of those write tools directly and asserts a 403 carrying error="insufficient_scope". That output is what goes in the review binder. The expensive part lands afterward: running the inventory across every MCP server your teams have quietly wired up, agreeing a tier vocabulary that survives contact with a second team, and standing up the gateway that enforces it. None of this addresses prompt injection, which remains a separate problem with separate controls.

Originally published at indragustiprasetya.com

Stop Kubernetes Admission Control Failing Open: 11 Tips

Indra Gusti Prasetya — Fri, 11 Sep 2026 05:08:39 +0000

Short version: an admission webhook with failurePolicy: Ignore lets the request through when it times out, and the API server records that in apiserver_admission_webhook_fail_open_count instead of denying anything. To prove enforcement you need three artifacts: the list of every Ignore webhook with its timeout, an alert on that counter using type="admit" for mutating webhooks, and the CEL-expressible rules moved into ValidatingAdmissionPolicy where there is no network call to lose.

The reader I have in mind is the one who has to put the answer in writing. A control-effectiveness questionnaire asks whether privileged pods can be admitted. An SRE has to explain a pod that exists in a namespace where policy said it could not. The usual answer is a screenshot of Kyverno or Gatekeeper in Enforce mode, which proves the policy object exists and says nothing about the seconds when the webhook behind it was unreachable. Admission control is also the wrong layer for a whole class of questions, so pair it with controls that keep working when the API server is busy: a default-deny egress policy stops pod exfiltration regardless of what got admitted, and Kubernetes v1.37 tightened the control plane itself, which is why mirror pods may no longer reference secrets.

The path a request takes, and the branch that produces a written object with no denial:

flowchart LR
  A["kubectl apply"] --> B["Mutating phase\nwebhooks + MutatingAdmissionPolicy"]
  B --> C{"Webhook answers\nin time?"}
  C -->|"no, failurePolicy Ignore"| F["Object written to etcd"]
  C -->|"yes"| D["Validating phase\nwebhooks + ValidatingAdmissionPolicy"]
  D --> F

Before the list, the mapping worth keeping next to your policy repo. Rows where the evidence column reads "the policy exists" are the expensive ones.

Control question	Mechanism	Enforcement point	Evidence you can query	Blind window
Can a privileged pod be created?	ValidatingAdmissionPolicy (CEL)	In-process, kube-apiserver	Audit annotation, or a denial	CEL runtime error with `failurePolicy: Ignore`
Is this image signed?	Kyverno or Gatekeeper webhook	External pod over the network	Webhook logs plus the fail-open counter	Webhook unreachable and `Ignore`
Do workloads created last year comply?	Background scanning	None	Policy report	Everything between scans
Can the enforcement be deleted?	RBAC on `admissionregistration.k8s.io`	API authorization	RBAC listing, audit delete events	Any holder of `delete` or `*`
Is policy active while the API server starts?	Manifest-based admission config (KEP-5793)	File on the control-plane node	API server fails readiness on an invalid manifest	Clusters before v1.37

Eleven checks, and what each one proves

Tips 1 to 4 establish what is enforced right now. Tips 5 to 8 name what each control refuses to answer. Tips 9 to 11 cover what survives a control-plane restart.

Inventory every webhook permitted to fail open, with its timeout, before claiming anything is enforced. The v1 API defaults failurePolicy to Fail, so every Ignore in your cluster was chosen by a person or a Helm chart. Service-mesh injectors, certificate controllers and image-mutation sidecars ship Ignore deliberately to avoid wedging the cluster, and that choice is defensible until the same configuration also carries your security rules. Every cluster I have run this against had at least one Ignore nobody present remembered choosing:

   kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o json \
     | jq -r '.items[] | .metadata.name as $c | .webhooks[]
              | select(.failurePolicy == "Ignore")
              | [$c, .name, (.timeoutSeconds // 10)] | @tsv'

Alert on apiserver_admission_webhook_fail_open_count, and use type="admit" for mutating webhooks. The counter increments each time a request is allowed through because the webhook errored while its policy said Ignore. Kubernetes PR #127898 corrected the help text in October 2024: the type label carries admit or validating, where the documentation had long promised mutating. A dashboard filtered on type="mutating" returns an empty series forever and reads as a clean bill of health, which is the worst possible failure mode for a control you are attesting to.

   sum by (name, type) (rate(apiserver_admission_webhook_fail_open_count[5m]))

Treat a fail-open spike as a signal to investigate, and know the one thing that inflates it. kubernetes/kubernetes#118829 documents a client cancelling a request mid-webhook-call being counted as a fail-open event. Somebody hitting Ctrl-C on a slow kubectl apply therefore moves the same counter that a real outage moves. Correlate against apiserver_admission_webhook_request_total and the webhook pod's own readiness before you open an incident ticket.
Move every rule CEL can express into ValidatingAdmissionPolicy, because an in-process check has no network to lose. VAP evaluates inside kube-apiserver, which shrinks the failure modes to CEL compile errors and runtime errors, and its failurePolicy governs only those (Kubernetes docs). Start with the rules that read a single object: privileged containers, hostNetwork, missing resource limits, forbidden image registries. Those are the four that show up in questionnaires and they need no external state.
Run validationActions: [Audit] first, then hunt the annotation the audit log actually writes. Failures in Audit mode land in the audit event under the key validation.policy.admission.k8s.io/validation_failure, and that annotation is the queryable artifact you hand over as evidence of coverage before flipping to Deny. The API rejects Deny and Warn together, so the useful pre-production pairing is [Warn, Audit]: authors see the message, you keep the record.

   jq -r 'select(.annotations["validation.policy.admission.k8s.io/validation_failure"])
          | [.requestReceivedTimestamp, .user.username, .objectRef.namespace,
             .annotations["validation.policy.admission.k8s.io/validation_failure"]] | @tsv' audit.log

Write down what VAP is silent on in the same document that claims the win. CEL in the API server makes no external calls and holds no state, so image signature verification, cross-resource lookups beyond a paramKind, resource generation and cleanup all stay with Kyverno or Gatekeeper. So does the category of resources that already exist: admission control has never evaluated an object created before the policy was written, which is what background scanning covers, and what runtime detection covers after that. Deciding which sensor owns the post-admission half is its own exercise, and I have argued the trade-offs in Falco vs Tetragon vs Tracee.
On Gatekeeper v3.20 or later you have two enforcement points, so confirm they share a scope. VAP generation reached beta and default-on in v3.20 with --default-create-vap-for-templates and --default-create-vap-binding-for-constraints both true (Gatekeeper docs). The sharp edge: a VAP is generated only for ConstraintTemplates that carry the K8sNativeValidation CEL engine, engine priority is fixed, and there is no fallback to Rego. A Rego-only template keeps running purely on the webhook while your dashboard reports VAP enabled.

   kubectl get constrainttemplates -o json \
     | jq -r '.items[] | [.metadata.name,
         ((.spec.targets[0].code // []) | map(.engine) | join(",") | if . == "" then "rego-only" else . end)] | @tsv'

Design around the shape of ApplyConfiguration before you retire a mutating webhook. MutatingAdmissionPolicy has been stable since Kubernetes v1.36, and its ApplyConfiguration patches use server-side apply semantics, which forbids modifying atomic structs, maps or arrays (Kubernetes docs). Surgical edits to fields Kubernetes marks atomic need JSONPatch instead. Set reinvocationPolicy: IfNeeded where your policy has to re-run after another policy mutates the object, otherwise ordering decides your outcome.
Prove that nobody outside the platform team can delete the enforcement. A ValidatingAdmissionPolicyBinding is an ordinary API object: delete the binding and the policy stops applying, with no denial, no warning, and one audit entry that looks like housekeeping. Enumerate who holds delete on the admissionregistration.k8s.io group, wildcards included, then check which of those subjects are still authenticating with long-lived credentials (the same argument as killing static ServiceAccount tokens).

   kubectl get clusterroles -o json | jq -r '
     .items[] | .metadata.name as $r | .rules[]?
     | select((.apiGroups[]? | . == "admissionregistration.k8s.io" or . == "*")
           and (.verbs[]? | . == "delete" or . == "*")) | $r' | sort -u

Close the bootstrap and self-protection gaps with manifest-based admission config, beta in v1.37. KEP-5793 states that file-configured admission policies and webhooks must be active before the API server begins processing requests, and that manifest-based controls cannot be bypassed or modified through the Kubernetes API. Objects loaded this way carry a .static.k8s.io name suffix, and an invalid manifest stops the API server reaching ready, which is a loud failure by design. Kubernetes v1.37 (Garhwal) was released on 26 August 2026, so treat this as a control-plane change with a rollback plan, tested on one control-plane node at a time.
```
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
staticManifestsDir: /etc/kubernetes/admission-static
```
Build the coverage matrix, one row per question you will be asked. Use the five columns in the table above: the control question, the mechanism, the enforcement point, the queryable evidence, and the window where it is blind. Filling it in takes a half day of reading manifests and produces a document that outlives every dashboard in this list, because the column that matters is the blind window and no tool populates it for you.

Wrap-up

The habit that carries the rest: never accept "the policy is in Enforce mode" as evidence. Ask for the artifact, meaning the metric series, the audit annotation, or the RBAC list, and where no artifact exists, record the blind window in place of the reassurance.

The counterargument is worth airtime. Teams running failurePolicy: Fail on every webhook have paid for it with a cluster that refused all writes when the policy pod crash-looped, and Ignore on the injector is how they stopped that recurring. That position buys availability and costs the ability to answer question one. Pick deliberately, write down which you picked, and date it.

Tips 1 through 5 fit inside a week with the staff you have. Tips 7 through 10 touch the control plane, need a maintenance window, and need an owner for the evidence trail afterwards. When that second half keeps slipping, a scoped admission-control review is the usual way in: inventory every webhook and policy, separate the rules CEL can hold from the rules that need an external engine, produce the audit queries your control-effectiveness answers point at, and hand back the blind-window list with a cost against each fix. Somebody still has to approve the API server restart.

Originally published at indragustiprasetya.com

Pin GitHub Actions by SHA: 10 Gaps the Pin Leaves Open

Indra Gusti Prasetya — Fri, 11 Sep 2026 02:08:20 +0000

Short version: pinning uses: to a full 40-character SHA stops a maintainer rewriting a tag under you, and it does nothing about a commit that entered the upstream repository legitimately. That is the path ChainDrop took on 4 August 2026 into packages with roughly two billion monthly installs, carrying cryptographically valid provenance the whole way. The ten checks below cover the rest; the first five are a week of existing-staff work.

The reader I have in mind is the person who has to write the answer down. A customer security questionnaire asks whether third-party CI code is pinned and verified. An auditor asks for the artifact that enforces it. An incident channel asks whether a compromised package version was ever resolved into a build. Datadog's State of DevSecOps 2026 found 71% of organisations leave GitHub Actions completely unpinned and only 4% pin every public action to a commit hash, so most teams are answering those three questions from memory. The same trust assumption bites outside CI too: the arrayref attack showed cargo's build.rs runs any code the moment a legitimate version bump lands.

Here is the path that beat the pin.

flowchart TD
  A["Maintainer account compromised"] --> B["Malicious commit lands in keyv repo"]
  B --> C["Tag v6.0.0 points at that commit"]
  C --> D["Legitimate release.yaml workflow runs"]
  D --> E["OIDC exchange mints short-lived npm token"]
  E --> F["keyv@6.0.0 published with valid provenance"]
  F --> G{"Consumer runs npm install"}
  G -->|"provenance verifies"| H["preinstall stealer executes"]

Datadog Security Labs timestamped the first unsigned commit at 09:02:37 UTC and the malicious publish at 09:35:00 UTC, a 32-minute window (full writeup). Snyk's analysis of the same compromise states the mechanism directly: the malicious source was present in the tagged repository state, so the legitimate release workflow built and attested the malicious artifact.

Before the list, the mapping I ask every team to keep next to their workflows:

Control	What it asserts	What it is silent on
SHA pin	The bytes behind this ref did not change under a tag	Whether the commit was reviewed, or lives in the repo you think it does
`actions.lock`	The pinned commit is reachable from a branch in the named repo	Actions reached via reusable workflows in other repositories
npm provenance	Which workflow, at which commit, built the tarball	Whether that commit was authorised
`gh attestation verify --signer-workflow`	A named workflow file produced this artifact	What that workflow's own dependencies did at runtime
Egress allowlist	Where the runner is permitted to talk	What the job does with secrets it already holds

Ten checks, and what each one actually proves

Confirm every pinned SHA lives in the upstream repository, not in a fork of it. GitHub resolves a commit against the shared object database of the entire fork network, so a hex string from anyone's fork answers a request to actions/checkout. Alex Vaines demonstrated in March 2026 that a pull request can keep uses: actions/checkout@ unchanged, swap only the hex, and execute forked code while the reviewer sees an ordinary SHA bump (the write-up). zizmor has an audit for exactly this substitution:

   zizmor --min-severity=medium .github/workflows/

The finding you want is impostor-commit. Treat a hit as an incident, since there is no benign reason for a fork-only commit to appear in a uses: line.

Adopt the Actions lockfile while it is still a preview. GitHub's gh-actions-lock extension writes .github/workflows/actions.lock and enforces check 1 at runtime: a locked action must have a branch the commit is reachable from, and repository redirects are refused.

   gh extension install github/gh-actions-lock
   gh actions-lock

It is pre-1.0, the file format can change between releases, and it skips local ./ action references, so onboard one low-traffic repository first and read the extension's own README before you wire it into required checks.

Inventory unpinned uses across the org, then state what the inventory misses. Two blind spots survive a per-repo scan: Docker-based actions pinned by image tag, and actions pulled in by reusable workflows that live in a different repository. zizmor's unpinned-images audit covers the first; nothing covers the second automatically, so enumerate callers by hand.

   gh repo list ORG --limit 1000 --json nameWithOwner --jq '.[].nameWithOwner' \
     | xargs -I{} gh api "repos/{}/contents/.github/workflows" --jq '.[].name' 2>/dev/null

This is the same discipline as generating an SBOM with Syft and filtering Grype with VEX: the list is only useful once you can say out loud which components it never sees.

Make gh attestation verify assert a specific signer. The bare two-argument form accepts any workflow in any repository under the owner you named, which means an artifact built by a workflow nobody reviewed still passes. The flags that turn it into a gate, per the gh manual:

   gh attestation verify ./dist/app.tar.gz --owner ORG \
     --signer-workflow ORG/platform/.github/workflows/release.yml \
     --predicate-type https://slsa.dev/provenance/v1

Check your CI logs for the short form today. In my experience it is usually there, copied from a quickstart during the week someone first enabled attestations.

Grep for the developer-machine execution paths, because the package manager stopped being the only one. The keyv payload shipped a .vscode/tasks.json with "runOn": "folderOpen" and a .claude/settings.json SessionStart hook, both invoking setup.mjs, documented by Snyk. Opening the folder was sufficient; no install required.

   rg -n --hidden -g '**/.vscode/tasks.json' -g '**/.claude/settings.json' \
      -e '"runOn"[[:space:]]*:[[:space:]]*"folderOpen"' -e '"SessionStart"' .

Add both paths to CODEOWNERS in every repository. Ten minutes of work buys a permanent review requirement on the two files nobody reads in a diff. The install-time half of this problem is separate, and npm's allowScripts lives in the package.json schema rather than .npmrc, which trips up most teams enabling it for the first time.

Split build from publish and put a human on the publish environment. Keep permissions: id-token: write in the publish job alone, then gate that job behind a GitHub Environment with required reviewers. A compromised push to the default branch then stops short of minting a registry token. The cost is honest: every release waits on an approver, which is a release-engineering trade rather than a free win. The token-scoping half of this is covered in the notes on GitHub Actions OIDC to AWS.
Ask provenance only which workflow built the tarball. npm provenance proves the workflow identity and the source commit, and it says nothing about whether that commit was authorised, which is precisely why the ChainDrop artifacts verified. Budget the authorisation question separately through install-time behavioural scanning or a cooldown window before new versions are allowed to resolve. Route on provenance, decide on behaviour.
Put hosted runners behind an egress allowlist you control now. GitHub's Actions security roadmap, published 26 March 2026, lists a native egress firewall for hosted runners alongside dependency locking, scoped secrets and an immutable Actions data stream, and most of it is still preview. ChainDrop anchored its command-and-control in an Ethereum smart contract so that domain blocklists would rot, per Datadog, which settles the allowlist-versus-denylist argument for CI. Run one repository in block mode for two weeks and expect roughly a week of tuning; the default-deny egress pattern for Kubernetes pods transfers almost directly.
Move the pinning rule into org-level Actions policy. GitHub Actions policy has supported blocking specific actions and requiring SHA pinning since the 15 August 2025 changelog. Policy is an artifact an auditor accepts; a wiki page and a review convention are not. New repositories inherit the setting at creation, which is the part that survives your attention moving to something else.
Rehearse the "were we affected" answer against a real indicator set. Datadog published a stage-two SHA-256 of 9fc2570b7cef51c1b8df116d144d11ff4096357be7d2c4c6367cfc2509cf1bcc, the contract address 0xE1f2395ee43e45A1556EC6438a88c31B83493103, and repositories named Shai-Hulud: Here We Go Again. Test the lookup before an incident channel is waiting on it:
```
npm ls keyv flat-cache file-entry-cache cacheable-request cacheable \
  @cacheable/utils cache-manager ecto --all
```
The blind spot, stated plainly: that reads today's tree. Reconstructing which version a build resolved six months ago depends on retained run logs, and default Actions retention expires well before a slow-burn compromise surfaces. Anything a compromised runner could reach counts as exposed, including every secret your OIDC trust policies hand out at job start.

Wrap-up

The habit that carries the rest: write one sentence next to each control saying what it asserts. "Pinned by SHA" asserts the bytes did not change under a tag. "Provenance verified" asserts which workflow built the artifact. Neither sentence contains the word "safe" or the word "reviewed", and the distance between what you believe you proved and what you actually proved is where the ChainDrop consumers were standing on 4 August.

The counterargument deserves airtime. Maintainers who resist pinning point out that a fully pinned fleet stops receiving security patches the moment Dependabot coverage lapses, and stale pins have burned plenty of teams. That position buys real patch velocity, and it costs the ability to answer question one at all. Pick deliberately and record which you picked, with a date.

Tips 1 through 5 fit inside a week with the staff you have. Tips 6 through 10 change release process, spend money on egress tooling, and need an owner for the evidence trail afterwards. When the second half keeps getting deferred, a scoped pipeline-provenance review is the usual way in: inventory every uses: across the org, verify each pin against its upstream branch, produce the attestation policy your questionnaire answers can point at, and hand back the gap list with a cost attached to each fix. It will not make the finding disappear by itself. Someone still has to approve the release gate.

Originally published at indragustiprasetya.com

Cyber Resilience Act Reporting: 10 Tips for Sept 11

Indra Gusti Prasetya — Wed, 09 Sep 2026 02:18:35 +0000

Short version: from 11 September 2026, Article 14 of Regulation (EU) 2024/2847 obliges manufacturers of products with digital elements to report actively exploited vulnerabilities and severe incidents through ENISA's Single Reporting Platform on a 24-hour, 72-hour and 14-day clock. The obligation reaches products you placed on the market years ago, the platform ships without an API, and picking the wrong coordinating CSIRT invalidates the notification.

This is for whoever owns product security at a company selling software, firmware or connected hardware into the EU. If your legal team has the CRA filed as a December 2027 problem, they are describing the conformity half. The reporting half starts in two days, and most of the operational work below is unglamorous inventory and timestamp discipline rather than engineering. If your release process already emits an SBOM per artifact, you are further along than you think.

flowchart TD
  A["Manufacturer becomes aware"] -->|24h| B["Early warning via SRP"]
  B --> C["Coordinating CSIRT"]
  C --> D["ENISA"]
  C --> E["Other relevant CSIRTs"]
  A -->|without undue delay| F["Impacted users"]
  B -->|72h| G["Full notification"]
  G -->|14 days after fix| H["Final report"]

Ten tips before the clock starts

Inventory the products you stopped selling, because they are in scope too. Article 69(3) of the Regulation extends Article 14 to every in-scope product placed on the market before 11 December 2027, with no carve-out for discontinued lines, end-of-life models or firmware nobody has touched since 2019, a reading the CRA-tracking site cyberresilienceact.eu spells out in detail. Those legacy products owe you nothing on Annex I essential requirements, conformity assessment or CE marking. They owe the report. Build one table and keep it where the incident runbook lives:

| Product | Versions in the field | Legal entity that placed it | Build infra still exists? | Advisory channel |
|---|---|---|---|---|
| Gateway X | fw 4.2.1 to 4.7.0 | DE GmbH | yes | portal + email list |
| Sensor Y (EOL 2021) | fw 2.9.3 | DE GmbH | no | release notes only |

The rows with "no" in column four are the ones that will hurt, and for those a binary scan of the shipped artifact is the only component list you are going to get, so decide which scanner you point at it before the incident. Auditing an old estate against a fresh obligation table is the same exercise as repointing a control crosswalk when the IDs move: the table is cheap, and the value is in the query that finds every row you forgot.

Determine your coordinating CSIRT on paper before you need it. ENISA's Single Reporting Platform FAQ is blunt that selecting the wrong CSIRT designated as coordinator invalidates the notification, and the platform asks you for it at registration. The test is your main establishment, meaning where decisions about your products' cybersecurity are predominantly taken, which is frequently a different city from your registered office. Where it is genuinely unclear, the fallback chain runs through largest EU establishment, then authorised representative, then importer, then distributor, then user-base concentration. Write the reasoning down, date it, and check the Member State entry against ENISA's coordinating CSIRT list, published on 4 September 2026.
Create EU Login accounts with MFA today and register on the platform later. Platform access runs on personal EU Login accounts with multi-factor authentication, with one Primary Assigned Representative per manufacturer who can invite up to 20 Secondary ARs. Account creation at https://ecas.ec.europa.eu/cas/login needs no approval and takes minutes. Do it for a filer and at least one deputy now, so nobody is doing an authenticator enrolment at hour 19 of a 24-hour window.
Know that an unvalidated account is capped at 20 notifications. ENISA runs manufacturer validation in parallel with reporting, so a pending validation does not block your first filing, per the same FAQ. The ceiling bites later: an unvalidated AR may submit up to 20 notifications before validation becomes mandatory. For a single-product manufacturer that limit is theoretical. For a large portfolio having a bad quarter it is a queue worth clearing while nothing is on fire.
Compose the notification outside the platform, then transcribe it. There is no API at initial release, every field is typed by a human into a web form, and platform drafts are not visible to colleagues, so two people cannot co-author one report inside the tool. Keep a skeleton in the shared document your incident process already uses:

   notification_type:   early warning
   awareness_utc:       2026-09-14T22:41Z
   awareness_basis:     PSIRT confirmed exploitation from customer PCAP
   awareness_owner:     <name>
   product:             <name> / firmware 4.2.1 to 4.7.0
   coordinating_csirt:  <Member State CSIRT>, per main-establishment note 2026-09-05
   suspected_malicious: yes

Record your own awareness timestamp, because the platform records a different one. Article 14 starts the 24-hour clock when the manufacturer becomes aware, while early platform builds capture incident detection, and those two moments diverge whenever triage takes a day to conclude that exploitation is real. Keep a timestamped note of the moment your organisation formed that belief plus the evidence behind it. That note is what you show a regulator, and it is the field an auditor asks for first.
Ignore the platform's 72-hour counter and run your own. The counter displays a due date 48 hours after your 24-hour submission, so a report filed at hour 20 can show as overdue while you are still inside the legal window. Compute both deadlines from awareness and put them in the incident tracker:

   # GNU coreutils date; both deadlines derive from awareness, never from submission
   date -u -d '2026-09-14 22:41 UTC + 24 hours' +%Y-%m-%dT%H:%MZ   # 2026-09-15T22:41Z
   date -u -d '2026-09-14 22:41 UTC + 72 hours' +%Y-%m-%dT%H:%MZ   # 2026-09-17T22:41Z

Define "actively exploited" as a triage rule bound to signals you can query. Severity alone does not fire the obligation; exploitation does. Article 14(5) defines a severe incident as one that negatively affects the product's ability to protect data or functions, or that has led to malicious code being introduced or executed, so translate that into named sources a duty engineer can check inside an hour: specific EDR or WAF rule IDs covering the affected code path, telemetry or honeypot hits on the vulnerable endpoint, customer-supplied indicators, and upstream PSIRT advisories for components you ship. If the runtime signal does not exist yet, that is a tooling decision to make now rather than during triage, and the Falco, Tetragon and Tracee tradeoffs determine which of those rule IDs you can even write. A trigger definition with no query behind it produces an argument at 2 a.m.
Wire the user-notification duty into the same runbook. Article 14(8) requires you to inform impacted users about the vulnerability or incident and the mitigations available, and the coordinating CSIRT may notify your users itself if you fail to do so in time. Your advisory channel has become a regulated control surface. Confirm you can publish a machine-readable advisory and reach the customers running affected versions on the same day you file the early warning; if the only distribution path is a release-notes page, that gap is the expensive one to discover live.
Separate the reporting budget from the conformity budget, and classify products anyway. Reporting readiness is a runbook, a CSIRT determination and staffing. Conformity is Annex I engineering, technical documentation and CE marking, due 11 December 2027 for new and substantially modified products, with the Commission's CRA reporting policy page as the canonical timeline. Classify each product now under Commission Implementing Regulation (EU) 2025/2392, which fixes the technical descriptions of the important and critical categories and settles them by core functionality, since that classification decides your 2027 assessment route and costs nothing to establish today. In my experience the split matters organisationally as much as financially: the reporting work lands on the security team and the conformity work lands on product engineering, on different clocks.

The honest counterargument: a small manufacturer with one product and one EU establishment can do tips 1 through 3 in an afternoon and defer the rest until something happens. That position buys real time back, and it is defensible. It costs you the weekend coverage question, because the 24-hour window is wall clock and includes Saturdays.

Wrap-up

The habit that matters is the timestamp. Everything else here is paperwork you can produce under pressure, but the moment your organisation became aware cannot be reconstructed after the fact, and it is the single input deciding whether a report was late. Add an awareness_utc field and a mandatory awareness_basis line to your incident template this week, populate them during triage instead of during the write-up, and derive both deadlines from that field with the date -u -d one-liner above rather than from whatever the platform displays. Cheap control, and it only works if it is already running on the bad day.

Scoping a product inventory across a legacy portfolio, writing the main-establishment memo and rehearsing a filing against a platform with no API is a few weeks of work for someone who has done it before, and it is the point where reading stops being enough.

Originally published at indragustiprasetya.com

OWASP LLM Top 10 2026: Fix Your Crosswalk, 8 IDs Moved

Indra Gusti Prasetya — Tue, 08 Sep 2026 03:07:58 +0000

OWASP published the Top 10 for LLM Applications 2026 (v1.0) on 3 August 2026, and eight of the ten identifiers now point at a different risk than they did in 2025. Only LLM01 and LLM02 survived unchanged. Every crosswalk, policy clause, and customer questionnaire answer that cites a bare LLM0x is now ambiguous, and the risk that moved furthest, Excessive Agency from #6 to #3, is the one your auditor will push hardest on. These tips are for the person who has to answer the questionnaire, defend the control set at renewal, or sign the ISO 42001 statement of applicability for a system that holds credentials and calls tools, the surface I wrote about in MCP OAuth 2.1.

Bare ID	Meant in 2025	Means in 2026
LLM01	Prompt Injection	Prompt Injection
LLM02	Sensitive Information Disclosure	Sensitive Information Disclosure
LLM03	Supply Chain	Excessive Agency
LLM04	Data and Model Poisoning	Supply Chain
LLM05	Improper Output Handling	Data and Model Poisoning
LLM06	Excessive Agency	Unbounded Consumption
LLM07	System Prompt Leakage	Misinformation
LLM08	Vector and Embedding Weaknesses	Hidden Context Exposure
LLM09	Misinformation	Vector and Embedding Weaknesses
LLM10	Unbounded Consumption	Improper Output Handling

One warning before the list: any assistant trained before August will answer you in 2025 numbering with complete confidence, so verify every ID against the resource page above rather than against a chat window.

The tips

Grep your policy estate for bare IDs and count them before you promise a timeline. The remap cost is a number you can produce in a minute, and producing it first turns "we should update that" into scoped work. Run this across policies, contracts, SOA spreadsheets exported to text, and your questionnaire answer library:
```
rg -Pc 'LLM(0[1-9]|10)\b(?!:20)' --glob '!node_modules' . | sort -t: -k2 -nr
```
The (?!:20) lookahead skips anything already year-qualified as LLM06:2025, so what remains is exactly the set of citations that changed meaning under you. -P is required for the lookahead; without it ripgrep refuses the pattern. Fix each hit by adding the year, never by bumping the number, because the old clause described the old risk and your evidence was collected against it.
Decide which OWASP list you are being assessed against, and say so in writing. OWASP now maintains the LLM Applications Top 10 2026 (LLM01 to LLM10) and a separate Top 10 for Agentic Applications 2026 (ASI01 to ASI10), the latter released 9 December 2025 per OWASP's own resource page. The agentic list covers goal hijack, tool misuse, privilege abuse, memory poisoning, cascading failures, and rogue agents, all of which only exist once the model holds credentials. If you ship agents and answer only against the LLM list, you have answered the easier question and the assessor will find that out later.
```
flowchart TD
  A["What are you being asked to prove?"] --> B{"Does the system call tools or act?"}
  B -->|"No: prompt in, text out"| C["LLM Top 10 2026 v1.0\nLLM01-LLM10, 3 Aug 2026"]
  B -->|"Yes: goals, memory, credentials"| D["Agentic Top 10 2026\nASI01-ASI10, 9 Dec 2025"]
  D --> E{"Can you show runtime evidence?"}
  E -->|"No"| F["Agent Control Standard v0.1\nAgBOM plus OTel/OCSF traces"]
  E -->|"Yes"| G["Map each ASI to one queryable artifact"]
  C --> G
```
Know why Excessive Agency moved, because it changes what counts as evidence. The 2026 ranking weighted expert vote at 75% and real incident data at 25%, drawn from 6,639 classifiable records out of 7,714 pulled from public vulnerability and AI-harm databases, as Help Net Security and ReversingLabs both reported from the launch. Excessive Agency climbed on incident counts, so reassurance about your design intent reads poorly against it. Bring numbers from the runtime layer you already operate: tool calls per session, blocked write attempts, approvals requested and denied. If you have no such layer yet, Falco, Tetragon, and Tracee are where the syscall-level half of that evidence comes from.
Retire prompt-filter percentages as your answer to LLM03:2026. OWASP's framing at launch was that you build the system around the model so that when it is fooled, and it will be, nothing important breaks. A filter accuracy figure answers a containment question with a prevention number, and reviewers who read the 2026 text notice. Show four limits instead: a per-session tool-call ceiling, a per-agent spend cap, a wall-clock timeout, and a circuit breaker that trips on consecutive tool failures.
```
if session.tool_calls >= LIMITS["max_tool_calls"] \
   or session.spend_usd >= LIMITS["max_spend_usd"] \
   or session.consecutive_tool_errors >= 3:
    raise AgentHalted(session.id)   # emit an OCSF event, page nobody, fail closed
```
The network half of the same control is a default-deny egress policy on the namespace the agent runs in, which caps the damage of a tool call you did not anticipate.
Treat LLM08 Hidden Context Exposure as a scope expansion for DLP and re-run classification. System Prompt Leakage was retired and replaced by Hidden Context Exposure, which covers everything assembled into the window: retrieved documents, agent memory, tool schemas, tool responses, and application state. The item most teams forget is the tool schema. Parameter descriptions routinely carry internal endpoint names, table names, and business rules, and all of it leaves on a single "list your tools" turn. Dump your registered schemas and read them as an attacker would before you claim coverage.
Instrument the context assembler so the LLM08 answer is a query. A reviewer asking what your model can leak deserves a search result rather than a paragraph. Emit one span event per injected source carrying a classification label, then answer by querying for any trace where a source labelled internal appears in a session whose response crossed the tenant boundary. Without that event you can only assert, and an assertion is what gets you a finding at the next audit.
Do not accept output sanitisation as a vendor's headline control. Improper Output Handling fell from #5 to #10. It stayed on the list for good reason, since model output still reaches shells, SQL, and browsers, but a pitch that leads with output filtering is selling you coverage of the tenth-ranked risk while the third-ranked one sits unowned. Ask where the agency limits live, who can raise them, and what the audit trail looks like when someone does, before you ask about filters.
Answer ASI03 with per-agent identity and be ready for the delegation follow-up. Identity and Privilege Abuse is where most agent programmes fail an assessment, because one shared service account backs every agent in the fleet. The defensible shape is a distinct workload identity per agent plus a short-lived, user-scoped token per action, which is the same migration described in bound ServiceAccount tokens. Expect the assessor to ask what happens when agent A calls agent B, so have the token exchange path drawn before the call.
Use the Agent Control Standard for inventory today and price its roadmap honestly. ACS was donated to the OWASP GenAI Security Project and announced on 1 September 2026, alongside the news that the community passed 30,000 members. Read the repository before you plan around it: the public preview is v0.1, Apache-2.0 for code and CC BY-SA 4.0 for docs, and it ships definitions plus observability schemas today, meaning OpenTelemetry and OCSF tracing and an Agent Bill of Materials expressed in CycloneDX, SWID, or SPDX. The published milestones put guardian-agent instrumentation at v1, AgBOM mappers at v2, and deny and modify operations at v3. Anyone selling ACS-based blocking this quarter is ahead of the specification. Adopt the AgBOM now as your inventory artifact, generated by the same tooling that already produces your Syft SBOMs, and keep enforcement where it currently works.
Give every risk one artifact and one query, then date-stamp the crosswalk. Auditors accept "control, artifact, query" and argue with prose. Build the table as risk ID with year, control, artifact, and the exact command or saved search that produces it. Set two review dates rather than one, because the two lists run on different clocks: the agentic list shipped December 2025 and the LLM list August 2026, so a single annual review will always miss one of them.

FAQ

Does LLM06 still mean Excessive Agency? No. In the 2026 list LLM06 is Unbounded Consumption. Excessive Agency is LLM03:2026. Cite the year and the name together so the sentence survives the next revision.

Which list applies if my app only summarises documents? The LLM Applications Top 10 2026. The agentic list applies once the system pursues goals, keeps memory across turns, or holds credentials to act.

Can I use the Agent Control Standard to block agent actions now? Not from the standard itself. v0.1 ships definitions and observability schemas, with deny and modify operations scheduled for v3 in the published milestones.

Wrap-up

Write the year every time you cite an OWASP identifier. LLM06 was Excessive Agency for a year and is now Unbounded Consumption, and a contract clause that says only LLM06 will eventually be read by someone who was not in the room when it was written. Run the grep from tip 1 this week and publish the remap table internally, so you know the size of the job before a customer asks you to have already finished it.

Originally published at indragustiprasetya.com

kube-proxy IPVS to nftables: Fix 11 Traps in 1.40

Indra Gusti Prasetya — Mon, 07 Sep 2026 02:11:55 +0000

Kubernetes 1.40 does two things to kube-proxy in the same release. KEP-5495 flips the KubeProxyIPVS feature gate to Default: false, so IPVS mode stops starting unless you override it, and KEP-5343 makes nftables the default proxy mode, so any cluster that never set mode: gets moved underneath it. If you run IPVS today on a cluster with LoadBalancers, a NodeLocal DNSCache, and an alert rule someone wrote in 2021, the upgrade picks your dataplane for you.

These eleven tips are the behaviour differences that show up in production rather than in the release notes, each with the command that finds it before the rollout does.

The 11 traps

Settle three inputs before you touch a ConfigMap, then run the decision per node pool. Kernel version, whether your CNI already replaces kube-proxy, and how many Services you run determine the path, and they often differ between pools. If you are on Cilium with kubeProxyReplacement enabled there is nothing to migrate, though the upgrade itself has its own set of traps before BGP drops.

flowchart TD
  A["kube-proxy mode: ipvs today"] --> B{"Kernel 5.13+ on every node?"}
  B -->|"No"| C["Set mode: iptables now\nbump the node image before 1.40"]
  B -->|"Yes"| D{"CNI replaces kube-proxy?"}
  D -->|"Yes, Cilium kubeProxyReplacement"| E["Delete the DaemonSet\nnothing to migrate"]
  D -->|"No"| F["Canary pool on mode: nftables\nthen fleet"]

The "bump the node image" branch is a separate project with its own failure modes, including kubelet refusing to start with "not run on a host using cgroup v1" on a newer base image. Budget it as one.

Price the escape hatch: KubeProxyIPVS=true buys you exactly three releases. KEP-5495 lays out the schedule: 1.35 logs a deprecation warning, 1.37 introduces the gate at Default: true, 1.40 flips it to false and kube-proxy exits with an error unless you set it back, 1.43 locks the gate and deletes pkg/proxy/ipvs, 1.46 removes the gate entirely. Setting it in 1.40 is a legitimate stall with a hard expiry on the 1.43 upgrade, and there is no further extension to negotiate for.
Write mode: explicitly this week, even if you plan to stay on iptables. KEP-5343 states the failure directly: a cluster on a kernel too old for nftables that upgrades to 1.40 "without having explicitly set mode: iptables in their config" will start kube-proxy in nftables mode and fail. One line of config removes an entire class of upgrade-day incident, which is cheap next to the ones you find out about from a pager.

# kube-system/kube-proxy ConfigMap, config.conf
mode: "nftables"   # or "iptables", never blank

Adding --proxy-mode=nftables to the DaemonSet args does nothing. When kube-proxy starts with --config, command-line flags are ignored (kubernetes/kubernetes #98302), and kubeadm's DaemonSet runs --config=/var/lib/kube-proxy/config.conf. Check which one you have, then confirm the mode actually changed from the logs rather than trusting the rollout. Upgrade changes that silently no-op are a recurring shape in this ecosystem, in the same family as unrecognized format int32 in 1.34.

kubectl -n kube-system get ds kube-proxy \
  -o jsonpath='{.spec.template.spec.containers[0].command}'
kubectl -n kube-system logs ds/kube-proxy | grep -i 'proxier\|nftables'

Check the kernel on every node before you edit anything, because the ConfigMap is one object for the whole cluster. nftables mode needs kernel 5.13+ and nft 1.0.1+, and kube-proxy refuses to start below that. Sorting the Node objects by kernel version gives you the fleet answer in one command.

kubectl get nodes -o custom-columns=\
NAME:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion | sort -k2

The blind spot: that string is what kubelet saw when it started, so a node that later rebooted into a rollback GRUB entry still reports the newer kernel. Spot-check those with uname -r on the host. nftables.skipKernelVersionCheck is a development option that skips the check without shipping a newer nft, so it papers over the problem instead of solving it.

NodePort quietly stops answering on every node address except the primary. iptables and IPVS default nodePortAddresses to all local IPs; nftables defaults to primary, meaning the node's primary IPv4 and IPv6 per the Node object (PR #123105). A hardware load balancer pointed at a secondary NIC starts failing health checks the moment the DaemonSet rolls, and the Service looks healthy from inside the cluster the entire time. Keep the old behaviour deliberately, or list the CIDRs you actually front:

nodePortAddresses: ["0.0.0.0/0", "::/0"]

127.0.0.1:<nodePort> is gone and will not come back as a default. KEP-3866 dropped loopback NodePort deliberately to avoid the route_localnet sysctl, which is the CVE-2020-8558 surface. KEP-6032 adds it back as an opt-in userspace TCP proxy behind KubeProxyNFTablesLocalhostNodePorts: TCP only, rejecting UDP and SCTP, and only when loopback appears in nodePortAddresses. Find the callers before the rollout by grepping host-network manifests, systemd units, and monitoring configs for 127.0.0.1:3[0-2][0-9][0-9][0-9]. The same inventory is worth having anyway when you get around to default-deny egress.
Expect intermittent "connection reset by peer" on long-lived connections, and answer it with tcpBeLiberal. iptables mode installs a DROP rule for packets that conntrack marks INVALID, and PR #120412 made that rule conditional so kube-proxy skips it when --conntrack-tcp-be-liberal is set. nftables mode never installs the rule at all, which means the resets described in issue #117924 can resurface on connections that idle through a conntrack window. Watch the per-CPU counters on a canary node first, then set the option:

conntrack -S | tr ' ' '\n' | grep invalid

conntrack:
  tcpBeLiberal: true

Your kube-proxy failure alert stops firing without ever going red. kubeproxy_sync_proxy_rules_iptables_restore_failures_total is exported in iptables and IPVS modes only; nftables mode exports kubeproxy_sync_proxy_rules_nftables_sync_failures_total under a different name. A counter alert on a metric that stopped existing stays silent forever, which is worse than no alert because the dashboard still looks green. Rewrite the rule ahead of the rollout and pair it with an absence check so a future rename pages you:

absent(kubeproxy_sync_proxy_rules_nftables_sync_failures_total) == 1

Metrics are on :10249/metrics, health on :10256/healthz.

Scale-test at your real Service count, because nftables has a reported cliff. kubernetes/kubernetes #135639, opened 2025-12-06 against 1.32 to 1.34 on Amazon Linux 2 and 2023 and Bottlerocket, reports sync times of 13m15.34210662s at 1000+ Services alongside kernel soft lockups (watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [nft:222752]). Check the current state of that issue against your exact patch version, then measure rather than trust either side of the argument: record the p99 of kubeproxy_sync_proxy_rules_duration_seconds on IPVS as your baseline and compare it on the canary pool under the same Service count.
Leave your NodeLocal DNSCache configuration exactly as it is. The Kubernetes docs give two recipes: under IPVS the node-local-dns pod listens only on the link-local address and kubelet's --cluster-dns points there, because the CoreDNS ClusterIP is already claimed by the kube-ipvs0 dummy interface. Under iptables and nftables the pod can additionally bind the ClusterIP. The IPVS-flavoured setup keeps working after the switch, and tidying it up means changing --cluster-dns on every kubelet plus a restart, which is a second and larger migration hiding inside the first. Give it its own change ticket, after the dataplane is stable.

Wrap-up

The objection you will actually have to argue is the scheduler one. KEP-5495 points out that many IPVS users have not realised the schedulers "aren't actually useful in Kubernetes": every node keeps its own IPVS table, so lc counts connections on that node alone, and conntrack pins every subsequent packet of a flow regardless. A team that says "we need least-connection balancing" has been getting a per-node approximation of random for years. That is a real cost of leaving IPVS only if you run one node.

Verify the switch instead of assuming it. After the canary rolls, ipvsadm -Ln returns nothing, ip link show kube-ipvs0 reports the device is gone, and nft list table ip kube-proxy prints a real ruleset. Hold the canary through one full traffic peak before touching the rest of the fleet, because the differences that hurt appear under load from a client nobody remembered.

The habit worth adopting today is tip 3: pin mode: explicitly in every cluster you own this week. It costs one line per cluster, and it decides whether you choose your dataplane or 1.40 chooses it for you.

Originally published at indragustiprasetya.com

Fix docker save "no suitable export target found

Indra Gusti Prasetya — Fri, 04 Sep 2026 02:08:45 +0000

A five-platform image that pulled without a single warning, docker image ls --tree printing linux/amd64 in the child rows, and the daemon refusing anyway:

Error response from daemon: no suitable export target found: image with reference
kserve/llmisvc-controller:v0.20.0 was found but does not provide the specified
platform (linux/amd64)

Short version: under Docker 29's containerd image store, that one string covers four different states of your local store, and the docs describe exactly one of them. docker buildx imagetools inspect tells you what the registry publishes, docker image ls --tree tells you what your disk actually holds, and a 0B in the CONTENT SIZE column is the difference between the two. The export that survives every cause is a pull by child digest.

The report above is docker/cli #7278, filed on 3 September 2026 against Docker 29.7.2 by an operator who wanted an amd64 tar for a Kind cluster.

What Docker 29 actually keeps when you pull one platform

Docker Engine 29.0.0 (10 November 2025) made the containerd image store the default for fresh installs and added --platform to docker image save and docker image load in the same release, per the 29 release notes. Those two changes interact badly.

Under the containerd store, a pull records the full OCI index and every child descriptor, then fetches blobs only for the platform you asked for. The manifest for linux/s390x is on your disk. Its layers never arrived. Every tool that walks the index sees seven platforms; every tool that walks content sees one. The index-of-manifests shape is the same one that breaks registries in other places, which is why Cosign v3 signatures start failing on Harbor and why scanner behaviour diverges per architecture when you are choosing between Trivy and Grype for a given job.

flowchart LR
  R["Registry index\n5 platforms"] -->|pull one platform| S["Local content store"]
  S --> M["5 child manifests\nall recorded"]
  S --> B["Blobs for amd64 only\nother rows read 0B"]
  M --> T{"docker save\nplatform lookup"}
  B --> T
  T -->|platform absent from index| C1["Cause 1"]
  T -->|manifest yes blobs no| C2["Cause 2"]
  T -->|shared layer bug| C3["Cause 3"]
  T -->|flag omitted| C4["Cause 4"]
  T -->|export succeeds| X["tar is an OCI index"]
  X -->|ctr import all-platforms| K["content digest not found"]

The bookkeeping says the platform exists and the storage disagrees. It is the same class of failure as Kubernetes DRA handing one GPU to a VM and a pod at once: the record of the resource and the resource itself drifted apart, and only one of them is authoritative when something tries to use it.

Four causes behind one string

Cause	Error wording	Separating command	Fix
1. Platform never published	`does not provide the specified platform`	`docker buildx imagetools inspect <ref>` omits it	Build the variant yourself; there is nothing to export
2. Manifest present, blobs absent	`does not provide the specified platform`	`docker image ls --tree` shows `0B` `0B` on that row	Pull the child manifest by digest
3. Shared-layer content-store bug	`does not provide any platform`	Two `--platform` pulls sharing layers, then a push	Pull by digest, or pull that image on its own
4. Host platform missing	`NotFound: content digest sha256:...: not found`	Failing command has no `--platform` and the host arch row is `0B`	Pass `--platform` explicitly

Cause 1 is the only one Docker documents. The save reference states that "An error is produced if the given platform is not present in the local image store," which is accurate and almost never what you are hitting.

Cause 2 is #7278, and before it docker/cli #6457 on 28.4.0, closed as not planned. The signal lives in docker image ls --tree, still flagged experimental in the CLI. Child rows carry DISK USAGE and CONTENT SIZE:

alpine:latest      beefdbd8a1da   10.6MB   3.37MB
├─ linux/riscv64   80cde017a105   10.6MB   3.37MB
├─ linux/amd64     33735bd63cf8   0B       0B
├─ linux/arm64/v8  9cee2b382fe2   0B       0B

Two zeros mean a descriptor with no content behind it. The daemon's phrasing, "was found but does not provide the specified platform," is a statement about the reference lookup. Operators read it as a statement about the manifest, and the wording collapses two different facts into one sentence.

Cause 3 is moby #52897, carrying status/confirmed against 29.5.3 with containerd v2.2.4. Pull rancher/mirrored-coredns-coredns:1.14.3 with --platform linux/amd64, pull rancher/mirrored-metrics-server:v0.8.1 the same way, then tag and push the second one:

image with reference localhost:5000/test:0617 was found but does not provide any platform

Pull the second image on its own and it works. Shared layers between the two are the trigger, and the classic image store is unaffected. Watch the noun: any platform instead of the specified platform. That single word is the cheapest discriminator in the family and it costs you nothing to read.

Cause 4 is moby #50173, a plain docker save with no --platform at all, on an amd64 host, after pulling only linux/arm64/v8:

unable to create manifests file: NotFound: content digest sha256:0800…: not found

The reporter's reading is that the export path silently substitutes the host platform rather than exporting everything present. It sits in the maintainers' Containerd-as-default project with kind/bug, next to moby #44578, which records the same host-platform default in docker image inspect and docker image rm.

There is a fifth, mostly closed: release 29.6.0 (18 June 2026) shipped a fix for "image selection with the containerd image store on amd64 hosts when images provide amd64 variant-specific manifests." If you are below 29.6.0 and the publisher ships linux/amd64/v3, upgrade before you triage anything else.

Which one am I hitting?

Two commands, in this order:

# 1. What does the registry actually publish?
docker buildx imagetools inspect kserve/llmisvc-controller:v0.20.0

# 2. What does the local store actually hold?
docker image ls --tree
docker image inspect --platform linux/amd64 kserve/llmisvc-controller:v0.20.0  # API 1.49+

Missing from step 1 is cause 1. Present in step 1 with 0B in step 2 is cause 2. Real bytes in step 2 with an error that says "any platform" is cause 3. No --platform on the failing command and a 0B host arch row is cause 4.

Why kind load dies on a digest you never pulled

kind load docker-image shells out to docker save and pipes the stream into ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs - inside the node container. --all-platforms walks every child descriptor in the index, reaches the six with no blobs, and dies:

ctr: content digest sha256:…: not found

That is kind #4066 on kind 0.30 with Docker 29.1.2, and again on v0.32.0 in kind #4224. The digest belongs to an architecture the developer never asked for, which is why the error is close to ungoogleable for the person staring at it. Air-gapped delivery has the same shape (pull, save, ship the tar, load) with the same trap: the tar is now an OCI index, and the receiving runtime may or may not tolerate that. Local Docker dev loops accumulate these version-boundary failures, the same way pnpm deploy stopped finding modules inside Docker on 11.19.

Is switching the store off a real fix?

The workaround repeated across all these threads is features.containerd-snapshotter=false in /etc/docker/daemon.json. It works, and it has a price nobody quotes. You move the daemon back to the store Docker is retiring, you strand every image already in the containerd store behind a wall you cannot see while the disk stays consumed, and you give up multi-platform handling.

The engineer who disagrees runs a stable CI fleet on a pinned engine and wants zero variance until a maintenance window. That position is defensible for exactly as long as the pin holds, which in my experience is until someone rebuilds a runner image and picks up a fresh 29.x install with the containerd store back on by default. Buying six quiet months and paying for them with an unplanned migration is a trade worth making only if you have the migration scheduled.

What to change before your next kind load

Check your engine version first. Below 29.6.0, upgrade before triaging: the amd64 variant-selection fix landed there and will resolve a subset of these outright.
Read the noun in the error. "the specified platform" points at content presence, cause 2. "any platform" is moby #52897 and needs a digest pull.
Run the two-command split (buildx imagetools inspect, then image ls --tree) before changing any daemon config. 0B in CONTENT SIZE is the only place the CLI states plainly that a platform row has no blobs behind it.
Pin by child digest wherever you pinned by tag plus --platform. This is the form that survives all four causes and is more reproducible in CI regardless:

DIGEST=$(docker buildx imagetools inspect --raw kserve/llmisvc-controller:v0.20.0 \
  | jq -r '.manifests[] | select(.platform.os=="linux" and .platform.architecture=="amd64") | .digest')
docker pull kserve/llmisvc-controller@$DIGEST
docker save kserve/llmisvc-controller@$DIGEST -o out.tar

For kind, skip the wrapper and drop the flag that causes the index walk:

docker save redis:7-alpine | docker exec -i kind-worker \
  ctr --namespace=k8s.io images import --digests -

Audit every pull → save → load path you own, including air-gapped tar delivery. Grep your scripts for --all-platforms on the import side; that flag is the trigger.
Review scripts that key off IMAGE ID. moby #51779 reports that image IDs under the containerd store now derive from the index digest rather than the platform-specific config digest, so anything comparing IDs across a store migration will lie to you in the same maintenance window.

FAQ

Why does docker image ls --tree show a platform that docker save says is missing?
The tree view walks index descriptors; the export path walks content. A pull records all child manifests and fetches blobs for one platform, so the row exists with 0B in both size columns.

What is the difference between "does not provide the specified platform" and "does not provide any platform"?
The first means the daemon looked for one platform and found no content for it. The second is the confirmed shared-layer bug in moby #52897 on 29.5.3 with containerd v2.2.4, where an image with real content reports no platforms at all.

Does disabling containerd-snapshotter delete my images?
No. It hides them. The containerd store keeps its content on disk and consumes the space while the classic store is active, and the images reappear when you switch back.

Why does kind load docker-image fail with a digest I never pulled?
kind imports with ctr images import --all-platforms, which iterates every child descriptor in the saved index, including the ones whose layers were never fetched.

How do I pin a single-platform image in CI so this stops happening?
Resolve the child digest with docker buildx imagetools inspect --raw plus jq, then pull and save image@sha256:.... The store then holds one platform with nothing dangling.

Originally published at indragustiprasetya.com