DEV Community: Indra Gusti Prasetya

Fix Kyverno CVE-2026-4789: Admission Controller SSRF

Indra Gusti Prasetya — Tue, 04 Aug 2026 12:55:47 +0000

Kyverno is the tool most teams reach for to enforce Kubernetes least privilege: block privileged pods, require signed images, deny risky RBAC. So there's a particular irony in CVE-2026-4789. The policy engine you deployed to close namespace-isolation gaps became a way to walk straight through them. Per the GitHub Security Advisory (GHSA-rggm-jjmc-3394, published April 13, 2026), the flaw is a server-side request forgery in Kyverno's CEL HTTP library, carrying CVSS 8.5 and affecting versions 1.16.0 and later.

The shape of it is a textbook confused deputy. Kyverno's CEL policies can call http.Get() and http.Post(). Those functions, per the advisory, "do not enforce any URL restrictions." A user who can only create namespaced policy objects (NamespacedValidatingPolicy, NamespacedMutatingPolicy, and the rest) can embed an arbitrary URL in a policy expression. When that policy evaluates, the request originates not from the low-privilege user but from the Kyverno admission controller pod, which sits on the cluster network with broad reach. Kubernetes RBAC governs API calls. It does not govern raw outbound TCP from a pod. So the deputy makes the call the user never could.

If you already run Kyverno to block privileged pods with a ValidatingPolicy, this is the uncomfortable part: the same delegation model you trust for enforcement is the one that hands out the SSRF primitive.

Two CVEs, one design lesson

This isn't the only Kyverno namespace-isolation break disclosed this cycle. CVE-2026-22039, a separate bug in the apiCall context feature, is scored CVSS 10.0 by Minimus, who traced it to user-controlled annotations being substituted into a urlPath that Kyverno then fetches using its own admission-controller ServiceAccount. Same root pattern, different code path.

Both bugs come down to a library asymmetry, not a logic typo. Per Orca's analysis, Kyverno's resource.Lib enforces namespace boundaries. The http.Lib was initialized with no namespace parameter and no destination checks at all. The two libraries encode two different trust assumptions, and the HTTP one quietly assumed every policy author was a cluster admin. That assumption held right up until multi-tenancy made it false.

What an attacker actually reaches

The blast radius is what makes this worth acting on this week rather than filing for the next maintenance window. Orca Security, which reported the SSRF, demonstrated three concrete targets reachable from the admission controller's network position:

Cloud metadata at 169.254.169.254. Unauthenticated on IMDSv1. AWS IAM role keys sit at /latest/meta-data/iam/security-credentials/, GCP service-account tokens at /computeMetadata/v1/instance/service-accounts/, Azure managed-identity tokens at /metadata/identity/oauth2/token. Network position alone is the credential.
Cross-namespace internal services. Anything resolvable at *.svc.cluster.local, including kube-system services a tenant namespace should never touch.
Exfiltration through error messages. Orca's proof-of-concept smuggles the HTTP response body back out through the admission webhook's denial message: admission webhook denied the request: SSRF_LEAKED: secret=STOLEN.... The stolen data never needs a separate channel. It rides the API rejection back to the caller.

That last one deserves a beat. Most SSRF write-ups assume the attacker needs a way to see the response. Here the response comes home for free, wrapped in the very rejection that looks, in your audit log, like the policy engine doing its job.

Who's exposed: any multi-tenant cluster that lets teams create their own Kyverno policies in their own namespaces, which is exactly the delegation model the namespaced policy types were built to enable. The people you trusted with only their namespace can now read your cloud credentials. On a young cluster with IMDSv1 still enabled and node roles scoped generously, that is game over, not a finding. Leaked node-role credentials are also the ugly kind to clean up, because you're into a full credential rotation across whatever those roles could touch.

The patched-version story is messier than one number

This is where I'd slow down before typing a helm upgrade. The fixed-version picture is not clean, and reading only the GitHub advisory will leave you short.

The GitHub advisory lists 1.17.0 as the fixed release. Orca reported no clean patched release as of their writeup and demonstrated the exploit on 1.16.2 (Helm chart 3.6.2). And byteiota reports that the complete set of controls, meaning HTTP restricted to cluster-scoped policies by default plus a default blocklist for loopback and metadata ranges, only lands in Kyverno 1.18, released May 5, 2026.

So treat 1.18 as the version that actually carries the full default-off behavior. Do not assume a 1.17.x bump alone hardens the defaults. Maintainer PR #15789 (merged to main April 6, 2026) is what inverts the default: it disables the HTTP library for namespaced policies unless you explicitly opt in with --allowHTTPInNamespacedPolicies, and it puts loopback (127.0.0.0/8) and link-local/metadata (169.254.0.0/16) on a default blocklist. Re-enable HTTP for namespaced policies and you re-open most of the hole.

CVE-2026-22039 has its own patch line. Minimus reports it fixed in Kyverno 1.16.3 and 1.15.3, affecting 1.16.2-and-earlier and 1.15.2-and-earlier. Because it's a different code path, the apiCall context rather than the CEL http lib, patching one does not patch the other. Check both against your running version.

One more number that should shape your urgency. Orca's timeline: reported January 29, 2026; CVE assigned March 24; interim mitigation PR #15729 on March 26; maintainer fix merged April 6. That is roughly a three-month window where the SSRF was live and unpatched. Assume exposure predates your awareness, and check logs accordingly rather than trusting that "we only just heard about it" means "we were only just exposed."

Upgrading fixes the default. It does not fix a malicious author.

Here's the honest framing, and the part that separates a patch from actual safety. The upgrade flips the default so a careless policy author can't wander into the SSRF. It does nothing about a policy author who is actively malicious or whose credentials are compromised, if that author can flip --allowHTTPInNamespacedPolicies back on or exploit a path you haven't patched.

Containment is a different control from prevention, and you want both. A default-deny egress NetworkPolicy on the Kyverno namespace would have starved this exploit even on a fully vulnerable build. It doesn't stop the initial policy from evaluating. It stops the packet from reaching 169.254.169.254, which is the difference between a leaked secret and a logged, blocked request. Network-layer containment is what saves you on the days your version pin is wrong, and given the version story above, some of your clusters are probably pinned wrong right now.

What to do before your next Kyverno upgrade window

Order matters here. Close the CVSS 10.0 first, then the SSRF, then the containment that protects you regardless.

If you're on the 1.15/1.16 line, jump to 1.16.3 or 1.15.3 immediately to close CVE-2026-22039. It's CVSS 10.0 and a different code path from the SSRF, so this is its own step, not a side effect of the bigger upgrade.
Upgrade to Kyverno 1.18 or later for the SSRF default-off behavior plus the metadata/loopback blocklist. Do not stop at 1.17.x and assume the defaults are hardened; byteiota puts the full default behavior in 1.18.
After upgrading, confirm you did not set --allowHTTPInNamespacedPolicies. The flag exists precisely to re-enable the risky behavior. If a team genuinely needs HTTP in a namespaced policy, that's a review conversation, not a running default.
Apply a default-deny egress NetworkPolicy to the Kyverno namespace and explicitly block 169.254.169.254 at the network layer. This is the control that holds even on a vulnerable build.
Enforce IMDSv2 on AWS nodes with http-tokens: required. IMDSv2 demands a PUT-issued token before any credential read, which a naive http.Get() SSRF can't perform. That neutralizes the single highest-value SSRF target on AWS.
Audit who can create namespaced policy objects. Run kubectl get clusterrolebindings,rolebindings -A -o wide and check for create on the Namespaced*Policy CRDs. Everyone on that list held the SSRF primitive on a vulnerable cluster. Scope it to the teams that truly need self-service policy. Tightening this also feeds directly into broader non-human identity governance, since the admission controller's ServiceAccount is exactly the kind of over-trusted workload identity that gets abused here.

While you're in the upgrade, watch one more clock. byteiota notes ClusterPolicy is slated for removal in v1.20 (October 2026) with no automated migration tool. The upgrade you're already doing for this CVE is the natural moment to start the manual split into ValidatingPolicy, MutatingPolicy, and the rest, rather than eating two disruptive Kyverno migrations in one quarter.

Sources

Originally published at indragustiprasetya.com

Rotate API Keys Without Downtime: Dual-Key Pattern

Indra Gusti Prasetya — Fri, 31 Jul 2026 11:20:21 +0000

Almost every team knows it should rotate credentials. The reason it doesn't is fear. The last time someone swapped a key in production, a service lost its auth mid-request and paged the on-call at 2 a.m., so the key now sits there for two years. That stale key becomes the one credential nobody touches, which is exactly the one an attacker keeps.

The fix is the dual-key grace-period pattern: for a short window, both the old and the new credential are valid at once, so consumers roll over on their own schedule and nothing breaks. Simple idea. The trap is that the ordering flips depending on whether you are rotating a bearer credential (an API key, an AWS access key, a webhook secret) or a signing key (a JWT/JWKS private key), and getting that ordering backwards is the single most common way a "safe" rotation still takes down auth. If you are already moving CI off long-lived secrets, pair this with killing static AWS keys in favor of OIDC; rotation is the discipline for the keys you cannot yet delete. And if you own a fleet of service accounts and machine tokens, the same pattern is the backbone of non-human identity governance. These tips are for the people who own the credentials but do not want to own the incident.

The tips

Rotate additively, never in place. Deploy the new key before you touch the old one. The whole pattern rests on one rule: a working credential is never removed before its replacement is proven under real traffic. The sequence is always create, deploy, verify, revoke, never revoke then create. AWS's own IAM rotation guidance follows exactly this five-step order: create the second key, update the app, disable (not delete) the old key, validate, then delete. That "disable first" step is your undo button. A deactivated key reactivates in seconds; a deleted one is gone.
On AWS, treat the two-active-keys limit as the feature it is. IAM caps you at two access keys per user, and that cap is not a nuisance. It is precisely enough to hold an old and a new key side by side during rollover. Run the rotation as CLI so it is scriptable and shows up in CloudTrail:

   aws iam create-access-key --user-name svc-deploy
   # deploy the new key to the workload, confirm traffic, THEN:
   aws iam update-access-key --access-key-id AKIAOLD --status Inactive --user-name svc-deploy
   # wait out the grace period watching for AccessDenied, THEN:
   aws iam delete-access-key --access-key-id AKIAOLD --user-name svc-deploy

If create-access-key fails because two keys already exist, that is a stuck prior rotation. Clean it up before you start a new one.

Before you rotate, find out who is actually using the key. The scary part of rotation is not knowing which consumer will break. AWS answers this directly. aws iam get-access-key-last-used --access-key-id AKIAOLD returns the last-used date, the region, and the service name that called it last. Pull an IAM credential report and read the access_key_x_last_used_date column across every user first. A key whose last-used date is N/A or months old is safe to disable on the spot. You are not rotating it, you are decommissioning a credential nobody uses, which is the cheapest win in the whole exercise.
Size the grace period to your slowest consumer, then make it as short as that allows. The overlap window has to outlast the longest interval any client waits before it picks up the new key: a cron job that runs nightly, a mobile app that only refreshes on launch, a cached config that redeploys weekly. Most provider docs land somewhere between 24 hours and 7 days for bearer keys. Longer is not safer. Every extra hour is an hour a possibly-leaked old key still works. Pick the smallest window that clears your slowest known consumer, then write that number into the runbook so the next person does not guess.
For JWT signing keys, invert the order. Publish the new public key before you sign with it. This is the gotcha that catches teams who assume signing keys behave like API keys. They do not. A verifier fetches your JWKS and caches it. If you start signing tokens with a new kid before that key is published and caches have refreshed, verifiers hit "key not found" and reject perfectly valid tokens. The correct sequence: add the new key to the JWKS endpoint, wait at least one full JWKS cache TTL so every verifier has it, then flip signing to the new key, and keep the old public key published until the last token it signed has expired.

   grace period (JWKS) = JWKS cache TTL  +  max token lifetime  +  safety buffer

Always sign with a kid, and never pull a public key while live tokens still reference it. The kid (key ID) header is what lets a verifier pick the right key out of a JWKS that legitimately holds two keys mid-rotation. Seeing two keys at your JWKS endpoint during overlap is normal and expected, not a bug to "fix." Retire the old public key only once a full max-token-lifetime has passed since you stopped signing with it. Pull it earlier and you invalidate tokens that are still inside their valid window. This is the same care agent identity built on SPIFFE and OAuth depends on when workloads verify each other's tokens.
Let webhook providers hold both secrets for you, and verify against an array. Webhook signing secrets rotate the same way, and good providers automate the overlap for you. Stripe's "Roll secret" keeps the current secret valid for up to 24 hours while the new one is live, signing each event with both during the window. Your endpoint's job is to verify against a list, not a single value, which is what the official SDKs support:

   const secrets = [process.env.STRIPE_WHSEC_NEW, process.env.STRIPE_WHSEC_OLD];
   // try each; accept the event if any signature validates

Deploy with both set, confirm deliveries still succeed, then drop the old one from the environment.

Verify with real traffic, not a health check, before you revoke. "The new key works" has to mean production requests are succeeding on it, not that a synthetic probe returned 200. Watch the actual signal for your credential type: AccessDenied counts in CloudTrail for IAM keys, 401/403 rates at the API gateway for API keys, signature-verification failures in your webhook logs, "key not found" rejections for signing keys. Cut over only when that error line stays flat under load. A green probe against an empty code path has fooled more rotations than any provider bug.
Automate the schedule, but keep a manual break-glass path. Rotation that depends on a human remembering is the rotation that quietly stops happening. Use AWS Config's access-keys-rotated rule (or your secrets manager's built-in rotation) to enforce a max key age of 90 days, and 30 to 60 for sensitive workloads. Keep the manual create/disable/delete runbook current and tested anyway, because the day you actually need to rotate fast is the day a key leaks, and that is the worst possible moment to be debugging your own automation.
Be honest about what rotation does and does not do. Rotation shrinks the window a leaked credential stays useful. It is containment, not detection. It does nothing to tell you a key leaked, and nothing to stop exfiltration while the key is still valid. So pair every rotation policy with a leak alert: secret scanning on your repos, anomalous-use detection on the credential. Treat an unexpected region or service in get-access-key-last-used as a signal to rotate now, off-schedule, not at the next 90-day tick. Rotation limits the blast radius. Something else has to spot the blast.

Wrap-up

If you take one habit from all of this, make it the ordering rule: the new credential is proven in production before the old one is revoked, and for signing keys it is published before it is used. Every zero-downtime rotation is just that rule applied carefully, with a grace period sized to your slowest consumer. Script it into a runbook once, and the credential nobody rotates stops being a category on your infrastructure.

Sources

Originally published at indragustiprasetya.com

Fix npm EALLOWSCRIPTS in project-scoped installs

Indra Gusti Prasetya — Tue, 28 Jul 2026 10:22:21 +0000

You did the responsible thing. You turned on npm's new install-script allowlist so a freshly-published malicious version can't run postinstall code on your build box. Then you cloned a repo, ran npm install, and it died on a flag you never typed:

npm error code EALLOWSCRIPTS
npm error --allow-scripts is not allowed in project-scoped installs.
Add the entries to the "allowScripts" field in package.json, or to .npmrc, instead.

Read that last line twice. It tells you to move a setting to where you already put it. That is not a helpful error, that is a bug talking. Here is what actually broke and how to get a green build back without gutting the control you just added.

What the error really is

The feature is real and it's good. As of npm 11.17.0 and the npm 12.x line, install scripts are gated behind an explicit allowlist: an allowScripts field in package.json, managed with npm approve-scripts. If you shipped through the earlier rollout and hit the "N packages have install scripts not yet covered by allowScripts" warning, that's the same feature. I wrote up that warning and its fix in the npm allowScripts install-scripts guide, and it's worth reading first because this is a different failure with a different cause. That one was a warning nudging you to approve. This one is a hard EALLOWSCRIPTS stop.

EALLOWSCRIPTS almost never comes from your command line. It fires while npm prepares a git dependency: a package pulled from git+https://… or a GitHub shorthand that carries a prepare build step. Per npm/cli issue #9783, npm prepares that dependency by spawning an inner npm install. The outer config gets forwarded to that child as npm_config_* environment variables. The inner process then reads an allow-scripts value that came from your persistent user-level .npmrc, decides it's a forbidden project-scoped CLI policy, and kills the whole install. The value it's complaining about is an environment variable it inherited from itself. Nobody passed --allow-scripts on that command.

So the message is wrong in a specific way. It treats an env-inherited setting as if you typed it as a flag, then tells you to relocate it to a place it's already living. npm's own maintainers list "correct the error message to avoid contradictory guidance" as a minimum fix in #9783. When the tool admits the text is contradictory, stop trusting the text.

Why this bites teams that did the right thing

The trap is that the security guidance builds the failure for you. npm's docs and its warnings point you toward persisting the allowlist so it survives across projects:

npm config set allow-scripts=<pkg> --location=user

Persist it that way, then clone any repo that pulls a package from git with a prepare step, and npm install dies with EALLOWSCRIPTS. You never scoped anything to that project. The config you set globally, on purpose, to be safer, is exactly what leaks into the git-dependency preparation and trips the check.

For a platform or DevSecOps team the shape of this is familiar. A pipeline goes red after a routine npm bump. The error blames a flag nobody in the repo passed. Someone greps the codebase for --allow-scripts, finds nothing, and burns an afternoon. Then the "obvious" fix surfaces: delete the .npmrc entry, watch the build go green, move on. That's the second-order bite. The hardening control does its job on the outer install and breaks the inner one, so the fastest path to green quietly removes a supply-chain defense you added deliberately. It's the same pattern I keep flagging: a control that works right up until it makes someone rip it out. The npm Trusted Publishing analysis is the cousin case, a defense that ships and still leaves the attack path open.

And the exposed population is growing, not shrinking, because the npm 12 install-script lockdown is reaching more teams every week.

The versions are current, not historical

This matters when you're deciding whether to wait it out. #9783 reproduces on npm 11.17.0, 12.0.1, and the main branch as of this writing. It is live. The reproduction is small: point at a git dependency that has a prepare script, set allow-scripts in a user-level .npmrc, then npm install a consumer of it. You don't need an exotic setup. A GitHub-shorthand dependency with a build step is enough.

The fix npm wants is upstream: strip npm_config_allow_scripts from the child environment during git-dependency preparation, and stop conflating env-inherited values with CLI flags. Until that release lands, the workaround is yours to run.

The sibling bug that makes approve-scripts untrustworthy

There's a related inconsistency worth knowing before you lean on npm approve-scripts as your source of truth. In npm/cli #9562 (npm 11.17.0, Node 24.16.0), npm ci --strict-allow-scripts rejected fsevents@2.3.3 with 1 package(s) have install scripts not covered by allowScripts. But npm approve-scripts --allow-scripts-pending never listed fsevents at all, because it's optional: true and os: ["darwin"]. The tool that builds your allowlist and the command that enforces it disagree.

Read what that does to a workflow. You run the approval tool on your Mac, it shows a clean list, you commit package.json, you feel done. Then CI on Linux (or a different OS matrix) rejects an optional dep the approval tool never surfaced. A "clean" approval run is not proof the enforcement pass will accept the same tree on another platform. For platform-specific optional deps, single-machine approval is a false positive.

What approve-scripts can and can't reach

One more boundary. Per the npm docs and issue #9463, npm approve-scripts maintains the allowScripts field inside a project's package.json. Run it against a global install and it errors with EGLOBAL. So for global CLIs and npx, you cannot pre-build a committed allowlist. You approve at install time with install-level flags instead. That's not a bug, it's a scope limit, but it catches people who assume one allowlist mechanism covers every install path. It doesn't. Project-scoped deps get allowScripts in package.json; global tools and npx get a flag on the install command.

If you're already treating fast-moving dependencies as a threat surface, this pairs with dependency cooldowns as a defense against fast supply-chain attacks: the allowlist controls what runs, the cooldown controls when a new version is even eligible to run.

How to get a green build back without losing the control

Work this in order. The first two steps fix the outage; the rest keep you from trading security for green.

When EALLOWSCRIPTS hits a normal npm install, hunt for a git dependency first, not a stray flag. Open package.json and scan every dependency for git+, a raw GitHub URL, or an owner/repo shorthand. The one with a prepare script is where the inner install fires. Grepping your source for --allow-scripts is a dead end; the value is an inherited env var, not a flag in the repo.
Stop persisting allow-scripts in a user or global .npmrc. That is the leak. Per #9783, remove the persistent setting so it can't be forwarded as npm_config_allow_scripts into git-dependency preparation, and pass --allow-scripts only on the specific install that genuinely needs it. Do not delete your allowlist to make the build pass. Those are different things: one is a persisted global env policy, the other is your durable per-project control.
Keep the project-scoped allowScripts field as your real control. It lives in package.json, it's managed by npm approve-scripts, it survives this bug, and it shows up in a pull request where a reviewer can actually see what you allowed. Global .npmrc policy gives you none of that and triggers the failure. Prefer the committed field.
Never trust a single-platform approval for optional deps like fsevents. Cross-check npm approve-scripts --allow-scripts-pending against what npm ci --strict-allow-scripts actually rejects on each target OS, per #9562. If your CI matrix spans macOS and Linux, run the enforcement pass on both before you call the allowlist complete.
For global tools and npx, bake the flag into the install command. approve-scripts returns EGLOBAL there (#9463), so don't expect a persisted allowlist to cover them. Approve at install time.
Pin your npm version and watch #9783. The genuine fix is upstream env-scrubbing. Until a release ships it, treat the workaround above as standing policy and re-test after every npm bump, because a version change is exactly what turns this back on. The same lockdown discipline pays off elsewhere: killing static credentials in CI, for instance, follows the same pattern in GitHub Actions OIDC to AWS.

The thing to hold onto: this error is the control working in the wrong place, not proof the control is wrong. Move the setting off the global env, keep it in package.json, and you get both a green build and the supply-chain gate you turned on in the first place.

Sources

Originally published at indragustiprasetya.com

Why npm Trusted Publishing Didn't Stop the Attack

Indra Gusti Prasetya — Fri, 24 Jul 2026 11:16:45 +0000

Two of 2026's cleanest npm supply-chain compromises both passed provenance verification. Sit with that for a second. The malicious @tanstack/* and @asyncapi/* packages carried valid Sigstore and SLSA attestations, shipped through legitimate OIDC trusted publishing, and returned a green check to anyone who bothered to verify. No stolen npm token. No hijacked maintainer account. I have told readers of this site to adopt trusted publishing, and I still would. It did not stop either attack, and pretending it did is how the next team gets hit.

The signature was real. The source wasn't.

On May 11, 2026, between 19:20 and 19:26 UTC, an attacker published 84 malicious versions across 42 @tanstack/* packages, per the maintainers' own postmortem. Two months later, on July 14 (07:10 to 08:30 UTC), four @asyncapi/* packages shipped five trojanized releases: @asyncapi/generator 3.3.1, generator-helpers 1.1.1, generator-components 0.7.1, and specs 6.11.2, with a combined weekly download count in the low millions, per Chainguard and StepSecurity. Both sets carried attestations pointing at the genuine source repo.

Chainguard wrote the sentence worth taping to your monitor: "Provenance proves where a package came from, not that its source was trustworthy at the time." Verifying the attestation on any poisoned release would have passed. That is the whole problem. Provenance is an authenticity control, not a statement about intent. It is the same failure mode I dug into with A2A agent cards that are signed but lying: a valid signature wrapped around a hostile payload is still, cryptographically, a valid signature.

If your mental model was "provenance verifies, therefore safe," these two incidents just retired it.

pull_request_target is the door both attackers walked through

Here is the part that should worry you more than the malware itself: both compromises opened with the same first move. A pull_request_target workflow that checks out and effectively runs untrusted fork code.

That trigger is the classic "Pwn Request." Unlike a plain pull_request event, pull_request_target runs in the context of the base repo, with the base repo's secrets and a write-scoped GITHUB_TOKEN. The catch is that the code being tested comes from the fork. So the attacker controls the script, and your repo hands that script its secrets. In the AsyncAPI case, per Chainguard and Wiz, that combination let the attacker lift a service account's privileged personal access token and then trigger the real release pipeline from a legitimate position inside the org.

The detail that should make every platform lead wince: Chainguard reports the flaw "had already been flagged internally fifty-eight days before the attack, with a fix sitting unmerged." The vulnerability was known. The PR just sat there. That gap between spotting a CI foot-gun and actually merging the fix is exactly where these attacks live, and it is a management problem as much as a technical one.

Then they read the OIDC token out of runner memory

TanStack's chain went a step further, and it is the part that quietly breaks a lot of assumptions. From the pull_request_target foothold, the attacker used GitHub Actions cache poisoning across the fork-to-base trust boundary, then extracted the OIDC token from the runner process at runtime.

The postmortem is precise about why that works: the runner "mints [the OIDC token] lazily, in memory, when id-token: write is set." So an attacker-controlled build step that runs before publish just reads the token out of the process and publishes directly. The trusted-publisher binding was never broken. It never needed to be. The maintainers are blunt that the attacker "never stole an npm credential, never compromised a maintainer account, never bypassed branch protection, and never broke the OIDC trusted-publisher binding." Every control held. Malware shipped anyway.

This is the honest way to frame what OIDC trusted publishing does and does not do, and it is the same discipline I applied to killing static AWS keys with GitHub Actions OIDC. OIDC removes a stealable long-lived secret and binds publishing to a specific workflow. Real, worth having. What it cannot do is vouch for the contents of the run that mints the token. Influence any step before publish and the token is minted for you, the registry accepts the push, and provenance faithfully records the compromised build. Containment is not prevention. Naming that distinction is the whole game.

The payload waits for import, not install

Most npm supply-chain defenses assume the malware fires during npm install, in a postinstall hook. This one doesn't.

Per Microsoft's write-up, the AsyncAPI payload executes on package import, not on install. That single design choice sidesteps install-script allowlisting entirely, the exact defense I walked through in the npm 12 allowScripts lockdown. Your allowScripts gate never sees it, because there is no install script to gate. The code runs the first time something calls require() or import on the package, which in a dependency of a dependency can be anywhere in your app or build.

From there it pulled a botnet framework (reported as "Miasma") from IPFS, with multi-channel command and control spread across HTTP, Nostr, IPFS, BitTorrent, libp2p, and Ethereum. That spread is deliberate. Block one channel and five remain. The one defense that still bites here is time: dependency cooldowns delay any brand-new version from reaching your build, which buys the hours you need for detection to catch up. Install-script allowlisting buys you nothing against an import-time payload.

So is provenance just theater now?

No, and it is worth answering the objection head-on rather than strawmanning it. Provenance still kills the entire stolen-long-lived-token attack class, which was the most common npm compromise for years. It still gives you a transparency-log audit trail. It still lets you pin trust to a named workflow instead of a person's laptop. StepSecurity's researcher flagged the TanStack packages within roughly 20 to 26 minutes, and public attestations plus transparency logs are part of why anomalies surface that fast.

The mistake is not adopting provenance. The mistake is reading a green attestation as proof the build was benign, then moving supply-chain risk to the bottom of the backlog. TanStack had 2FA, no long-lived publish tokens, and OIDC scoped to a specific workflow and branch. All of it held. The malware shipped through the front door you left propped open in CI.

What to audit before your next release

Do these in order. Each one traces to a specific failure above.

Grep every repo that can publish for pull_request_target and workflow_run. If a workflow with either trigger checks out PR head code and has access to secrets or id-token: write, that is the exact hole both attacks used. Split it: run untrusted build and test on plain pull_request (no secrets), and do privileged publish only on a trusted trigger. Never run fork-controlled code in a job that holds publish scope.
Pin every action to a full commit SHA, not a tag. Floating tags let a re-tagged or compromised action slip a new step into your build. SHA pinning removes that lever. Both postmortems land on this.
Gate the publish job. Use GitHub environment protection with required reviewers on the release environment, set persist-credentials: false on checkout, add repository_owner and branch guards, and grant id-token: write to the single job that publishes, never workflow-wide.
Shrink the OIDC token's window. Because it lives in runner memory during the run, allow no untrusted steps between mint and publish, purge Actions caches that cross the fork-to-base trust boundary, and alert on any publish outside your expected release windows.
On any CI compromise, rotate everything the runner could reach. npm tokens, GitHub tokens, SSH keys, and cloud credentials. The AsyncAPI payload harvested exactly those. Then add default-deny egress controls and watch for outbound IPFS, Nostr, or unexpected C2 from build and application processes, not just postinstall hooks.

The one-line version for your next architecture review: OIDC trusted publishing moved the attacker's target off your npm token and onto your CI workflow. If you hardened the first and left the second full of pull_request_target, you did not close the gap. You relocated it.

Sources

Originally published at indragustiprasetya.com

Block Privileged Pods with a Kyverno ValidatingPolicy

Indra Gusti Prasetya — Tue, 21 Jul 2026 12:57:59 +0000

You will end up with one stable Kyverno ValidatingPolicy that denies any pod carrying a privileged: true container, written against the API that replaced ClusterPolicy in 1.18. The catch that makes this worth a walkthrough: copy an old example onto a current cluster and it can apply cleanly while enforcing nothing.

Every blog, and every AI assistant, hands you a Kyverno ClusterPolicy with validationFailureAction: Enforce. On a current cluster that answer is wrong in three quiet ways. ClusterPolicy was deprecated in Kyverno 1.17 (announced February 2026) with removal already scheduled. The replacement ValidatingPolicy graduated to stable in Kyverno 1.18 (announced May 5, 2026 on the CNCF blog). And the field that turns enforcement on got renamed. Paste a stale snippet onto 1.18 and, in the worst case, it applies with no error and blocks nothing.

Admission control like this stops the object from ever being created. If you also want to kill a privileged process at the kernel after something slips past policy, that is Tetragon's runtime enforcement job, a separate layer. Think of the policy below as a companion to defenses like running pods in user namespaces with hostUsers: false, not a replacement for them.

This tutorial builds the correct thing: a stable ValidatingPolicy (API policies.kyverno.io/v1) that denies privileged pods, covers init and ephemeral containers, and survives a container with no securityContext at all. That null case is the one that makes the "obvious" CEL expression throw or silently pass. It targets platform and security engineers running Kyverno on Kubernetes 1.30+.

Prerequisites

A Kubernetes cluster on 1.30 or newer. The ValidatingPolicy CEL engine builds on the upstream ValidatingAdmissionPolicy machinery that stabilized in 1.30.
Kyverno 1.18+ installed. Older releases only expose ValidatingPolicy under policies.kyverno.io/v1alpha1; the v1 version arrives in 1.18.
kubectl with admin rights, and optionally the Kyverno CLI (kyverno) for offline or CI testing.
Cluster admin. You are registering an admission policy that affects every namespace you match.

Step-by-step

1. Install (or confirm) Kyverno 1.18+

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Now confirm you actually got a 1.18+ image. The app version, not the chart version, is what matters here:

kubectl get deploy -n kyverno kyverno-admission-controller \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

You need a tag of v1.18.x or later. Anything older and the policies.kyverno.io/v1 API below simply won't exist.

2. Confirm the ValidatingPolicy CRD is actually installed

This is the step everyone skips, and it is the cause of the most-Googled failure for this feature. Per Kyverno issue #12441, installing the Helm chart with default CRD settings does not always create the ValidatingPolicy CRD:

kubectl get crd | grep -E 'validatingpolicies.policies.kyverno.io'

If that returns nothing, your kubectl apply in step 4 fails with no matches for kind "ValidatingPolicy" (see Common pitfalls). Reinstall with the policy CRDs enabled in the chart's CRD values before continuing. Do not downgrade the API version to make the error go away.

3. Write the ValidatingPolicy

# disallow-privileged.yaml
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-privileged-containers
spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups:   [""]
        apiVersions: ["v1"]
        operations:  ["CREATE", "UPDATE"]
        resources:   ["pods"]
  variables:
    - name: allContainers
      expression: >-
        object.spec.containers +
        object.spec.?initContainers.orValue([]) +
        object.spec.?ephemeralContainers.orValue([])
  validations:
    - expression: >-
        variables.allContainers.all(c,
          c.?securityContext.?privileged.orValue(false) == false)
      message: "Privileged mode is disallowed. Every container must set securityContext.privileged to false or leave it unset."

Four things here are the whole point of the article.

kind: ValidatingPolicy, not ClusterPolicy. The new type is a superset of the upstream Kubernetes ValidatingAdmissionPolicy, which is why it uses matchConstraints.resourceRules (Kubernetes' shape) instead of Kyverno's old match.any.resources.kinds.

validationActions: [Deny], not validationFailureAction: Enforce. Different field, different value. Deny blocks the request at admission; Audit and Warn are the non-blocking modes. The base policy in Kyverno's library ships as Audit, so change it to Deny or it enforces nothing. This one rename is why so many copied examples pass review and protect nothing.

CEL optional chaining (.? plus .orValue(false)). A container with no securityContext has no privileged field. The naive c.securityContext.privileged == false throws a CEL evaluation error on those pods; c.?securityContext.?privileged.orValue(false) resolves the missing path safely to false. This mirrors the exact expression Kyverno ships in its pod-security-vpol baseline policy.

allContainers merges init and ephemeral containers. Check only spec.containers and a privileged initContainer walks straight in. It is the single most common miss in hand-rolled pod-security rules, and it matters because a privileged container is the shortest path to a container escape.

4. Apply it

kubectl apply -f disallow-privileged.yaml
kubectl get validatingpolicy disallow-privileged-containers

The policy is cluster-scoped and takes effect on the next matching admission request. No restart needed.

5. (Optional) Gate it in CI before it ever reaches the cluster

kyverno apply disallow-privileged.yaml --resource privileged-test.yaml

The Kyverno CLI evaluates the same policy offline, so you can fail a pull request on a privileged manifest instead of discovering it at deploy time.

Verify it works

Try to create a privileged pod:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: privileged-test
spec:
  containers:
    - name: app
      image: nginx:1.27
      securityContext:
        privileged: true
EOF

Expected output, the request rejected at admission:

Error from server: error when creating "STDIN": admission webhook
"vpol.validate.kyverno.svc-fail" denied the request:
Privileged mode is disallowed. Every container must set
securityContext.privileged to false or leave it unset.

Now flip privileged: true to false (or delete the securityContext block) and re-apply. The pod should create normally. Run both halves. The bad manifest blocked and the good one admitted is what proves the Deny action and the null-safe expression are each doing their job. One check without the other tells you nothing.

Common pitfalls

no matches for kind "ValidatingPolicy". The full error reads error: resource mapping not found for name: "disallow-privileged-containers" ... no matches for kind "ValidatingPolicy" in version "policies.kyverno.io/v1". The CRD isn't installed (Kyverno issue #12441) or you're on Kyverno < 1.18. Fix it with step 2, not by blindly changing the API version.
Copying v1alpha1 from the policy library. Kyverno's own published pod-security-vpol pages still carry apiVersion: policies.kyverno.io/v1alpha1 and minversion: 1.14.0. On a 1.18 cluster prefer v1. The v1alpha1 alias may still resolve, but new manifests should target the graduated version so they don't break when the alpha alias is eventually dropped.
Policy applies but nothing is blocked. You left validationActions: [Audit] (the library default) or carried over validationFailureAction: Enforce, which ValidatingPolicy does not read. Only validationActions: [Deny] blocks.
A privileged init or ephemeral container slips through. Your expression iterated only object.spec.containers. Use the allContainers variable that concatenates all three lists.
CEL error on pods with no securityContext. Without .? and orValue, evaluation errors on those pods, and depending on your webhook failurePolicy the request may be allowed through. Keep the optional chaining.
Honest scope. This stops the privileged flag, and only that. It does not cover allowPrivilegeEscalation, added Linux capabilities, hostPID or hostNetwork, or host-path mounts. Each of those needs its own validation. A single "block privileged" rule is not a Pod Security Standards restricted profile, the same way a default-deny egress rule limits exfiltration without stopping the initial compromise. Naming what a control leaves open is the difference between a real defense and a checkbox.

Wrap-up

You now have a stable, CEL-based ValidatingPolicy that denies privileged pods across the cluster, handles the null-securityContext case, and covers init and ephemeral containers. It is written against the API that replaced the deprecated ClusterPolicy, not the one every stale example still shows.

Because ClusterPolicy removal is already on Kyverno's roadmap, treat this as the migration template for your other rules: Deny in place of validationFailureAction: Enforce, matchConstraints.resourceRules in place of match.any, and optional chaining everywhere a field can be absent. The natural next policy is disallowing allowPrivilegeEscalation and unlisted capabilities, which lets you stage a full baseline-to-restricted rollout with the same type.

Sources

Kyverno docs, ValidatingPolicy
CNCF, Announcing Kyverno release 1.18
Kyverno policy library, Disallow Privileged Containers in ValidatingPolicy
Kyverno blog, Announcing Kyverno Release 1.17
Kyverno GitHub, Issue #12441: default Helm CRD values don't create ValidatingPolicy

Originally published at indragustiprasetya.com

Fix MCP OAuth 2.1 Before the July 28 Rewrite

Indra Gusti Prasetya — Tue, 14 Jul 2026 11:27:41 +0000

The Model Context Protocol is about to ship the biggest rewrite of its authorization layer since it launched. Six authorization-hardening SEPs, a stateless protocol core, an Extensions framework, all locked May 21 and publishing July 28, per the Model Context Protocol blog. There is one problem with celebrating it. Almost nobody is running the auth layer it rewrites.

That gap is the actual story. The November 2025 revision already made OAuth 2.1 the mandated model for any HTTP-based MCP server: PKCE, RFC 8707 Resource Indicators, mandatory audience validation, the works. Then an Astrix Security survey cited by Microsoft's App Service team found that only 8.5% of MCP servers use OAuth. The rest lean on static API keys, personal access tokens, or nothing. Microsoft's own writeup puts 25% of public servers at no authentication and 53% on long-lived static keys. So the community is shipping its second major auth revision to an ecosystem where nine in ten servers never adopted the first.

This is not the tool-poisoning problem I've written about before, and it is not the stdio RCE baked into the SDKs. This is the transport-level question underneath both: who is even allowed to call your MCP server, and whether the token they hand you was ever meant for you. It is the least glamorous corner of MCP security. Right now it is also the most neglected.

The spec nobody's running

Read the 2025-11-25 authorization spec and the requirements are not subtle. OAuth 2.1 with PKCE using S256. Clients MUST refuse to proceed if the authorization server does not advertise code_challenge_methods_supported. Servers act as OAuth 2.1 resource servers and MUST validate that a presented token was issued specifically for them as the audience.

Now hold that against 8.5% adoption and 53% on static keys. That is roughly an eleven-to-one gap between what the spec demands and what the field runs. And it is not a gap you close by reading more spec. A server that authenticates with a shared API key cannot do audience binding at all, because there is no audience to bind. The token is a bearer secret, full stop. Whoever holds it is trusted.

I keep coming back to the 25% figure, the servers with no authentication whatsoever. In any other part of infrastructure that number would be a scandal. Here it barely registers, because MCP servers still feel like dev toys to the people standing them up, right up until one of them is wired to a production GitHub org.

Why audience binding is the whole game

MCP servers occupy the worst possible seat for a confused-deputy attack. They hold credentials to real backends (GitHub, databases, cloud APIs) and they take instructions from an LLM that will faithfully relay whatever a web page, an email, or an upstream agent tells it to. The model does not know it is being manipulated. The server does not know the model was.

The spec draws the line exactly where it should. A presented token MUST have been issued for this server as its audience, and the server MUST NOT pass a client's token through to an upstream API. Skip the first check and any valid token from a neighboring service becomes a skeleton key: a token minted for service A sails into your server, which never asks who it was for, and now A's caller is acting as you. This is the same class of blast-radius problem that pushed teams toward per-workload identity with SPIFFE and OAuth instead of one shared key, and it is why static-key MCP servers are structurally unfixable without moving to OAuth first.

Here is the honest boundary, though, and the spec is refreshingly clear about it. The no-passthrough rule and audience validation stop token reuse across services. They do not stop a compromised server from acting on a request that was legitimately scoped to it. Audience binding is containment, not prevention. It turns a stolen neighbor-service token from a skeleton key into a 401. It does not make your server safe to point at prompt-injected input. Anyone selling it as the latter is overselling.

What the July 28 SEPs actually change

The July 28 hardening is real work by capable people, and it is refinement on a foundation most servers have not poured yet.

SEP-2468 requires validating the iss parameter per RFC 9207 to close authorization-server mix-up attacks. SEP-2352 binds credentials to the issuing authorization server, so migrating servers forces re-registration rather than silently trusting a new issuer. SEP-2350 and SEP-2351 clarify scope accumulation and discovery. Every one of these assumes you already speak OAuth. For the 53% on static keys, they are irrelevant until the far bigger jump happens first.

The release candidate also breaks things, and this is where the calendar gets sharp. A stateless protocol core. A 12-month deprecation window for what it replaces. Tier 1 SDKs expected to ship support inside a 10-week validation window. If you pin an SDK to the RC without reading the migration notes, credential re-binding on migration and the stateless core will surprise you at the worst time. This is a version-pinning decision, and it deserves to be made on purpose, the same discipline that keeps a Cosign or supply-chain toolchain from breaking on a point release.

The resource parameter almost everyone omits

If you read one line of the spec, make it this one. MCP clients MUST send the RFC 8707 resource parameter, the canonical server URI (for example resource=https%3A%2F%2Fmcp.example.com), in both authorization and token requests, regardless of whether the authorization server supports it. That single parameter is what lets a server verify a token's audience later. It is also the exact line most homegrown clients leave out, because everything appears to work fine without it. It appears to work because nothing is checking. The day a server starts enforcing audience, the clients that never sent resource break, and the clients that did keep working. Cheap insurance, skipped for no reason but inattention.

The enterprise trap: ID-JAG without audience checks

The part enterprises actually want is the centralized model. Per InfoQ, MCP is adding enterprise-managed authorization built on the Identity Assertion JWT Authorization Grant (ID-JAG), letting an organization broker one login across every connected MCP server instead of each server minting its own trust. One identity plane, real revocation, audit in one place. That is the pattern security teams have been asking for.

It also collapses the instant a downstream server skips audience validation. A centrally issued assertion is precisely the kind of token that gets replayed across resources that never check whom it was for. Roll out ID-JAG across servers where some fraction still do not validate audience, and you have not built single sign-on. You have built single-point-of-token-replay with a nicer dashboard. The centralized model raises the value of the check at the exact moment most servers are not doing it.

Client ID Metadata Documents and the SSRF they invite

The RC changes how strangers establish trust, too. For clients and servers with no prior relationship, the new default is a Client ID Metadata Document: an HTTPS-URL client_id that points to a JSON metadata document, with Dynamic Client Registration demoted to backwards-compatibility. It kills a lot of fragile registration dances.

To its credit, the spec names the new risk instead of hiding it. An authorization server that fetches those client_id URLs is now making outbound requests to attacker-influenced addresses, which is a textbook SSRF surface. CIMDs also cannot, on their own, prevent localhost redirect impersonation. So the mechanism that simplifies trust bootstrapping hands the authorization server a fetch it has to sandbox. If you operate one, treat CIMD URL fetching like any other server-side request to untrusted input: allowlist, block internal ranges, cap redirects.

What to check before July 28

Work these in order. Each ties to a specific number or SEP above.

Find out if you are even in the 8.5%. If your server authenticates with a shared API key or a PAT, the July 28 SEPs do not apply to you yet. Your first move is OAuth 2.1, not SEP-2468. Treat 8.5% as your baseline honesty check, and treat the 25%-no-auth figure as the deadline you have already missed.
Make audience validation a hard gate today. Confirm your server rejects any token whose audience claim is not its own canonical URI, and confirm it never forwards a client token upstream (it should obtain its own token as an OAuth client). This is the one control that turns a stolen neighbor-service token into a 401 instead of a breach.
Add the resource parameter to every client now. It is required by the 2025-11-25 spec, it costs one line, and it is the prerequisite for audience binding to mean anything. Servers that do not enforce it yet will ignore it. Servers that do will start protecting you the moment they flip enforcement on.
Decide your SDK pin deliberately. With a 10-week Tier 1 window and breaking changes (stateless core, credential re-binding on migration), choose whether you adopt inside the window or wait for the July 28 final. If you run a proxy server with static client IDs, budget for the per-client consent the spec now demands, or you reopen the confused-deputy path you just closed.
If you are rolling out ID-JAG, gate it on downstream audience checks. Do not enable centralized authorization until every server behind it validates audience. Ship them together, or the assertion becomes a replayable master key. Static-key servers that route OIDC or brokered tokens should move to short-lived, workload-scoped credentials before they see a single ID-JAG token.

The July 28 rewrite is good work. It is also aimed at a version of the ecosystem that mostly does not exist yet. Fix the foundation the 8.5% number exposes, and the new SEPs become the upgrade they were meant to be. Skip it, and you are hardening a door on a house with no walls.

Sources

Originally published at indragustiprasetya.com

Trivy vs Grype 2026: Pick by the Job, Not Speed

Indra Gusti Prasetya — Fri, 10 Jul 2026 19:02:00 +0000

Most of the pages ranking for "Trivy vs Grype" answer the wrong question. They line the two tools up on a single-image speed benchmark, declare Grype 30 to 40 percent faster, and call that a decision. It isn't. These are not two builds of the same product. They do two different jobs, and swapping one in for the other is exactly how you end up with a gap in your pipeline that nobody notices until an auditor does.

And here's the part the benchmark posts skip entirely: in the first quarter of 2026, both tools had a defining incident. One went blind. The other got backdoored. Neither shows up in a feature matrix, and both change how you should be running these scanners right now.

They are not competitors, they are different jobs

Trivy, from Aqua Security, is the all-in-one DevSecOps scanner. One binary covers container images, filesystems, git repos, VM images, and live Kubernetes clusters, plus IaC misconfiguration checks, secret scanning, and license scanning. You point it at almost anything and it tells you what's wrong.

Grype, from Anchore, does one thing. It matches packages against known CVEs. No IaC, no secrets, no license compliance. It is the vulnerability-matching half of a pipeline, and its other half is Syft, Anchore's SBOM generator. Syft builds the bill of materials, Grype scores it.

So the real question was never "which is faster." It's "what job am I filling." If you want a single tool for CVEs and misconfig and secrets and cluster posture, that's Trivy. If you're building an SBOM-first supply chain and want a clean, auditable matcher on top of a generated bill of materials, that's Syft plus Grype. Choosing Grype to cover Trivy's IaC scanning, or Trivy to replace a dedicated SBOM flow, means quietly dropping capability you may be assuming you have.

The speed gap is real and mostly a distraction

Multiple 2026 comparisons do put Grype ahead: roughly 30 to 40 percent faster on pure vulnerability matching, around 0.7 seconds versus 1.2 seconds per scan with a warm cache. Fine. But on a cold first scan of a 500 MB image, both land near 8 to 9 seconds, and in a pipeline that already burns minutes building and pushing layers, half a second of matcher time decides nothing.

I've never once had a platform team come to me and say the image scan was the bottleneck in their build. The bottleneck is the base image download, the layer export, the registry round-trip. Picking your security scanner on a 0.5s delta is optimizing the wrong number. If speed genuinely matters to you, it's usually because you're running the scan hundreds of times an hour, and at that point cache behavior and DB freshness matter far more than a warm-cache micro-benchmark.

Grype quietly went blind on March 6

Here's the incident nobody benchmarks. Per Anchore's community announcement, Grype DB schema v5 reached end of life on March 6, 2026, and publishing of v5 updates was disabled around March 9. Any Grype older than v0.88.0 stopped receiving new vulnerability data on that date.

Read that carefully, because the failure mode is nasty. The old Grype doesn't error. It keeps running, keeps exiting zero, keeps printing findings against a frozen database. A pinned-and-forgotten Grype baked into a CI image three quarters ago is now a scanner that cannot see a single CVE disclosed after early March, and your build stays green the whole time. That's the worst kind of security control: one that reports success while doing nothing.

The upgrade is not optional. Grype on schema v6 also ships something genuinely useful, which softens the pill.

What v6 actually buys you: EPSS and KEV in the database

Per Anchore's schema write-up, DB v6 embeds the CISA KEV catalog and EPSS scores directly in the database. So the output now carries an EPSS score like 0.97112 at percentile: 0.9989 sitting next to the CVSS severity. That's the difference between "this is rated High" and "this has a 97 percent modeled probability of exploitation and is already in the known-exploited catalog." One of those you can triage by. The other is a label.

The v6 archive also dropped from 210 MB to 65 MB compressed, a 69 percent cut. If you run air-gapped scanners or bandwidth-constrained runners that pull the DB on every job, that's a real operational win, not a footnote.

The scanner that finds compromised software got compromised

Now the Trivy side, and it's a darker story. Per Aqua Security's own incident write-up (GitHub discussion #10425), on March 19, 2026 a threat actor used a compromised credential to publish malicious versions of Trivy. The bad v0.69.4 went out across GHCR, ECR Public, Docker Hub, deb, rpm, and get.trivy.dev, and the latest tag pointed at it for roughly three hours, 18:22 to about 21:42 UTC. The trivy-action GitHub Action was poisoned for about twelve hours, setup-trivy for about four. Malicious v0.69.5 and v0.69.6 followed on Docker Hub on March 22 to 23.

The detail that should bother you most is the cause. Aqua notes this followed incomplete containment of an earlier March 1 incident: they had rotated secrets, but "the process wasn't atomic and attackers may have been privy to refreshed tokens." A partial rotation left a window, and the window got used. That is a textbook supply-chain lesson, and it landed on a supply-chain tool.

Aqua also paused updates to vuln-list, trivy-db, and trivy-java-db during the investigation, then restored them. So if your pipeline pulled the DB inside that window, don't assume it caught up on its own. Confirm it refreshed.

The honest limit: neither tool would have caught this

This is where I want to be blunt, because the marketing around image scanning encourages a dangerous assumption. Neither Trivy nor Grype would have flagged the poisoned Trivy binary. A backdoored release is not a package with a published CVE. Image scanning is disclosed-vulnerability detection, full stop. It tells you which known CVEs affect your packages. It does not prove the binary you pulled is the one the maintainer built.

Treating a scanner as build-provenance verification is the gap that Q1 2026 exposed. The tool you install to catch compromise is itself a dependency at your most sensitive control point, and if you pull it as latest, it's an unpinned one. The fix for compromised software was never the vulnerability scanner. It's signature and provenance verification, sitting alongside the scanner, doing the job the scanner was never built to do.

Running both scanners is a legitimate strategy in regulated shops: two independent matchers with different databases catch more than either alone. But be honest about the cost. After Q1 2026, keeping both toolchains current, both DBs fresh, both binaries pinned and verified, is real ongoing work. Two half-maintained scanners are worse than one you actually keep up.

What to fix this week

Pick by job first, then harden the runtime. Concrete steps, in the order they bite:

Upgrade every Grype to at least v0.88.0 and confirm schema v6. Anything older went blind on March 6, 2026 and is scanning against a dead database while exiting zero. Grep your CI images for pinned Grype versions today.
Add a DB-freshness gate. Fail the build if the vulnerability database is older than a threshold you set (a day or two for a matcher that ships daily). A frozen DB must never pass silently. This is the single check that would have caught the March 6 blind spot on its own.
Pin Trivy to a known-good version. Aqua flags v0.69.3 as the safe immutable release around the incident. Pin trivy-action and setup-trivy to safe tags too, not floating ones. latest at a security control point is a liability.
Verify the binary you install. Check the checksum or the Cosign signature on the Trivy you pull, in the same job that installs it. If you pulled anything in the March 19 to 23 window, re-verify against a clean release.
Confirm your Trivy DB refreshed after the incident pause. Updates to trivy-db and friends were suspended during the investigation. If you cached the DB then, force a refresh.
Pair scanning with provenance. Add signature and SBOM-attestation verification next to the scan. Q1 2026 proved the scanner is part of your attack surface, and no CVE matcher will ever tell you the tool itself was swapped.

Choose Trivy or Grype on the job you need covered. Then treat both as what they are: dependencies you pin, verify, and keep fresh, because the quarter just showed you what happens when you don't.

Sources

Originally published at indragustiprasetya.com

GitHub Actions OIDC to AWS: 10 Tips to Kill Static Keys

Indra Gusti Prasetya — Tue, 07 Jul 2026 12:48:27 +0000

An AWS_SECRET_ACCESS_KEY living in GitHub Actions secrets is a long-lived credential that never rotates, gets exfiltrated by one poisoned dependency, and shows up in nobody's audit trail. OIDC federation deletes it. GitHub mints a short-lived JWT per run, AWS STS trades that token for temporary credentials, and there is no static key left to steal. Simple thesis, sharp edges. One of those edges changes on July 15, 2026, and a trust policy that looks correct today will quietly stop matching. These tips are for the engineer wiring this into a real repo who wants it scoped right the first time, not lifted from a tutorial that grants repo:org/* and calls the job done.

The tips

Register the OIDC provider once per account, and drop the thumbprint. You add token.actions.githubusercontent.com as an IAM OIDC provider once in each AWS account. Per the AWS IAM docs, the thumbprint is now optional: IAM validates GitHub's JWKS endpoint against its own library of trusted root CAs and only falls back to a configured thumbprint when the cert is not signed by a trusted CA. So delete that SHA1 string you copied from a 2022 blog post. It does nothing now.

aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com

The first thing that breaks is a missing id-token: write. Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity is the single most-reported failure on the configure-aws-credentials repo (issues #318, #961, #1137). Half the time the trust policy is fine and GitHub never minted a token at all, because the job lacked permission to request one. Set it at job scope, not just as a workflow default that a matrix job can shadow.

permissions:
  id-token: write   # required to mint the OIDC token
  contents: read

Pin the audience to sts.amazonaws.com. The official action requests aud: sts.amazonaws.com, so your trust policy asserts it with StringEquals. GitHub's own guidance is blunt about the stakes: you must define at least one condition, or any repository on GitHub can request a token that assumes your role. The audience is not that protective condition. It is table stakes.
Pin the sub claim to a branch or environment, never a wildcard. StringLike with repo:octo-org/octo-repo:* hands the role to every branch, every PR, and every fork's merge ref. For anything that touches production, use StringEquals on an exact sub. The formats you actually need: ref:refs/heads/main for a branch, ref:refs/tags/v1.2.3 for a tag, environment:prod for a deployment environment, pull_request for PR runs.

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:octo-org/octo-repo:ref:refs/heads/main"
  }
}

Prepare for the July 15, 2026 immutable subject-claim change now. Per GitHub's April 23, 2026 changelog, the default sub claim is gaining immutable owner and repo IDs. Any repository created, renamed, or transferred after July 15, 2026 will mint a sub shaped like repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main. A policy pinned to the old name-only string stops matching the moment someone renames the repo, with no error until the deploy fails. If your sub conditions are exact-match on names, audit them before the deadline. This is exactly the gotcha a model trained before April 2026 gets wrong.
Pin repository_id, not the name, to kill the rename-squat confused deputy. The reason behind the immutable change is real, and worth understanding rather than just working around. Delete or rename a repo, and someone can register a new repo that reclaims the freed name and mint a token whose name-based sub still satisfies your trust policy. GitHub exposes a stable numeric repository_id that never gets recycled. Add it as a second condition: identity lives in the immutable ID, branch scoping stays in sub.

"StringEquals": {
  "token.actions.githubusercontent.com:repository_id": "456789",
  "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
}

Pin the action to a full 40-character commit SHA. configure-aws-credentials is at v6.1.0 as of this writing (v5.1.1 shipped 2025-11-24). A floating @v4 tag can be re-pointed by anyone who compromises the tag, and this action handles your credentials. It is the last place you want a mutable reference. Pin the SHA, leave the human-readable version in a comment, and do the same for every third-party action in the credential path.

- uses: aws-actions/configure-aws-credentials@<full-40-char-sha>  # v6.1.0
  with:
    role-to-assume: arn:aws:iam::123456789012:role/github-deploy
    aws-region: us-east-1
    role-session-name: gha-${{ github.run_id }}

Set role-session-name so CloudTrail can tell you which run did what. Skip it, and every assumed session looks identical in CloudTrail. Stamp the run ID or repo into the session name and it becomes an audit trail: an unexpected S3 write traces straight back to one workflow run. Pair it with a short role-duration-seconds. The default is one hour, but a deploy job that finishes in three minutes has no business holding a token for sixty. Drop it to 900 and a leaked token expires before anyone can use it.
One narrow role per repo-and-environment, least privilege on the permissions policy. The trust policy decides who can assume the role. The permissions policy decides what they can do, and this is where most setups quietly over-grant. Do not attach AdministratorAccess to a deploy role, ever. Scope it to the exact actions and resource ARNs the job touches, and keep separate roles for prod and staging so a staging workflow can never reach production. Gate the prod role behind a GitHub Environment with required reviewers, then pin its sub to environment:prod.
When it fails, decode the token instead of guessing. The AccessDenied message never tells you which claim mismatched, which is maddening the first time and routine after that. Print the actual JWT payload from inside the failing job and compare it byte for byte against your trust policy conditions.

curl -sH "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
  "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" \
  | jq -r '.value' | cut -d. -f2 | base64 -d 2>/dev/null | jq .

Reusable workflows hide a trap here: the sub reflects the calling repo, not the reusable one. If you need to trust the reusable workflow itself, condition on the job_workflow_ref claim instead of sub, or the match never fires and you will stare at the trust policy for an hour wondering why.

Wrap-up

If you do one thing, do this: define at least one exact-match condition on both aud and an identity claim, and make that identity claim the immutable repository_id, not the repo name. That single habit buys you least-privilege scoping today, survives the July 15, 2026 immutable-claim rollout, and closes the rename-squatting hole the change exists to fix. The static key you delete afterward is the entire point. A credential that does not exist cannot leak.

Sources

Originally published at indragustiprasetya.com

Kubernetes Default-Deny Egress Stops Pod Exfiltration

Indra Gusti Prasetya — Fri, 03 Jul 2026 12:54:03 +0000

Every Kubernetes cluster ships with the same quiet default: a pod can reach anything on the internet the second it starts. Ingress gets all the scrutiny, because that is where an attacker knocks. Egress, the traffic leaving your pods, is treated as plumbing. That treatment is the bug.

The default nobody audits

The upstream NetworkPolicy documentation is blunt about it. A pod is non-isolated for egress until some policy selects it, and until that happens every outbound connection is allowed. No policy, no limit.

Play that out. One pod, freshly scheduled, can open a socket to your database, to the cloud metadata endpoint at 169.254.169.254, to your secrets store, and to any host on the public internet, with nothing standing in the path. Most teams write an ingress policy, feel covered, and never notice that the return direction is wide open. Ingress tells you who may reach the pod. It says nothing about where the pod may reach.

Why the 2026 worms need that open door

The supply-chain worms this year made the cost concrete, and they did it without a single clever exploit. The Shai-Hulud family and the copycats trailing it arrive as a trusted dependency. An npm or PyPI package you already pull. It runs at install time, inside your build pod, harvests whatever credentials are in reach, and then opens an outbound connection to carry them out.

Per Microsoft's May 20, 2026 writeup of the Mini Shai-Hulud variant that hit the @antv packages, the payload scraped CI/CD credentials off the runner and exfiltrated them, then propagated through publishing workflows. Datadog Security Labs documented the 2.0 variant going further: multi-platform credential theft spanning GitHub, AWS, Vault, npm, Kubernetes, and 1Password, GitHub Action runner memory scraping, and dual-channel exfiltration that included writes to public GitHub dead-drop repositories.

Look at the shape of it. Code execution is the entry. But the theft only becomes a breach at the moment of exfiltration, when the payload dials out. And that dial-out runs, by default, completely unblocked.

Containment, not prevention. Say it out loud.

Default-deny egress will not stop the malicious package from executing. It will not stop the token from being read out of memory. Anyone who sells it as prevention is lying to you.

What it does is turn "compromised pod" into "compromised pod that cannot phone home." It shrinks the blast radius. The stolen token exists, sitting in a process, with nowhere to go. That is a containment control, and containment is worth a great deal when prevention has already failed at the dependency layer, which is exactly where these worms operate. Honest framing matters here because the wrong framing gets the control ripped out the first time someone points out it "didn't stop the malware."

DNS is the first thing you break

Here is where most rollouts die. You apply default-deny egress, and within seconds pods cannot resolve names. Nearly everything fails, and it fails in a way that is genuinely miserable to debug, because the app logs say "connection timeout," not "your NetworkPolicy ate my DNS lookup."

The deny itself is two lines of intent:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: build
spec:
  podSelector: {}
  policyTypes:
  - Egress

An empty podSelector: {} selects every pod in the namespace, policyTypes: [Egress] isolates them for outbound, and the absence of any egress: rule means nothing is permitted. Everything out is denied.

That is why the DNS allow has to ship in the same change, never as a follow-up:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: build
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Deploy the deny and the DNS allow together, in one commit. Split them across two commits and you buy yourself a phantom outage and a panicked revert.

Native NetworkPolicy cannot match a domain name

Once DNS works, the next wall is real and it is a limitation, not a preference. The built-in policy ipBlock selector matches CIDR ranges only. There is no FQDN support in upstream NetworkPolicy. None.

So if the honest egress list for your build pod is "must reach api.github.com and one S3 bucket," you cannot express that in native policy. You are reduced to enumerating IP ranges that GitHub and AWS rotate without telling you. That does not scale, and a stale CIDR allowlist fails in both directions: it blocks legitimate traffic when the ranges shift, and it silently permits whatever new tenant moved into an old range.

This is the point where the CNI stops being an implementation detail and becomes part of the security control. Cilium's DNS-aware policy runs a DNS proxy that watches lookups and programs egress rules for the resolved addresses, so you write intent instead of arithmetic:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-github-egress
  namespace: build
spec:
  endpointSelector: {}
  egress:
  - toFQDNs:
    - matchName: "api.github.com"

Calico offers equivalent domain-based egress rules. If your CNI cannot do FQDN egress, that is now a security decision you are making, not just a networking one, and it should be on the record as such.

The strongest objection, and the answer

The best counterargument against all of this is real, so I will state it properly rather than knock down a strawman. A determined attacker can tunnel exfiltration through a domain you have already allowlisted. GitHub, for instance. And the Shai-Hulud 2.0 dead-drop-to-public-repo channel is precisely that: the data leaves through github.com, a host your build pod is supposed to reach.

True. It does not stop that path. But "does not stop everything" is not "does not help," and the gap between those two is the whole value. Forcing exfiltration through a narrow, logged, allowlisted set of destinations does two things at once. It kills the long tail of arbitrary attacker-controlled endpoints outright, and it converts the remaining attempts into something you can alert on. A control that turns invisible theft into a policy-violation log entry has earned its place, even though it cannot cover the case where the thief hides inside your own approved traffic.

Where to start, on your build namespaces first

Do these in order. Each step ties to something specific above.

Pick one build namespace in staging, not the whole cluster. Egress breaks apps in ways ingress never does, so apply default-deny to a single workload's namespace and let it soak for a few days before you touch anything production-shaped.
Ship the deny and the DNS allow in the same commit. Your first two objects are default-deny-egress and allow-dns-egress on UDP/TCP 53 to kube-system. Deploy them as one change. Separating them is the single most common reason a rollout gets reverted in a panic.
Lock down the pods that run untrusted code before your app pods. CI runners, dependency-update bots like Renovate, and AI-agent sandboxes execute third-party code on every build, and their legitimate egress list is short, which makes them both the highest-value target and the easiest to scope. Start there.
Reach for FQDN policy the moment you need a real external service. Use Cilium toFQDNs or Calico domain rules instead of hand-built CIDR lists. If your CNI cannot do it, record that as an accepted security gap, do not paper over it with a static IP allowlist that will rot.
Alert on the deny, do not just enforce it. Set the trigger concretely: a pod that has never made an outbound connection suddenly generating egress denies to an unknown host is a possible install-time payload. Pull it for inspection. That denied connection is your earliest and cheapest signal that a dependency went bad, and it costs you nothing to watch for it once the policy is live.

Egress default-deny is not new and it is not clever. It is the boring control that the 2026 worms quietly bet you never enabled. Prove them wrong on your build namespaces, this week, and work outward from there.

Sources

Originally published at indragustiprasetya.com

Falco vs Tetragon vs Tracee: Pick the Right One

Indra Gusti Prasetya — Tue, 30 Jun 2026 10:44:37 +0000

Three eBPF projects land on every Kubernetes runtime security shortlist in 2026: Falco, Tetragon, and Tracee. They get compared as if they were three brands of the same product. They are not. They answer three different questions, and any benchmark that ranks them on one axis is measuring the wrong thing.

The expensive mistake is treating them as interchangeable. A team runs Falco for a year, gets annoyed that it "only alerts," rips it out for Tetragon "because Tetragon can block," and discovers six weeks later that it has traded a paging problem for an outage problem. Those are not the same risk decision. Let me separate the three before anyone reaches for a feature matrix.

Falco answers: did something suspicious just happen, and who do I page?

Falco is a detection engine. A rules engine matches syscall events and Kubernetes audit events against a ruleset, and routes the hits, to Slack, PagerDuty, or a SIEM, through Falcosidekick. That is the whole shape of it: see the event, match the rule, fire the alert.

It is also the most mature of the three by a clear margin. Falco graduated within the CNCF on February 29, 2024, the only one of these three that is a graduated CNCF project. It was created by Sysdig in 2016 and was the first runtime security project to enter the CNCF Sandbox, in 2018. That lineage matters less for raw capability and more for the boring things that decide procurement: governance sign-off, a credible expectation of long-term maintenance, and a security team's willingness to bet a control on it.

One thing people miss: Falco does not block by default. Its job ends at the alert. If you want it to stop something, you wire that alert into an admission controller or a response action yourself. The detection is the product. The response is your integration.

Tetragon answers: can I stop this in the kernel before it completes?

Tetragon is the enforcement tool. It is built as part of the Cilium project, originally by Isovalent (now part of Cisco), and its differentiator is in-kernel enforcement with no userspace round trip.

The mechanism is worth understanding because it explains both the appeal and the danger. Per Tetragon's enforcement documentation, it stops an action by overriding the return value of a kernel function and sending a signal such as SIGKILL to the offending process. It does this by attaching eBPF programs to kprobes, tracepoints, and LSM hooks. So it can kill a process or fail a syscall before that syscall returns. There is no "alert fires, controller reacts, maybe we catch it" gap. The action just does not complete.

That is genuinely powerful. It is also the riskiest mode you can run in production, which is the part the marketing buries.

The inversion nobody puts on the slide

In-kernel enforcement sounds strictly better than alerting. It is not. It is more dangerous, and the danger scales with exactly the feature people buy it for.

Think about what a false positive costs in each tool. A false positive in Falco pages an engineer who looks at it, sighs, and tunes the rule. Annoying. Survivable. A false positive in Tetragon enforcement mode sends SIGKILL to a legitimate process in your hot path. Now a tuning error is an outage, and it happened inline, in the kernel, before anything in userspace could second-guess it.

Detection is forgiving of imperfect rules. Enforcement is not. The capability everyone reaches for first is the one that demands the most rule maturity before you can trust it with a kill signal. Pick your tools on a CPU-overhead benchmark and you have optimized the cheapest variable while ignoring the one that actually bites: how many hours of tuning each mode needs before it is safe to leave on.

Tracee answers: what exactly happened, so I can reconstruct it?

Tracee is the forensics tool. Built by Aqua Security's research team (Nautilus), it pairs a deep event collector, tracee-ebpf, with a signatures engine, tracee-rules. Its design point is event depth. It captures far more context per event than the other two, which is precisely what you want at 2 a.m. when you are rebuilding an attack timeline or working out how far a supply chain compromise reached.

There is a tension baked into that depth, and it is honest to name it. Per-event richness is the reason Tracee is valuable for incident response, and it is also why deep tracing carries more overhead than aggressive in-kernel filtering. You pay for the detail. The trick is to pay for it on the hosts where reconstruction actually matters, not fleet-wide at full verbosity on nodes you will never forensically examine.

The constraint that decides it before features do: kernel version

Before any feature comparison, inventory your kernels. This narrows the field faster than any capability checklist.

Tetragon documents testing on LTS kernels back to 4.19. Falco retains a kernel-module fallback for older kernels where modern eBPF features, BTF and CO-RE on 5.8 and up, are not available. If you run a fleet with mixed or genuinely old kernels, or a module-restricted environment, that fact alone may eliminate options before you ever open a feature matrix. I have watched a careful tool selection collapse the moment someone finally ran uname -r across the actual fleet instead of the shiny new nodes. Do that first.

The honest counterpoint: isn't one tool simpler to run?

The strongest objection to all of this is operational: three tools is three sets of rules, three upgrade cycles, three things to keep tuned. Real cost. So why not consolidate?

Because the jobs do not actually overlap enough to collapse. Falco gives you broad detection coverage and routing. Tetragon gives you surgical kill-it-now enforcement on a few paths you understand cold. Tracee gives you the depth to investigate after the fact. Force one tool to do all three and you get a detection layer you are afraid to put in blocking mode, or an enforcement tool you are running at forensic verbosity and paying for it in overhead. The consolidation saves operational surface and loses the thing each tool was good at. The common production shape is hybrid for a reason: broad detection, narrow enforcement, on-demand forensics, each scoped so you are not running every rule at full depth on every node.

How to choose

Start from the question, not the tool. Then work down this list.

Write down the question first. Need to know and route alerts: Falco. Need to stop a specific known-bad action in the kernel: Tetragon. Need to reconstruct an incident in detail: Tracee. A "which is best" question with no stated job is unanswerable, and vendors love that ambiguity.
Check kernel versions before features. Run uname -r across the real fleet. Any nodes below 5.8, or a module-restricted environment, and your choice is decided: Falco's module fallback or Tetragon's 4.19 support may be your only viable paths. This gate comes before everything else.
If you can only run one, run detection. A tool that reliably tells you what happened is more valuable, and far less dangerous, than one configured to act on rules you have not earned trust in yet. Start with Falco. Add enforcement after the detection layer is quiet.
Never deploy enforcement on day one. Run every Tetragon policy in observe mode first. Confirm zero false positives on the target path across a real traffic window (include a deploy, a batch job, a traffic spike, whatever your hot path actually does). Only then enable the SIGKILL action, and only for that one path. Promote path by path: a known cryptominer binary, a container-escape primitive, an unexpected write to a credential path. Never enable a ruleset-wide kill at once. Treat an enforcement policy the way you treat a firewall deny rule in prod.
Scope Tracee to where it pays. Full event depth on bastion hosts, CI runners, and crown-jewel workloads. Not full verbosity cluster-wide, or you are buying forensic detail on hosts you will never investigate.
Don't pick on the overhead benchmark. The number that decides safety is tuning hours to trust, not CPU percent. Budget the tuning time explicitly before you commit to a blocking mode.

The short version: these three are not three tiers of one product. They are three jobs. Match the tool to the job, gate on your kernels, and earn your way into enforcement instead of starting there.

Sources

Falco Graduates within the CNCF: https://falco.org/blog/falco-graduation/
Tetragon Enforcement documentation: https://tetragon.io/docs/concepts/enforcement/
Tracee (Aqua Security) repository: https://github.com/aquasecurity/tracee
Tetragon (Cilium) repository and releases: https://github.com/cilium/tetragon

Originally published at indragustiprasetya.com

Cosign v3: Sign and Verify Images, Fix Harbor Breaks

Indra Gusti Prasetya — Mon, 29 Jun 2026 07:55:18 +0000

By the end of this you will have a container image signed with Cosign v3, a passing verification both with a key pair and keyless from GitHub Actions, and the two fallback flags that keep you working when your registry or admission controller hasn't caught up to the v3 defaults.

Here is the thing the release notes bury. Cosign v3 shipped on October 8, 2025 (per the Sigstore blog), and the upgrade flips three opt-in flags to on-by-default at the same time: --new-bundle-format, --trusted-root, and --use-signing-config. The pitch is "fewer flags, one standardized format." The cost is that your signatures now land in your registry in a shape older tooling cannot see, and a cosign verify call that passed last quarter can fail, or hang, against an air-gapped registry.

This walks the full sign-and-verify loop and foregrounds the version-specific breakage you only learn by hitting it: Harbor not detecting the new bundle, verify reaching out to the TUF CDN even when you handed it a local key, and --tlog-upload=false turning into a hard error instead of a quiet no-op. It is for platform and supply-chain engineers already running admission-time image verification who are moving up from v2.x.

Prerequisites

cosign v3.0.4 or later. Check with cosign version. The v3.0.x line through v3.1.1 is where the default flip and the early breakage fixes live.
Docker or any OCI client, plus push access to a registry. To exercise the new format end to end, your registry must support OCI 1.1 referrers. GHCR and recent registries do; Harbor needs 2.15.0 (see Common pitfalls).
For the keyless path: a GitHub Actions runner with id-token: write permission. No long-lived keys.
An image you can push, referenced by digest. Signing a mutable tag signs whatever the tag happened to point at, which is rarely what you mean.

Step-by-step

1. Confirm your version and the default that changed

cosign version
# cosign: v3.0.4 (or later)

In v3 the new bundle format is on by default. The Sigstore blog states --new-bundle-format, --trusted-root, and --use-signing-config all moved from opt-in to default-on. You no longer pass them. You now pass their negation when something downstream breaks.

2. Generate a key pair (skip if you go keyless)

cosign generate-key-pair
# writes cosign.key (encrypted private) and cosign.pub

Simplest path for a private registry. For CI, prefer keyless (Step 5) so there is no private key sitting somewhere to leak.

3. Resolve the digest, then sign by digest

IMG=ghcr.io/yourorg/app
DIGEST=$(docker buildx imagetools inspect "$IMG:latest" --format '{{.Manifest.Digest}}')
cosign sign --key cosign.key "$IMG@$DIGEST"

On v3 this writes a signature in the standardized bundle format (media type vnd.dev.sigstore.bundle.v0.3+json, per the Harbor issue thread) and stores it as an OCI 1.1 referring artifact instead of the legacy sha256-<digest>.sig tag. That storage change is the root of most upgrade surprises. Nothing about your signing command looks different. Where the signature lands does.

4. Verify with the public key

cosign verify --key cosign.pub "$IMG@$DIGEST" | jq .

A clean run prints the verified bundle JSON. If this hangs or errors on the network even though you supplied a local key, jump to Common pitfalls: v3.0.2 still tried to reach the TUF CDN in air-gapped setups (sigstore/cosign issue #4550).

5. Sign keyless from GitHub Actions

permissions:
  id-token: write   # required: mints the OIDC token Fulcio trusts
  packages: write
steps:
  - uses: sigstore/cosign-installer@v3
  - run: cosign sign --yes "ghcr.io/${{ github.repository }}@${DIGEST}"

No --key. Cosign exchanges the workflow's OIDC token for a short-lived Fulcio certificate and records the signature in Rekor. There is no private key to steal because the certificate is ephemeral. This is the path I'd push any team toward for release artifacts.

6. Verify keyless by identity

cosign verify \
  "$IMG@$DIGEST" \
  --certificate-identity="https://github.com/yourorg/app/.github/workflows/release.yml@refs/heads/main" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" | jq .

Per the Cosign verification docs, one of --certificate-identity or --certificate-identity-regexp is mandatory for keyless flows, alongside --certificate-oidc-issuer. Pin the exact workflow path and ref. A regexp that matches any workflow in the org defeats the entire point of identity-based verification.

7. Wire verification into admission (Kyverno or policy-controller)

Keep enforcing at the cluster edge, but confirm your policy engine speaks the new bundle format before you flip the default fleet-wide. Test one namespace first:

kubectl run probe --image="$IMG@$DIGEST" -n verify-test

If the pod is rejected with a signature-not-found error while cosign verify on the CLI passes, your controller is still hunting for the legacy .sig tag. See Common pitfalls.

Verify it works

CLI verification should exit 0 and print a JSON bundle carrying the certificate subject and the Rekor entry. Then confirm independently that the signature actually exists as a referrer in the registry:

cosign tree "$IMG@$DIGEST"
# lists the attached signature/attestation artifacts

If cosign verify passes but your registry UI shows no signature, that is the OCI 1.1 referrer visibility gap, not a failed signature. The signature is there. The registry just can't render it yet.

Common pitfalls

Harbor (and other registries) do not show the signature. Signatures created with the new bundle format carry media type vnd.dev.sigstore.bundle.v0.3+json, which Harbor's signature detection did not recognize. goharbor/harbor issue #22401 targets the fix for Harbor 2.15.0. Until your registry supports it, sign with the legacy layout: cosign sign --new-bundle-format=false --key cosign.key "$IMG@$DIGEST".
Air-gapped verify still phones home. In sigstore/cosign issue #4550, v3.0.2 kept trying to reach the TUF CDN even with a local key on a Nexus-only network. The working offline shape is cosign verify --key cosign.pub --offline --new-bundle-format=false --trusted-root trusted_root.json --local-image <dir>. Offline validation of the new protobuf bundle had not landed in an early v3 release, so --new-bundle-format=false is the reliable disconnected path today.
--tlog-upload=false now errors instead of silently skipping. Per the v3.0.4 release notes, disabling transparency-log upload is no longer allowed when --use-signing-config (now default) is set. Cosign fails before it writes the bundle. To sign without Rekor, also pass --use-signing-config=false.
--bundle is now required where it was optional. The flag that names the output bundle file moved from optional to required in v3, so any script that omitted it will error on the bump.
CI cache signing regressed. moby/buildkit issue #6737 reports GitHub Actions cache signing breaking since cosign 3.0.4. If your build cache step fails right after the upgrade, pin the installer to a known-good version rather than chasing it live in CI.
Signing a tag, not a digest. Always resolve and sign @sha256:.... A tag is mutable, and you will eventually verify a different image than the one you signed. This one bites quietly, months later.

Wrap-up

You have a Cosign v3 signing and verification loop that works with a key pair and keyless from GitHub Actions, plus a clear read on the three behaviors that quietly changed underneath you: the new bundle format, OCI 1.1 referrer storage, and signing-config-driven Rekor upload. The two levers worth memorizing are --new-bundle-format=false and --use-signing-config=false, which keep you signing while Harbor, your air-gapped registry, and your admission controller catch up to the defaults.

Do this before you enforce anywhere: confirm your policy engine (Kyverno or Sigstore policy-controller) verifies the new bundle format in a test namespace, and version-pin cosign-installer in CI so a point release can't silently change your supply-chain guarantees overnight.

Sources

Originally published at indragustiprasetya.com

Why the 2026 RAM Shortage Spiked DDR5 Prices 60% a Quarter

Indra Gusti Prasetya — Wed, 24 Jun 2026 10:21:54 +0000

Price a Kubernetes node pool in January, re-quote it in June, and the memory line has roughly tripled. Nobody on the vendor side will give you a straight reason. The industry has a name for it, used without much irony: the RAMpocalypse. The real story underneath is duller and worse. The world's DRAM fabs are being structurally re-pointed at AI, and that reallocation is now landing in everyone's capacity plan, not just the hyperscalers who started it.

The 20% number that lies to you

The figure everyone quotes is the reassuring one. Per TrendForce's December 2025 forecast, AI will consume roughly 20% of global DRAM wafer capacity in 2026, led by HBM and GDDR7. A fifth of the fabs. Sounds survivable.

It is not, and the reason is the whole article: HBM does not turn wafers into usable bits at anything close to the rate commodity memory does.

Here is the part that bites. Per Tom's Hardware, citing the supply-chain analysis behind the shortage, one gigabyte of HBM consumes roughly three times the wafer capacity of one gigabyte of DDR5. That is yield loss from die stacking plus the extra process steps. So when a fab shifts a wafer from DDR5 to HBM, it is not a one-for-one trade. It is closer to three bits of commodity DDR5 and LPDDR5 vanishing for every one bit of HBM that ships. The "20% of capacity" headline is technically about wafer starts. The damage to the commodity bit pool, the actual RAM going into your servers, laptops, and phones, is disproportionately larger than 20%. That gap is why "AI is only a fifth of the fab" and "your server memory contract jumped 60% in a quarter" are both true in the same breath.

This is now an infrastructure-budget problem, not a PC story

For a while you could file this under consumer-PC inflation and ignore it. That window closed.

Per TrendForce's March 31, 2026 forecast, conventional DRAM contract prices were expected to rise 58% to 63% quarter-on-quarter in Q2 2026. NAND Flash contract prices in the same window: up 70% to 75% quarter-on-quarter. Contract prices, not spot. That distinction matters more than it sounds. Spot is the noisy number traders chase. Contract is what your procurement team and your cloud provider actually sign, which means it flows straight into instance pricing and hardware quotes a quarter or two later.

So your storage budget moves with the same tide. NAND up 70-plus percent QoQ means SSDs and storage tiers inflate alongside RAM. If you patch your memory forecast and leave the storage forecast at last year's numbers, you have only fixed half the hole.

Why suppliers want it this way

The most uncomfortable detail is that none of this is an accident, and accidents are the only kind of shortage that resolves on its own.

TrendForce notes that suppliers are prioritizing server DRAM for its superior profitability and signing long-term agreements (LTAs) with cloud service providers, who will pay more to lock in supply for AI server build-outs. Read that again from the fab's side. Samsung, SK hynix, and Micron are looking at a choice between low-margin commodity DDR5 and high-margin HBM plus guaranteed multi-quarter CSP contracts. They picked the money. TrendForce's December 2025 analysis framed it plainly: DDR5 profitability is intensifying the capacity crowding. The squeeze is a pricing strategy. Strategies do not reverse in ninety days because your refresh budget is uncomfortable.

And HBM's slice keeps growing. Figures attributed to TrendForce across the coverage put HBM at roughly 23% of total DRAM wafer output in 2026, up from about 19% in 2025. Every point HBM gains is commodity DDR5 leaving the market. The trend line points the wrong way for anyone buying ordinary RAM.

The fab can't just pivot back

There is a hope buried in a lot of procurement conversations: prices spike, suppliers chase the spike, capacity floods back, prices crater. It is the classic memory cycle, and it has happened before.

It is shakier this time because the lines are not interchangeable. HBM needs its own production tools, masks, and advanced packaging. That equipment sits where DDR5 or LPDDR5 lines would otherwise run. A fab cannot flip a tool back to commodity output over a weekend, and the capital is already committed to the high-margin product. This is why analysts keep using the word structural rather than calling it a temporary allocation choice. The bottleneck is built into the equipment plan.

The duration estimates match that. SK hynix's CEO has reportedly estimated the shortage running until 2030. Even the optimistic industry reads point to late 2027 before supply meaningfully eases. Either way you are planning around a condition, not waiting out a blip.

It compounds on an already-high base

2026's jump is not starting from a calm baseline. Per the compiled industry record, DRAM rose roughly 172% across 2025 before these 2026 contract increases even landed. Memory's share of a PC bill of materials has reportedly climbed from the mid-teens toward roughly a third over the same stretch. So the 58% to 63% QoQ figure is a percentage increase on top of a number that already doubled-and-then-some last year. For server fleets, where you are buying memory by the terabyte, that compounding is the difference between a noticeable line item and a board-level conversation.

The counterargument, and why I don't buy it this time

The honest objection: memory is famously cyclical, and every shortage in history has ended in a glut. People who lived through the 2018 and 2023 down-cycles will tell you cheap RAM always comes back. They are right about the past.

But every prior cycle was driven by demand swings on an interchangeable commodity product. When demand fell, the same lines that made the expensive RAM made the cheap RAM, and prices collapsed. This cycle is driven by suppliers deliberately reallocating fab capacity toward a higher-margin, non-interchangeable product under multi-year contracts. The thing that broke the old gluts, instant fungibility of supply, is exactly what HBM lacks. Until that allocation reverses, and the people signing the LTAs are betting years of capacity that it will not, the cheap-RAM assumption is the riskiest line in your capacity plan.

What to do before your next refresh quote

Concrete moves, each tied to a number above. Do these this quarter, not next.

Re-baseline any capacity model older than two quarters. If your cost-per-node, cost-per-pod, or cost-per-GB-cached math predates the Q2 2026 contract jump, it is understating memory by 50% or more. Re-quote with current contract pricing before you commit a single refresh PO or new cluster.
Audit Kubernetes memory requests against actual usage. Overprovisioned requests and idle headroom were free insurance when DRAM was cheap. At 58% to 63% QoQ, that slack is a measurable line item. Pull requests-vs-usage from your metrics, find pods sitting at 30% of their request, and reclaim the gap. This is the fastest dollar you will save with zero hardware spend.
Lock forward terms now if you run on-prem or colo. CSPs are signing LTAs precisely because spot exposure is brutal. Call your memory and SSD vendors about forward pricing or a fixed-term agreement before the next quarterly reset. Waiting one more quarter is a bet against a documented 58-to-75% trend.
Budget NAND in the same pass. Storage is up 70% to 75% QoQ per TrendForce, so SSD tiers move with DRAM. Update the storage forecast in the same spreadsheet, same meeting. Do not ship a RAM-only correction.
Re-examine memory-as-a-crutch architecture. In-memory caches, oversized JVM heaps, and "just add RAM" scaling all got a recurring quarterly tax. Where a design leans on cheap memory to dodge engineering work (a giant cache instead of a smarter query, headroom instead of right-sizing), that trade just inverted. Spend the engineering time now.
Plan multi-year, not next-quarter. With LTAs locking allocation and HBM at ~23% of wafer output and climbing, treat this as structural through at least 2027, possibly to 2030 on SK hynix's own estimate. Revisit pricing every quarter and stop modeling a return to 2024 memory costs. It is not coming on your planning horizon.

The uncomfortable summary for anyone who signs infrastructure budgets: memory stopped being a rounding error and became a strategic input, priced by people whose interests run directly against yours. Plan accordingly.

Sources

Originally published at indragustiprasetya.com