DEV Community

sachin k
sachin k

Posted on

A Package You Use Was Just Compromised. Now What?

#ai #programming #cybersecurity #npm

There are plenty of good posts about preventing supply-chain attacks: **ignore-scripts=true, minimumReleaseAge,** frozen lockfiles, provenance checks. Read them. Apply them.

But all of those posts end at the same cliff. What happens when prevention fails?

You open Slack and see: "lib-x versions 4.2.1 to 4.2.3 were compromised, malicious postinstall, rotate your creds." Your heart rate doubles. What do you actually type into your terminal right now?

Nobody writes that post, so I did. A 60-minute playbook, in order, with the exact commands. It assumes npm/pnpm/yarn or pip/uv, but the logic ports anywhere.

Minute 0-10: Answer one question first. Did I ever resolve the bad version?

Don't rotate anything yet. Don't panic-delete node_modules. First establish whether you were ever exposed, because "we depend on lib-x" and "we installed lib-x 4.2.2" are very different incidents.

Check what's installed right now

`# npm / pnpm / yarn: shows every resolved copy, including transitive
npm ls lib-x --all
pnpm why lib-x
yarn why lib-x`
Enter fullscreen mode Exit fullscreen mode 
# Python
`pip show lib-x
uv pip list | grep lib-x`
Check the lockfile, including its history
Enter fullscreen mode Exit fullscreen mode

Your current lockfile might be clean while last Tuesday's wasn't. The exposure window is every commit where the lockfile referenced a bad version:

`# Every lockfile commit that ever mentioned the package
git log --follow -p -- package-lock.json | grep -n -B2 -A2 '"lib-x"'`
Enter fullscreen mode Exit fullscreen mode

Faster: which commits introduced/removed the bad version string 

git log -S '4.2.2' --oneline -- package-lock.json pnpm-lock.yaml yarn.lock
Enter fullscreen mode Exit fullscreen mode

Python equivalents 

`git log -S 'lib-x==4.2.2' --oneline -- requirements.txt uv.lock poetry.lock`
Enter fullscreen mode Exit fullscreen mode

Don't forget branches and open PRs

A Renovate or Dependabot PR sitting open with the bad version has already been installed by CI on every push to that branch:

`# Search the bad version across ALL branches
git grep '4.2.2' $(git for-each-ref --format='%(refname)' refs/remotes) -- '*lock*'`
Enter fullscreen mode Exit fullscreen mode

Decision point. If the bad version never appeared anywhere, you're done. Write two sentences in your incident channel, add the prevention controls, go back to work. If it appears in any lockfile, on any branch, at any point in the disclosed window, keep reading. You are now in incident mode.

Minute 10-20: Could the payload have executed?

Compromised is not the same as executed. Figure out which class of payload you're dealing with. Advisories usually say.

Class A: install-time payload (preinstall/postinstall, setup.py). It ran on every machine that installed it. Every dev laptop, every CI runner, every Docker build. If you run with ignore-scripts=true (or pnpm's default script blocking, or Bun), your laptops may be fine. Check CI separately though, because CI configs often differ from local ones.

`# Did this environment allow scripts?
npm config get ignore-scripts        # want: true
pnpm config get ignore-scripts`
Enter fullscreen mode Exit fullscreen mode

Class B: runtime payload (malicious code inside the module itself). Installing it was harmless. Importing it was not. Now the question is which processes loaded it: your app in prod, your test suite in CI, a build script?

`# Was it actually imported anywhere, or just present in the tree?
grep -rn "require(['\"]lib-x" src/ scripts/
grep -rn "from ['\"]lib-x" src/
# Python
grep -rn "import lib_x\|from lib_x" .`
Enter fullscreen mode Exit fullscreen mode

Write down your blast radius in one line. Something like: "Class A payload, scripts enabled in CI but not laptops, CI ran 14 installs between Aug 12 and Aug 18." Everything after this is scoped by that sentence.

*Minute 20-35: Contain. Rotate what the payload could reach.
*
This is the step people do in the wrong order. Rotate before cleanup, because your cleanup commits will trigger CI, and if CI secrets are burned you're re-exposing fresh code to a compromised environment.

Recent real-world payloads (the Shai-Hulud-style worms) go straight for tokens. Assume the payload harvested every credential readable from the environment it ran in.

If it ran in CI:
All repo/org secrets in that CI system (GitHub Actions secrets, GitLab variables)

Cloud credentials. Prefer OIDC federation over static keys going forward, but rotate any static AWS_/GCP_ keys now.
NPM_TOKEN / PYPI_TOKEN. Top priority if you publish packages. This is how one compromise becomes a worm: your token publishes the next infected version.

`# npm: list and revoke tokens
npm token list
npm token revoke <id>`
Enter fullscreen mode Exit fullscreen mode 
`# GitHub: check for tokens/keys you didn't create
gh api /user/keys
gh auth status`
Enter fullscreen mode Exit fullscreen mode

If it ran on laptops:

~/.npmrc and ~/.pypirc tokens
SSH keys, gh CLI tokens, long-lived cloud creds in ~/.aws/credentials
.env files in project directories. These are exactly what the payloads grab.

Also check for persistence. Several npm worms added malicious GitHub Actions workflows or new repos using stolen tokens:

`# Recently created/modified workflows across your org
gh api "search/code?q=org:YOUR_ORG+path:.github/workflows&sort=indexed" \
  --jq '.items[].repository.full_name' | sort -u`

`# Repos created during the exposure window
gh repo list YOUR_ORG --limit 200 --json name,createdAt \
  --jq '.[] | select(.createdAt > "2026-08-12")'`
Enter fullscreen mode Exit fullscreen mode

*Minute 35-50: Eradicate the version, and every cache holding it
*
Fixing package.json is the easy 10%. The tarball is also sitting in caches that will happily re-serve it.

`1. Force a safe version everywhere, including transitive deps
jsonc
// package.json (npm)
"overrides": { "lib-x": "4.2.0" }

// pnpm: pnpm-workspace.yaml (or package.json pnpm.overrides)
// overrides:
//   lib-x: 4.2.0

// yarn
"resolutions": { "lib-x": "4.2.0" }
bash
# Python: pin explicitly, even if it's transitive
echo "lib-x==4.2.0" >> requirements.txt   # or constraints.txt
uv lock --upgrade-package lib-x==4.2.0`
Enter fullscreen mode Exit fullscreen mode

Then regenerate and diff the lockfile by hand before committing. You're looking for the bad version disappearing and nothing suspicious appearing.

  1. Purge every cache layer 
`# Local
rm -rf node_modules
npm cache clean --force
pnpm store prune
yarn cache clean
pip cache purge && uv cache clean
`
# CI caches. This is the one everyone forgets.
# GitHub Actions: delete caches so restored node_modules can't resurrect the payload
gh cache list
gh cache delete --all

# **Docker:** layer caches can pin the bad tarball indefinitely
docker builder prune --all
Enter fullscreen mode Exit fullscreen mode

If you run a registry proxy (Artifactory, Verdaccio, Nexus), purge the bad version there too. Otherwise every future install in your org re-downloads it from your own mirror.

*3. Rebuild and redeploy anything built in the window
*
Any artifact (Docker image, Lambda zip, frontend bundle) built while the bad version was resolvable is suspect. Rebuild from the fixed lockfile, redeploy, and if you keep image digests, note which digests were built in the window.

Minute 50-60: Verify, then write it down

`# Confirm the bad version is gone from the tree
npm ls lib-x --all
# Confirm the lockfile can't drift
npm ci        # or: pnpm install --frozen-lockfile
# Confirm integrity/signatures on what you now have
npm audit signatures`
Enter fullscreen mode Exit fullscreen mode

Then write a short postmortem. Five lines, in the incident channel, today:

Exposure window: first commit to fix commit, with timestamps
Blast radius: which environments installed it, whether scripts or imports executed
What was rotated (and what deliberately wasn't, and why)
Caches purged, artifacts rebuilt
The prevention gap that let it in, with one linked follow-up ticket

That last line is where all those prevention articles finally become relevant: ignore-scripts=true, a release-age cooldown (minimum-release-age in npm 11.10+, on by default in pnpm 11, npmMinimalAgeGate in Yarn 4.10+), frozen-lockfile installs in CI, and OIDC instead of static cloud keys. Prevention posts tell you to do these things. An incident tells you which one you actually needed.

The 10-second version to bookmark

`1. git log -S '<bad-version>' -- <lockfiles>     -> was I ever exposed? (all branches!)
2. Install-time or runtime payload?              -> scope: laptops vs CI vs prod
3. Rotate: publish tokens > CI secrets > cloud keys > laptop creds
4. Check persistence: new workflows, new repos, new tokens
5. overrides/resolutions pin + purge npm/pnpm/CI/Docker caches
6. Rebuild artifacts from the window; redeploy
7. npm ci + npm audit signatures                 -> verify
8. Five-line postmortem + one prevention ticket`
Enter fullscreen mode Exit fullscreen mode

Prevention articles get written because prevention is comfortable. Incident response gets skipped because it's the part where you were already wrong. The window between "compromise published" and "compromise detected" will never be zero, which means this playbook isn't a nice-to-have. It's the half of supply-chain security we all stopped writing about.

If you've lived through one of these, drop your war story in the comments. Especially the cache layer you forgot.

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

The “now what” phase is where supply-chain plans usually get exposed. Teams need more than a patched version number: affected paths, secret rotation decisions, build provenance, and a way to prove which deployments included the bad package. Incident inventory matters as much as dependency scanning.