DEV Community

Posting Dude
Posting Dude

Posted on

Chrome Blocked 8 eMudhra CA Keys on Oct 17. A Node Release From Oct 14 Still Trusts Them.

On Saturday, October 17, at 18:05 UTC, a researcher reported to the Chrome Root Program that the private keys of eight eMudhra intermediate CA certificates could be recovered by anyone. The keys had been generated with weak randomness. With one of them you can sign a certificate for any domain, and clients that still trust that intermediate will accept it.

Chrome blocked the keys 100 minutes later with a CRLSet push. Firefox blocked them the same day. Microsoft followed on the 20th, Apple on the 22nd, and Chrome removed the root above them, emSign ECC Root CA - G3, from its root store on October 27.

That covers browsers. Your backend is not a browser. It calls Stripe, sends webhooks and talks to your email provider through Node, Python or Go, and a lot of those don't read CRLSets. They check against a list of roots that shipped with the runtime or the package.

So I tested it. Same Debian 13 box, with the patched system ca-certificates (20250419+deb13u1) installed. A Node binary released on October 14 still connected to a site signed by a compromised intermediate.

What actually happened, briefly

  • Eight intermediates under emSign ECC Root CA - G3 had keys that could be recovered straight from the public certificates. eMudhra says it revoked all eight on October 18, that seven were never used in production, and that the one in production had issued zero certificates.
  • No abuse has been reported. Cloudflare said it found no exploitation on its services, and the test page below says plainly that nobody knows whether non-browser clients were targeted. Browsers also enforce Certificate Transparency, so a forged cert that was never logged fails there anyway.
  • Most non-browser clients check neither revocation nor CT. For them, the only fix is a trust store that no longer includes those roots.

Dennis Jackson from Mozilla put up a test page with live endpoints signed by the compromised intermediates, plus a list of which packages have shipped fixes. If your client connects and gets the text "That's bad!", it still trusts them. The -noct hosts use leaf certs that were never logged to CT, which is the case an attacker would actually use.

The test

A 13-line script that calls fetch() on the four test hosts:

// check-emudhra.mjs
const hosts = ['dv-g3b', 'dv-g3b-aaa', 'dv-g3b-noct', 'dv-g3b-aaa-noct'];
console.log(`node ${process.version}`);
for (const h of hosts) {
  const url = `https://${h}-emudhra.we-have-a-problem.com/`;
  try {
    const res = await fetch(url, { signal: AbortSignal.timeout(15000) });
    const body = (await res.text()).trim().slice(0, 40);
    console.log(`${h.padEnd(16)} TRUSTED   ${res.status} "${body}"`);
  } catch (e) {
    console.log(`${h.padEnd(16)} rejected  ${e.cause?.code ?? e.message}`);
  }
}
Enter fullscreen mode Exit fullscreen mode

Real output, run this morning:

node v24.22.1
dv-g3b           rejected  CERT_REVOKED
dv-g3b-aaa       rejected  CERT_REVOKED
dv-g3b-noct      rejected  CERT_REVOKED
dv-g3b-aaa-noct  rejected  CERT_REVOKED

node v24.22.0
dv-g3b           TRUSTED   200 "Your client successfully connected to th"
dv-g3b-aaa       rejected  UNABLE_TO_GET_ISSUER_CERT_LOCALLY
dv-g3b-noct      TRUSTED   200 "Your client successfully connected to th"
dv-g3b-aaa-noct  rejected  UNABLE_TO_GET_ISSUER_CERT_LOCALLY
Enter fullscreen mode Exit fullscreen mode

The full body of that 200 is "Your client successfully connected to the site. That's bad!". Node 24.22.0 came out on October 14. Node 24.22.1 is the October 21 security release, which updated the bundled roots to NSS 3.131.1 and added a check that rejects chains containing blocked public keys. That second change is most likely where the CERT_REVOKED comes from.

I ran the same thing in Python with requests, which uses certifi's bundle and not the OS:

certifi 2026.10.23
dv-g3b           rejected  unable to get local issuer certificate
dv-g3b-noct      rejected  unable to get local issuer certificate

certifi 2026.07.22
dv-g3b           TRUSTED   200 'Your client successfully connected to th'
dv-g3b-noct      TRUSTED   200 'Your client successfully connected to th'
Enter fullscreen mode Exit fullscreen mode

(The two -aaa hosts, which use Sectigo cross-signs, were rejected by every client on this box, old or new, so I've left them out here.)

curl on the same machine rejected all four, because it uses the patched system bundle. So the OS was fixed and two of the three clients on it weren't.

Why patching the OS didn't help

Node ships its own copy of the Mozilla root list inside the binary. requests reads certifi's cacert.pem out of your virtualenv. Updating ca-certificates touches neither.

One trap I didn't expect: node --use-system-ca didn't help on the old binary either. It still trusted the bad host. The Node docs say that flag uses the system store "along with" the bundled one, so the old bundled root is still there. --use-openssl-ca, which swaps the bundled store for OpenSSL's system store, did reject all four on the old binary. Upgrading Node is still the real fix. The flag just tells you where the trust is coming from.

Where the old copies are hiding

For a small SaaS or a side project, I'd check these:

  • Docker images built before October 21. The Node or Python inside is whatever was current at build time. If you pin node:24-slim by digest, it doesn't move until you move it.
  • Anything on Node 20. It's end-of-life, so it doesn't get the new root list at all. Vercel's advisory says the same thing and tells people to move to 22 or newer.
  • A pinned certifi in requirements.txt, poetry.lock or uv.lock. Patched from 2026.10.19.
  • CI runners and self-hosted cron boxes, which tend to drift.
  • Your Mac. As of the test page's October 28 update, the curl, git and python3 Apple ships still trust the compromised certificates.

The project you haven't touched since summer is the most likely to still have the old list. If that's you, I wrote a list of what to check first when you restart an abandoned side project, and an expired runtime is one of the items on it.

An 18-line check for any machine or image

This prints 1 if emSign ECC Root CA - G3 is still in the system bundle, in the node you're running, or in your certifi:

#!/usr/bin/env bash
# roots-check.sh: 1 = root still trusted (bad), 0 = gone
NODE=${NODE:-node}; PY=${PY:-python3}
needle='emSign ECC Root CA - G3'

count_in_bundle() {
  openssl crl2pkcs7 -nocrl -certfile "$1" | openssl pkcs7 -print_certs -noout \
    | grep '^subject=' | grep -c "$needle"
}

echo "system bundle:          $(count_in_bundle /etc/ssl/certs/ca-certificates.crt)"
echo "node $($NODE -v) bundled: $($NODE -e '
  const { X509Certificate } = require("crypto");
  const hits = require("tls").rootCertificates
    .filter(pem => new X509Certificate(pem).subject.includes(process.argv[1]));
  console.log(hits.length)' "$needle")"
CF=$($PY -c 'import certifi; print(certifi.where())')
echo "certifi $($PY -c 'import certifi; print(certifi.__version__)'):     $(count_in_bundle "$CF")"
Enter fullscreen mode Exit fullscreen mode

Output for the current versions and then the old ones:

system bundle:          0
node v24.22.1 bundled: 0
certifi 2026.10.23:     0

system bundle:          0
node v24.22.0 bundled: 1
certifi 2026.07.22:     1
Enter fullscreen mode Exit fullscreen mode

To check a container, mount the script and run it inside, something like docker run --rm -v "$PWD:/c" your-image bash /c/roots-check.sh. Slim images often lack openssl or python, so just keep the line that applies (for a Node image, the node -e part is all you need).

Is your own site's certificate affected?

This is the other half, and for most indie sites the answer is no. If your cert comes from Let's Encrypt or Google Trust Services, like most small sites, nothing changes for you. It matters if you bought an ECC certificate that chains to emSign ECC Root CA - G3. The University of Michigan, for example, had to replace certificates issued through CERTInext with ECC keys, and its sites behind Cloudflare in Full (strict) mode started returning 526 errors on October 20.

To see which root you're on, look at the issuer line and every row of the chain, not just the expiry date. openssl s_client -connect yourdomain.com:443 -showcerts shows them, or you can paste the domain into a free SSL checker that lists the issuer and the full chain. If "emSign" shows up anywhere in that chain, get a new certificate from a different CA before your visitors start seeing warnings.

The fix list

  1. Node: 22.23.4, 24.22.1 or 26.12.1 or later. Get off Node 20.
  2. certifi 2026.10.19 or later, and update the lockfile too.
  3. Rebuild images with --pull --no-cache so you don't reuse the old layer, then redeploy. Restart long-running processes too. A Node process that's been up since September is still running the old binary with the old roots.
  4. Run check-emudhra.mjs (or curl on the test host) from inside the deployed thing, not from your laptop. Cloudflare's write-up makes the same point: a passing browser test doesn't mean your background worker is fixed.
  5. If something starts failing with certificate errors after you patch, do not "fix" it with NODE_TLS_REJECT_UNAUTHORIZED=0 or verify=False. That turns off the protection you just installed.

This is the second certificate change this month that only shows up when you check what's actually running instead of what the config says. The other one is Let's Encrypt moving to 64-day certificates, where the danger is hardcoded renewal thresholds.

If you pin your base image by digest (which you should for other reasons), what makes you rebuild it? A schedule, Renovate/Dependabot bumping the digest, or only when something breaks?

Top comments (0)