DEV Community: ntoledo319

"Error: The module was compiled against a different Node.js version using NODE_MODULE_VERSION — fixing native addon failures after a Lambda upgrade"

ntoledo319 — Mon, 03 Aug 2026 07:17:02 +0000

You upgrade a Lambda function from nodejs20.x to nodejs22.x, redeploy, and the first cold start dies with something like this:

Error: The module '/var/task/node_modules/sharp/build/Release/sharp-linux-x64-115.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 115. This version of Node.js requires
NODE_MODULE_VERSION 127. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).

The numbers vary by package (115 and 127 are Node 20 and Node 22 respectively), but the shape is always the same: a .node binary was compiled for one ABI and the runtime is now a different one.

Why this happens

Every Node.js major version assigns a new NODE_MODULE_VERSION — an integer baked into compiled .node files that the runtime checks at load time:

Node.js version	NODE_MODULE_VERSION
Node.js 20	115
Node.js 22	127

Native addons — sharp, bcrypt, better-sqlite3, canvas, grpcio, and anything else that ships a precompiled .node file — link against the ABI of the Node version they were built on. When the runtime changes, the loader reads the embedded version, sees it doesn't match the running Node, and refuses to load the binary.

This is intentional: the ABI mismatch means the binary literally cannot be safely loaded. The error is a hard stop at cold-start time, not a warning.

The problem surfaces specifically after a Lambda runtime bump because your CI pipeline typically builds the deployment package once (on Node 20, say) and keeps redeploying the same artifact. The binary in /var/task/node_modules/<pkg>/build/Release/*.node was compiled for Node 20's ABI and the new nodejs22.x runtime won't touch it.

The fix

1. Rebuild the deployment package on the target Node version

The root fix is always: build where you deploy. Tear down node_modules and reinstall on a machine (or container) running Node 22:

rm -rf node_modules
npm install

Or, if you can't change the build machine, force a rebuild:

npm rebuild

Then re-bundle and redeploy. Crucially, do this inside the Lambda base image if you have native dependencies, so the binary also links against the right glibc:

docker run --rm -v "$PWD":/var/task \
  public.ecr.aws/lambda/nodejs:22 \
  npm install

This produces a binary that is both the right ABI and linked against the right glibc — avoiding a second failure down the line.

2. If the package has no Node 22 prebuilt, upgrade it

Some packages ship prebuilt binaries keyed to specific Node major versions. If your pinned version has no Node 22 prebuild, the install falls back to compiling from source — which can fail in a CI environment without the right build toolchain. The fix is to upgrade to a version that does ship Node 22 prebuilts:

sharp: upgrade to >= 0.33.0 (moved to Node-API, no longer tied to a specific ABI)
bcrypt: >= 5.1.1 has Node 22 prebuilt support
better-sqlite3: >= 11.0.0 for Node 22; consider >= 13.0.0 which switched to N-API for future-proof ABI compatibility

3. For dead packages, replace them — don't rebuild them

Some native packages have no Node 22 build and no intention of shipping one:

Dead package	Replacement
`node-sass`	`sass` (Dart Sass — pure JS, no native build needed)
`fibers`	Remove it; Node's async primitives have superseded it
`grpc`	`@grpc/grpc-js` (pure JS gRPC client)

These replacements are also smaller and faster in Lambda cold-start terms because they have no native compilation step.

4. Lock your build Node version in CI

Add an explicit engine constraint so a future CI runner upgrade can't silently produce an incompatible artifact:

"engines": {
  "node": "22.x"
}

Or pin the Node version in your CI environment file (.nvmrc, .node-version, GitHub Actions node-version: '22'). The goal: whatever Node version runs npm install in CI must match the Lambda runtime.

Catching it before the cold start

The mismatch is silent until a cold start fires in production. The fastest way to surface it earlier: import the native module in a smoke test that runs inside the Lambda base image (public.ecr.aws/lambda/nodejs:22), not just on your developer machine.

If you'd rather get a full picture of which Lambda functions carry native-addon ABI risk — alongside deprecated-runtime exposure — without auditing each deployment package by hand, the free EOLkits scanner checks an account in about 30 seconds, nothing uploaded. Full reference: eolkits.com/fix/node-module-version-mismatch.

"Why did my AWS deploy break with zero code changes?"

ntoledo319 — Sat, 01 Aug 2026 07:17:03 +0000

Every other class of "it broke" starts with a diff. This one doesn't. git log shows nothing since last week, CI was green then, it's red now, and the error doesn't look like anything your team wrote.

That's the tell. A deploy that breaks with zero code changes is almost always one of two things: a fixed AWS calendar cutoff you crossed passively, or a dependency/base-image that moved out from under you without a corresponding commit in your repo. Neither shows up in git blame. Here's how to tell them apart.

First: is it a fixed AWS date?

Two calendar cutoffs cover almost every "nothing changed but it broke" report on Lambda and EC2/ECS/EKS right now.

Lambda runtime blocks. AWS enforces these on the account, not on your deploy history — the API call itself gets rejected once the date passes, independent of when you last touched the function. As of today, nodejs16.x, nodejs18.x, nodejs20.x, python3.8, python3.9, and python3.10 all share the same two dates:

Block function create: Feb 1, 2027 — CreateFunction (or a Terraform/CDK/SAM deploy that provisions a new function or alias) on one of these runtimes is rejected outright after this date.
Block function update: Mar 3, 2027 — after this, UpdateFunctionCode/UpdateFunctionConfiguration on an existing function using one of these runtimes is rejected too. The function keeps running and keeps invoking; you just can't ship a change to it anymore.

(python3.11 is on a later cluster: block-create Jul 31 2027, block-update Aug 31 2027 — don't assume every runtime lines up with the Q1-2027 dates above.)

If your error is coming back from the Lambda control plane itself — not from inside your code, not from a cold start — rather than a runtime crash, this is almost certainly it. Check the runtime string on the function the deploy touched.

Amazon Linux 2 end of life. AL2 reached end of standard support June 30, 2026 — already past. This one doesn't reject an API call the way the Lambda blocks do; instead, AL2 stops getting security patches, and separately, any user-data or AMI-bake step that calls yum/amazon-linux-extras starts failing as AL2's package repos and Extras Library shift under it. If the failure is at instance boot or AMI bake time rather than at application runtime, check the base AMI before anything else.

Cross-check either of these in 10 seconds against the same table AWS enforces — paste your runtimes or click the ones you use in the free EOL checker (client-side, nothing uploaded); it flags exactly which cutoff applies and how many days you have left.

If it's not a fixed date: three things that move without a commit

Not every silent break is calendar-driven. These three change underneath you with no corresponding line in your diff:

An unpinned or :latest-tagged base image moved forward. A Docker base image reference that isn't pinned to a digest can resolve to a newer patch build on a fresh pull — bringing a newer OS or language patch version than what last built successfully, with nothing in your Dockerfile diff to show for it.
A transitive dependency resolved differently. A fresh npm ci or pip install in CI re-resolves anything not exactly pinned. A transitive package (not your direct requirement) picking up a new minor version can drop or rename an API your code calls indirectly — again, invisible in your own diff.
An IaC provider bumped its own defaults. A routine terraform init -upgrade or CDK library bump can change what a resource defaults to when you didn't explicitly set it — including, on Lambda resources, which runtime a construct assumes if a version pin was left implicit.

None of these are bugs in the classic sense — they're all "something you depend on changed, and your repo correctly recorded that you didn't ask it to stay still."

Narrowing it down

Read where the error actually comes from. A rejection with no invocation ever happening, straight from the deploy/API call, points at a Lambda block date. A failure during instance boot or AMI bake points at AL2. A runtime crash after a successful cold start points at a stdlib/dependency change — see the runtime-upgrade error map for the exact error text against the specific fix.
Check what's actually pinned. git diff won't show you a moving base image tag or an unpinned transitive dependency — docker inspect on the built image, or a fresh npm ls/pip freeze diffed against your last known-good build, will.
If you're not sure which functions or AMIs are even exposed, the free scanner checks a whole account in about 30 seconds and flags both the calendar-cutoff risk and known dependency landmines before the next deploy trips one — nothing uploaded. I maintain it, disclosing that plainly since it's the one link here that isn't a diagnostic tool or a fix page.

"The AWS runtime-upgrade error map: which errors you'll hit, in what order, for each migration path"

ntoledo319 — Fri, 31 Jul 2026 07:17:03 +0000

Every deprecated-runtime migration produces the same shape of pain: you bump one version number, redeploy, and get hit with errors that look unrelated to each other but are all downstream of the same jump. The errors aren't random — for a given upgrade path, they show up in a fairly predictable order.

This is a map, not a new deep dive. Each line below is a real, verbatim error with its own fix already written up — this just sequences them by which upgrade produces them, so you know what's coming before you hit it instead of debugging one surprise at a time.

Path 1: Python 3.9 (or earlier) → Python 3.12 on Lambda

You're doing this because AWS blocks python3.9 creates/updates Feb 1 / Mar 3 2027 (and earlier versions are already blocked). Python 3.12 removed a batch of stdlib modules in one release (PEP 632/594), so this path front-loads the most breakage:

ModuleNotFoundError: No module named 'distutils' — removed in 3.12. Fix →
ModuleNotFoundError: No module named 'imp' — removed in 3.12. Fix →
AttributeError: module 'collections' has no attribute 'Mapping' — moved to collections.abc (removed from collections in 3.10, so this can also hit on the way to 3.10/3.11). Fix →
ModuleNotFoundError: No module named 'smtpd' — removed in 3.12. Fix →
ModuleNotFoundError: No module named 'asyncore' — removed in 3.12, usually alongside smtpd. Fix →
DeprecationWarning: datetime.datetime.utcnow() is deprecated — not fatal, but noisy, and worth fixing in the same pass. Fix →
/lib64/libc.so.6: version 'GLIBC_2.28' not found — if you ship any native dependency (cryptography, numpy, psycopg2, pydantic-core), moving to the AL2023-based Python 3.12 runtime (glibc 2.34) from an AL2-based one (glibc 2.26) can surface this separately from the stdlib removals above. Fix →

Path 2: Python 3.11/3.12 → Python 3.13

A second, smaller PEP 594 wave hits on the next hop:

ModuleNotFoundError: No module named 'cgi' — removed in 3.13. Fix →
ModuleNotFoundError: No module named 'telnetlib' — removed in 3.13. Fix →
ModuleNotFoundError: No module named 'crypt' — removed in 3.13. Fix →
ModuleNotFoundError: No module named 'lib2to3' — removed in 3.13. Fix →

And one that hits on the way into 3.11, if you skipped straight past it:

AttributeError: module 'asyncio' has no attribute 'coroutine' — the legacy @asyncio.coroutine decorator, removed in 3.11. Fix →

Path 3: Node.js 16/18 → Node.js 20/22 on Lambda

This path is less about stdlib removals and more about what the runtime stopped bundling and what OpenSSL 3 stopped allowing:

Error: Cannot find module 'aws-sdk' — AWS SDK v2 stopped shipping preinstalled from nodejs18.x onward; only @aws-sdk/* v3 is preinstalled now. This is usually the first error you hit, at cold start. Fix →
Error: error:0308010C:digital envelope routines::unsupported — OpenSSL 3 (bundled from Node 17+) rejecting a legacy hash, often from an older build tool (webpack 4). Fix →
error:1E08010C:DECODER routines::unsupported — OpenSSL 3 refusing to load a private key stored in a legacy format (PKCS#1, or encrypted with a weak cipher). Fix →
Error: The module was compiled against a different Node.js version using NODE_MODULE_VERSION — any native addon (sharp, bcrypt, better-sqlite3) needs a rebuild for the new ABI. Fix →
Node Sass does not yet support your current environment — node-sass specifically, since LibSass is dead and ships no Node 22 prebuild. Fix →
TypeError: crypto.createCipher is not a function — createCipher/createDecipher were removed outright in Node 22 (not just deprecated). Fix →
[DEP0040] DeprecationWarning: The 'punycode' module is deprecated — not fatal, usually from a transitive dependency, loudest on Node 22. Fix →

If none of the above matches and you're just staring at a bare Runtime.ImportModuleError: Cannot find module, the cause could be any of #1 or #4 above, an esbuild 0.22+ bundling default change, or a Lambda layer built on the wrong OS/arch — see the dedicated triage guide to tell them apart before guessing.

Path 4: Amazon Linux 2 → Amazon Linux 2023

AL2 reaches end of life June 30, 2026. This path isn't a Lambda runtime change at all — it's the base OS on EC2/ECS/EKS nodes — but it produces the same "one version bump, five unrelated-looking failures" shape:

amazon-linux-extras: command not found — the Extras Library mechanism doesn't exist on AL2023 at all. Fix →
Error: Unable to find a match: <package> — package renamed, version-namespaced, moved to SPAL, or dropped. Fix →
Failed to start ntpd.service: Unit ntpd.service not found — AL2023 uses chronyd. Fix →
Failed to start iptables.service: Unit iptables.service not found — AL2023 defaults to nftables. Fix →
/usr/bin/env: 'python2': No such file or directory — AL2023 ships no Python 2 at all. Fix →

Why this exists as one page

Each fix above already has its own full write-up — root cause, verified fix steps, primary source. What's usually missing when you're mid-migration isn't any single fix, it's knowing what's coming next so you can batch the work instead of playing whack-a-mole one redeploy at a time. Bookmark whichever path applies to you and work down it in order.

If you'd rather have this run against your actual account instead of reading it, the free EOLkits scanner checks in about 30 seconds, nothing uploaded — I maintain it, disclosing that plainly since it's the one link in this piece that isn't a fix.

"[DEP0040] DeprecationWarning: The `punycode` module is deprecated — what to do about it on Node.js 22 Lambda"

ntoledo319 — Thu, 30 Jul 2026 07:17:03 +0000

Upgrade a Lambda function to nodejs22.x and CloudWatch fills up with a warning that wasn't there before:

(node:8) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

Nobody on the team imports punycode. It's not in package.json. The warning still shows up on every cold start.

Why it appears now

Node.js has shipped a built-in punycode module since 0.x — it converts internationalized domain names between Unicode and ASCII (xn--) form. Node deprecated the built-in copy in favor of the identical userland package back in Node 7, but the warning was quiet (--pending-deprecation only) for years. Node 22 raises the volume: DEP0040 fires on stderr by default whenever anything in the process still resolves require('punycode') — and it usually isn't your code doing the requiring.

The most common source is a transitive dependency that hasn't been updated: older major versions of whatwg-url (used by jsdom, node-fetch polyfills, and various URL-parsing packages), some url-parsing utilities, and legacy AWS SDK v2 internals all require('punycode') directly instead of require('punycode/') (note the trailing slash — that one character routes to the npm package instead of the deprecated Node built-in).

Finding the source

node --trace-deprecation your-handler.js

--trace-deprecation prints a full stack trace with the warning instead of just the one-line message, which usually points straight at the offending require() inside node_modules.

If you'd rather not run the handler locally, grep for the bare require:

grep -rn "require('punycode')" node_modules --include="*.js" | grep -v "punycode/"

Anything that matches without the trailing slash is pulling in the deprecated built-in.

The fix

If it's your own code:

npm install punycode

// Before — resolves to Node's deprecated built-in
const punycode = require('punycode');

// After — resolves to the maintained userland package
const punycode = require('punycode/');

The trailing slash is not a typo — it's how Node's module resolver distinguishes the userland package (node_modules/punycode/) from the built-in of the same name.

If it's a dependency: upgrade it. Recent major versions of whatwg-url, jsdom, and the AWS SDK v3 packages have all moved off the built-in. npm ls punycode (or npm why punycode on npm 9+) shows which installed package is pulling it in, so you know exactly what to bump.

If you can't upgrade the dependency yet: this is a warning, not a breaking error — the function still runs correctly. It's safe to ship a Node 22 migration with the warning present and clean it up separately. Don't let it block the runtime upgrade itself; deprecated-runtime block dates are the deadline that actually matters. But don't ignore it indefinitely either — punycode is a real removal candidate in a future Node major, at which point it stops being a warning and starts being a crash.

Silencing it (not recommended as a permanent fix)

node --no-deprecation your-handler.js

or set NODE_NO_WARNINGS=1 in the Lambda environment variables. This hides the symptom — the transitive dependency is unchanged and will eventually need upgrading regardless. Useful for keeping noisy logs quiet during a migration window; not a substitute for actually tracking down the source.

The free EOLkits scanner flags deprecated Lambda runtimes and known dependency-level breaks like this one from your package.json / lockfile — in your browser, nothing uploaded; I maintain it. Full reference: eolkits.com/fix/node-punycode-module-deprecated.

"amazon-linux-extras: command not found — it's gone on Amazon Linux 2023"

ntoledo319 — Wed, 29 Jul 2026 07:17:03 +0000

You moved an AMI, Dockerfile, or user-data script from Amazon Linux 2 to Amazon Linux 2023, and a line that used to work now dies immediately:

bash: amazon-linux-extras: command not found

Not a missing package, not a stale repo — the amazon-linux-extras mechanism itself doesn't exist on AL2023. There's nothing to install to bring it back.

Why this happens

Amazon Linux 2 shipped one old baseline of everything in its base repo, so AWS layered the Extras Library on top — amazon-linux-extras enable nginx1.20, amazon-linux-extras install docker, and so on — to get newer versions of common packages without breaking the base image. Amazon Linux 2023 dropped that constraint (and the tool with it): its base dnf repos ship current versions directly, so there's no separate "extras" layer to enable. Anything still calling amazon-linux-extras — a golden-AMI build step, a Dockerfile RUN, a cloud-init runcmd — has no command to run.

The fix

Stop calling amazon-linux-extras on AL2023. What it used to gate now falls into one of three places:

# 1. Most former "extras" packages are just default dnf packages now
sudo dnf install -y docker

# 2. Version-specific ones are version-namespaced package names
sudo dnf install -y python3.11
sudo dnf install -y nginx1.24
sudo dnf install -y postgresql15

# 3. The rest live in SPAL (Supplementary Packages for Amazon Linux) —
#    enable it, then install as normal
sudo dnf install -y python3.11  # example: SPAL packages install like any other dnf package once the repo is present

Check which bucket your package landed in with a plain search before assuming it's gone:

dnf search <package-name>
dnf list --available | grep <package-name>

If dnf search and dnf list --available both come up empty, it's likely an EPEL-only package now (see the related Unable to find a match error for that case) rather than an extras casualty.

Fixing the automation, not just the instance

The one-off fix is deleting the amazon-linux-extras line from whatever's broken right now. The durable fix is finding every place that assumption is baked in before it breaks the next build:

grep -rl "amazon-linux-extras" Dockerfile* user-data/ ansible/ packer/ cloud-init/

Replace each hit with the direct dnf install (or version-namespaced equivalent) for that package, and drop the amazon-linux-extras enable ... step entirely — AL2023 has no topic to enable against.

This is one piece of the broader Amazon Linux 2 → 2023 migration (package manager, firewall, time sync, and Python version all shift at once) — the full checklist is at eolkits.com/fix/amazon-linux-2-eol. If you want to know which instances in your account are still on AL2 before support ends, the free EOLkits scanner checks in about 30 seconds — nothing uploaded, and I maintain it.

"Failed to start ntpd.service — Amazon Linux 2023 replaced it with chrony"

ntoledo319 — Tue, 28 Jul 2026 07:17:03 +0000

You moved an AMI, launch template, or Ansible playbook from Amazon Linux 2 to Amazon Linux 2023, and time-sync provisioning that used to just work now fails:

Failed to start ntpd.service: Unit ntpd.service not found.

The instance boots fine, dnf works, but whatever enabled or started ntpd.service — user-data, cloud-init, a Packer build, a config-management run — has nothing to attach to.

Why this happens

Amazon Linux 2 shipped ntp/ntpd for time synchronization. Amazon Linux 2023 doesn't install that package or unit by default — it standardizes on chrony (chronyd) instead, the same time-sync daemon most modern Linux distros have already moved to. Anything that assumes ntpd.service exists — a systemctl enable --now ntpd line baked into a golden AMI, a cloud-init runcmd, an Ansible task targeting ntp.conf — has no unit to enable on AL2023.

The fix

# Install and enable chrony (often already present on AL2023 AMIs — check first)
sudo dnf install -y chrony
sudo systemctl enable --now chronyd

# Verify it's syncing
chronyc sources
chronyc tracking

If you had custom NTP servers configured in /etc/ntp.conf, migrate the server/pool lines to /etc/chrony.conf (chrony's syntax is close enough that most entries port over as-is — just double-check any iburst/prefer flags still parse).

Fixing the automation, not just the instance

A one-time fix on a running instance doesn't help if the same user-data or Ansible role provisions the next hundred instances off a stale AL2 assumption. Grep provisioning code for the old unit name and update it once at the source:

grep -rl "ntpd.service\|ntp\.conf\|dnf install.*ntp[^d]\|yum install.*ntp[^d]" user-data/ ansible/ packer/

Replace systemctl enable --now ntpd with systemctl enable --now chronyd, and drop any dnf install ntp line — that package isn't what AL2023 uses for this.

"/usr/bin/env: 'python2': No such file or directory — Amazon Linux 2023 has no Python 2"

ntoledo319 — Mon, 27 Jul 2026 07:17:39 +0000

You moved an instance, AMI, or launch template from Amazon Linux 2 to Amazon Linux 2023, and something that used to run cleanly now fails on first boot:

/usr/bin/env: 'python2': No such file or directory

Or, if the script called the interpreter directly:

-bash: /usr/bin/python2: No such file or directory

dnf works, the instance is healthy, but any script with a #!/usr/bin/python2 shebang, a python2 script.py invocation, or a tool that shells out to python2/pip2 under the hood has nothing to run against.

Why this happens

Amazon Linux 2 shipped Python 2.7 as part of the base image, because plenty of legacy system tooling and older cloud-init modules still expected it. Amazon Linux 2023 made a clean break: there is no Python 2 package at all — not python2, not python27, nothing installable via dnf. AL2023 ships Python 3 (currently 3.9 as the default python3, with versioned packages like python3.11/python3.12 available) and nothing older.

This isn't a deprecation warning you can silence — the interpreter binary simply isn't in the AL2023 package repos. Anything that assumes it exists breaks the moment you provision on the new AMI:

A user-data script with a #!/usr/bin/env python2 shebang
A legacy cron job or systemd unit calling python2 /opt/app/run.py
Config management (Ansible, Chef, Puppet) with a module that shells out to python2/pip2 internally
A vendored tool or SDK that was never ported off Python 2

The fix

There is no supported way to install Python 2 on Amazon Linux 2023 — the only real path is porting the code to Python 3.

Step 1 — point the shebang/interpreter at Python 3:

# before
#!/usr/bin/python2

# after
#!/usr/bin/env python3

Step 2 — install a specific Python 3 minor version if your code depends on one:

sudo dnf install -y python3.11
# or: python3.12, depending on what you've tested against

Step 3 — update anything that invokes python2/pip2 explicitly:

grep -rl "python2\|pip2" user-data/ cron.d/ ansible/ systemd/

Replace each hit with python3/pip3 (or the versioned binary from step 2), and re-test — Python 2→3 porting issues (print statements, unicode vs str, dict.iteritems(), integer division) are a separate, well-documented migration, not an AL2023-specific one; the Python 3 porting guide covers the language-level changes.

Step 4 — check for silent failures, not just missing-binary errors:

If the Python-2-dependent step is wrapped in a script that swallows its exit code (|| true, a bare except:), you may not see the "No such file" error at all — just a feature that quietly stopped working. Grep first, don't wait for a support ticket.

"Node Sass does not yet support your current environment — what breaks when you upgrade to nodejs22.x"

ntoledo319 — Mon, 27 Jul 2026 07:17:03 +0000

You upgrade a Lambda function, a Docker base image, or your CI pipeline to Node.js 18 or 22 — and the next npm install or Docker build dies like this:

Error: Node Sass does not yet support your current environment:
  Linux 64-bit with Unsupported runtime (v22)

For more information on which environments are supported please see:
https://github.com/sass/node-sass/releases/tag/v9.0.0

Or, if the binary was already installed on an older runtime and the container now runs Node 22:

Error: The module '/app/node_modules/node-sass/vendor/linux-x64-127/binding.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 108. This version of Node.js requires
NODE_MODULE_VERSION 127.

Both are the same root cause. Here's what's happening and how to fix it permanently.

Why this happens

node-sass is a Node.js wrapper around LibSass — a C++ implementation of the Sass compiler. Because it's a native addon, it must ship a precompiled binary for each Node.js major version and platform combination. When it can't find one at install time, it falls back to compiling from source with node-gyp, which usually fails in CI or Lambda build environments that don't have the right build tools.

The core problem: LibSass and node-sass are end-of-life. The Sass team deprecated LibSass in October 2020 and archived the sass/node-sass GitHub repository in July 2024. No new prebuilt binaries are released for Node.js 18, 20, or 22. The package is frozen.

This surfaces specifically during Lambda runtime upgrades because:

nodejs16.x shipped Node.js 16 (ABI version 93) — old enough that node-sass had a prebuild.
nodejs18.x ships Node.js 18 (ABI 108). No node-sass prebuild.
nodejs22.x ships Node.js 22 (ABI 127). No node-sass prebuild, and the repo is archived.

The moment you flip the runtime and rebuild your deployment package, the install fails or the previously-built native binary refuses to load.

The fix: migrate to Dart Sass

The sass package is Dart Sass — the official, actively-maintained Sass implementation. It compiles to pure JavaScript: no native addon, no prebuilt binary, no ABI mismatch, no node-gyp. It installs cleanly on every Node.js version from 12 onward.

Step 1 — swap the package:

npm uninstall node-sass
npm install --save-dev sass

That's the whole migration for most projects.

Step 2 — update webpack's sass-loader (if you use it):

Older sass-loader versions would try node-sass first. With node-sass removed, it auto-detects sass — but if you have an explicit implementation option in your webpack config, update it:

// webpack.config.js — before
{
  loader: 'sass-loader',
  options: { implementation: require('node-sass') }
}

// webpack.config.js — after
{
  loader: 'sass-loader',
  options: { implementation: require('sass') }
}

Step 3 — Angular CLI:

Angular projects that specify node-sass in the stylePreprocessorOptions or have it as a dependency can swap to sass with no config change — Angular CLI auto-detects the installed implementation.

Step 4 — react-scripts / Create React App:

CRA uses sass as the recommended processor. Remove node-sass, install sass, and the build picks it up automatically.

Compatibility: is it a drop-in?

For the overwhelming majority of real-world SCSS codebases, yes. The only behavioral differences you might hit:

/ division in Sass — Dart Sass warns that / is deprecated as a division operator inside calc-style expressions (use math.div() from sass:math instead). This is a deprecation warning, not a breakage — your build still works.
!global in nested scopes — a rarely-used pattern; Dart Sass handles it consistently with the spec.
Indented syntax (.sass files) — fully supported.

If your tests pass after the swap, you're done. Run npx sass --version to confirm you're on Dart Sass (it prints X.Y.Z compiled with dart2js).

Lambda-specific check: build the package on the right runtime

After the migration, make sure your deployment package is built on the same Node.js version Lambda will run:

# Build inside the Lambda container image to guarantee ABI match
docker run --rm -v $(pwd):/app -w /app \
  public.ecr.aws/lambda/nodejs:22 \
  npm ci --production

Because sass is pure JS, this is less critical than before — but it's still good practice for any Lambda package containing native modules.

Timeline: why this hits now

Runtime	Node.js	Last node-sass prebuild	Phase 2 (create-block)
`nodejs16.x`	16	✓ v9.0.0	Blocked
`nodejs18.x`	18	✗ archived	Blocked
`nodejs20.x`	20	✗ archived	~late 2026
`nodejs22.x`	22	✗ archived	—

AWS blocks creating new functions on deprecated runtimes first, then blocks updating existing ones. nodejs16.x is already fully blocked. Teams migrating away from it are hitting this node-sass wall now.

The free EOLkits scanner detects deprecated Lambda runtimes and known packages that require rebuilt binaries after a runtime upgrade — in your browser, nothing uploaded. Full per-error fixes: eolkits.com/fix/node-sass-deprecated-unsupported.

"ModuleNotFoundError: No module named 'smtpd'" / "'asyncore'" — Python 3.12's PEP 594 stdlib cleanup

ntoledo319 — Sun, 26 Jul 2026 07:17:03 +0000

You moved a Lambda function (or any service) to the python3.12 runtime and something that worked fine on 3.11 now dies at import time with one of:

ModuleNotFoundError: No module named 'smtpd'

ModuleNotFoundError: No module named 'asyncore'

Nothing in your code changed. These modules are just gone.

Why this happens

smtpd, asyncore, and asynchat were all deprecated in Python 3.6 with explicit DeprecationWarnings, and all three were removed outright in Python 3.12 as part of PEP 594 — the standard library's long-running "dead batteries" cleanup (the same PEP that also dropped distutils, imp, and several others).

They tend to surface in a few specific places:

smtpd — a local DebuggingServer or test SMTP server used in integration tests, or a script that fakes outbound mail for local development.
asyncore / asynchat — older network code written before asyncio existed, or a dependency (often something mail- or FTP-related) that never migrated off the callback-based event loop these modules implemented.

Either way, the pattern is the same one behind most Python-3.12 Lambda breakage: code that was already deprecated-but-working for years suddenly hard-fails the moment the runtime bump turns the warning into a missing import.

Fix — smtpd

Replace it with aiosmtpd, the maintained asyncio-based successor:

pip install aiosmtpd

# Before (removed in 3.12)
from smtpd import DebuggingServer
import asyncore
server = DebuggingServer(('localhost', 1025), None)
asyncore.loop()

# After
import asyncio
from aiosmtpd.controller import Controller

class PrintHandler:
    async def handle_DATA(self, server, session, envelope):
        print(envelope.content.decode('utf8', errors='replace'))
        return '250 Message accepted for delivery'

controller = Controller(PrintHandler(), hostname='localhost', port=1025)
controller.start()

If you only need the removed module back verbatim rather than rewriting call sites, the standard-smtpd package on PyPI restores it as a drop-in shim — a faster stopgap, not the long-term fix.

Fix — asyncore / asynchat

Port the code to asyncio, which is the supported replacement for the transport/protocol model asyncore implemented:

# Before (removed in 3.12): asyncore.dispatcher subclass with handle_read/handle_write
# After: an asyncio.Protocol
import asyncio

class MyProtocol(asyncio.Protocol):
    def data_received(self, data):
        # equivalent to handle_read
        ...

    def connection_lost(self, exc):
        # equivalent to handle_close
        ...

If the import comes from a third-party dependency rather than your own code, upgrade it first — most actively maintained libraries that used asyncore shipped an asyncio-based release well before 3.12 removed it. For a genuinely unmaintained dependency, vendoring the pure-Python asyncore.py/asynchat.py from a Python 3.11 source tree is a working (if inelegant) stopgap while you plan the real port.

Catch every PEP 594 removal before the deploy, not after

Both of these tend to arrive together, and rarely alone — a codebase old enough to import smtpd or asyncore is usually old enough to still import imp or rely on distutils too. Smoke-test the whole surface on 3.12 before flipping the runtime label:

python3.12 -c "import your_module"

Running that against every entry point (handler, layer, and any script the deploy pipeline invokes) surfaces the full removal list at once, instead of one ModuleNotFoundError per cold start.

Free EOLkits scanner flags every deprecated Python/Node Lambda runtime in an account in about 30 seconds, nothing uploaded — I maintain it. Full references: eolkits.com/fix/python-no-module-named-smtpd and eolkits.com/fix/python-no-module-named-asyncore.

"Failed to start iptables.service — Amazon Linux 2023 dropped it for nftables"

ntoledo319 — Sat, 25 Jul 2026 07:17:03 +0000

You moved an AMI, launch template, or Ansible playbook from Amazon Linux 2 to Amazon Linux 2023, and a step that used to just work now fails:

Failed to start iptables.service: Unit iptables.service not found.

The instance is healthy, dnf works, but whatever enabled or started iptables.service — user-data, cloud-init, a Packer build, a config-management run — breaks on first boot.

Why this happens

On Amazon Linux 2, the iptables-services package provided an iptables.service unit that saved and restored firewall rules across reboots (iptables-save / iptables-restore under systemd). Amazon Linux 2023 doesn't install that package or unit by default — it standardizes on nftables as the default firewall backend instead. Anything that assumes iptables.service exists — a systemctl enable --now iptables line baked into a golden AMI, a cloud-init runcmd, an Ansible task — has nothing to attach to on AL2023.

The iptables command itself isn't gone (there's an iptables-nft compatibility layer), but the systemd unit your automation expects is.

The fix, two ways

Move to native nftables (recommended for anything new):

# Convert existing iptables rules to nftables syntax
sudo iptables-save > /tmp/rules.v4
sudo iptables-translate -f /tmp/rules.v4 > /etc/nftables/rules.nft   # per-rule; wrap in a table/chain if starting fresh

# Enable and start the nftables service instead of iptables.service
sudo systemctl enable --now nftables

# Verify
sudo nft list ruleset

iptables-translate converts rules one at a time — for anything beyond a handful of rules, review the generated nftables syntax rather than trusting a blind bulk conversion.

Keep iptables syntax via the compatibility layer (faster migration, same commands):

sudo dnf install -y iptables-nft

Rules written with iptables/ip6tables commands are transparently stored in the nftables backend. You keep your existing iptables -A ... scripts, but persistence is managed by the nftables service, not iptables.service — update any systemctl enable iptables reference to systemctl enable nftables.

Fixing the automation, not just the instance

The one-time instance fix doesn't help if the same user-data or Ansible role provisions the next hundred instances. Grep your provisioning code for the old assumption and update it once at the source:

grep -rl "iptables.service\|iptables-services" user-data/ ansible/ packer/

Replace systemctl enable --now iptables with systemctl enable --now nftables (or the iptables-nft equivalent), and drop any dnf install iptables-services line — that package doesn't exist on AL2023.

"Error: Unable to find a match — fixing dnf package lookups after migrating off Amazon Linux 2"

ntoledo319 — Fri, 24 Jul 2026 07:17:38 +0000

You moved an instance, AMI, or user-data script from Amazon Linux 2 to Amazon Linux 2023, and a package install that's worked for years suddenly fails:

Error: Unable to find a match: python3-devel

(or whatever package your script names). The instance boots fine, dnf works, but this one package is gone.

Why this happens

Amazon Linux 2023's repositories aren't the same tree as AL2's — they were rebuilt, and along the way packages got renamed, version-namespaced, moved to a supplementary repo, or dropped outright:

Version-namespaced. AL2023 ships multiple versions of things side by side instead of one default, so the plain name is gone in favor of a versioned one: python3 → python3.11 (or python3.12), nginx → nginx1.24, postgresql → postgresql15.
Moved out of the base repo. Packages that lived in amazon-linux-extras on AL2 (which doesn't exist on AL2023 at all) are now either default dnf packages, version-namespaced ones, or live in SPAL (Supplementary Packages for Amazon Linux) — a repo you have to enable separately.
Moved to EPEL. Some AL2 packages were never Amazon's to begin with and only ever came from EPEL; if your script assumed they'd resolve from the base AL2023 repos, they won't until EPEL is enabled.
Actually removed. A handful of packages just aren't packaged for AL2023 at all.

Scripts that hardcode an AL2-era name and run yum install <name> (which on AL2023 is a symlink straight to dnf) hit this the moment they run against the new repo set.

Finding the real name

Don't guess — ask dnf directly:

# Search by keyword
dnf search python3-devel

# Search by the actual binary/file you need, if you know it
dnf provides '*/pip3'
dnf provides '*/nginx'

dnf provides is the more reliable of the two when you know the file the old package used to install (a binary, a .so, a config path) but not what the new package is called.

The fix, by cause

Version-namespaced package: install the versioned name directly.

dnf install -y python3.11 python3.11-devel   # or python3.12
dnf install -y nginx1.24
dnf install -y postgresql15 postgresql15-server

Used to come from amazon-linux-extras: that command doesn't exist on AL2023 — there's nothing to enable it against. Check whether the package is now a default dnf package, a version-namespaced one, or lives in SPAL:

dnf repository-packages amazon-linux-extras-equivalent-repo list  # naming varies by topic; check SPAL docs for the exact repo id

In practice most former amazon-linux-extras topics (nginx, php, docker CLIs, etc.) are just direct or version-namespaced dnf installs on AL2023 now — the extras layer was there because AL2's base repo shipped one old version of everything; AL2023's doesn't have that constraint.

Genuinely EPEL-only: enable EPEL for AL2023, then install normally.

dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
dnf install -y <package>

(Confirm the EPEL release package that matches AL2023's current EPEL support before pinning a version — the AL2023 user guide documents which EPEL release to use.)

Actually dropped, no replacement: vendor a static binary, build from source against AL2023's toolchain, or containerize the workload instead of installing it as a system package.

Before you standardize on a name

AL2023 versions packages more aggressively than AL2 did, which means today's python3.11 might not be tomorrow's default. Check a package's support window before hardcoding it into a golden AMI or long-lived user-data script:

dnf install -y dnf-plugin-support-info
dnf support-info <package>

That tells you how long the specific package version is supported — useful for avoiding a repeat of this exact migration in a couple of years.

"Runtime.ImportModuleError: Cannot find module — a triage guide for Lambda runtime upgrades"

ntoledo319 — Fri, 24 Jul 2026 07:17:03 +0000

You bumped a Lambda function to a newer runtime, redeployed, and the first invocation dies at cold start:

Runtime.ImportModuleError: Error: Cannot find module 'some-package'

(Python's version reads Unable to import module 'handler': No module named 'some_package' — same underlying category of failure.)

The unhelpful part: this exact message is the symptom of at least four different root causes, and none of them are code bugs — your handler didn't change. Here's how to tell them apart before you start guessing.

The triage

1. Is the missing module aws-sdk (lowercase, no @) and did you move to nodejs18.x or later?

That's not a bundling problem — AWS stopped preinstalling AWS SDK v2 in the runtime starting with nodejs18.x. Only @aws-sdk/* v3 modular clients ship now. This has a dedicated fix with the full v2→v3 migration: eolkits.com/fix/node-cannot-find-module-aws-sdk.

2. Is the module genuinely in package.json/requirements.txt, and did you switch bundlers or bump esbuild recently?

esbuild 0.22+ changed its default to treat node_modules as external — meaning a real dependency can still get silently excluded from the deployment ZIP. If you use SAM, CDK, or Serverless Framework with the esbuild build method, force it back to inline:

# SAM template.yaml
Metadata:
  BuildMethod: esbuild
  BuildProperties:
    Packages: bundle

or esbuild --packages=bundle directly. Confirm by unzipping your build artifact and checking whether the module directory is actually present under node_modules/ — don't trust the build log alone.

3. Does the error only happen through a Lambda Layer, and did you build that layer on your laptop or a generic CI image?

A layer built on the wrong base image is a structure/path problem, not a missing-package problem: Node expects nodejs/node_modules/<package> inside the layer zip, Python expects python/<package>. If the layer was zipped with a different top-level folder, or built for the wrong architecture (x86_64 vs arm64), the runtime won't find it even though the package is technically "in" the layer. Rebuild inside the actual Lambda base image for your target runtime and architecture:

docker run --rm -v "$PWD":/opt public.ecr.aws/lambda/nodejs:22 \
  npm install --prefix /opt/nodejs

4. Does the module load fine on your machine but fail only on Lambda, and does it involve a native binary (.node file, or a Python package like psycopg2, numpy, pydantic-core, grpcio)?

That's usually not a "cannot find" problem at the module-resolution level at all — it's the loader finding the file and then failing to link it, which Node/Python often still surfaces as an import error. This is a glibc/architecture mismatch between where the binary was compiled and the runtime's base OS. Full diagnosis and fix: eolkits.com/fix/lambda-glibc-version-not-found.

If none of the four fit

Check the boring things first, in order: does the Handler setting (file.export for Node, file.function for Python) still match where the file actually lives in the deployment package after the runtime change moved your build output layout? Did the runtime change also change the working directory the bundler assumes (/var/task vs /opt)? A surprising fraction of "phantom" ImportModuleErrors are a handler path pointing at a file that no longer exists at that path post-rebuild, not a missing dependency at all.

Catching this before a cold start in production does

All four causes above are detectable from the deployment artifact itself, before you ever invoke the function — which matters because ImportModuleError only surfaces at cold start, so a broken deploy can sit "successful" for minutes or hours before anyone notices.

If you're tracking deprecated-runtime exposure across an account already, the same scan pass is a natural place to also flag native-dependency ABI mismatches and stale bundler configs before a redeploy trips one of these: the free EOLkits scanner checks an account in about 30 seconds, nothing uploaded — I maintain it.