DEV Community: Anton Golub

yarn.lock: may the `--force` be with you

Anton Golub — Tue, 29 Sep 2026 06:56:40 +0000

Part 2 brought Yarn level with npm audit fix; this part goes past it. --force is the one move npm already had — the rest is new: gate a fix on the Node you target, the licenses you allow, the module format it must keep, and scope it to production, or to the one workspace you're touching.

yarn audit [still does not] fix
yarn.lock: you can't sed a graph
yarn.lock: may the --force be with you

TL;DR

"Lowest version that clears the advisory" (Part 2) is necessary, not sufficient: that fix — or the closure it drags in — can need a newer Node than you target, a license you don't allow, or fall outside a range you pinned. --immutable won't catch it — it checks the lockfile reproduces, not that the result runs or complies.
Landing a fix is a dial. --force bends ranges and rewrites package.json to force a patch in; the opt-in constraints — --engines, --license, --package-type — pull the other way, applying a fix only if it won't introduce a new break, and --safe flips the automatable ones on together. The contract is guarantee-or-flag: a compatible fix or an attributed skip, never a silent incompatible one.
Two more knobs narrow which advisories, not just how a fix resolves: --production skips dev-only vulnerabilities (they never ship), --workspace scopes a monorepo fix to the packages you name.
Every skip is attributed — the axis, the package, the versions it tried — so you can widen, exclude, or accept. Constraints, --safe, scoping, and precise suppression (--ignore a CVE, --exclude a package) are all things npm audit fix doesn't do: this is where yaf leads it, not mirrors it.

The fix that installed clean and broke anyway

Part 2 ended on a strong claim: the patched lockfile passes yarn install --immutable, byte-for-byte what Yarn would compute. That invariant is real, and it's the whole design — but it's worth being clear about what it doesn't promise.

Immutable mode is a reproducibility oracle. It verifies that the lockfile agrees with itself and with your manifests — same versions, same checksums, no drift. It does not verify that the versions it locked will run. Berry doesn't check a dependency's engines at install at all (npm merely warns, EBADENGINE); licenses aren't its concern either; and a range you declared is checked for consistency, not for whether the fix was a good idea.

A security bump moves a package to its lowest patched version — and that version's dependencies, re-resolved the Part-2 way to the highest satisfying version, pull in a transitive whose recent major quietly dropped old Node support. lru-cache@11 wants Node 20; glob@11, rimraf@6 — same. Your project targets Node 18. yarn install is delighted: the graph resolves, the checksums match, --immutable exits 0. Then CI runs the suite on Node 18 and something three levels down throws SyntaxError on a syntax it doesn't recognize, or the process exits on an engines assertion. The fix was "correct" by every check we'd built — and it broke the build.

We hit exactly this on a real project: clean remediation, green install, and it died on Node 18 in CI. The lockfile was right; the fix was wrong for that project.

So a fix has a footprint beyond the advisory — and the real question is how much of your project it's allowed to reshape to land. That's a dial. At one end, --force: bend a consumer's range, rewrite a manifest, take the major bump — make it fit. At the other, the constraints and --safe: apply a fix only if it leaves your Node, your licenses, and your declared ranges alone. The rest of this part is those two ends, and the default that sits between them.

`--force`: break to fix

The forceful end of the dial is the oldest flag in the tool, and the bluntest. By default a fix has to fit: it must stay within every surviving consumer's declared range and within the range you declared for a direct dependency in package.json. When it can't — the only patched version is a semver-major that overshoots a parent's ^1, or a dependency you pinned exactly — the default declines and reports, because quietly applying it would hand you a lockfile your own manifests forbid:

Skipped (package.json pins these to a range the fix can't satisfy; re-run with --force …):
  lodash (pinned → "4.17.11" in packages/foo/package.json)

--force overrides that. It takes the cross-major bump past a consumer that wanted the old line, and — the newer half of this — rewrites the direct-dependency range in your package.json so manifest and lockfile agree again ("lodash": "4.17.11" → "4.18.0", "~4.17.0" → "~4.18.0", preserving your operator). In a monorepo it finds the pin in the workspace that declared it and rewrites that file, not the root. Both moves — cross the range, rewrite the pin — are exactly where the community plugin (Part 1) stops: it fixes within a package's range and, when the only patch lies outside it, reports no compatible patched version and gives up on it. --force is the out-of-range flag its roadmap wanted.

It works the same across formats: Berry models workspace→dependency edges in the graph, so its compatibility gate (Part 2) can already see a workspace consumer; Yarn Classic can't, so reading the manifests is the uniform path — and it adds the --force rewrite the edge-only gate never had. Same move as npm audit fix --force, done out loud.

It earns its Star Wars reputation, though. The case against npm audit fix --force is concrete: it installs versions outside your declared ranges — semver-major, 1.2.0 straight to 5.0.0 if that's where the patch lives — so a function you call gets renamed or removed and the app won't start, or starts and quietly returns wrong answers. One forced top-level bump cascades into dozens of moved transitives, each its own chance to break. It can even oscillate — downgrade, upgrade, downgrade — into a lockfile that differs from one run to the next. And a green test suite is thin cover: the real break tends to hide in an integration seam or an edge case that surfaces months later, sometimes as a fresh vulnerability in a version nobody has battle-tested yet.

yaf's --force is that instinct, narrowed. It bumps to the lowest version that clears the advisory, never the latest, so the jump is the smallest one that works; it moves the vulnerable package and re-resolves its closure, not the whole tree; and it stays bounded by the Part 2 reproducibility check and whatever constraints you've set. So --force here means bend the ranges that block this fix, not take the resolver off its leash. Sometimes the real fix is the major bump and you'd rather the tool take it and say so than hand you a red pipeline — but it's opt-in, loud, and paired with an opposite.

`--safe`: skip to stabilize

That was the forceful end. The other runs on the opposite instinct: when landing a fix would break something new, skip it — leave the vulnerability flagged rather than trade it for a fresh breakage. It's the newer half, and still finding its edges. Before its mechanics, the policy, because it too departs from what npm does, on purpose.

Run npm audit fix and a bump that needs a newer engine still gets applied; you get an EBADENGINE warning and a lockfile that installs. That's the package-manager default: resolve the highest thing that satisfies semver, warn, move on. It's a reasonable default for an always-on installer.

It's the wrong default for an opt-in remediation flag. If you go out of your way to tell the tool "this fix has to run on Node 18," you don't want a warning buried in install output three CI steps later — you want one of two things: a fix that actually meets the bar, or a clear refusal that names what it couldn't satisfy.

So the constraints are off unless you ask; ask, and highest-satisfying-that-also-passes replaces highest-satisfying-and-warn. When nothing passes, the default is to skip that one fix and report it — leave the vulnerability, tell you exactly why — with --on-conflict=stop to turn any such skip into a hard error for a CI gate that must not proceed on a half-fix.

Engines: will the fix run on the Node you target?

yarn-audit-fix --engines.node='>=18'

Reads as: apply a fix only if its completed closure runs on Node ≥ 18. Every newly-pulled transitive is now selected as the highest in-range version whose own engines.node admits your floor. The lru-cache@11 from the opening is rejected for needing Node 20; if an in-range lru-cache@10 line runs on 18, that's what lands. If nothing in range clears the bar, the fix is skipped and attributed (below).

Three ways to name the target, because "what Node do you support" has three real answers:

A range — --engines.node='>=18'. Explicit, unambiguous, what CI should use.
Bare — --engines.node, meaning the Node running the CLI. Cheapest, and the report flags that it may differ from your project's target (the classic footgun: you're on 22, prod is on 18).
floor — --engines.node=floor, meaning the highest engine floor your project already requires. We read the root package.json, every workspace manifest, and every hoisted package in node_modules, take the maximum engines lower bound, and target that. The invariant it encodes: a fix must not raise the floor your tree already stands on. If the tree already needs 18, a fix is fine as long as it doesn't quietly push you to 20.

The check is lenient in the npm sense — a package that declares no engines makes no claim and is accepted (rejecting the silent majority would be useless). And it's not just for node: --engines.npm='>=9' gates the same way on any engine your closure declares.

One subtlety that matters for correctness: the gate covers the fix version itself, not only its closure. Completion re-resolves the transitives, but the package you're bumping to is placed directly — so a fix that lands on a version whose own engines exceed your target would slip past a closure-only check. A pre-apply pass runs the same constraints against the fix version before it's committed; if the fix itself doesn't fit, it's skipped with "the fix version itself is not permitted." Closure and seed, both gated.

Licenses: does the fix pass your policy?

yarn-audit-fix --license.allow=MIT,ISC,Apache-2.0,BSD-3-Clause
# or the other direction:
yarn-audit-fix --license.deny=GPL-3.0

Same machine, different axis: apply a fix only if every new package it pulls in carries an allowed SPDX license (or none of a denied one). A security bump that drags a copyleft transitive into a proprietary codebase is exactly the kind of thing you'd rather catch at remediation time than at legal review.

There's a data wrinkle worth surfacing, because it drove a real change in the pipeline. The registry's abbreviated package document — the compact "corgi" metadata the completion walk fetches for speed — carries engines but not license. So the engine gate is free (it reads metadata we already have), while the license gate has to fetch the full manifest for a candidate to see its license field. The tool exposes that full-manifest fetch through the same scope-aware, host-bound registry client the rest of the pipeline uses, and the license gate reaches for it only when a candidate actually needs licensing — cheap gates first, so a version an in-memory engine check already rejected never triggers a license round-trip.

v1 compares single SPDX ids. An SPDX expression — (MIT OR Apache-2.0) — isn't something a token match can decide: under OR, one allowed license should clear the whole thing, but a naive scan trips on the disallowed half. So rather than guess, an unresolvable expression is treated as unevaluable and, under the "guarantee-or-flag" contract, flagged rather than silently accepted.

Package format: does the fix stay require-able?

yarn-audit-fix --package-type=cjs

A third axis, built on a limit worth stating plainly: we can't verify a change fully, but we can check its consistency. A good chunk of the ecosystem has gone ESM-only — a package flips to "type": "module", drops its CommonJS entry, and now require('that-package') throws ERR_REQUIRE_ESM. A security bump that lands on such a version, or drags one in transitively, breaks every CommonJS require of it at runtime — cleanly, past every install check.

We can't run your code to know whether that break is real for you. But we can check the thing that makes it possible: does the fix's closure stay require-able? --package-type=cjs rejects a new package that's ESM-only — "type": "module" with no CJS main and no require/default export condition. A package that makes no type claim is CommonJS by default and passes; a dual-mode one (an ESM build plus a require export) passes; only the genuinely ESM-only versions are held back.

Unlike engines and license, this isn't a library built-in — and that's deliberate. True require(ESM) compatibility is per-edge and Node-version-gated (newer Node keeps relaxing it), so a node-local yes/no is an approximation, not a proof. lockgraph ships engines and license as built-in conditions and takes custom ones through the same seam; yaf plugs its module-format check in there — checking entry-point consistency, which it can, and staying upfront that it isn't checking behaviour, which it can't.

Scope: which advisories, not just which fix

--force and --safe tune how a fix lands. A separate control tunes how much gets fixed at all — because not every advisory is worth acting on in every context, and the smallest change that clears the risk is usually the right one.

--production fixes only advisories reachable from your production dependencies. A vulnerability that lives entirely under devDependencies — in a test runner, a bundler, some linter's transitive — never ships, so patching it churns the lockfile for a risk your users never run. --production leaves those flagged and fixes the rest. The split can't be read from the lockfile — yarn.lock records no dev/prod distinction; every edge is just depends on. So it's computed from each package.json, walked over the graph: start at the production dependencies, follow the edges, and a package is in scope only if production actually reaches it.

--workspace=<glob> narrows a monorepo fix to the packages you name — --workspace=packages/api, --workspace='@scope/*'. Remediating one service's advisories in a fifty-package repo without touching the other forty-nine keeps the diff reviewable and the blast radius small. A workspace pulled in as a dependency of one you selected contributes only its production closure, so a neighbour's dev tree never leaks into your fix.

Two more narrow by identity rather than reach. --ignore drops a specific advisory before the fix starts — a GHSA or npm-advisory-id glob — so a CVE you've assessed and accepted (a dev-only false positive, a disputed report) never triggers a bump. --exclude holds a package still by name and optional version range: --exclude='lodash@>=2 <3' leaves that line alone wherever it sits in the tree, however deep. Neither has an equivalent in npm audit fix — to fix everything except one CVE, or to freeze one nested package, you'd reach for overrides or a third-party audit wrapper.

They all compose into one precise instruction — --production --workspace=packages/api --safe, with --ignore / --exclude trimming the exceptions you've already vetted — so the fix set is exactly the one you meant, not everything the audit happened to return.

Attribution: a skip you can act on

A gate that silently drops fixes is worse than no gate — you'd never know a vulnerability was left behind, or why. So every skip carries its reason, composed from the constraint you set, the axis that rejected, the package that blocked, and the versions it tried:

Constraints: node >=18
Skipped (constraints — no fix keeps the closure within the policy [node >=18];
         relax it, --exclude the package, or accept the newer dep):
  some-tool@1.4.0 → 2.0.0 — needs lru-cache@^11.0.0
    - lru-cache@11.5.2: engines.node '20 || >=22' does not support node 18.0.0 (target '>=18')
    - lru-cache@11.0.0: engines.node '20 || >=22' does not support node 18.0.0 (target '>=18')

From that you know precisely what to do: widen the target if 20 is actually fine, --exclude some-tool to accept the residual risk deliberately, or pin lru-cache yourself. The vulnerability is still there — but it's visible and explained, which is the whole difference between "skipped" and "swept under the rug." The license axis reads the same, with license MIT not permitted in the reason slot; a rejected fix version reads "the fix version itself is not permitted."

For a pipeline, the same account comes as data: --dry-run --json prints what would be upgraded and what would be skipped — each skip with its reason (consumer-range, override-pin, manifest-pin, constraint, out-of-scope) — as one JSON object, before anything is written. A CI step can gate on what a fix would leave behind, not just on what it would change.

Under the hood

Three properties hold it in line: off by default, no effect on the reproducibility guarantee, and no claim beyond what it checks.

Off means off. With no constraint set, the apply path is exactly Part 2's: rebind everything, complete the closure once, byte-identical output — the golden and real-world fixtures prove it. The constraint machinery is a branch you only enter by opting in.

Tentative apply, immutable graphs. When constraints are active, each fix is applied and its closure completed on a trial graph, and committed only if it clears every gate; a fix that hits a dead end is discarded and the graph is left exactly as it was, then the next fix is tried. That's clean precisely because the graph library is immutable — modify and complete return new graphs rather than mutating in place, so "try this fix, and if it doesn't pass, pretend it never happened" is just don't keep the result. No rollback, no partial state. The constraints themselves ride the library's own resolution seam — the same step that already honored resolutions (Part 2) also runs your acceptance gates — so "don't pick an engine-incompatible version" is a property of resolution, next to "don't violate a known pin," not a filter bolted on afterward.

It's a bet, not a proof. The safe end rests on one wager: that checking each fix against its own declared metadata — the version it lands on, and the ones it drags in — is enough, without a solver backtracking the whole tree. That check is node-local, which buys a single guarantee: it can only be too cautious, never too permissive. It won't hand you an engine-incompatible or ESM-only pin. At worst it over-reports, refusing a fix a backtracking resolver might have landed by picking a lower parent upstream. That case is narrow, and the skip report names it, so you close it by hand. So far it holds; the corners get named and narrowed one at a time.

Where this leaves the tool

Part 2's claim was the fix reproduces. Part 3's is a step further: the fix reproduces **and fits** — runs on the Node you target, passes the license policy you set, agrees with the ranges you declared, across every workspace in a monorepo. All opt-in, all guarantee-or-flag, all attributed when they can't.

The lowest patched version clears the advisory. Whether it's the right version, only your project can answer — so the tool leaves that call to you, and flags the ones where the answer is "not this one."

There's a single stance under all of it. --force says apply everything, break what you must. The constraints are its mirror image — there's even a --safe flag that flips the automatable ones on together: apply what fits, leave what doesn't for a deliberate hand. Update what you safely can now, and schedule the engine bump, the license swap, or the ESM migration as its own change rather than smuggling it inside a security patch. The tool's job is to make the split legible: what it fixed, what it couldn't, and why.

If Yarn ships a native audit fix one day, we'll retire ours happily — constraints and all. Until then, it fixes your lockfile, and names the fixes it can't land without breaking something else.

Sources / links

The tool: yarn-audit-fix (npm) — --engines.<engine>, --license.allow/.deny, --package-type, --safe, --on-conflict, --production, --workspace, --json, and the --force manifest rewrite.
The library: lockgraph — the complete step's constraint seam (engines / license conditions, plus custom ones) and the immutable graph operations the tentative apply relies on; the per-format specs (including the missing dev/prod marker) live under docs/spec/formats.
Background: Why npm audit fix --force is risky, SPDX license list, EBADENGINE in npm

yarn.lock: you can't `sed` a graph

Anton Golub — Tue, 29 Sep 2026 06:55:05 +0000

After the first workarounds, a lockfile-graph library appeared — and it changed what a fix even is: not a text edit, but graph surgery. How lockgraph rebuilt yarn-audit-fix from the inside, down to the unglamorous work of emitting a file that still installs.

yarn audit [still does not] fix
yarn.lock: you can't sed a graph
yarn.lock: may the --force be with you

TL;DR

A lockfile fix is not string replacement. Bump a package and its transitive closure can change, its checksums change, and the file still has to reproduce under --immutable. Miss any of those and you've shipped a lockfile that doesn't install.
lockgraph parses every Yarn lockfile — Classic (v1) through Berry (v4–v10, Yarn 2 onward) — into one dependency graph; the fix is a transformation on that graph, serialized back to the original format. Registry-only, spawn-free — no npm/yarn child process.
The hard part is the closure: re-resolve the upgraded node's dependencies the way yarn install would — honoring a project's declared resolutions pins rather than silently resolving around them — and recompute checksums, so the result passes yarn install --immutable.
Field proof: a real monorepo build broke because a simple-git bump needed two sub-packages it never received. The graph model adds them.

Recap: the wall

Part 1 ended on the difficulty that actually matters:

Bump one entry and its transitive dependencies should change too — but Yarn doesn't reload a patched entry's dependencies for you. A correct fix has to re-resolve subtrees against the registry and keep the lockfile internally consistent and deterministic. With PnP and Yarn's strict linker, a naive in-place edit yields an install that doesn't reproduce.

A lockfile isn't a list of versions; it's a resolved graph — every package, every edge, pinned and checksummed so an install is byte-reproducible. "Fix the vulnerability" means edit the graph and keep it resolved. A sed one-liner gives you a file that lints fine and installs wrong: you can't sed the graph underneath it.

So we stopped editing text and started editing the graph.

One graph for every Yarn lockfile

Underneath the tool is a separate, single-purpose library — lockgraph — whose only job is to turn lockfiles into a graph and back:

detect(text)            → which format is this?
parse(text, format)     → a dependency Graph (nodes = packages, edges = ranges)
…transform the graph…
stringify(graph, format) → text in the SAME format, quirks preserved

The formats it round-trips for us: Yarn Classic yarn.lock (v1) and Yarn Berry (the v4 through v10 schemas, Yarn 2 onward). One model, many serializers. That's the leverage: the fix is written once, against the graph, and every Yarn schema gets it. The serializer's job is to be boring — emit the exact dialect Yarn expects, down to key ordering and quoting, so a clean tool run produces a diff that's only the fix.

The same library also round-trips npm, pnpm, bun, and Deno locks through one model, but that breadth is its own story. Here, every lock is a yarn.lock.

A node is a resolved package (lodash@4.17.20, with its integrity, its dependency ranges). An edge is "this descriptor (lodash@^4) resolves to that node." Once the file is a graph, a security fix is just graph surgery: move a node, rewire its edges, complete what's now missing, drop what's now orphaned.

The pipeline

Five steps, all over HTTP to a registry, no subprocess.

1. Advisories, straight from the registry

The library hands us the package set that actually needs auditing — graph.registryPackages() — which is locator-aware: it skips workspace members, link: / portal: / patch: / file: / git: entries, and anything whose version isn't real semver. Those can't have a registry advisory and shouldn't be POSTed.

That set goes to the registry's bulk audit endpoint (/-/npm/v1/security/advisories/bulk), chunked, over plain HTTPS — the same endpoint yarn npm audit proxies, talked to directly. Auth is resolved from your project config (.npmrc, .yarnrc.yml, env), host-bound and https-only: a token declared for one registry is never attached to a request to another host, and never sent over http. Scoped packages route to their own registry with their own credential.

What comes back is normalized into a report: severity and CVSS, the GHSA/CVE refs, and — because advisory ranges are the awkward data from Part 1 — a patched range derived from the vulnerable one. --audit-level and --ignore (npm id or GHSA) filter here. When a package carries several advisories they merge: the vulnerable ranges OR together, the patched ranges AND together.

2. Choosing the fix version

For each vulnerable node we ask the registry for the lowest published version that clears the merged advisory set. Lowest, not latest — the smallest move that makes you safe is the least likely to break you.

"Lowest that clears the merged set" has a subtlety worth showing. Take lodash at 4.17.20 with five advisories. If four are fixed by 4.17.21 but one is only fixed in 4.18.0, the AND of their patched ranges is >=4.18.0, and the lowest version satisfying that is 4.18.0. So the tool lands on 4.18.0 — not because it prefers newer, but because that's the floor where every advisory is cleared at once.

Then the compatibility gate, which is the default safety rail: a fix is applied only if the new version still satisfies every surviving consumer's declared range. If clearing the advisory requires a cross-major jump that breaks a parent's ^x range, the default declines and reports it rather than quietly breaking your build — and --force is there for when you've decided to take the major bump.

There's a second rail alongside consumer ranges: a project's own resolutions. Pin a transitive version deliberately, and a fix that ignored that pin would emit a lockfile contradicting its own package.json. So resolutions bound the remediation too, and yaf takes the same line npm audit fix --force takes with a user's override. A pin holding a version the fix can't satisfy is authoritative: yaf leaves it and reports it, never silently rewriting your pin. A pin whose range still admits the fix is honored — the fix resolves within it. Either way the emitted lock agrees with what yarn install would produce. This is where modeling pays off: the graph library treats resolutions as first-class constraints rather than a special case bolted onto the patcher, so "don't violate a known pin" is a property of the resolution step, not an afterthought.

3. The transformation

Now the graph surgery, in three moves.

Rebind (modify with a replaceVersion operation): add the patched node and redirect the edges that pointed at the old one onto it. The operation reports back what it disturbed — which nodes it just added, which it left orphaned — and that report seeds the next step.

Complete the closure (complete): this is the step the naive tools skip and the one Part 1 called the hard part. The new version's own dependencies may differ from the old one's. So we walk the newly-added nodes, fetch each one's metadata, and for every dependency it declares we either reuse the resolution the lock already records for that descriptor or resolve a new one and wire it in. And those new nodes have new dependencies of their own — so it's re-resolution, then re-resolution of whatever that dragged in, and again, until a pass turns up nothing new.

Crucially, new descriptors resolve to the highest published version that satisfies the range — because that's exactly what a fresh yarn install would pick. Match that policy and the lockfile you emit equals the lockfile Yarn would compute; diverge from it and --immutable rejects you. (More on that below.)

Here's the field case, concretely. A monorepo build broke with:

Error: Cannot find module '@simple-git/args-pathspec'

simple-git past 3.34 had split its argument handling into two new packages. A version-only bump moved simple-git and left the lockfile believing it had no new dependencies — so the install never fetched them. Before:

"simple-git@npm:^3.10.0":
  version: 3.24.0
  dependencies:
    "@kwsites/file-exists": ^1.1.1
    # …no @simple-git/* — they didn't exist yet at 3.24

After the graph fix:

"simple-git@npm:^3.10.0":
  version: 3.36.0
  dependencies:
    "@kwsites/file-exists": ^1.1.1
    "@simple-git/args-pathspec": ^1.0.3   # ← re-resolved + wired
    "@simple-git/argv-parser": ^1.1.0     # ←

"@simple-git/args-pathspec@npm:^1.0.3":   # ← added to the graph
  version: 1.0.4
"@simple-git/argv-parser@npm:^1.1.0":     # ←
  version: 1.1.1

Prune (pruneOrphans, an option of the same completion pass): the bump can also strand a subtree — the old version's unique dependencies, now referenced by nothing. Those get garbage-collected. With one guard that took a round of debugging to get right: a real lockfile already contains "danglers" — entries with no incoming edge that Yarn nonetheless keeps. If the prune is too eager and GCs those, yarn install --immutable notices the file changed and fails (YN0028). So the prune starts only from what the fix itself stranded — nodes that had an incoming edge before the bump and have none after — and cascades from there. A dangler that was already there never enters the sweep.

4. Making it reproduce: checksums

A Berry entry carries an integrity checksum; a freshly-added node doesn't have one yet, and a lock with the line missing isn't finished — the next install writes it in and leaves your tree dirty. So the last step (refurbish) fetches each new package's tarball and recomputes the hash.

That's harder than it sounds. Berry's checksum isn't the tarball's hash: it's computed over the package's cache archive, a zip whose bytes depend on how Yarn compressed it — the lock's cacheKey. To reproduce it you rebuild that archive the way Yarn would: with a pure-JS deflate where the layout is known to be reproducible, and with Yarn's own zip layer (@yarnpkg/libzip) where it isn't. Before trusting either, the tool proves the recipe on a package your lock already has a checksum for — rebuild it, compare — and fills the new ones only on a match.

When that proof fails — no fetchable tarball, or a compression profile it can't match byte-for-byte — it doesn't write a wrong checksum: it omits those lines and says so, naming the reason:

Could not compute checksums for N package(s) (tarball-unavailable: N) —
run `yarn install` to finish the lockfile.

A plain yarn install recovers every one of them in a single pass, changing nothing else. It's the one place the user's own yarn can re-enter the loop — only for checksums the tool couldn't prove — and it's the same two-step the maintainers' own muscle memory already expects: apply the fix, then install.

5. The invariant

Put the pieces together and you get the property the whole design is for:

The patched lockfile resolves exactly as Yarn would and, with every checksum in, passes yarn install --immutable with no rewrite.

That holds because of three choices that have to agree: completion resolves to the highest satisfying version (= what yarn install picks), the prune leaves pre-existing danglers alone (= what Yarn keeps), and checksums are recomputed and proven, never guessed (= what Yarn verifies). Break any one and immutable mode catches it.

Does it actually reproduce? A real-world test

The simple-git break above came from a real qiwi/mware build (PR #275). So that's where we test the claim, on the real repo, at the commit before the fix:

Check out the pre-fix commit in a throwaway worktree. simple-git is 3.24.0; the @simple-git/* packages are absent.
Run yarn-audit-fix. It bumps simple-git to 3.36.0 (plus fifteen other fixes), completes the closure — the two sub-packages appear and are wired in — and recomputes the checksum of every package it added.
Verify with the user's own Yarn: straight to yarn install --immutable, on a cold cache.

The result that matters: --immutable exits 0 and leaves the file byte-for-byte as it was, no YN0028. And every checksum the tool computed is the one Yarn writes itself — let Yarn fill them in on the same resolutions and it lands on the identical file, byte for byte. The graph the tool produced is the graph a clean install would have produced, and so is the file.

A note on portability

Registry-only means there's no second package manager to install and nothing platform-specific to shell out to — it's just HTTP and graph manipulation. The graph library assumes as little as it can: fetch is polyfilled (via node-fetch-native) rather than taken for granted, and it runs on Node 14.18. The one piece borrowed from Yarn — its zip layer, the price of byte-exact checksums — is what sets the tool's floor at Node 18.12. yarn-audit-fix declares no engines of its own, so installing it never raises an EBADENGINE on its own behalf.

The side effect: real lockfiles are the test suite

Because the parser is its own library, separate from the fixer, it can run across a large corpus of real-world lockfiles — and real lockfiles are stranger than any fixture you'd think to write. Every odd one in the wild becomes a test you didn't author.

That's how a quiet auth bug surfaced. A .yarnrc.yml declaring a scoped registry with a token was dropping it silently: the registry URL was stored under one normalized form and looked up under another, so the two never matched, and requests to that scope went out unauthenticated with nothing to flag it. No fixture would have guessed the exact URL shape that triggered it; a real config had it. Fixed once in the shared parser, it's gone everywhere that parser runs.

Where this goes

None of this is an argument that Yarn should have shipped audit fix years ago — Part 1 is sympathetic to why it didn't. It's a demonstration that the hard part is tractable outside the package manager: treat the lockfile as a graph, re-resolve instead of replace, and check yourself against --immutable.

There's a catch that reproducibility alone doesn't cover, though: a lockfile can install perfectly and still be the wrong fix — one that needs a newer Node than you target, or a license you don't allow, or a version outside a range you pinned on purpose. Making the fix not just reproducible but right for your project is its own problem.

Sources / links

The library: lockgraph — parse / stringify, and the modify / complete / refurbish operations the pipeline drives; the cross-format conversion + loss reference is CONVERT.md.
The tool: yarn-audit-fix (npm).
Earlier notes: Yarn audit fix: workaround, The missing yarn audit fix for Yarn 2+ Berry

yarn audit [still does not] fix

Anton Golub — Tue, 29 Sep 2026 06:54:07 +0000

Why a native yarn audit fix is a hard problem, the tools and services that help in the meantime, and how we try to lend the Yarn ecosystem a hand — with remediation rebuilt on a lockfile graph.

yarn audit [still does not] fix
yarn.lock: you can't sed a graph
yarn.lock: may the --force be with you

TL;DR

yarn audit (and yarn npm audit in Berry) reports vulnerabilities; turning that into an automatic fix has never been part of Yarn. npm has npm audit fix, pnpm has pnpm audit --fix — Yarn, Classic through 4, doesn't.
Not for lack of asking (the request has been open since 2019) — it's a hard problem on a finite maintenance budget, and "fixing" a lockfile is re-resolution, not search-and-replace.
SaaS bots (Dependabot, Renovate, Snyk, Socket) solve a different shape — asynchronous, hosted, PR-gated — and leave a gap for local, one-command remediation.
We've tried to fill that gap, in the open: yarn-audit-fix does remediation as a deterministic lockfile-graph transformation — registry-only, no npm/yarn subprocess — emitting a lockfile that still passes yarn install --immutable.

`audit` ≠ `audit fix`

Years ago I ran into something every Yarn user eventually does: yarn audit happily prints a wall of CVEs, and then you're on your own for the fix. I wrote up the first workaround back then — Yarn audit fix: workaround — which did something slightly horrifying but effective: convert yarn.lock → package-lock.json (via Synp), run npm audit fix --package-lock-only, convert back. A year later, when Yarn 2 (Berry) changed everything, I followed up with The missing yarn audit fix for Yarn 2+ Berry, which dropped the conversion dance and patched the lockfile directly from yarn npm audit --json.

Those posts are old. The problem they describe is not.

The naming confuses people, so let's be precise:

npm audit fix — bumps vulnerable packages to patched versions within your declared ranges, in place, locally, in one command. With --force it will also do semver-major jumps (more on why that's risky below).
pnpm audit --fix — pnpm took the overrides route: it writes pinned non-vulnerable versions into overrides (and, since v11, --fix=update can update the lockfile instead). See the pnpm docs.
yarn audit fix — isn't a command. There's no built-in verb that takes the audit report and applies it: Classic's yarn audit and Berry's yarn npm audit filter and format the report (--level, --severity, --exclude, --ignore, --json) but never touch the lockfile.

For a single app you can shrug and run npm in a temp dir. For a Yarn-only monorepo — workspaces, constraints, PnP, a yarn.lock that's the single source of truth — that escape hatch closes.

Years of asking — for something hard

This isn't an obscure request; the paper trail is long:

2019 — yarn#7075 [feat] yarn audit fix, on Classic.
2021 — berry#3582 [Feature] yarn npm audit --fix, whose author volunteered to implement it.
2025 — berry#6959, reframing the gap as a leaky abstraction; closed as a duplicate of the open request.

A community member even built a plugin — sargunv/yarn-plugin-npm-audit-fix — and offered to upstream it. It hasn't landed in core, and the plugin's repository has since been archived. Keeping it out of core is fair: a remediation engine inside a package manager competes with the installer, PnP, constraints, and the plugin API for a small team's time — a higher bar than a standalone tool.

The request has outlived Classic, 2, 3 and 4 and every lockfile schema between them — exactly the gap a community stopgap fills.

Where the difficulty comes from

It's tempting to read years of an open issue as a quiet no. It isn't: the feature is plainly wanted — it just hasn't been built yet, because building it well is hard. Architecture, finite maintainer bandwidth, awkward data; any one would slow a feature down, and together they explain the timeline.

1. Audit is a thin proxy, by design. yarn npm audit --json is largely a pass-through to the registry's audit endpoint. Yarn returns the server's verdict; it doesn't own a remediation model. That's the "leaky abstraction" from #6959: the report reflects the registry's worldview, which isn't the same as Yarn's resolution graph. Reasonably, Yarn's maintainers have scoped remediation to yarn up --recursive and resolutions rather than the audit command — leaving auto-fix as plugin territory.

2. The advisory data is awkward to build on. Advisory ranges aren't always valid semver (>1.0.0 <2.0.0 and friends need coercion before you can compare). And the signal itself is still settling: audit IDs aren't stable across runs, ignores used to misfire for a package included multiple times, and the endpoint has returned 400s on ordinary lockfiles. Reliable auto-fix needs a stable input, and much of that input lives upstream of Yarn.

3. "Fixing" a lockfile is re-resolution, not string replacement. Bump one entry and its transitive dependencies should change too — but, as I found writing the Berry tool, Yarn doesn't reload a patched entry's dependencies for you. A correct fix has to re-resolve subtrees against the registry and keep the lockfile internally consistent and deterministic. With PnP and Yarn's strict linker, a naive in-place edit yields an install that doesn't reproduce. The community plugin takes the pragmatic route: ask Yarn's own resolver for the highest version within a package's declared range, and if it clears the advisory, pin that version and re-install. That works right up to the ceiling where most deep transitives live — when the only patched version is a cross-major bump out of range, the highest in-range candidate still doesn't clear the advisory, so it reports no compatible patched version and gives up on that package. Its own roadmap names the rest of the climb: walk up the tree to fix via parents, update manifests for direct deps, add a flag for out-of-range bumps. None of it is trivial.

4. Most vulnerabilities are transitive and deep. The vulnerable package is rarely one you installed. When it's three levels down, the parent controls its version, and the only real fix is often a cross-major bump that breaks the consumer's declared range. Doing that safely — and knowing when not to — is the actual hard part.

5. npm's audit fix shows the trap. Even copying npm wouldn't be a clean win. In range, npm audit fix often reports "fixed 0 of N" because the patched version is out of range; reach for --force and it can silently do semver-major upgrades and break builds; and a declared overrides entry could block it from replacing even eligible dependencies until npm 11.2. A built-in yarn audit fix would inherit all of this — plus PnP.

The remediation landscape (and the gap for a local fix)

So how do teams remediate on Yarn? Two tiers, and the gap they leave.

Built-in / local (manual)

yarn up <pkg> --recursive per advisory, or pin transitive versions via resolutions, then re-install. This is the maintainer-endorsed path and it works — but it's manual, per-package, and easy to get subtly wrong at scale.

The bots (asynchronous, hosted)

Dependabot — GitHub-native, zero-config, opens PRs from the GitHub Advisory DB. No reachability analysis; remediation is "here's a PR, you review and merge."
Renovate — the most configurable updater, multi-platform. It's an update tool, not a scanner — you layer SCA on top.
Snyk — its own advisory DB (often ahead of NVD), automated fix PRs, monitoring, IDE plugins. Commercial, cloud.
Socket.dev — a different axis: behavioral / supply-chain analysis (malicious install scripts, network access, typosquats) that CVE scanners miss. Detection-first.

These are excellent, and most serious teams should run one. But they share a shape: asynchronous, hosted, repo-bound, review-gated. They assume a cloud-connected VCS, a bot identity, and a PR cadence.

The gap they leave

What none of them gives a Yarn user is the thing an npm user gets from npm audit fix: a local, synchronous, one-command, scriptable "fix my yarn.lock now." That shape matters exactly where many enterprises live:

air-gapped / regulated environments where SaaS bots and a chatty GitHub App are non-starters;
a CI gate that must remediate in the same job, not three PRs later;
pre-commit / local dev loops where you want the fix before you push;
Yarn monorepos, where bot support for workspaces + PnP + resolutions is uneven.

That gap is why community tools exist at all — yarn-audit-fix, the sargunv plugin, and other takes on it. The need is real and the core feature is hard, so the ecosystem fills in around it. That's open source working as intended.

How we try to help

We've kept at it, in the open. yarn-audit-fix tracks every Yarn lockfile schema that shipped — Classic (v1), Berry v4–v6 (Yarn 2/3), and Yarn 4+ (Berry v8/v9/v10) — detecting the version at runtime and parsing each into one shared dependency graph.

Two things have shifted since those early posts.

Where the data comes from. The first versions shelled out: convert the lockfile, run someone else's audit fix, parse the CLI output back. The current line talks to the registry directly — it reads advisories from the registry's bulk audit endpoint, resolves fix versions (and the dependencies they pull in) from package metadata, and fetches tarballs to recompute integrity hashes. No npm or yarn subprocess anywhere — no second package manager to install, no CLI output format to keep chasing, just HTTP to a registry you already trust, using your existing scoped .npmrc / .yarnrc.yml auth (host-bound, https-only).

How the fix is computed. Remediation is a deterministic transformation over that graph (lockgraph): move the vulnerable node to the lowest published version that clears the advisory, rebind the edges that pointed at it — and then re-resolve that new version's own dependency closure from the registry. That last step is the one flagged as the hard part above: a patched package's transitive deps can differ from the version you had — simple-git past 3.34, for instance, splits out @simple-git/args-pathspec and @simple-git/argv-parser — and Yarn won't reload them for you. Pull them in and the result is a lockfile that actually reproduces: the resolutions are the ones a clean yarn install would compute, and the lock passes yarn install --immutable. A compatibility gate keeps the default safe — a fix applies only when it still satisfies every surviving consumer's declared range, with the riskier cross-major bumps behind an explicit --force.

Building on a shared model has a nice side effect: when the real-world corpus turns up a parser gap, we file it and it gets fixed — the ecosystem gets a little better for everyone, not just for us.

This is meant as a contribution, not a verdict. If Yarn ships a native audit fix one day, we'll happily retire ours and help however we can. Until then, consider this a community on-ramp.

The graph model itself — the edge-redirect patch, the closure re-resolution, the compatibility gate, and how it stays faithful to every Yarn lockfile format from Classic to Berry v10 — is the next part.

Sources

Yarn issues: yarn#7075, berry#3582, berry#6959, berry#4117, berry#6438, berry#5104
Community plugin: sargunv/yarn-plugin-npm-audit-fix
npm / pnpm: npm/cli#7987, pnpm audit docs, pnpm#8943, Why npm audit fix --force is risky
Tooling comparisons: SCA tools compared, Dependabot vs Renovate vs Snyk
Yarn versions / EOL: endoflife.date/yarn
Earlier notes: Yarn audit fix: workaround, The missing yarn audit fix for Yarn 2+ Berry

CVE-2024-29415: problem solution

Anton Golub — Tue, 16 Sep 2025 14:39:58 +0000

an illustrative note on the relationship logic between Enterprise and OSS based on a specific issue

If you've landed on this page, it's likely that your codebase relies on a widely deployed package that is currently struggling to maintain and is therefore vulnerable to certain scenarios. You have several strategies to follow:

Ask the author for a fix
Suggest a fix
Apply patches locally
Override internal registry versions
Consider migrating to other solutions
Just live with it

It's always a question of rational choice and cost of effort. In our case, the dependency tree profile required too much cascading fixes including external components, so we decided to take matters into our own hands — create a fully compatible alternative and apply it to our systems.

We found that the address classification error (private/public) was due to a lack of strictness in the parser implementation, and the format error was interpreted as a sign of being in (or out of) the range. An additional complication was that the logic was divided into several auxiliary functions, but was combined differently in the compositions of parsing addresses of variouus notations.

Algorithm

In the first attempt, we simply made the pattern conditions strict, but as expected, we encountered the problem of reduced performance. It became immediately clear that stricter conditions would require a different algorithmic approach.
Here we moved step by step: from split() to a string traverser, until we were satisfied with the ops/s metric.

The second significant change was related to the internal structure of stored addresses: Buffers have been replaced with bigints, which allowed us to greatly simplify and speed up the logic of comparisons and format translation. So it became easy to implement proper and full support for IPv6 for mask() and subnet() methods.

Address.from('0.0.0.0').big // 0n

The third major aspect of algorithmic improvements is related to the refactoring of special IP address ranges declarations. Instead of a set of magic conditions in helpers, a single source of truth is allocated now, and productive predicate functions are preinitialized.

https://en.wikipedia.org/wiki/Reserved_IP_addresses

const SPECIALS: Record<Special, string[]> = {
  loopback: [
    '127.0.0.0/8',    // IPv4 loopback
    '::1/128',        // IPv6 loopback
  ],
  // ...
]
const SPECIAL_MATCHERS: ((addr: Address) => Special | undefined)[] = []
for (const [cat, cidrs] of Object.entries(SPECIALS)) {
  for (const cidr of cidrs) {
    for (const x of ipv6fySubnet(cidr)) {
      const subnet = Address.cidrSubnet(x)
      SPECIAL_MATCHERS.push((addr: Address) => addr.family === subnet.family && subnet.contains(addr) ? (cat as Special) : undefined)
    }
  }
}

Compatibility

The key point was to keep backward compatibility. To ensure this, we completely borrowed the test suite from the upstream repo. For clarity, we also additionally investigated how similar other IP tools are in their operating logic. The conclusion is obvious: without additional modifications, it is impossible to achieve equivalent operating results in all scenarios. It was also important for us to make sure that the library functionality would not differ in different versions of runtimes and would be compatible with the browser environment. Polyfills and an extensive palette of smoke tests are right for this.

Drop-in

As we noted at the beginning, the process of cascading dependency updates is expensive, and some tools of the JS ecosystem provide workarounds for this. And the package should strive for seamlessness: so we've added an additional pkg entry point to support legacy IDE-hinted require('ip/lib/ip.js) statements.

- const ip = require('ip')
+ const ip = require('@webpod/ip')

Temporary workaround to avoid refactoring is using overrides / resolutions in your package.json:

{
  "overrides": {
    "ip": "@webpod/ip"
  }
}

Next steps

There is no discussion here. The only correct way is to push changes to upstream, reduce the heterogeneity of dependencies. I hope that maintainers, when they have enough time, will support the community's efforts in this direction. Until that happens, there is @webpod/ip

zx@lite: minimalistic shell scripting with TS/JS superpowers

Anton Golub — Thu, 19 Jun 2025 14:33:00 +0000

Ditch clunky bash syntax. Script smarter — with fewer dependencies and full TypeScript support.

⚡ What is `zx@lite`?

The original zx makes shell automations delightful in modern JS runtimes. The @lite version, introduced in v8.5.0, gives you optimal performance, devX without the extra API bloat:

✅ Core zx functionality ($, cd, ps)
✅ First-class TypeScript support
⚡ ~7× smaller than full zx (~124 KB vs ~875 kB)
🧱 Static build with zero dependencies.
🔌 Wide compat range: Node.js 12+, Deno 1.30+, Bun 1.0+, GraalVM 17+ on Win, Mac or Linux

Ideal for:

Fast script prototyping
DevOps and CI automations
Docker images
Small CLI tooling

🧾 Setup

npm install zx@lite

TypeScript is an option but highly recommended for safety and better IDE support.

npm install -D typescript @types/node

Attach tsconfig.json like this:

{
  "compilerOptions": {
    "target": "esnext",
    "module": "esnext",
    "moduleResolution": "nodenext",
    "strict": true
  }
}

🧪 Examples

Let’s say you want to clone several repos and list their directories. In bash, this would be a mess of &, wait, and error handling. In zx@lite, it’s much cleaner:

#!/usr/bin/env zx
import { $ } from 'zx'

// Run 3 commands in parallel
await Promise.all([
  $`git clone https://github.com/google/zx zx-clone`,
  $`git clone https://github.com/webpod/webpod webpod-clone`
])

const output = await $`ls -1`
console.log('Cloned directories:\n', output.lines())

Native parallelism with Promise.all()
Clean output handling via stdout, lines(), json() helpers
Controlled error flow: configurable exception throwing behavior

Combine zx@lite with runtime features. Need transport? Just bring in native fetch:

#!/usr/bin/env zx
import { $ } from 'zx'

const repo = 'google/zx'
const latest = await (await fetch(`https://api.github.com/repos/${repo}/releases/latest`)).json()
const v = latest.tag_name
const zip_file = `${repo.replace('/', '-')}-${v}.zip`

await $`curl -o ${zip_file} -L ${latest.zipball_url}`
await $`mkdir -p releases/${v}`
await $`unzip ${zip_file} -d releases/${v}`

console.log('✅ Done')

⚠️ What’s missing in `zx@lite`?

Feature	`zx`	`zx@lite`
`$`	✅	✅
`cd`	✅	✅
`ps`	✅	✅
`glob`	✅	❌
`YAML`	✅	❌
`fs-extra`	✅	❌
`minimist`	✅	❌

Check the full API comparison for details.

📦 Upgrade path

Switching between lite and full is trivial:

# Go full speed:
npm install zx

# Back to lean mode:
npm install zx@lite

📚 References

🧠 Final thoughts

zx@lite is everything you need — and nothing you don’t — for modern scripting with real programming ergonomics. Stop writing fragile bash. Start scripting like a developer.

💬 Got ideas, snippets, or zx@lite tips? Drop them below or feel free to share in the project repo: discussions, issues.

How and why do we bundle zx?

Anton Golub — Mon, 06 May 2024 08:12:21 +0000

When zx first appeared, it was a tiny esm script that just proposed a new idea for how child_process.spawn API could be enhanced with string template literals.

const result = await $`call smth --with any ${args}`

As we now know, this was exactly what JS folks had been looking for for many years :-). The main goal was to improve the developer experience, so we focused on finding the best combination of utils to solve everyday problems, expand the area of application.

Short history

The project went through 8 major releases, but in the context of the assembly issues, there are only 3 significant ones.

Packed as CommonJS

Despite the fact that the future belongs to ESM, many users have to remain in the notation of their legacy codebase (without any irony). So we have added compatibility with the require API for them.

https://github.com/google/zx/commit/303feea3cd0d95e8d733ee72b96b3316ea5827aa

Dropped CJS support

On the other hand, there were libraries on which zx depend: and only the latest modern major versions received patches, including vulnerabilities fixes. In fact we inherited CVE issues transitively, which was completely unacceptable. And we switched back to ESM.

https://github.com/google/zx/commit/5578d67ebae2049d430bf6489784f6a2aa47b0b1

Migrated to TypeScript

As the code base grew, it became harder for us to ensure consistency of internal contracts. Adding typing was the solution, but it also made us dependent on transpilation and we could no longer control the package artifact in detail.

https://github.com/google/zx/commit/8c2aea89fe7c65767d29c52bfb48f6a46af01864

Problems

Size

Filling the zx with new functionality, we continued to expand the dependency tree, which required their own dependencies, etc. Installation time increased significantly. And at some point it became impossible to ignore: zx#712, zx#669

1.15.2	2.1.0	3.1.0	4.3.0	5.3.0	6.2.5	7.2.3
1.38 MB	3.01 MB	3.01 MB	3.89 MB	10.5 MB	10.7 MB	16.3 MB

Compatibility

When we define node engine requirement we limit not users, but ourselves in what platform API we can use. But this in no way prevents our dependencies from pursuing their own policies which affects zx.

Stability & Security

Reproducible setups are important. Especially for utilities of our type. We can try to pin down versions, but if there is a caret (^, ~) or an asterisk (*) at the secondary level, this introduces node_modules variability. Lockfiles can mitigate the problem, but not for npx/yarn dlx execution case.

Bundle

At first we wanted to just get rid of all the helper utilities. Keep only the kernel, but this would mean a loss of backward compatibility. We needed some efficient code processing instead with recomposition and tree-shaking. We needed a bundler. But which one? Our testing approach relies on targets, not sources. We rebuilt the project frequently, speed was critical requirement. In essence, we chose a solution from a couple of among all available alternatives: esbuild and parcel. Esbuild won. Specifically in our case, it proved to be more productive and customizable.

js

Customizable means that we found a way to adapt the tool for our tasks. And a lot had to be improved, including writing our own plugins:

Esbuild provides code-splitting with dynamic imports to reuse common code chunks. But not is static form. The manual chunks API is missing.
esbuild-plugin-entry-chunks
By default, esbuild injects helpers into each cjs module. It's fine, when you have just a few, but definitely not when there are many.
esbuild-plugin-extract-helpers
We also wanted to bring back cjs support, but at the same time avoid duplicating code in two versions of the bundles.
esbuild-plugin-hybrid-export
Esbuild and its plugins mostly focus on sources processing, but sometimes additional modification is also necessary for dependencies or bundles:

Polyfill injects: esbuild#2840, esbuild#3517, esbuild#3099
Custom patches: esbuild#3360
Dynamic banners: esbuild#3291
Importable helpers: esbuild#1230.

esbuild-plugin-transform-hook

d.ts

While we were fighting against the modules, we forgot one small detail - their built-in typings. Esbuild can't do this at all yet. Unbelievable, but the tsc, native TS compiler, also does not provide a typings concat feature. Got around this problem: we've introduced a utility to combine typings of zx own code, and applied some monkey patches for external libdefs squashed via dts-bundle-generator.

Results

There is no silver bullet, all tools and approaches have limitations. Summary:

Install size reduced: 16Mb → 1Mb
Reproducible npx invocations
Hybrid package: ESM and CJS
Wide range of supported Node.js versions 12...22
Easy to audit: complete code in one place

Perhaps our efforts are not commensurate with the profit. But we gained invaluable experience. Now we understand the entire zx code in detail down to each char.

zx build setup

google / zx

A tool for writing better scripts

zx

#!/usr/bin/env zx

await $`cat package.json | grep name`

const branch = await $`git branch --show-current`
await $`dep deploy --branch=${branch}`

await Promise.all([
  $`sleep 1; echo 1`,
  $`sleep 2; echo 2`,
  $`sleep 3; echo 3`,
])

const name = 'foo bar'
await $`mkdir /tmp/${name}`

Bash is great, but when it comes to writing more complex scripts, many people prefer a more convenient programming language. JavaScript is a perfect choice, but the Node.js standard library requires additional hassle before using. The zx package provides useful wrappers around child_process, escapes arguments and gives sensible defaults.

Install

npm install zx

Documentation

Read documentation on google.github.io/zx.

License

Apache-2.0

Disclaimer: This is not an officially supported Google product.

View on GitHub

config: apply extends

Anton Golub — Mon, 05 Feb 2024 08:00:02 +0000

As we know, the forms and approaches to describing configuration files are extremely varied. I discussed this in the previous note. But there are several techniques and practices that may one day become standards.

One of this – the extends directive. Many tools provide this feature, but it works a little differently in each place. For example, tsconfig applies deep merge to compilerOptions, while prettier concatenates overrides array sections, commitlint joins plugin, etc. So the authors have to implement these nuances on site every time, and it's tiring and annoying a bit. I believe we will come up with a more efficient way to handle this routine.

Proposal

import { populate, populateSync } from '@topoconfig/extends'

const tsconfig = await populate('tsconfig.json', {
  compilerOptions: 'merge'
})
const prettierCfg = populateSync('.prettierrc.json', {
  overrides: 'merge'
})

20 minutes adventure

At first glance, the extends may seem like a simple combination of Object.assign and fs.readFile. Well, technically it is, but not quite so. Let's see how the real scenario unfolds. To simplify the details, we will use pseudocode.

The starting point:

input = {
  foo: 'bar',
  extends: './base.json'
}

pick extends

resource = input.extends

load the referenced ./base.json

extra = JSON.parse(fs.readFileSync(resource, 'utf8'))

merge input and extra

result = {
  ...input,
  ...extra,
  extends: undefined,
}

How about ./base.js? Reasonable request.

provide js files support

input = { extends: './base.js' }

require hurries to the resque:

load = (ref) => isJs(ref) ? require(ref) : JSON.parse(fs.readFileSync(ref, 'utf8'))

./base.mjs?

Ooops. Here the API switches to async mode to handle mjs/esm inputs. Of course, we can patch require with esm package, or load module via a process.spawnSynced nodejs. We can, but we won't

load = isJs(ref)
  ? (await import(ref))?.default
  : fs.promises.readFile(ref, 'utf8')

What if the value is passed without an extension, like base?

add support for shared npm configs
provide configurable resolve

resolve = (ref, cwd) =>
  ref.startsWith('.') || ref.startsWith('/') || path.extname(ref)
    ? path.resolve(cwd, ref)
    : ref

But what if extends is a nested object? It definitely is.

apply everything recursively:

handle = input = ({
  ...input,
  ...handle(load(input.extends)),
  extends: undefined,
})

In this case, circular references are something to pay attention to.

add index to avoid infinite loops

memo = new Map()

memoizedLoad = ref => {
  if (memo.has(ref)) {
    return memo.get(ref)
  }
  memo.set(ref, load(ref))
  return load(ref)
}

Relative resource path should use the closest parent dirname as a base to work properly.

resolve cwd for each extends reference

base = path.resolve(process.cwd(), _opts.cwd ?? '.')
[cwd, _config] = typeof config === 'string'
  ? [path.resolve(base, path.dirname(config)), path.basename(config)]
  : [base, config]

Multiple extends maybe?

let extends hold several references

input = {
  foo: 'bar',
  extends: ['./extra.json', './mixin.js']
}
extras = [input.extends].flat().map(load)

How to read toml, yaml, ini, env? Obviously, custom formats are required too.

provide parse option

parse = (ref, contents) => {
  if (ref.endsWith('.yaml') || ref.endsWith('.yml')) 
      return parseYaml(contents)
  if (ref.endsWith('.json')) 
      return JSON.parse(contents)
  throw new Error(`Unsupported format: ${ref}`)
}

How to resolve value conflicts: merge / override?

provide rules to handle prop-specific join strategies

rules = {
  '*': 'merge',
  foo: 'override'
}
rule = getRule(propPath, rules)
result[key] = isObject(value) && rule === 'merge'
  ? extend({
    sources: [value],
    rules,
    prefix: p,
    index
  })
  : value

What about arrays?

implement array merge as concat

if (rule === 'merge') {
  arr.push(...sources.flat(1))
} else {
  arr.length = 0
  arr.push(...sources.slice(-1).flat(1))
}

It would be nice to avoid mutations.

introduce clone

safeLoad = compose(memoize, clone, load)

Conclusion

Summarizing:

search extends statements
load & parse referenced resources
- support async and sync flows
- use cloning to avoid mutations
- .js, .cjs, .mjs, .json
- provide support for custom formats
- resolve cwd for the nested resource by its closest parent resource
- support custom resolvers
walk recursively
- handle circular references
provide sync and async API
assemble the result
- handle arrays / plain objects
- apply prop-specific merge

Additionally:

build as cjs and esm
provide types
handle windows specific issues
add dozens of tests
handle different runtimes specifics (node, deno, bun)

There is an obvious benefit to implementing this algorithm on your own – it clarifies and makes controllable every aspect of config processing in place.
However, we do not always have enough time to do this exercise. Therefore, maybe someone will find this shortcut useful:

config: enhancing declarations with graphs

Anton Golub — Tue, 03 Oct 2023 11:14:44 +0000

Typically application configs are not a bottleneck or development issue. But everything can suddenly change when the context of expands with infrastructure domain, external sources interactions, environments.

Config mess

Many years ago configs were pretty simple. They looked more or less like .properties-files or INI-files, simple kv-maps with sections or composite keys to bring some kind of context:

# https://docs.oracle.com/cd/E23095_01/Platform.93/ATGProgGuide/html/s0204propertiesfileformat01.html

# You are reading a comment in ".properties" file.
! The exclamation mark can also be used for comments.
# Lines with "properties" contain a key and a value separated by a delimiting character.
# There are 3 delimiting characters: '=' (equal), ':' (colon) and whitespace (space, \t and \f).
website = https://en.wikipedia.org/
language : English
topic .properties files
# A word on a line will just create a key with no value.
empty

; last modified 1 April 2001 by John Doe
[owner]
name = John Doe
organization = Acme Widgets Inc.

[database]
; use IP address in case network name resolution is not working
server = 192.0.2.62     
port = 143
file = "payroll.dat"

At the same time, another part of the configuration was supplied from the environment variables or CLI parameters reflecting the idea of dynamic settings.

Now we use dotenv-files, ironic :

# https://hexdocs.pm/dotenvy/0.2.0/dotenv-file-format.html
S3_BUCKET=YOURS3BUCKET
SECRET_KEY=YOURSECRETKEYGOESHERE

Even then, the resolution logic began to penetrate into the app layer.

// Just an illustration. This problem existed before JS was invented

const config = require('config')
const logLevel = process.env.DEBUG ? 'trace' : config.get('log.level') || 'info'
//...
const dbConfig = config.get('Customer.dbConfig')
db.connect(dbConfig, ...)

if (config.has('optionalFeature.detail')) {
  const detail = config.get('optionalFeature.detail')
  //...
}

When centralized configuration management came, the settings has been moved partially to the remote storage. Local pre-config (entrypoints, db credentials) was used to get the rest. Configuration assembly itself has become multi-stage.

Later, specialized systems such as vault made new additions: now env holds an access token and defines an entrypoint by running mode to make a POST request to reveal credentials profile to mix this data to the entire config.

Here's how uniconfig obtains secrets from the vault storage:

{
  "data": {
    "secret": "$vault:data"
  },
  "sources": {
    "vault": {
      "data": {
        "data": {
          "method": "GET",
          "url": "$url:",
          "opts": {
            "headers": {
              "X-Vault-Token": "$token:auth.client_token"
            }
          }
        },
        "sources": {
          "url": {
            "data": {
              "data": {
                "data": {
                  "name": "$pkg:name",
                  "space": "openapi",
                  "env": "$env:ENVIRONMENT_PROFILE_NAME",
                  "vaultHost": "$env:VAULT_HOST",
                  "vaultPort": "$env:VAULT_PORT"
                },
                "template": "{{=it.env==='production' ? 'https': 'http'}}://{{=it.vaultHost}}:{{=it.vaultPort}}/v1/secret/applications/{{=it.space}}/{{=it.name}}"
              },
              "sources": {
                "env": {
                  "pipeline": "env"
                },
                "pkg": {
                  "pipeline": "pkg"
                }
              }
            },
            "pipeline": "datatree>dot"
          },
          "token": {
            "data": {
              "data": {
                "method": "POST",
                "url": "$url:",
                "opts": {
                  "json": {
                    "role": "$pkg:name",
                    "jwt": "$jwt:"
                  }
                }
              },
              "sources": {
                "pkg": {
                  "pipeline": "pkg"
                },
                "jwt": {
                  "data": {
                    "data": {
                      "data": {
                        "tokenPath": "$env:TOKEN_FILE",
                        "defaultTokenPath": "/var/run/secrets/kubernetes.io/serviceaccount/token"
                      },
                      "template": "{{=it.tokenPath || it.defaultTokenPath}}"
                    },
                    "sources": {
                      "env": {
                        "pipeline": "env"
                      }
                    }
                  },
                  "pipeline": "datatree>dot>file"
                },
                "url": {
                  "data": {
                    "data": {
                      "data": {
                        "env": "$env:ENVIRONMENT_PROFILE_NAME",
                        "vaultHost": "$env:VAULT_HOST",
                        "vaultPort": "$env:VAULT_PORT"
                      },
                      "template": "{{=it.env==='production' ? 'https': 'http'}}://{{=it.vaultHost}}:{{=it.vaultPort}}/v1/auth/kubernetes/login"
                    },
                    "sources": {
                      "env": {
                        "pipeline": "env"
                      },
                      "pkg": {
                        "pipeline": "pkg"
                      }
                    }
                  },
                  "pipeline": "datatree>dot"
                }
              }
            },
            "pipeline": "datatree>http>json"
          }
        }
      },
      "pipeline": "datatree>http>json"
    }
  }
}

Meanwhile, formats have been evolving (JSON5, YAML), config entry points are constantly changing. These fluctuations, fortunately, were covered by tools like the cosmiconfig.

[
  'package.json',
  `.${moduleName}rc`,
  `.${moduleName}rc.json`,
  `.${moduleName}rc.yaml`,
  `.${moduleName}rc.yml`,
  `.${moduleName}rc.js`,
  `.${moduleName}rc.ts`,
  `.${moduleName}rc.mjs`,
  `.${moduleName}rc.cjs`,
  `.config/${moduleName}rc`,
  `.config/${moduleName}rc.json`,
  `.config/${moduleName}rc.yaml`,
  `.config/${moduleName}rc.yml`,
  `.config/${moduleName}rc.js`,
  `.config/${moduleName}rc.ts`,
  `.config/${moduleName}rc.cjs`,
  `${moduleName}.config.js`,
  `${moduleName}.config.ts`,
  `${moduleName}.config.mjs`,
  `${moduleName}.config.cjs`,
]

Configs are still trying to be declarative, but they can't. Templates appeared first.

template:
    metadata:
      annotations:
        cni.projectcalico.org/ipv4pools: '["${APP_NAME}"]'
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/secrets-injection-method: "env"
        vault.hashicorp.com/secrets-type: "static"
        vault.hashicorp.com/agent-inject-secret-${APP_NAME}: secret-v2/applications/${DEPLOYMENT_NAMESPACE}/${APP_NAME}
        vault.hashicorp.com/agent-inject-template-${APP_NAME}: |
          {{ with secret "secret-v2/applications/${DEPLOYMENT_NAMESPACE}/${APP_NAME}" }}
            {{- range $secret_key, $secret_value := .Data.data }}
            export {{ $secret_key }}={{ $secret_value }}
            {{- end }}
          {{ end }}
        vault.hashicorp.com/auth-path: ${AUTH_PATH}
        vault.hashicorp.com/role: ${APP_NAME}

Then templates inside templates. With commands and scripts invocations inside dynamic DSL wrapped into matrices.

      - uses: actions/cache@v3
        id: yarn-cache
        with:
          path: ${{ needs.init.outputs.yarn-cache-dir }}
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

      - name: Restore artifact from cache (if exists)
        uses: actions/cache@v3
        with:
          path: artifact.tar
          key: artifact-${{ needs.init.outputs.checksum }}

      - name: Check artifact
        if: always()
        id: check-artifact
        run: echo "::set-output name=exists::$([ -e "artifact.tar" ] && echo true || echo false)"

As we can see, syntax complexity increases as the cost of declarativeness. It's still unclear how this problem can be mitigated. Perhaps new specialized formats will appear or more strict forms (schemas) of using existing ones will be introduced.

Budget loss

Anyway, ::$([ is definitely not an optimal solution. Сonfusing, fragile and overcomplicated for the most developers. For example, here is how Python Engineer was fighting against kube.yaml:

fix vault in kube yaml Jul 04   XS      
fix vault in kube yaml Jul 04   XS
fix vault in kube yaml Jul 04   XS
fix vault in kube yaml Jul 04   XS
fix vault in kube yaml Jul 04   XS
fix vault in kube yaml Jul 04   XS
fix vault in kube yaml Jul 04   XS
fix vault in kube yaml Jul 04   XS
fix vault in kube yaml Jul 03   XS
fix vault in kube yaml Jul 03   XS
fix vault in kube yaml Jul 03   XS
fix vault in kube yaml Jul 03   XS
fix vault in kube yaml Jul 03   XS
fix vault in kube yaml Jul 03   XS
...

This is definitely not configuring but more guessing. On a company scale, such exercises are a significant waste of resources. And this experience is almost one-time only, which cannot be formalized and transmitted except by copy-paste. Every time we see the same thing, with a different number of attempts.

What we need

The overcomplexity problem seems to have arisen from the fact that we combined resolving, processing and accessing data into one structure. Although the entire theory of programming / CS instructs us to do exactly the opposite. Separation of concerns: imagine a config which explicitly divides value resolutions, compositions and operations.

{
  "data": "<how to expose values>",
  "sources": "<how to resolve values>",
  "cmds": "<available cmds/ops/fns>"
}

Let data to represent how the result structure may be built if all the required transformations were made — like a mapping.

{
  "data": {
    "a": {
      "b": "$b.some.nested.prop.value.of.b",
      "c": "$external.prop.of.prop"
    }
  }
}

Templating bases on regular substring replacements:

String.format("foo %s", "bar")                   // gives 'foobar'
// But positional contract is enhanced with named refmap
String.format("foo $a $b $a", {"a": "A", "b": "B"}) // returns 'foo A B A'
//            ↑ data chunks ↑ sources map

Let sources to describe how to obtain and process values for referencing in data map. Like reducing pipelines.

{
  "sources": {
    "a": "<pipeline 1>",
    "b": "<pipeline 2>"
  }
}

Let pipeline to compose actions in natural ~~human~~ dev-readable format like CLI:

cmd param > cmd2 param param > ... > cmd3

Let intermediate values be referenced by lateral (bubbling concept) or nested contexts.

{
  "sources": {
    "a" : "cmd param",
    "b": "cmd $a" // b refers to a
  }
}

Apply DAG walker for consistency checks and processing.

Working draft

https://github.com/antongolub/misc/tree/master/packages/topoconfig

Install

yarn add topoconfig@draft

API

import {topoconfig} from 'topoconfig'

const config = await topoconfig({
  // define functions to use in pipelines: sync or async
  cmds: {
    foo: () => 'bar',
    baz: async (v) => v + 'qux'
  },
  // pipelines to resolve intermediate variables
  sources: {
    a: 'foo > baz', // pipeline returns 'barqux'
    b: {            // b refers to b.data
      data: {
        c: {
          d: 'e'
        }
      }
    }
  },
  // output value
  data: {
    // $name.inner.path populates var ref with its value
    x: '$b.c.d',  // 'e'
    y: {
      z: '$a'     // 'barqux'
    }
  }
})

Real-world usage example may look like:

import {topoconfig} from 'topoconfig'

const config = await topoconfig({
  data: {
    foo:      '$a',
    url:      'https://some.url',
    param:    'regular param value',
    num:      123,
    pwd:      '\\$to.prevent.value.inject.use.\.prefix',
    a: {
      b:      '$b.some.nested.prop.value.of.b',
      c:      '$external.prop.of.prop'
    },
    log: {
      level:  `$loglevel`
    }
  },
  sources: {
    a:        'file ./file.json utf8',
    b:        'json $a',
    c:        'get $b > assert type number',
    cwd:      'cwd',
    schema:   'file $cwd/schema.json utf8 > json',
    external: 'fetch http://foo.example.com > get .body > json > get .prop > ajv $schema',
    extended: 'extend $b $external',
    loglevel: 'find $env.LOG_LEVEL $argv.log-level $argv.log.level info',
    template: `dot {{? $name }}
<div>Oh, I love your name, {{=$name}}!</div>
{{?? $age === 0}}
<div>Guess nobody named you yet!</div>
{{??}}
You are {{=$age}} and still don't have a name?
{{?}} > assert $foo`,
  },
  cmds: {
    // http://olado.github.io/doT/index.html
    dot:      (...chunks) => dot.template(chunks.join(' '))({}),
    extend:   Object.assign,
    cwd:      () => process.cwd(),
    file:     (file, opts) => fs.readFile(file, opts),
    json:     JSON.parse,
    get:      lodash.get,
    argv:     () => minimist(process.argv.slice(2)),
    env:      () => process.env,
    find:     (...args) => args.find(Boolean),
    fetch:    async (url) => {
      const res = await fetch(url)
      const code = res.status
      const headers = Object.fromEntries(res.headers)
      const body = await res.body()

      return {
        res,
        headers,
        body,
        code
      }
    },
    //...
  }
})

Notes

export type TData = number | string | { [key: string]: TData } | { [key: number]: TData }
export type TCmd = (...opts: any[]) => any
export type TCmds = Record<string | symbol, TCmd>
export type TConfigDeclaration = {
  data: TData,
  sources: Record<string, string | TConfigDeclaration>
  cmds?: TCmds
}

TConfigDeclaration defines two sections: data and sources:

data describes how to build the result value based on the bound sources: it populates $-prefixed refs with their values in every place.
sources is a map, which declares the algorithm to resolve intermediate values through cmd calls composition. To fetch data from remote, to read from file, to convert, etc.

{
  "data": "$res",
  "sources": {
    "res": "fetch https://example.com > get .body > json"
  }
}

cmd is a provider that performs a specific action.

type TCmd = (...opts: any[]) => any

directive is a template for defining a value transformation pipeline

// fetch http://foo.example.com > get body > json > get .prop
// ↑ cmd ↑opts                  ↑ pipes delimiter

Next steps

Add ternaries: cmd ? cmd1 > cmd2 ... : cmd
Handle or statement: cmd > cmd || cmd > cmd
Expose commands presets: import {cmds} from 'topoconfig/cmds' or @topoconfig/cmds
Provide lazy-loading for cmds:

{
  cmds: {
    foo: 'some/package',
    bar: './local/plugin.js'
  }
}

Support a pipeline factory as cmd declaration.

{
  cmds: {
    readjson: 'path $0 resolve > file $1 > json'
  }
}

Use vars as cmd refs:

{
  sources: {
    files: 'glob ./*.*'
    reader: 'detect $files'
    foo: 'file $files.0 > $reader'
  }
}

Bring smth like watchers to trigger graph re-resolution from the specified vertex

antongolub / misc

Experiment on maintaining a multi-project monorepository

<misc>

An experiment on maintaining a multi-project monorepository

Statuses

Blueprint/B marks the project as an idea w/o any implementation provided. Just a contract proposal.
PoC/C — proof of concept that shows the declared behavior in action.
Working draft/W — the project work is in progress. Some known corner cases are not covered, but it's already mostly usable.
Production ready/R — the implementation is stable, documented, tested and ready for use.
Deprecated/D — the project is no longer maintained.

Package	Description	Status
@antongolub/blank	Blank TS project
@antongolub/infra	Repo infra assets
@topoconfig/cmds	Topoconfig basic cmds preset	W
@topoconfig/extends	Flexible config extender	W
depseek	Seeks for dependency references in JS/TS code	W
depshot	Gathers deps snapshot by analyzing sources	B
envader	Occupies env vars for data storage	C
envimist	Applies minimist to process.env	W
esbuild-c	Empowers esbuild with config processing	C
esbuild-plugin-entry-chunks	Esbuild plugin to compose entryPoints as chunks	C
lcov-utils

…

View on GitHub

Refs and Inspirations

NestJS + esbuild workarounds

Anton Golub — Tue, 13 Sep 2022 20:31:07 +0000

Q: Do we need ESM for backend apps?
A: Yes. More and more pkgs are moving to the ESM format, this is inevitable.

Q: Do we need bundles?
A: Probably, yes.

If your Nestjs app is distributed as a container with node_modules inside, bundling can greatly reduce its size.

du -hs ./node_modules
495M   ./node_modules
du -hs ./output.js   
10M    ./output.js

Bundle is the best point for a thorough ISEC audit: to search for redos, suspicious API usages, protestware, etc.
The fewer fs I/O, the faster the application starts.
(TS) bundle absorbs the all required contents, so there is no tsc-esm transpilation issues

Q: esbuild, swc, rollup, parcel or babel?
A: esbuild.

Stack

:-(

const config =  {
  platform: 'node',
  target: ['node18', 'es2021'],
  format: 'esm',
  bundle: true,
}

Problems

https://github.com/nestjs/nest-cli/issues/1157
https://github.com/nestjs/swagger/issues/1450
https://github.com/evanw/esbuild/pull/509
https://github.com/evanw/esbuild/issues/566

0. Some Nestjs internals are lazy-loaded, so sometimes they may be omitted, but in another cases they should be bundled. How to handle? https://esbuild.github.io/api/#external

const config =  {
  ...
  external: [
    'mqtt',
    'amqplib',
    'class-transformer/storage'
  ]
}

1. openapi is not defined. https://github.com/nestjs/swagger/issues/1450

__decorate([
  Post('event-unsafe-batch'),
  HttpCode(200),
  openapi.ApiResponse({ status: 200, type: String }),
  __param(0, Body()),
  __param(1, Req()),
  __metadata("design:type", Function),
  __metadata("design:paramtypes", [Object, Object]),
  __metadata("design:returntype", Promise)
], EventUnsafeController.prototype, "logEventBatch", null);

2. openapi / class-validator DTOs are referenced by require API. https://github.com/microsoft/TypeScript/issues/43329

export class CspReportDto {
  static _OPENAPI_METADATA_FACTORY() {
    return { timestamp: { required: false, type: () => Object }, 'csp-report': { required: true, type: () => require("./csp.dto.js").CspReport } };
  }
}

3. NodeJS builtins are referenced via require API.

var require_async4 = __commonJS({
  "node_modules/resolve/lib/async.js"(exports, module2) {
    var fs2 = require("fs");

4. esbuild-compiled ESM bundle cannot refer to views/redoc.handlebars

const redocFilePath = path_1.default.join(__dirname, "..", "views", "redoc.handlebars");

5. _OPENAPI_METADATA_FACTORY class fields may be empty, so the swagger declaration cannot be properly rendered.

var Meta = class {
};
// →
var Meta = class {
  static _OPENAPI_METADATA_FACTORY() {
    return { appName: { required: false,  type: () =>  String }, appHost: { required: false,  type: () =>  String }, appVersion: { required: false,  type: () =>  String }, appNamespace: { required: false,  type: () =>  String }, appConfig: { required: false,  type: () =>  typeof (_a3 = typeof Record !== "undefined" && Record) === "function" ? _a3 : Object }, deviceInfo: { required: false,  type: () =>  typeof (_b3 = typeof Record !== "undefined" && Record) === "function" ? _b3 : Object }, userAgent: { required: false,  type: () =>  String }, envProfile: { required: false,  enum:  typeof (_c = typeof import_substrate2.EnvironmentProfile !== "undefined" && import_substrate2.EnvironmentProfile) === "function" ? _c : Object } }
  }
};

6. Extra type wrappers cannot be processed by openapi / class-validator / class-transformer

  __metadata("design:type", typeof (_d = typeof Array !== "undefined" && Array) === "function" ? _d : Object)
  __metadata("design:type", typeof (_e = typeof import_substrate2.LogLevel !== "undefined" && import_substrate2.LogLevel) === "function" ? _e : Object)
  // →
  __metadata("design:type", Array)
  __metadata("design:type", import_substrate2.LogLevel)

How to fix?

The right solution is certainly to improve the tools: create issues, discuss, suggest PRs.

How to fix it right here and right now?

@anatine/esbuild-decorators + old good monkey patching.
Fragile. Wrong. Terrible.

// build.js
import { build } from 'esbuild'
import path from 'node:path'
import { esbuildDecorators } from '@anatine/esbuild-decorators'

const cwd = process.cwd()
const outfile = path.resolve(cwd, 'output.js')
const tsconfig = path.resolve(cwd, 'tsconfig.json')
const entryPoints = [path.resolve(cwd, 'src/main/ts/index.ts')]
const config =  {
  platform: 'node',
  target: ['node18', 'es2021'],
  format: 'esm',
  bundle: true,
  keepNames: true,
  plugins: [
    esbuildDecorators({
      tsconfig,
      cwd
    }),
  ],
  tsconfig,
  entryPoints,
  outfile,
  external: [
    'kafkajs',
    'mqtt',
    'amqplib',
    'amqp-connection-manager',
    'nats',
    '@grpc/grpc-js',
    '@grpc/proto-loader',
    '@nestjs/websockets/socket-module',
    'class-transformer/storage'
  ]
}

await build(config)

node build.js && nestjs-esm-fix --target=./output.js

But it works.

npm-registry-firewall

Anton Golub — Wed, 13 Apr 2022 10:44:15 +0000

I couldn't google it, so I wrote my own: 

antongolub / npm-registry-firewall

npm registry proxy with on-the-fly filtering

Motivation

To mitigate security and legal risks.

Open Source is essential for modern software development. According to various estimates, at least 60% of the resulting codebase is composed of open repositories, libraries and packages. And keeps growing. Synopsys OSSRA 2021 report found that 98% of applications have open source components.

But open does not mean free. The price is the risk that you take:

Availability
Security
Legality / license

Let's consider these problems in the context of the JS universe.

Availability risks

JS packages are distributed in various ways: git repos, cdns and package registries. Regardless of the method, there are only two entry types that are finally resolved by any pkg manager: git-commit pointers and tarball links.

"dependencies": {
  "yaf" : "git://github.com/antongolub/yarn-audit-fix.git#commit-hash",
  "yaf2": "antongolub/yarn-audit-fix",
  "yarn-audit-fix" : "*"
}

yaf2@antongolub/yarn-audit-fix:
  version "9.2.1"
  resolved "https://codeload.github.com/antongolub/yarn-audit-fix/tar.gz/706646bab3b4c7209596080127d90eab9a966be2"
  dependencies:
    "@types/find-cache-dir" "^3.2.1"
    "@types/fs-extra" "^9.0.13"

"node_modules/yaf": {
  "name": "yarn-audit-fix",
  "version": "9.2.1",
  "resolved": "git+ssh://git@github.com/antongolub/yarn-audit-fix.git#706646bab3b4c7209596080127d90eab9a966be2",
  "license": "MIT",

"node_modules/yarn-audit-fix": {
  "version": "9.2.1",
  "resolved": "https://registry.npmjs.org/yarn-audit-fix/-/yarn-audit-fix-9.2.1.tgz",
  "integrity": "sha512-4biFNP4ZLOHboB2cNVuhYyelTFR/twlfmGMQ2TgJgGRORMDM/rQdQqhJdVLuKvfdMLFEPJ832z6Ws5OoCnFcfA==",
  "dependencies": {

So the implementation of mirroring is fundamentally quite simple: we just need to save and expose these assets from an alternative ssh/https entry point. Luckily this has already happened. The main repository for JS code is registry.npmjs.org. And at least 5 public replicas are always available as alternatives:

If this reliability level is not enough, you can easily run one more registry:

Security risks

Any code may not work properly. Due to error or malice. Keep in mind that most OSS licenses exclude any liability for damages. It's also important to always remember that oss code is not verified before being published.
These two circumstances sometimes give rise to dangerous incidents like colors.js or node-ipc.

The independent audit process is expensive, time consuming, so only setting a delay before using new pkg version might be effective countermeasure.

Legal risks

License agreement is an attribute of the moment: it can suddenly change and affect the development process (for example, husky-5). Uncontrolled use of new versions may have legal and financial consequences. Therefore, automated license checks should be part of CI/CD pipeline or the registry's own feature.

Key Features

Restricts access to remote packages by predicate: name, org, version (semver range), license, dateRange, username, age.
Multi-configuration: define as many port/context-path/rules combinations as you need.
Expressjs-inspired server implementation.
Has no deps. Literally zero.

Requirements

Node.js >= 14

Install

# npm
npm i npm-registry-firewall

# yarn
yarn add npm-registry-firewall

Usage

CLI

npm-registry-firewall /path/to/config.json

JS API

import {createApp} from 'npm-registry-firewall'

const app = createApp({
  server: {
    host: 'localhost',
    port: 3001,
  },
  firewall: {
    registry: 'https://registry.npmmirror.com',
    rules: [
      {
        policy: 'allow',
        org: '@qiwi'
      },
      {
        policy: 'deny',
        name: '@babel/*'
      },
    ]
  }
})

await app.start()

Config

{
  "server": {
    "host": "localhost",        // Defaults to 127.0.0.1
    "port": 3000,               // 8080 by default
    "secure": {                 // Optional. If declared serves via https
      "cert": "ssl/cert.pem",
      "key": "ssl/key.pem"
    },
    "base": "/",                // Optional. Defaults to '/'
    "healthcheck": "/health",   // Optional. Defaults to '/healthcheck'. Pass null to disable
    "keepAliveTimeout": 15000,  // Optional. Defaults 61000
    "headersTimeout": 20000,    // Optional. Defaults 62000
    "requestTimeout": 10000     // Optional. Defaults 30000
  },
  "firewall": {
    "registry": "https://registry.npmmirror.com",  // Remote registry
    "entrypoint": "https://r.qiwi.com/npm",        // Optional. Defaults to `${server.secure ? 'https' : 'http'}://${server.host}:${server.port}${route.base}`
    "base": "/",                // Optional. Defaults to '/'
    "rules": [
      {
        "policy": "allow",
        "org": "@qiwi"
      },
      {
        "policy": "allow",
        "name": ["@babel/*", "@jest/*", "lodash"] // string[] or "comma,separated,list". * works as .+ in regexp
      },
      {
        "policy": "deny",
        "name": "colors",
        "version": ">= v1.4.0"  // Any semver range: https://github.com/npm/node-semver#ranges
      },
      {
        "policy": "deny",
        "license": "dbad"       // Comma-separated license types or string[]
      },
      {
        "policy": "allow",
        "username": ["sindresorhus", "isaacs"] // Trusted npm authors.
      },
      {
        "policy": "allow",
        "name": "d",
        // `allow` is upper, so it protects `< 1.0.0`-ranged versions that might be omitted on next steps
        "version": "< 1.0.0"
      },
      {
        "policy": "deny",
        // Checks pkg version publish date against the range
        "dateRange": ["2010-01-01T00:00:00.000Z", "2025-01-01T00:00:00.000Z"]
      },
      {
        "policy": "allow",
        "age": 5    // Check the package version is older than 5 days. Like quarantine
      }
    ]
  }
}

Multi-config

// Array at the top level
[
  // Two servers (for example, http and https) share the same preset
  {
    "server": [
      {"port": 3001},
      {"port": 3002},
    ],
    "firewall": {
      "registry": "https://registry.yarnpkg.com",
      "rules": {"policy": "deny", "org": "@qiwi"}
    }
  },
  // One server has a pair of separately configured endpoints
  {
    "server": {"port": 3003},
    "firewall": [
      {"base": "/foo", "registry": "https://registry.npmjs.org", "rules": {"policy": "deny", "org": "@qiwi"}},
      {"base": "/bar", "registry": "https://registry.yarnpkg.com", "rules": {"policy": "deny", "org": "@babel"}}
    ]
  }
]

.npmrc

registry=https://localhost:3000
strict-ssl=false

npm view

npm-registry-firewall % npm view d versions                          
[ '0.1.0', '0.1.1' ]

output

$ node src/main/js/cli.js config.json
{"level":"INFO","timestamp":"2022-04-11T20:56:47.031Z","message":"npm-registry-firewall is ready for connections: https://localhost:3000"}
{"level":"INFO","timestamp":"2022-04-11T20:56:49.568Z","traceId":"44f21c050d8c6","clientIp":"127.0.0.1","message":"GET /d"}
{"level":"INFO","timestamp":"2022-04-11T20:56:50.015Z","traceId":"44f21c050d8c6","clientIp":"127.0.0.1","message":"HTTP 200 446ms"}

Contributing

Feel free to open any issues: bug reports, feature requests or questions.
You're always welcome to suggest a PR. Just fork this repo, write some code, put some tests and push your changes.
Any feedback is appreciated.

License

MIT

The missing `yarn audit fix` for Yarn 2+ Berry

Anton Golub — Sun, 12 Dec 2021 16:12:44 +0000

As everybody knows, audit --fix feature is out of scope of Yarn 1 Classic yarn/7075 and it also has not been implemented yet (2021-12-12) for Yarn 2+ berry/3582.

Let's just fix it.

tldr

npm_config_yes=true npx yarn-audit-fix@latest

1.

First of all, we need a lib to read/write yarnlock v2 files. @yarnpkg/lockfile seems the best choice, but it works with v1 only. Maybe nodejs-lockfile-parser?
Missed the mark again. It swaps checksums and does not provide dump/format API what is expected for the parser :). It turns out that we are missing yet another one lockfile processor. No problem. If we look closely, the new shiny yarn.lock v2 is a regular yaml with a little strange formatting like extra empty line delimiters, extra quotes, and so on.

import yaml from 'js-yaml'

export const parse = (raw: string): TLockfileObject => {
  const data = yaml.load(raw)
  delete data.__metadata

  return Object.entries(data).reduce<Record<string, any>>(
    (m, [key, value]: [string, any]) => {
      key.split(', ').forEach((k) => {
        m[k] = value
      })
      return m
    },
    {},
  )
}

export const format = (lockfile: TLockfileObject): string => {
  const keymap = Object.entries(lockfile).reduce<Record<string, any>>(
    (m, [k, { resolution }]) => {
      const entry = m[resolution] || (m[resolution] = [])
      entry.push(k)

      return m
    },
    {},
  )

  const data = Object.values(lockfile).reduce<Record<string, any>>(
    (m, value) => {
      const key = keymap[value.resolution].join(', ')
      m[key] = value

      return m
    },
    {
      __metadata: {
        version: 5,
        cacheKey: 8,
      },
    },
  )

  return `# This file is generated by running "yarn install" inside your project.
# Manual changes might be lost - proceed with caution!

${yaml.dump(data, {
  quotingType: '"',
  flowLevel: -1,
  lineWidth: -1,
})
  .replace(/\n([^\s"].+):\n/g, '\n"$1":\n')
  .replace(/\n(\S)/g, '\n\n$1')
  .replace(/resolution: ([^\n"]+)/g, 'resolution: "$1"')}`
}

2.

We need to fetch audit data. Yarn Berry integrated audit API is much better than previous Classic, which returned the report in the form of chunks.
yarn npm audit --all --recursive --json gives exactly what we need:

{
  "actions": [],
  "advisories": {
    "1004946": {
      "findings": [
        {
          "version": "4.1.0",
          "paths": [
            "ts-patch>strip-ansi>ansi-regex",
            "lerna>npmlog>gauge>ansi-regex",
            "lerna>@lerna/bootstrap>npmlog>gauge>ansi-regex",
            ...
          ]
        }
      ],
      "metadata": null,
      "vulnerable_versions": ">2.1.1 <5.0.1",
      "module_name": "ansi-regex",
      "severity": "moderate",
      "github_advisory_id": "GHSA-93q8-gq69-wqmw",
      "cves": [
        "CVE-2021-3807"
      ],
      "access": "public",
      "patched_versions": ">=5.0.1",
      "updated": "2021-09-23T15:45:50.000Z",
      "recommendation": "Upgrade to version 5.0.1 or later",
      "cwe": "CWE-918",
      "found_by": null,
      "deleted": null,
      "id": 1004946,
      "references": "- https://nvd.nist.gov/vuln/detail/CVE-2021-3807\n- https://github.com/chalk/ansi-regex/commit/8d1d7cdb586269882c4bdc1b7325d0c58c8f76f9\n- https://huntr.dev/bounties/5b3cf33b-ede0-4398-9974-800876dfd994\n- https://github.com/chalk/ansi-regex/issues/38#issuecomment-924086311\n- https://app.snyk.io/vuln/SNYK-JS-ANSIREGEX-1583908\n- https://github.com/chalk/ansi-regex/issues/38#issuecomment-925924774\n- https://github.com/advisories/GHSA-93q8-gq69-wqmw",
      "created": "2021-11-18T16:00:48.472Z",
      "reported_by": null,
      "title": " Inefficient Regular Expression Complexity in chalk/ansi-regex",
      "npm_advisory_id": null,
      "overview": "ansi-regex is vulnerable to Inefficient Regular Expression Complexity",
      "url": "https://github.com/advisories/GHSA-93q8-gq69-wqmw"
    },

We take only significant fields: vulnerable_versions, module_name, patched_versions

export const parseAuditReport = (data: string): TAuditReport =>
  Object.values(JSON.parse(data).advisories).reduce<TAuditReport>(
    (m, { vulnerable_versions, module_name, patched_versions }: any) => {
      m[module_name] = {
        patched_versions,
        vulnerable_versions,
        module_name,
      }
      return m
    },
    {},
  )

3.

Almost done. Now we need to replace vulnerable packages versions in lockfile with the advisories, taking into account semver-compatibility and to remove the prev checksum fields. This brilliant idea was suggested by G. Kosev. Important note Seems like dependencies of the patched entries are not reloaded automatically by Yarn, so they need to be requested from the registry by hand: yarn npm info react --fields dependencies --json.

export const patchEntry = (
  entry: TLockfileEntry,
  name: string,
  newVersion: string,
  npmBin: string,
): TLockfileEntry => {
  entry.version = newVersion
  entry.resolution = `${name}@npm:${newVersion}`

  // NOTE seems like deps are not updated by `yarn mode='--update-lockfile'`, only checksums
  entry.dependencies =
    JSON.parse(
      invoke(
        npmBin,
        ['view', `${name}@${newVersion}`, 'dependencies', '--json'],
        process.cwd(),
        true,
        false,
      ) || 'null',
    ) || undefined

  delete entry.checksum

  return entry
}

export const _patch = (
  lockfile: TLockfileObject,
  report: TAuditReport,
  { flags, bins }: TContext,
  lockfileType: TLockfileType,
): TLockfileObject => {
  if (Object.keys(report).length === 0) {
    !flags.silent && console.log('Audit check found no issues')
    return lockfile
  }

  const upgraded: string[] = []

  for (const depSpec of Object.keys(lockfile)) {
    // @babel/code-frame@^7.0.0
    // @babel/code-frame@npm:^7.0.0

    const [, pkgName, desiredRange] =
      /^(@?[^@]+)@(?:\w+:)?(.+)$/.exec(depSpec) || []

    const pkgAudit = report[pkgName]
    if (!pkgAudit) continue
    const pkgSpec = lockfile[depSpec]
    if (sv.satisfies(pkgSpec.version, pkgAudit.vulnerable_versions)) {
      const fix = sv.minVersion(pkgAudit.patched_versions)?.format()
      if (fix === undefined) {
        console.error(
          "Can't find satisfactory version for",
          pkgAudit.module_name,
          pkgAudit.patched_versions,
        )
        continue
      }
      if (!sv.satisfies(fix, desiredRange) && !flags.force) {
        console.error(
          "Can't find patched version that satisfies",
          depSpec,
          'in',
          pkgAudit.patched_versions,
        )
        continue
      }
      upgraded.push(`${pkgName}@${fix}`)

      patchEntry(pkgSpec, pkgName, fix, bins.npm)
    }
  }

4.

The final step is to update the checksums.

yarn install mode='--update-lockfile'

Refs

Actually you don’t need 'semantic-release' for semantic release

Anton Golub — Sun, 23 May 2021 12:32:10 +0000

I’m a big fan of semantic-release since it appeared. I followed its development, studied its inners. I made in-house reports, held workshops and finally brought semrel to our build infrastructure. I wrote plugins, plugin-factories and testing-tools for it. For several years now, I've been trying to combine semantic releases and monorepositories in many OSS projects:

Etc, etc, so on. I’m just trying to say, that semrel had a significant impact on my professional life.

Semrel goal

The main purpose of semantic-release is to transform semantic (conventional) commits into build artifacts and deployments. With version bumping, changelogs, tagging, pkg publishing. “Fully-automated release” — is the true. There are also dozens on plugins, so you’ll most likely find a solution for any standard case. It really saves times.

But sometimes

You may need a minor tweak up. For example, push some pkg to both public and internal registries. Ooops. "...publishing to two different registry is not a good idea". In this case you can not rely on stable, convenient and tested in millions runs semrel/npm plugin, and you have to just write a pair of commands by hand with semantic-release/exec instead:

echo "//npm-registry.domain.com/:_authToken=${TOKEN}” >> .npmrc
echo "\`jq '.name="@scope/pkg-name”’ package.json\`" > package.json
npm config set registry https://npm-registry.domain.com
npm publish --no-git-tag-version

Another instance — disabling git notes fetching. "Afraid that won't be possible".

Of course, you may fork semrel and remove the mentioned line. Or create a plugin/hook, that will override loaded execa module with patched one version, than just skips git notes invocation (this is really frustrating, I did smth similar). Or… {{ another crazy workaround goes here }}.

This is a watershed moment. Once you start to fight against the tool, it's time to just pick another one. The new dilemma:

Spend days and days for searching, tuning and testing analogs.
Write your own semantic-release.

My opinionated suggestion: if your case is very simple or, conversely, very complex, the second option will be optimal. Release script — is not a rocket science!

140 lines alternative

Let's take a look at what exactly each release consists of, if we discard the high-level tool contracts. I use zx in the examples, but it could be execa or native child_process.exec too.

1. Git configuration

To make a commit you need a committer: just name and email that will be associated with author. Also PAT or SSH token is required to push the commit.

const {GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, GITHUB_TOKEN} = process.env
if (!GITHUB_TOKEN || !GIT_COMMITTER_NAME || !GIT_COMMITTER_EMAIL) {
  throw new Error('env.GITHUB_TOKEN, env.GIT_COMMITTER_NAME & env.GIT_COMMITTER_EMAIL must be set')
}

const gitAuth = `${GIT_COMMITTER_NAME}:${GITHUB_TOKEN}`
const originUrl = (await $`git config --get remote.origin.url`).toString().trim()
const [,,repoHost, repoName] = originUrl.replace(':', '/').replace(/\.git/, '').match(/.+(@|\/\/)([^/]+)\/(.+)$/)
const repoPublicUrl = `https://${repoHost}/${repoName}`
const repoAuthedUrl = `https://${gitAuth}@${repoHost}/${repoName}`
await $`git config user.name ${GIT_COMMITTER_NAME}`
await $`git config user.email ${GIT_COMMITTER_EMAIL}`
await $`git remote set-url origin ${repoAuthedUrl}`

2. Commit analysis

Conventional commits are just a prefixed strings in git log. We should define some rules on how to associate messages substrings with corresponding release types:

const semanticTagPattern = /^(v?)(\d+)\.(\d+)\.(\d+)$/
const releaseSeverityOrder = ['major', 'minor', 'patch']
const semanticRules = [
{group: 'Features', releaseType: 'minor', prefixes: ['feat']},
{group: 'Fixes & improvements', releaseType: 'patch', prefixes: ['fix', 'perf', 'refactor', 'docs']},
{group: 'BREAKING CHANGES', releaseType: 'major', keywords: ['BREAKING CHANGE', 'BREAKING CHANGES']},
]

Then we search for the prev release tag, that satisfies semver pattern:

const tags = (await $`git tag -l --sort=-v:refname`).toString().split('\n').map(tag => tag.trim())
const lastTag = tags.find(tag => semanticTagPattern.test(tag))

And make commits cut from the found ref:

const newCommits = (lastTag
  ? await $`git log --format=+++%s__%b__%h__%H ${await $`git rev-list -1 ${lastTag}`}..HEAD`
  : await $`git log --format=+++%s__%b__%h__%H HEAD`)
  .toString()
  .split('+++')
  .filter(Boolean)
  .map(msg => {
    const [subj, body, short, hash] = msg.split('__').map(raw => raw.trim())
    return {subj, body, short, hash}
  })

Now we just need to parse them:

const semanticChanges = newCommits.reduce((acc, {subj, body, short, hash}) => {
  semanticRules.forEach(({group, releaseType, prefixes, keywords}) => {
    const prefixMatcher = prefixes && new RegExp(`^(${prefixes.join('|')})(\\(\\w+\\))?:\\s.+$`)
    const keywordsMatcher = keywords && new RegExp(`(${keywords.join('|')}):\\s(.+)`)
    const change = subj.match(prefixMatcher)?.[0] || body.match(keywordsMatcher)?.[2]

    if (change) {
      acc.push({
        group,
        releaseType,
        change,
        subj,
        body,
        short,
        hash
      })
    }
  })
  return acc
}, [])

Ta-da. Semantic changes:

semanticChanges= [
  {
    group: 'Fixes & improvements',
    releaseType: 'patch',
    change: 'perf: use git for tags sorting',
    subj: 'perf: use git for tags sorting',
    body: '',
    short: 'a1abdae',
    hash: 'a1abdaea801824d0392e69f9182daf4d5f4b97db'
  },
  {
    group: 'Fixes & improvements',
    releaseType: 'patch',
    change: 'refactor: minor simplifications',
    subj: 'refactor: minor simplifications',
    body: '',
    short: 'be847a2',
    hash: 'be847a26e2b0583e889403ec00db45f9f9555e30'
  },
  {
    group: 'Fixes & improvements',
    releaseType: 'patch',
    change: 'fix: fix commit url template',
    subj: 'fix: fix commit url template',
    body: '',
    short: '3669edd',
    hash: '3669edd7eb440e29dc0fcf493c76fbfc04271023'
  }
]

3. Resolve next version:

const nextReleaseType = releaseSeverityOrder.find(type => semanticChanges.find(({releaseType}) => type === releaseType))
if (!nextReleaseType) {
  console.log('No semantic changes - no semantic release.')
  return
}
const nextVersion = ((lastTag, releaseType) => {
  if (!releaseType) {
    return
  }
  if (!lastTag) {
    return '1.0.0'
  }

  const [, , c1, c2, c3] = semanticTagPattern.exec(lastTag)
  if (releaseType === 'major') {
    return `${-~c1}.0.0`
  }
  if (releaseType === 'minor') {
    return `${c1}.${-~c2}.0`
  }
  if (releaseType === 'patch') {
    return `${c1}.${c2}.${-~c3}`
  }
})(lastTag, nextReleaseType)

const nextTag = 'v' + nextVersion

4. Generate release notes

const releaseDiffRef = `## [${nextVersion}](${repoPublicUrl}/compare/${lastTag}...${nextTag}) (${new Date().toISOString().slice(0, 10)})`
const releaseDetails = Object.values(semanticChanges
.reduce((acc, {group, change, short, hash}) => {
const {commits} = acc[group] || (acc[group] = {commits: [], group})
const commitRef = `* ${change} ([${short}](${repoPublicUrl}/commit/${hash}))`

      commits.push(commitRef)

      return acc
    }, {}))
    .map(({group, commits}) => `
### ${group}
${commits.join('\n')}`).join('\n')

const releaseNotes = releaseDiffRef + '\n' + releaseDetails + '\n'

5. Update CHANGELOG.md

Attach releaseNotes to file. Just one string.

await $`echo ${releaseNotes}"\n$(cat ./CHANGELOG.md)" > ./CHANGELOG.md`

6. Update package version

await $`npm --no-git-tag-version version ${nextVersion}`

7. Git release.

Create commit. Create tag. Push them.

const releaseMessage = `chore(release): ${nextVersion} [skip ci]`
await $`git add -A .`
await $`git commit -am ${releaseMessage}`
await $`git tag -a ${nextTag} HEAD -m ${releaseMessage}`
await $`git push --follow-tags origin HEAD:refs/heads/master`

8. GitHub release

Just a single curl POST to gh rest api.

const releaseData = JSON.stringify({
  name: nextTag,
  tag_name: nextTag,
  body: releaseNotes
})
await $`curl -u ${GIT_COMMITTER_NAME}:${GITHUB_TOKEN} -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${repoName}/releases -d ${releaseData}`

9. Npm publish

await $`npm publish --no-git-tag-version`

Need several registries? NP.

await $`npm config set registry https://registry.npmjs.org`
await $`npm publish --no-git-tag-version`
await $`echo "\`jq '.name="@${repoName}"' package.json\`" > package.json`
await $`npm config set registry https://npm.pkg.github.com`
await $`npm publish --no-git-tag-version`

Conclusions

This solution does not cover corner cases and has significant limitations of usage. Ultimately, you don't care if other tools have 99.99999% applicability until they ignore just one specific case — yours. But now you have completely taken back release flow control. You're able to improve and modify this snippet as you wish and whenever you like.

release.mjs
gh release.yaml
release log

semrel-extra / zx-semrel

`zx`-based release script as `semantic-release` alternative (PoC)

zx-semrel

zx -based release script as semantic-release alternative (PoC)

Sometimes bloody enterprise enforces you not to use any third-party solutions for sensitive operations (like release, deploy, so on) Old good script copy-paste hurries to the rescue!

Btw, here's an adaptation for monorepos: zx-bulk-release

Requirements

macOS / linux
Node.js >= 14.13.1
git >= 2.0
zx >= 1.6.0

Key features

Zero dependencies
Zero configuration
Pretty fast
Tiny, less than 140 lines with comments
Reliability, safety, simplicity and maintainability (sarcasm)

Functionality

Poor conventional commits analysis
CHANGELOG.md generation
package.json version bumping
Git release commit creation
GitHub Release
Package publishing to both npmjs and gh registries

🚀 Usage

Copy
Tweak up, inject tokens, etc
Run

curl https://raw.githubusercontent.com/semrel-extra/zx-semrel/master/release.mjs > ./release.mjs
zx ./release.mjs

or this like if zx is not installed:

# Just replace GIT* env values with your own
GIT_COMMITTER_NAME=antongolub GIT_COMMITER_EMAIL=mailbox@antongolub.ru GITHUB_TOKEN=token npx zx ./release.mjs

or just run it without any edits though npx:

…

View on GitHub

DEV Community: Anton Golub

yarn.lock: may the `--force` be with you

TL;DR

The fix that installed clean and broke anyway

--force: break to fix

--safe: skip to stabilize

Engines: will the fix run on the Node you target?

Licenses: does the fix pass your policy?

Package format: does the fix stay require-able?

Scope: which advisories, not just which fix

Attribution: a skip you can act on

Under the hood

Where this leaves the tool

Sources / links

yarn.lock: you can't `sed` a graph

TL;DR

Recap: the wall

One graph for every Yarn lockfile

The pipeline

1. Advisories, straight from the registry

2. Choosing the fix version

3. The transformation

4. Making it reproduce: checksums

5. The invariant

Does it actually reproduce? A real-world test

A note on portability

The side effect: real lockfiles are the test suite

Where this goes

Sources / links

yarn audit [still does not] fix

TL;DR

audit ≠ audit fix

Years of asking — for something hard

Where the difficulty comes from

The remediation landscape (and the gap for a local fix)

Built-in / local (manual)

The bots (asynchronous, hosted)

The gap they leave

How we try to help

Sources

CVE-2024-29415: problem solution

Algorithm

Compatibility

Drop-in

Next steps

zx@lite: minimalistic shell scripting with TS/JS superpowers

⚡ What is zx@lite?

🧾 Setup

🧪 Examples

⚠️ What’s missing in zx@lite?

📦 Upgrade path

📚 References

🧠 Final thoughts

How and why do we bundle zx?

Short history

Packed as CommonJS

Dropped CJS support

Migrated to TypeScript

Problems

Size

Compatibility

Stability & Security

Bundle

js

d.ts

Results

google / zx

A tool for writing better scripts

zx

Install

Documentation

License

config: apply extends

Proposal

20 minutes adventure

Conclusion

config: enhancing declarations with graphs

Config mess

Budget loss

What we need

`--force`: break to fix

`--safe`: skip to stabilize

`audit` ≠ `audit fix`

⚡ What is `zx@lite`?

⚠️ What’s missing in `zx@lite`?