Why we moved our Backstage platform from Yarn to pnpm

What Git worktrees, parallel coding agents, and a cache pointing at the wrong directory taught us about the assumptions hiding inside node_modules.

I’m a developer at bol, where we have been building and running our internal developer platform on Backstage for more than five years.

After that much time, a Backstage platform becomes more than a collection of plugins. It accumulates its own ways of working: CI conventions, local startup tooling, container images, Git worktree helpers, and shortcuts that make sense inside one engineering organization.

One of our most productive choices has been using Git worktrees for parallel development with coding agents. An agent can fix a bug in one worktree while another handles a small improvement elsewhere, without branches fighting over the same working directory. When several small tasks are moving at once, this can multiply the amount of useful work we finish.

That workflow also made our package manager part of the architecture. We wanted a more forward-looking setup, and pnpm looked like a better fit: a shared content-addressable store, efficient reuse across checkouts, and a dependency layout managed by the package manager itself.

Our goal was simple: let independent worktrees move in parallel, while package content is reused safely from one shared store.

So we moved from Yarn 4 to pnpm. I expected the lockfile and commands to be the main work. They were only the entrance to the migration.

The migration looked finished until CI ran

The visible switch was straightforward: pin pnpm 12.3.0, commit pnpm-lock.yaml, replace the Yarn commands, and update the contributor guides. Then CI ran and downloaded 4,226 packages from scratch.

The next run did the same. The log read reused: 0, downloaded: 4226. The download itself took about 18 seconds, but the native node-gyp builds added several minutes on top. We had changed package managers, and the pipeline was behaving as if no cache existed.

It turned out that the cache did exist. It was looking in the wrong place.

The store had two addresses

Our container image carried a global pnpm configuration that sent the store to /builds/.pnpm-store. GitLab, meanwhile, was trying to archive .pnpm-store inside the project checkout. pnpm wrote outside that directory, the archiver found no matching files, and every job that followed started cold.

The fix was small: set the location explicitly inside CI, and print the effective value so the job log can prove it.

.gitlab-ci.yml

variables:
  npm_config_store_dir: '.pnpm-store'

before_script:
  - pnpm config set store-dir "$CI_PROJECT_DIR/.pnpm-store"
  - pnpm store path

That was the first real lesson of this migration: never reason about a package-manager cache from configuration files alone. Ask the running job where its store actually is.

We also stopped retrying script failures automatically. A dropped connection may deserve a retry. A --frozen-lockfile mismatch will fail six times for exactly the same reason, and bill you for six runners on the way.

When a working cache became the slow part

Once the store was finally cached, the next problem surfaced. The archive now held the pnpm store and every node_modules tree: 1.42 GB spread across roughly 601,000 files.

Four jobs pulled that archive. Saving, transferring, and extracting it became a substantial part of the pipeline. Worse, our installs still rebuilt the dependency layout and the native modules anyway, so those cached node_modules trees were not buying the time we expected.

After keeping the pnpm store and dropping node_modules from this cache, its size fell by about 72% and its file count by about 59%. Figures are approximate observations taken during the migration.

We removed node_modules, plugins/*/node_modules, and packages/*/node_modules from the cache paths. The archive fell to roughly 400 MB and 244,000 files—about six minutes saved per pipeline, by our estimate.

The useful lesson was to measure the dependency phase we actually had: restore, install, native builds, save. pnpm’s own CI documentation warns that caching its store is not guaranteed to make installation faster; the right policy depends on your runner, your network, and your workload. [1]

The fastest retry is the one you do not schedule

We switched installs to --frozen-lockfile --prefer-offline, made transfer progress visible in the logs, and chose faster cache compression. Each change removed a little uncertainty. That mattered, because the migration was no longer one problem: it was a chain in which every fix exposed the next bottleneck.

Then pnpm met our custom worktree machinery

The hardest surprise was local development. Our worktree bootstrap contained about 350 lines of custom symlink-farm logic, all of it built around the node_modules structure we had used with Yarn.

The helper mirrored third-party dependencies from the main checkout and redirected workspace packages to each worktree’s own source. It was clever, fast when its assumptions held, and deeply coupled to a filesystem layout that the package manager was free to change.

pnpm uses a virtual store and links packages into the dependency graph. [2] In our setup, the old mirroring code met pnpm links where it expected ordinary directories, and we began seeing ENOTDIR failures. The optimization that made worktrees convenient had become the thing preventing worktrees from working.

The best fix deleted the clever part

Instead of teaching our symlink farm every detail of pnpm’s layout, I deleted it. The central helper went from 370 lines to 31, and the whole bootstrap became one command:

pnpm install --frozen-lockfile

Each checkout gets an installation created by pnpm, while package content can still be reused through the shared store.

This was the point where the migration began to feel successful. We had not recreated the previous mechanism under a new name. We had returned ownership of dependency layout to the package manager.

There was a tradeoff. Our replacement verifier is much lighter: it checks for node_modules/.pnpm instead of walking dangling links and proving that every workspace package resolves correctly. Less maintenance code is valuable, but I still want an integration check that edits a plugin in a secondary worktree and confirms the running application uses that source.

Existing laptops remembered the old world

A clean checkout worked. Some existing developer checkouts did not.

They still contained the project-local .pnpm-store created by our earlier configuration. Once local development moved to the global store, pnpm detected the mismatch and relinked roughly 4,800 packages on repeated starts—about four minutes, for a change that should not have required a dependency install at all.

We added a one-time migration to startup: detect the obsolete local store, remove it together with the root node_modules and the dependency hash, then run one clean install. Later starts with an unchanged lockfile take the existing hash-skip path and return almost immediately.

That distinction matters. pnpm did not install 4,800 packages in zero seconds. Our startup code learned when no install was needed.

A migration must include yesterday’s state

This changed how I think about developer-tool migrations. Testing a fresh clone is necessary, but your colleagues do not all have fresh clones. They have old caches, generated files, global configuration, half-finished branches, and worktrees created before the migration.

The cleanup also had to be environment-specific. CI intentionally kept a project-local store so GitLab could archive it. Local development treated that same directory as stale. The name of a folder is not enough context to decide whether it should be deleted.

Slow networks exposed another edge

On slower office and VPN connections, pnpm’s default request concurrency could lead to timeouts. We time a request for a small artifact in our registry; when a successful request takes more than three seconds, bootstrap adds --network-concurrency=1, five fetch retries, and a longer maximum retry timeout.

The idea helped, but the implementation taught us something as well: a failed probe never reaches the slow-success branch. A production version of this pattern should decide explicitly what a timeout, an authentication failure, and a slow success each mean.

The journey improved more than installation

Once we were looking closely at the pipeline, we found work that was only loosely related to pnpm but still affected the experience of the migration.

Jest coverage was one example. Its cache had grown to 5.2 GB. We switched to the V8 coverage provider and enabled inline source maps for @swc/jest, which brought the cache back to no more than 2 GB. [3] A later change removed Jest-cache transfer entirely and revisited test selection, worker limits, and when coverage runs.

These are reported outcomes from the wider CI work. The unit-test change includes test selection, coverage policy, worker, and cache changes, so it is not a pnpm-only benchmark.

Unit-test wall time moved from 17 minutes to roughly 10. That result is useful, but I would not present it as “pnpm made our tests 41% faster.” The amount and the kind of test work changed too.

We enabled incremental TypeScript compilation and baked the pinned pnpm version into the runtime image. We also had to make our native-build policy explicit: one image gained pkg-config and the libsecret development headers, while our pnpm workspace policy later disabled keytar’s build.

Installing native prerequisites and disabling a build are different choices. If the application needs the native module at runtime, a green install is not sufficient evidence that the decision is safe.

The results I would confidently share

• The dependency cache shrank from 1.42 GB to about 400 MB, and from roughly 601,000 files to 244,000.
• A worktree bootstrap helper shrank from 370 lines to 31, by replacing custom filesystem logic with pnpm install.
• Warm installs in the reported CI run reused 4,226 packages and downloaded none, once the store path was corrected.
• Unchanged local startup can skip dependency installation entirely, after the stale state is cleaned once.
• Jest cache size and unit-test duration both came down—though several of those causes sit outside the package-manager switch.

What I would tell the next team

Moving a mature Backstage monorepo to pnpm is not mainly a search-and-replace exercise. The package manager sits underneath a web of assumptions about directories, caches, startup order, native builds, and developer habits.

• Start with the workflow you want to enable. For us, that was dependable parallel development with Git worktrees and coding agents.
• Map every place that knows about dependencies: CI jobs, images, bootstrap scripts, worktree tools, docs, generators, and assistant instructions.
• Print the effective store path inside the real job image. Run the same lockfile twice and capture restore, install, native-build, and save time separately.
• Benchmark no cache, store-only caching, and your existing strategy. Measure elapsed pipeline time separately from the sum of concurrent job durations.
• Exercise a clean clone and an old checkout. Test stale stores, unchanged restarts, dependency changes, slow networks, and authentication failures.
• Change source code inside a secondary worktree and prove the running Backstage instance uses it. Directory existence alone is a weak health check.
• Prefer deleting compatibility machinery when the new package manager can own the same responsibility.

What happened on our journey

We began with a productivity idea: parallel agents working safely in separate Git worktrees. We chose pnpm because its shared store and installation model fit that direction. Then the migration forced us to confront every place where our platform still depended on the old world.

We fixed a cache that pointed to the wrong directory. We removed hundreds of thousands of unnecessary files from that cache. We deleted a custom symlink farm. We cleaned stale state from existing laptops, adapted to slow connections, and made native build policy explicit.

The biggest gain was not one benchmark. Our setup became easier to explain: pnpm manages dependencies, CI caches a deliberate store, and each worktree is an independent place for an engineer or a coding agent to work. That clarity is what lets the productivity benefit survive after the migration project is over.

Appendix: the code changes, end to end

Everything above is the story. This is the diff. Below are the concrete changes we made, in the order that mattered, so you can lift them into your own repository without repeating the week we spent finding them. Versions and paths are ours—adjust them to your setup.

Do step 5 first. If the store lands in the wrong directory, everything else here is wasted: the cache misses on every run, and every number you measure afterward is the wrong number.

1. package.json—declare the package manager

Pin the version once, so CI, laptops, and containers cannot disagree about it.

{
  "packageManager": "pnpm@12.3.0"
}

2. .npmrc—use the global store, prefer offline

-store-dir=.pnpm-store
+prefer-offline=true
+side-effects-cache=true

Removing store-dir is the line that matters. A project-local store conflicts with the global one, and it is exactly the leftover that step 11 has to clean up later.

3. tsconfig.json—enable incremental compilation

   "compilerOptions": {
+    "incremental": true,
+    "tsBuildInfoFile": "build/.tsbuildinfo"
   }

Add build/.tsbuildinfo to .gitignore.

4. Jest—switch the coverage provider to V8

jest.config.js (or the Jest block in package.json):

-"sourceMaps": false
+"sourceMaps": "inline"

Inline source maps are what V8 needs to map coverage back to your sources. Then change the CI test command:

-pnpm test:all --coverage
+pnpm test:all --coverage --coverageProvider=v8

Why: the Babel provider writes duplicate transform-cache entries—one instrumented, one not—which is what took our Jest cache from around 2 GB to over 5 GB. [3]

5. GitLab CI—make the store path win

The Docker image’s global pnpm configuration overrides .npmrc, so configuration alone will not win this argument. An environment variable will. Log the effective path in the same breath, so the job can prove where its store is:

variables:
  npm_config_store_dir: '.pnpm-store' # env var beats global config

before_script:
  - corepack enable pnpm
  - pnpm config set store-dir "$CI_PROJECT_DIR/.pnpm-store"
  - pnpm store path # log it, so you can verify it

6. GitLab CI—cache the store, not node_modules

 cache:
   paths:
     - '.pnpm-store/'
-    - 'node_modules/'
-    - 'plugins/*/node_modules/'
-    - 'packages/*/node_modules/'

pnpm re-links from the store on every install, so caching node_modules adds archive overhead with no install-time benefit. While you are in there, switch cache compression to fast—a large store is network-bound, not CPU-bound:

variables:
  CACHE_COMPRESSION_LEVEL: 'fast'
  TRANSFER_METER_FREQUENCY: '2s' # shows transfer times in the job log

7. GitLab CI—pass --prefer-offline in every job

-pnpm install --frozen-lockfile
+pnpm install --frozen-lockfile --prefer-offline

In every job that installs—install_deps, build, test-and-lint—not only the install job, so the pipeline still behaves when the cache is cold.

8. GitLab CI—stop retrying script failures

 retry:
   when:
     - runner_system_failure
     - stuck_or_timeout_failure
     - api_failure
-    - script_failure # six retries for one deterministic lockfile failure

9. CI image—give keytar what it needs, or turn its build off

 RUN apt-get install -y \
     python3 g++ build-essential \
+    pkg-config libsecret-1-dev

The alternative is adding keytar to allowBuilds: false. These are genuinely different decisions: if the application needs the native module at runtime, a green install is not evidence that skipping the build was safe.

10. Runtime image—bake Corepack in once

Dockerfile (base image):

ARG PNPM_VERSION=12.3.0
RUN corepack enable && corepack prepare pnpm@${PNPM_VERSION} --activate

Dockerfile (application image):

-RUN corepack enable && corepack prepare pnpm@12.3.0

Caveat: if your build environment cannot reach a private registry from RUN layers—Cloud Build, in our case—point this one step at the public npm registry.

11. Startup script—clean up the stale local store

If store-dir=.pnpm-store was ever in your .npmrc, every existing checkout still has a project-local store. pnpm sees the storeDir mismatch and re-links every package on every install—roughly four minutes, for us. One guard in the start script fixes it for the whole team, once:

const staleLocalStore = join(root, '.pnpm-store');
if (existsSync(staleLocalStore)) {
  rmSync(staleLocalStore, { recursive: true, force: true });
  rmSync(join(root, 'node_modules'), { recursive: true, force: true });
  if (existsSync(hashFile)) rmSync(hashFile);
}

Keep this environment-specific. CI deliberately keeps a project-local store so GitLab can archive it; only local development should treat that directory as stale. The name of a folder is not enough context to decide whether it should be deleted.

12. Install scripts—a slow-network guard

Time one small request to the registry, and back off the concurrency only when a successful request is slow:

_pnpm_extra_flags=()
_start_ms=$(date +%s%3N)
curl -fsSo /dev/null --max-time 5 \
  https://registry.npmjs.org/is-odd/-/is-odd-3.0.1.tgz 2>/dev/null && \
  _elapsed=$(( $(date +%s%3N) - _start_ms )) && \
  (( _elapsed > 3000 )) && \
  _pnpm_extra_flags+=(--network-concurrency=1 --fetch-retries=5 \
                      --fetch-retry-maxtimeout=120000) || true

pnpm install --frozen-lockfile "${_pnpm_extra_flags[@]+"${_pnpm_extra_flags[@]}"}"

Note the edge in our version: a probe that fails outright falls straight through to the normal path. Decide deliberately what a timeout, an authentication failure, and a slow success should each do.

13. Worktrees—do not install in secondary checkouts

If you have worktrees or slot-mode checkouts that share the main node_modules, do not run pnpm install in the secondary checkout. In hoisted mode, pnpm 12 hits ENOTDIR when it tries to create symlinks over the real directories the shared farm has already placed there.

What the numbers looked like

Metric	Before	After
install_deps on a new lockfile	~3.5 min (reused 0, downloaded 4226)	Seconds (reused 4226, downloaded 0)
CI cache size	1.42 GB / 601k files	~400 MB / 244k files
Per-pipeline cache overhead (4 jobs)	+~12 min	Baseline
Jest cache size	5.2 GB	≤ 2 GB
Unit-test wall time	17 min	~10 min
Local startup (executor JIT)	22–26 s silent gap	~2–3 s
Local install after a 1-line change	~4 min (stale store)	~0 s (hash skip)
CI Corepack setup	Every application build	Once per runtime image rebuild

These are observations from our pipeline, not a controlled benchmark. Several of them—the test numbers in particular—include changes that have nothing to do with the package manager. Measure your own before and after.

References

1. pnpm—Continuous Integration
2. pnpm—Symlinked node_modules structure
3. Jest—coverageProvider