DEV Community: Bogdan Nechyporenko

Four Ways to Survive a Network Split

Bogdan Nechyporenko — Sun, 13 Sep 2026 08:45:10 +0000

What Paxos, VR, Zab, and Raft teach us.

You may never implement consensus yourself. You still need its mental models to choose databases, coordination services, and infrastructure that fail the way you expect.

At 2:13 a.m., your monitoring says the service is healthy.

The API is answering. The database has replicas. Every dashboard is green enough to let you hope the pager was a mistake.

But two nodes disagree about who the leader is.

One accepted a write just before the network split. Another is about to accept a conflicting write. Both are alive. Both are behaving rationally with the information they have. And somewhere between them sits a question that a health check cannot answer:

When several machines remember different versions of reality, which one becomes the truth?

That is the problem consensus algorithms solve.

Most developers will never implement one—and probably should not. Yet many of us choose and operate systems built on them: databases, configuration stores, service-discovery platforms, schedulers, control planes, and distributed locks. If you understand the ideas underneath those tools, their tradeoffs stop looking like mysterious product limitations. They become predictable consequences of design.

This article looks at four protocols: Multi-Paxos, Viewstamped Replication, Zab, and Raft. The goal is not to reproduce their proofs or prepare you to write one over a weekend. It is to learn four ways of thinking about the same hard problem—and then use those mental models when evaluating real infrastructure.

Who this is for

This is for backend developers, platform engineers, SREs, architects, and technical leads who choose or operate distributed systems but do not build consensus protocols themselves.

You should leave with enough intuition to:

Ask better questions during a database or coordination-tool evaluation.
Understand why a system may reject traffic while several machines are still running.
Reason about leader changes, stale reads, write latency, quorum size, and recovery time.
Recognize which complexity a tool has absorbed—and which complexity it has handed back to you.

Suppose three replicas maintain the same state. A client sends SET plan = pro. The replicas must agree not only that the command happened, but where it belongs in the history. If command number 42 is SET plan = pro on one replica, command number 42 cannot be DELETE account on another.

The usual mechanism is a replicated log. Each entry is an ordered command. Once a quorum—typically a majority—has accepted an entry under the protocol's rules, the cluster can treat it as committed and apply it to a state machine.

A majority matters because any two majorities overlap. In a five-node cluster, a quorum of three can tolerate two crashed or unreachable nodes. Two separate groups cannot both form a majority, so a network partition cannot safely produce two independent committed histories.

That safety has a price. If no majority can communicate, a correctly designed cluster may stop accepting writes. The machines are up; the service is unavailable by choice. This is not necessarily a bug. It may be the system refusing to invent two truths.

One more boundary: the four protocols here are designed primarily for crash faults—nodes fail, restart, or become unreachable—not Byzantine behavior where a participant lies or sends maliciously contradictory messages.

Why exactly these four?

These four are useful together because they form a compact tour through the major design ideas behind crash-tolerant replicated state machines.

Protocol	The design question it highlights	The lesson worth keeping
Multi-Paxos	How can repeated decisions reuse stable leadership?	Separate safety from the optimizations that make a protocol fast.
Viewstamped Replication (VR)	How does a primary-backup system change leaders without losing committed work?	Recovery and view change are part of the protocol, not cleanup after it.
Zab	How do we preserve a strict prefix of history for a coordination service?	A workload-specific ordering contract can shape the entire protocol.
Raft	Can the same safety goals be expressed as understandable, enforceable rules?	Comprehensibility is an operational feature.

They are not the only important consensus algorithms, nor four entries in a leaderboard. They overlap substantially. The value of comparing them is seeing where each one places structure: in ballots, views, epochs, terms, logs, leaders, and recovery rules.

Multi-Paxos

Basic Paxos reaches agreement on one value. Roughly, a proposer first asks acceptors to promise not to accept older proposals, then asks them to accept a value under a numbered ballot. The subtle rules ensure that once a value can be chosen, a later ballot cannot replace it with a conflicting value.

Doing that full exchange for every log position would be expensive. Multi-Paxos adds the practical move: establish a stable leader for a ballot, then let that leader drive many log positions without repeating the prepare phase every time.

In steady state, the shape becomes:

A leader receives a command.
It proposes the command for the next log position.
A quorum of acceptors accepts it.
The value becomes chosen and can be learned or applied.

The important insight is not “Paxos is complicated.” It is that the protocol's safety core is more general than the leader-based system we usually run. Stable leadership is an optimization for progress and efficiency. When leadership becomes unstable, the system falls back into the harder work of establishing a newer ballot and discovering what may already have been chosen.

Where this mental model helps

Multi-Paxos is a good lens for highly optimized or custom replicated services. Implementations can pipeline, batch, use flexible quorum arrangements, and make other choices around the safety core. That flexibility is powerful, but it increases the amount of protocol detail an implementation team must get right.

When a vendor says its system is “Paxos-based,” the label is only the beginning. Ask what kind of Paxos, how leadership works, how log gaps are repaired, how membership changes are handled, whether reads use quorum or lease mechanisms, and what operators can observe during recovery.

What to remember: A general safety foundation can support many optimized implementations, so two “Paxos-based” products may behave very differently in production.

Viewstamped Replication (VR)

Viewstamped Replication starts from a primary-backup picture. At any moment, the replicas operate in a numbered view, and one replica is the primary for that view.

During normal operation:

The client sends a request to the primary.
The primary gives it the next operation number and sends a prepare message to backups.
Once enough replicas acknowledge it, the operation commits.
Replicas execute committed operations in order, and the primary replies to the client.

The interesting part begins when the primary appears to fail. Replicas move to a higher view, exchange information about their logs, and the new primary constructs a state that preserves committed operations. In the revisited protocol, the primary for a view is selected deterministically from the view number and group membership.

VR makes an architectural point that is easy to miss: a leader change is a state-transfer protocol. Electing a name is not enough. The new primary must know which history it is allowed to continue.

Where this mental model helps

VR is especially useful for understanding primary-backup storage and replicated services. Even when a product does not literally implement VR, its vocabulary—views, operation numbers, commit numbers, normal processing, view change, recovery—gives you a clean way to interrogate failover.

Ask: What state is transferred before a promoted replica serves writes? Can an out-of-date replica become primary? When does a client retry become a duplicate operation? How does the system distinguish a recovering replica from a participant in the current view?

What to remember: Failover is safe only when leadership and history move together.

Zab

Zab—ZooKeeper Atomic Broadcast—was built for Apache ZooKeeper's primary-backup architecture. ZooKeeper is not merely storing independent keys. It provides a coordination namespace where the order of changes matters: create a membership node, update configuration, delete a lock contender, trigger watchers.

Zab therefore emphasizes a totally ordered stream of state changes and divides the protocol into two broad modes:

Recovery, where a leader is established and replicas synchronize on a valid history.
Broadcast, where the leader proposes transactions, followers acknowledge them, and committed transactions are delivered in order.

Transactions carry a zxid, a monotonically ordered identifier containing an epoch-related component and a counter. A prospective leader cannot simply start appending after an election. It must complete the synchronization work that gives the ensemble a safe common prefix.

That is Zab's central lesson: sometimes the product's data model tells you which consensus property deserves the spotlight. ZooKeeper's hierarchical namespace and coordination semantics make ordered broadcast—not isolated agreement on unrelated values—the natural abstraction.

Where this mental model helps

You encounter Zab when using ZooKeeper directly or operating platforms that rely on it for metadata and coordination. The practical questions are not “Is Zab better than Raft?” but “Do ZooKeeper's semantics fit this job?”

ZooKeeper is well suited to small coordination data, configuration, membership, and synchronization primitives. It is not a general replacement for a high-volume application database. You should also examine read semantics separately from write ordering: a protocol can totally order updates while a product still offers local reads with freshness tradeoffs.

What to remember: Choose a system whose ordering contract matches the meaning of your data, not one whose algorithm name sounds strongest.

Raft

Raft was designed around understandability. It decomposes replicated-log consensus into leader election, log replication, and safety, then makes the leader-follower relationship deliberately asymmetric.

In normal operation:

A client sends a command to the leader.
The leader appends it to its log and sends AppendEntries RPCs to followers.
After the entry is safely replicated according to Raft's commit rule, the leader applies it and replies.
Followers learn the commit position and apply the same entries in order.

Terms act as logical eras of leadership. Each log entry records the term in which it was created. During an election, a voter rejects a candidate whose log is less up to date than its own. Once elected, the leader uses the previous log index and term in AppendEntries to detect divergence, then repairs followers by replacing conflicting uncommitted suffixes.

The nuance matters: a new leader does not win because it has every byte any replica has ever seen. It wins under rules designed to ensure that committed entries cannot be lost. Uncommitted entries may disappear, which is why a client timeout does not always tell you whether an operation committed.

Raft's greatest contribution may be sociotechnical: a protocol that engineers can explain, review, test, and debug is less likely to be implemented incorrectly. Understandability is not cosmetic when the code decides which data survives a failure.

Where this mental model helps

Raft appears under widely used infrastructure such as etcd and Consul; CockroachDB uses many Raft groups to replicate ranges of data. Knowing the shared foundation helps, but it does not make these products interchangeable. Their data models, transaction layers, read paths, placement, snapshots, reconfiguration, and operational tooling differ enormously.

What to remember: An understandable consensus core reduces one category of risk, but the surrounding distributed system still determines the user experience.

The four protocols in one operational picture

Lens	Multi-Paxos	Viewstamped Replication	Zab	Raft
Leadership era	Ballot	View	Epoch	Term
Normal-path coordinator	Stable proposer/leader in practical deployments	Primary	Leader	Leader
Replicated object	Sequence of consensus instances, commonly used as a log	Ordered operation log	Totally ordered transaction stream	Ordered log
Recovery emphasis	Discover and preserve already chosen values	Build a safe new view from replica state	Synchronize a safe history before broadcast	Elect an up-to-date candidate; repair follower suffixes
Design personality	General and optimization-friendly	Primary-backup made explicit	Coordination-workload-specific	Structured for understandability

This table is a map, not a benchmark. Performance depends on implementation, batching, storage, network topology, durability settings, read mode, workload, and cluster health. “Raft versus Paxos” is rarely a useful procurement question by itself.

How to apply this knowledge without implementing anything

Imagine you are choosing a distributed database for an order service. Product A advertises Raft. Product B advertises Paxos. The tempting conclusion is that the protocol name settles the decision. It does not.

Use the algorithms to generate better questions.

1. What is the unit of consensus?

Is there one log for the whole cluster, one group per shard, one group per data range, or a metadata consensus group plus separate data replication? This determines where contention appears and how failure domains interact.

2. Which operations actually pass through consensus?

Writes probably do. What about reads? Are they linearizable, lease-based, quorum-based, or served locally and potentially stale? Does a transaction spanning several consensus groups require another coordination layer?

3. What happens without a quorum?

Will the system stop writes, serve stale reads, fail over elsewhere, or expose a tunable consistency mode? A safe refusal can be preferable to silent divergence, but your product must be designed for that refusal.

4. What does “acknowledged” mean?

Was the operation accepted by memory, written to durable storage, replicated to a majority, committed, or applied to the state machine? Those are different milestones. Ask what survives a leader crash immediately after the client receives success.

5. How is leadership moved?

Look for election timeouts, planned leadership transfer, fencing, catch-up requirements, and behavior under asymmetric packet loss. Failover time is often a distribution, not a single number.

6. How does a slow replica recover?

Does it replay a log, install a snapshot, fetch a checkpoint, or rebuild from another store? Can recovery saturate the same network and disks serving production traffic?

7. How does membership change?

Adding and removing voters is consensus about who participates in consensus. Safe reconfiguration is not equivalent to editing a host list. Understand the product's supported procedure.

8. Can operators see protocol state?

You want metrics for leader changes, term/view/epoch changes, commit lag, proposal latency, quorum health, snapshot transfer, and rejected requests. A correct protocol hidden behind poor observability can still create a terrible incident.

What these algorithms do not solve for you

Consensus is a foundation, not a complete distributed system.

It does not automatically give you:

Multi-key or cross-shard transactions.
Exactly-once side effects.
A correct retry strategy for ambiguous client timeouts.
Good geographic latency.
Elastic scaling.
Protection against malicious replicas.
Correct application invariants.
Painless upgrades and disaster recovery.

For example, if a payment request times out during a leader change, consensus may preserve the command perfectly while the client cannot tell whether it committed. The application still needs an idempotency key and a safe retry contract.

Likewise, consensus can order reserve item and charge card without guaranteeing that the business workflow across two external systems is atomic. That is an application-level problem.

A small real-life decision exercise

Suppose you need three capabilities:

Service discovery and a small amount of strongly consistent configuration.
A globally scaled transactional database.
A custom replicated control plane.

The protocol perspective changes how you evaluate them:

For coordination, you inspect ZooKeeper, etcd, or Consul semantics: watch behavior, session or lease models, read consistency, quorum loss, and operational maturity. Zab versus Raft is context, not the final score.
For the database, you ask how many consensus groups exist, how transactions cross them, where leaders are placed, and what geographic topology does to commit latency.
For the custom control plane, you should strongly prefer a mature library or service over implementing a paper. If custom behavior is truly necessary, Multi-Paxos shows the optimization freedom available; Raft shows the value of constrained, reviewable state transitions.

In all three cases, knowing consensus helps you see the system you are actually buying.

The deeper lesson: failure behavior is part of the API

We often evaluate infrastructure by its happy-path interface: SQL syntax, key-value operations, SDK quality, or throughput in a benchmark. Distributed systems reveal their real contract when messages are delayed, leaders restart, disks stall, and clients retry.

Multi-Paxos teaches us to distinguish a safety core from its performance optimizations.
Viewstamped Replication teaches us that a new leader must inherit a safe history, not merely a title.
Zab teaches us that ordering should match the workload's meaning.
Raft teaches us that understandability can improve implementation and operations.

You do not need to implement any of them to benefit. You need to recognize the questions they force every distributed tool to answer:

The questions worth asking: Who may speak for the cluster? Which history survives? What can progress without a majority? And how will we know what happened after the network heals?

The next time a product page says “strongly consistent” or “powered by Raft,” do not stop at the label. Ask for the failure story.

Because at 2:13 a.m., the failure story is the product.

Why we moved our Backstage platform from Yarn to pnpm

Bogdan Nechyporenko — Sat, 12 Sep 2026 19:44:23 +0000

What Git worktrees, parallel coding agents, and a cache pointing at the wrong directory taught us about the assumptions hiding inside node_modules.

I’m a developer at bol, where we have been building and running our internal developer platform on Backstage for more than five years.

After that much time, a Backstage platform becomes more than a collection of plugins. It accumulates its own ways of working: CI conventions, local startup tooling, container images, Git worktree helpers, and shortcuts that make sense inside one engineering organization.

One of our most productive choices has been using Git worktrees for parallel development with coding agents. An agent can fix a bug in one worktree while another handles a small improvement elsewhere, without branches fighting over the same working directory. When several small tasks are moving at once, this can multiply the amount of useful work we finish.

That workflow also made our package manager part of the architecture. We wanted a more forward-looking setup, and pnpm looked like a better fit: a shared content-addressable store, efficient reuse across checkouts, and a dependency layout managed by the package manager itself.

Our goal was simple: let independent worktrees move in parallel, while package content is reused safely from one shared store.

So we moved from Yarn 4 to pnpm. I expected the lockfile and commands to be the main work. They were only the entrance to the migration.

The migration looked finished until CI ran

The visible switch was straightforward: pin pnpm 12.3.0, commit pnpm-lock.yaml, replace the Yarn commands, and update the contributor guides. Then CI ran and downloaded 4,226 packages from scratch.

The next run did the same. The log read reused: 0, downloaded: 4226. The download itself took about 18 seconds, but the native node-gyp builds added several minutes on top. We had changed package managers, and the pipeline was behaving as if no cache existed.

It turned out that the cache did exist. It was looking in the wrong place.

The store had two addresses

Our container image carried a global pnpm configuration that sent the store to /builds/.pnpm-store. GitLab, meanwhile, was trying to archive .pnpm-store inside the project checkout. pnpm wrote outside that directory, the archiver found no matching files, and every job that followed started cold.

The fix was small: set the location explicitly inside CI, and print the effective value so the job log can prove it.

.gitlab-ci.yml

variables:
  npm_config_store_dir: '.pnpm-store'

before_script:
  - pnpm config set store-dir "$CI_PROJECT_DIR/.pnpm-store"
  - pnpm store path

That was the first real lesson of this migration: never reason about a package-manager cache from configuration files alone. Ask the running job where its store actually is.

We also stopped retrying script failures automatically. A dropped connection may deserve a retry. A --frozen-lockfile mismatch will fail six times for exactly the same reason, and bill you for six runners on the way.

When a working cache became the slow part

Once the store was finally cached, the next problem surfaced. The archive now held the pnpm store and every node_modules tree: 1.42 GB spread across roughly 601,000 files.

Four jobs pulled that archive. Saving, transferring, and extracting it became a substantial part of the pipeline. Worse, our installs still rebuilt the dependency layout and the native modules anyway, so those cached node_modules trees were not buying the time we expected.

After keeping the pnpm store and dropping node_modules from this cache, its size fell by about 72% and its file count by about 59%. Figures are approximate observations taken during the migration.

We removed node_modules, plugins/*/node_modules, and packages/*/node_modules from the cache paths. The archive fell to roughly 400 MB and 244,000 files—about six minutes saved per pipeline, by our estimate.

The useful lesson was to measure the dependency phase we actually had: restore, install, native builds, save. pnpm’s own CI documentation warns that caching its store is not guaranteed to make installation faster; the right policy depends on your runner, your network, and your workload. [1]

The fastest retry is the one you do not schedule

We switched installs to --frozen-lockfile --prefer-offline, made transfer progress visible in the logs, and chose faster cache compression. Each change removed a little uncertainty. That mattered, because the migration was no longer one problem: it was a chain in which every fix exposed the next bottleneck.

Then pnpm met our custom worktree machinery

The hardest surprise was local development. Our worktree bootstrap contained about 350 lines of custom symlink-farm logic, all of it built around the node_modules structure we had used with Yarn.

The helper mirrored third-party dependencies from the main checkout and redirected workspace packages to each worktree’s own source. It was clever, fast when its assumptions held, and deeply coupled to a filesystem layout that the package manager was free to change.

pnpm uses a virtual store and links packages into the dependency graph. [2] In our setup, the old mirroring code met pnpm links where it expected ordinary directories, and we began seeing ENOTDIR failures. The optimization that made worktrees convenient had become the thing preventing worktrees from working.

The best fix deleted the clever part

Instead of teaching our symlink farm every detail of pnpm’s layout, I deleted it. The central helper went from 370 lines to 31, and the whole bootstrap became one command:

pnpm install --frozen-lockfile

Each checkout gets an installation created by pnpm, while package content can still be reused through the shared store.

This was the point where the migration began to feel successful. We had not recreated the previous mechanism under a new name. We had returned ownership of dependency layout to the package manager.

There was a tradeoff. Our replacement verifier is much lighter: it checks for node_modules/.pnpm instead of walking dangling links and proving that every workspace package resolves correctly. Less maintenance code is valuable, but I still want an integration check that edits a plugin in a secondary worktree and confirms the running application uses that source.

Existing laptops remembered the old world

A clean checkout worked. Some existing developer checkouts did not.

They still contained the project-local .pnpm-store created by our earlier configuration. Once local development moved to the global store, pnpm detected the mismatch and relinked roughly 4,800 packages on repeated starts—about four minutes, for a change that should not have required a dependency install at all.

We added a one-time migration to startup: detect the obsolete local store, remove it together with the root node_modules and the dependency hash, then run one clean install. Later starts with an unchanged lockfile take the existing hash-skip path and return almost immediately.

That distinction matters. pnpm did not install 4,800 packages in zero seconds. Our startup code learned when no install was needed.

A migration must include yesterday’s state

This changed how I think about developer-tool migrations. Testing a fresh clone is necessary, but your colleagues do not all have fresh clones. They have old caches, generated files, global configuration, half-finished branches, and worktrees created before the migration.

The cleanup also had to be environment-specific. CI intentionally kept a project-local store so GitLab could archive it. Local development treated that same directory as stale. The name of a folder is not enough context to decide whether it should be deleted.

Slow networks exposed another edge

On slower office and VPN connections, pnpm’s default request concurrency could lead to timeouts. We time a request for a small artifact in our registry; when a successful request takes more than three seconds, bootstrap adds --network-concurrency=1, five fetch retries, and a longer maximum retry timeout.

The idea helped, but the implementation taught us something as well: a failed probe never reaches the slow-success branch. A production version of this pattern should decide explicitly what a timeout, an authentication failure, and a slow success each mean.

The journey improved more than installation

Once we were looking closely at the pipeline, we found work that was only loosely related to pnpm but still affected the experience of the migration.

Jest coverage was one example. Its cache had grown to 5.2 GB. We switched to the V8 coverage provider and enabled inline source maps for @swc/jest, which brought the cache back to no more than 2 GB. [3] A later change removed Jest-cache transfer entirely and revisited test selection, worker limits, and when coverage runs.

These are reported outcomes from the wider CI work. The unit-test change includes test selection, coverage policy, worker, and cache changes, so it is not a pnpm-only benchmark.

Unit-test wall time moved from 17 minutes to roughly 10. That result is useful, but I would not present it as “pnpm made our tests 41% faster.” The amount and the kind of test work changed too.

We enabled incremental TypeScript compilation and baked the pinned pnpm version into the runtime image. We also had to make our native-build policy explicit: one image gained pkg-config and the libsecret development headers, while our pnpm workspace policy later disabled keytar’s build.

Installing native prerequisites and disabling a build are different choices. If the application needs the native module at runtime, a green install is not sufficient evidence that the decision is safe.

The results I would confidently share

The dependency cache shrank from 1.42 GB to about 400 MB, and from roughly 601,000 files to 244,000.
A worktree bootstrap helper shrank from 370 lines to 31, by replacing custom filesystem logic with pnpm install.
Warm installs in the reported CI run reused 4,226 packages and downloaded none, once the store path was corrected.
Unchanged local startup can skip dependency installation entirely, after the stale state is cleaned once.
Jest cache size and unit-test duration both came down—though several of those causes sit outside the package-manager switch.

What I would tell the next team

Moving a mature Backstage monorepo to pnpm is not mainly a search-and-replace exercise. The package manager sits underneath a web of assumptions about directories, caches, startup order, native builds, and developer habits.

Start with the workflow you want to enable. For us, that was dependable parallel development with Git worktrees and coding agents.
Map every place that knows about dependencies: CI jobs, images, bootstrap scripts, worktree tools, docs, generators, and assistant instructions.
Print the effective store path inside the real job image. Run the same lockfile twice and capture restore, install, native-build, and save time separately.
Benchmark no cache, store-only caching, and your existing strategy. Measure elapsed pipeline time separately from the sum of concurrent job durations.
Exercise a clean clone and an old checkout. Test stale stores, unchanged restarts, dependency changes, slow networks, and authentication failures.
Change source code inside a secondary worktree and prove the running Backstage instance uses it. Directory existence alone is a weak health check.
Prefer deleting compatibility machinery when the new package manager can own the same responsibility.

What happened on our journey

We began with a productivity idea: parallel agents working safely in separate Git worktrees. We chose pnpm because its shared store and installation model fit that direction. Then the migration forced us to confront every place where our platform still depended on the old world.

We fixed a cache that pointed to the wrong directory. We removed hundreds of thousands of unnecessary files from that cache. We deleted a custom symlink farm. We cleaned stale state from existing laptops, adapted to slow connections, and made native build policy explicit.

The biggest gain was not one benchmark. Our setup became easier to explain: pnpm manages dependencies, CI caches a deliberate store, and each worktree is an independent place for an engineer or a coding agent to work. That clarity is what lets the productivity benefit survive after the migration project is over.

Appendix: the code changes, end to end

Everything above is the story. This is the diff. Below are the concrete changes we made, in the order that mattered, so you can lift them into your own repository without repeating the week we spent finding them. Versions and paths are ours—adjust them to your setup.

Do step 5 first. If the store lands in the wrong directory, everything else here is wasted: the cache misses on every run, and every number you measure afterward is the wrong number.

1. `package.json`—declare the package manager

Pin the version once, so CI, laptops, and containers cannot disagree about it.

{
  "packageManager": "pnpm@12.3.0"
}

2. `.npmrc`—use the global store, prefer offline

-store-dir=.pnpm-store
+prefer-offline=true
+side-effects-cache=true

Removing store-dir is the line that matters. A project-local store conflicts with the global one, and it is exactly the leftover that step 11 has to clean up later.

3. `tsconfig.json`—enable incremental compilation

   "compilerOptions": {
+    "incremental": true,
+    "tsBuildInfoFile": "build/.tsbuildinfo"
   }

Add build/.tsbuildinfo to .gitignore.

4. Jest—switch the coverage provider to V8

jest.config.js (or the Jest block in package.json):

-"sourceMaps": false
+"sourceMaps": "inline"

Inline source maps are what V8 needs to map coverage back to your sources. Then change the CI test command:

-pnpm test:all --coverage
+pnpm test:all --coverage --coverageProvider=v8

Why: the Babel provider writes duplicate transform-cache entries—one instrumented, one not—which is what took our Jest cache from around 2 GB to over 5 GB. [3]

5. GitLab CI—make the store path win

The Docker image’s global pnpm configuration overrides .npmrc, so configuration alone will not win this argument. An environment variable will. Log the effective path in the same breath, so the job can prove where its store is:

variables:
  npm_config_store_dir: '.pnpm-store' # env var beats global config

before_script:
  - corepack enable pnpm
  - pnpm config set store-dir "$CI_PROJECT_DIR/.pnpm-store"
  - pnpm store path # log it, so you can verify it

6. GitLab CI—cache the store, not `node_modules`

 cache:
   paths:
     - '.pnpm-store/'
-    - 'node_modules/'
-    - 'plugins/*/node_modules/'
-    - 'packages/*/node_modules/'

pnpm re-links from the store on every install, so caching node_modules adds archive overhead with no install-time benefit. While you are in there, switch cache compression to fast—a large store is network-bound, not CPU-bound:

variables:
  CACHE_COMPRESSION_LEVEL: 'fast'
  TRANSFER_METER_FREQUENCY: '2s' # shows transfer times in the job log

7. GitLab CI—pass `--prefer-offline` in every job

-pnpm install --frozen-lockfile
+pnpm install --frozen-lockfile --prefer-offline

In every job that installs—install_deps, build, test-and-lint—not only the install job, so the pipeline still behaves when the cache is cold.

8. GitLab CI—stop retrying script failures

 retry:
   when:
     - runner_system_failure
     - stuck_or_timeout_failure
     - api_failure
-    - script_failure # six retries for one deterministic lockfile failure

9. CI image—give `keytar` what it needs, or turn its build off

 RUN apt-get install -y \
     python3 g++ build-essential \
+    pkg-config libsecret-1-dev

The alternative is adding keytar to allowBuilds: false. These are genuinely different decisions: if the application needs the native module at runtime, a green install is not evidence that skipping the build was safe.

10. Runtime image—bake Corepack in once

Dockerfile (base image):

ARG PNPM_VERSION=12.3.0
RUN corepack enable && corepack prepare pnpm@${PNPM_VERSION} --activate

Dockerfile (application image):

-RUN corepack enable && corepack prepare pnpm@12.3.0

Caveat: if your build environment cannot reach a private registry from RUN layers—Cloud Build, in our case—point this one step at the public npm registry.

11. Startup script—clean up the stale local store

If store-dir=.pnpm-store was ever in your .npmrc, every existing checkout still has a project-local store. pnpm sees the storeDir mismatch and re-links every package on every install—roughly four minutes, for us. One guard in the start script fixes it for the whole team, once:

const staleLocalStore = join(root, '.pnpm-store');
if (existsSync(staleLocalStore)) {
  rmSync(staleLocalStore, { recursive: true, force: true });
  rmSync(join(root, 'node_modules'), { recursive: true, force: true });
  if (existsSync(hashFile)) rmSync(hashFile);
}

Keep this environment-specific. CI deliberately keeps a project-local store so GitLab can archive it; only local development should treat that directory as stale. The name of a folder is not enough context to decide whether it should be deleted.

12. Install scripts—a slow-network guard

Time one small request to the registry, and back off the concurrency only when a successful request is slow:

_pnpm_extra_flags=()
_start_ms=$(date +%s%3N)
curl -fsSo /dev/null --max-time 5 \
  https://registry.npmjs.org/is-odd/-/is-odd-3.0.1.tgz 2>/dev/null && \
  _elapsed=$(( $(date +%s%3N) - _start_ms )) && \
  (( _elapsed > 3000 )) && \
  _pnpm_extra_flags+=(--network-concurrency=1 --fetch-retries=5 \
                      --fetch-retry-maxtimeout=120000) || true

pnpm install --frozen-lockfile "${_pnpm_extra_flags[@]+"${_pnpm_extra_flags[@]}"}"

Note the edge in our version: a probe that fails outright falls straight through to the normal path. Decide deliberately what a timeout, an authentication failure, and a slow success should each do.

13. Worktrees—do not install in secondary checkouts

If you have worktrees or slot-mode checkouts that share the main node_modules, do not run pnpm install in the secondary checkout. In hoisted mode, pnpm 12 hits ENOTDIR when it tries to create symlinks over the real directories the shared farm has already placed there.

What the numbers looked like

Metric	Before	After
`install_deps` on a new lockfile	~3.5 min (`reused 0`, `downloaded 4226`)	Seconds (`reused 4226`, `downloaded 0`)
CI cache size	1.42 GB / 601k files	~400 MB / 244k files
Per-pipeline cache overhead (4 jobs)	+~12 min	Baseline
Jest cache size	5.2 GB	≤ 2 GB
Unit-test wall time	17 min	~10 min
Local startup (executor JIT)	22–26 s silent gap	~2–3 s
Local install after a 1-line change	~4 min (stale store)	~0 s (hash skip)
CI Corepack setup	Every application build	Once per runtime image rebuild

These are observations from our pipeline, not a controlled benchmark. Several of them—the test numbers in particular—include changes that have nothing to do with the package manager. Measure your own before and after.

DEV Community: Bogdan Nechyporenko

Four Ways to Survive a Network Split

Who this is for

Why exactly these four?

Multi-Paxos

Where this mental model helps

Viewstamped Replication (VR)

Where this mental model helps

Zab

Where this mental model helps

Raft

Where this mental model helps

The four protocols in one operational picture

How to apply this knowledge without implementing anything

1. What is the unit of consensus?

2. Which operations actually pass through consensus?

3. What happens without a quorum?

4. What does “acknowledged” mean?

5. How is leadership moved?

6. How does a slow replica recover?

7. How does membership change?

8. Can operators see protocol state?

What these algorithms do not solve for you

A small real-life decision exercise

The deeper lesson: failure behavior is part of the API

Further reading

Why we moved our Backstage platform from Yarn to pnpm

The migration looked finished until CI ran

The store had two addresses

When a working cache became the slow part

The fastest retry is the one you do not schedule

Then pnpm met our custom worktree machinery

The best fix deleted the clever part

Existing laptops remembered the old world

A migration must include yesterday’s state

Slow networks exposed another edge

The journey improved more than installation

The results I would confidently share

What I would tell the next team

What happened on our journey

Appendix: the code changes, end to end

1. package.json—declare the package manager

2. .npmrc—use the global store, prefer offline

3. tsconfig.json—enable incremental compilation

4. Jest—switch the coverage provider to V8

5. GitLab CI—make the store path win

6. GitLab CI—cache the store, not node_modules

7. GitLab CI—pass --prefer-offline in every job

8. GitLab CI—stop retrying script failures

9. CI image—give keytar what it needs, or turn its build off

10. Runtime image—bake Corepack in once

11. Startup script—clean up the stale local store

12. Install scripts—a slow-network guard

13. Worktrees—do not install in secondary checkouts

What the numbers looked like

References

1. `package.json`—declare the package manager

2. `.npmrc`—use the global store, prefer offline

3. `tsconfig.json`—enable incremental compilation

6. GitLab CI—cache the store, not `node_modules`

7. GitLab CI—pass `--prefer-offline` in every job

9. CI image—give `keytar` what it needs, or turn its build off