
Three independent MPC nodes, each holding one key share. The backend never holds a key. It sends a 32-byte digest and gets a signature back.
A restored MPC cluster can boot up, report [READY] on all three nodes, pass every health check — and have silently lost every private key it was supposed to hold. Ours did exactly that, once, during a disaster-recovery drill. A signature check caught it, with about thirty minutes to spare. In a real emergency, on a bad night, we might not have.
That's the whole difference between a deploy guide and a production playbook, and it's why this one spends as much time restoring the cluster as standing it up.
This is the hands-on half of my previous article, *How to Store Your Users' Crypto Wallets So Nobody Can Steal Them. There I argued *why the private key should never exist as a whole. Here we build the thing: a self-hosted 2-of-3 MPC signing cluster you can run on your own infrastructure, initialize with a distributed key-generation ceremony, back up, and recover from a real disaster.
Everything below is from an actual deployment. The tools are open source, the numbers are measured, and the gotchas cost real hours. I've kept the architecture generic, a custodial Web3 backend that generates wallets and signs transactions for users, so you can map it onto your own stack.
If you've deployed distributed systems before and just want the landmines, jump to Section 5 (the startup-order trap that can force you to rebuild your message bus) and Section 8 (the silent restore bug from the opening). Everything else builds up to those two.
Table of Contents
- Architecture overview
- The stack
- Docker Compose — nine services
- Node configuration
- The initialization ceremony (DKG)
- One master wallet, infinite users
- The backup model
- The disaster-recovery drill
- Deploying across separate hosts
- Production hardening
- Scaling — splitting hosts and adding nodes
- Troubleshooting cheat sheet
- Build it so you don't need the luck
Section 1: Architecture overview
The whole point is one structural property: no single machine can produce a signature alone, and no machine ever holds the whole private key.
The shape that delivers it:
- Three MPC nodes, each on its own isolated host, each holding one key share
- A 2-of-3 threshold: any two nodes can sign, one can be down or compromised without breaking anything
- A message bus for the nodes to exchange protocol messages
- A service-discovery / KV store for peer routing and wallet metadata
- A backend that holds zero keys and only ever sends a 32-byte digest
The trust boundary is the security model. The recommended topology puts the three nodes on three separate hosts over a private encrypted network, so they fail independently. A local Docker Compose file mirrors the same wiring for development — same services, same connections — and the local layout maps straight onto the multi-host production one in Section 9.
Why not one box with three containers? Because the isolation is the security. Three containers share one kernel and one root, so host compromise = all three shares = the master key reconstructed. Co-located nodes give you the security of one machine, not true 2-of-3. There's an honest middle ground, because you won't always get three hosts on day one: run the nodes as separate users on a single memory-encrypted, attested VM, a Confidential VM (AMD SEV-SNP) or an enclave (AWS Nitro). Even host root can't read the nodes' memory there, and attestation proves the running image wasn't tampered with. It's a legitimate compensating control for a soft launch. Just be honest about what it is: it hardens the one box, it does not give you independent-failure 2-of-3. Treat multi-host as the target you graduate to.
Section 2: The stack
I didn't want to build an MPC server from scratch. The cryptography is the easy part now; the hard part is the server around it, networking, retries, encrypted storage, backups, health checks, a client SDK. That's easily a quarter of an engineering budget before you sign your first transaction.
So the stack sits on **Mpcium** (fystack), the most turnkey open-source MPC server there is, Apache-2.0, written in Go. It hands you a full t-of-n cluster out of the box:
- Mpcium — What it is: Open-source MPC server (Go) · Role here: Runs the nodes, the signing/keygen lifecycle, the SDK
-
tss-lib** (
fystack/tss-lib v2.0.3)** — What it is: Threshold-signature library (GG18/GG20) · Role here: The cryptographic core, backing Mpcium - BadgerDB — What it is: Embedded KV store, AES-256 encrypted · Role here: Each node's local key-share storage
- NATS JetStream — What it is: Message bus · Role here: Carries keygen/signing request + result events
- Consul KV — What it is: Service discovery + KV · Role here: Peer routing and wallet metadata
- PostgreSQL — What it is: Relational DB · Role here: User rows, including each user's derivation index
Out of the box you also get ECDSA (secp256k1) + EdDSA (Ed25519), Ed25519 mutual auth on every message, ECDH-encrypted peer channels, incremental encrypted backups, and a TypeScript client ([@fystack/mpcium-ts](https://www.npmjs.com/package/@fystack/mpcium-ts)) that drops into a NestJS backend.
Two things it deliberately doesn't have, and how we deal with them:
- No trusted-dealer key import. You can't import an existing private key. Section 6 makes that a non-issue by deriving all user wallets from one generated master instead of importing anything.
- No REST/gRPC API, it's NATS pub/sub + SDK only. Fine for a backend already on a message bus; wrap it if you need HTTP.
One thing worth doing before you trust any of this. Mpcium runs GG18/GG20, the family with the *BitForge* vulnerability class from the last article. It's acceptable here for two reasons. BitForge needs a malicious signing party, which is irrelevant when you run all three nodes. And more importantly, I checked the patch is actually in the code path. The tss-lib v2 line carries the two hardening proofs, Paillier-Blum in crypto/modproof/ and no-small-factor in crypto/facproof/, generated in GG18 keygen round 2 and verified in round 3. Mpcium pins the fystack/tss-lib v2.0.3 fork through a go.mod replace, so I grepped its whole tree for the flags that would disable those proofs (SetNoProofMod, SetNoProofFac) and found them called nowhere in the keygen path, only in the library's own tests. That's the level of due diligence a signing stack earns: don't trust the protocol name, read the code path.
(Starting fresh and want the newest protocol? [cggmp24](https://github.com/LFDT-Lockness/cggmp21) (the cggmp21 crate, which now implements CGGMP24 — patched, v0.7.0-alpha.2+) and Coinbase's Cure53-audited [cb-mpc](https://github.com/coinbase/cb-mpc) are the forward-looking cores. But they're libraries, not servers, so you're back to building the server yourself.)
Section 3: Docker Compose — nine services
The full stack is nine services, eight long-running plus a one-shot bootstrap. Here's how they fit, and the two traps that will bite you.
-
**app** — Image / build: your backend (NestJS) · Role: Custodial backend + MPC client. Starts last. -
**postgres** — Image / build:postgres:15-alpine· Role: User rows, including each user's derivation index -
**redis** — Image / build:redis:7-alpine· Role: Cache (not part of MPC state) -
**nats** — Image / build:nats:2.10-alpine(-js) · Role: Message bus (JetStream enabled) -
**consul** — Image / build:hashicorp/consul:1.15· Role: Service discovery + routing KV -
**mpcium-bootstrap** — Image / build:hashicorp/consul:1.15· Role: One-shot: seeds peer IDs into Consul, then exits -
**mpcium-node0/1/2** — Image / build: your Mpcium node image · Role: The three MPC nodes

depends_on enforces the order: Consul healthy → the one-shot bootstrap completes → nodes start → the backend connects last.
The bootstrap pattern
The nodes need to know about each other before they start. Rather than bake that in, a one-shot mpcium-bootstrap container writes the three peer UUIDs into Consul and exits 0. Every node then depends on it with service_completed_successfully:
services:
mpcium-bootstrap: # one-shot: seed peer UUIDs into Consul, then exit 0
image: hashicorp/consul:1.15
environment: [CONSUL_HTTP_ADDR=consul:8500]
command:
- sh
- -c
- |
consul kv put mpc_peers/node0 <node0-uuid> &&
consul kv put mpc_peers/node1 <node1-uuid> &&
consul kv put mpc_peers/node2 <node2-uuid>
depends_on: { consul: { condition: service_healthy } }
restart: "no"
mpcium-node0:
build: ./mpc/node # distroless image, runs as UID 65532
command: ["start", "-n", "node0"]
working_dir: /cluster/node0 # mpcium reads identity/ + peers.json CWD-relative
volumes:
- ./mpc/node0:/cluster/node0 # host dir: chown 65532:65532
- ./mpc/config/node0.yaml:/cluster/node0/config.yaml:ro
depends_on:
mpcium-bootstrap: { condition: service_completed_successfully }
nats: { condition: service_started }
consul: { condition: service_healthy }
restart: unless-stopped
# node1, node2 are identical. The `app` service depends on all three and starts LAST.
A few load-bearing lines in there:
-
restart: "no"onmpcium-bootstrap— it's a one-shot: seed the peer IDs, exit0, done. Restart it and it re-seeds on every boot. -
condition: service_completed_successfully— the nodes block until that clean exit, so peers always exist before a node reads them. - the
:roconfig mount — each node'sconfig.yamlis read-only; nothing at runtime should mutate it. -
working_dirand the volume's owner are both load-bearing enough to be their own traps, below.
(As of v0.3.5 there's a lighter option: pass --peers=peers.json to each node and it syncs any new peers into Consul on startup, skipping ones already there — so you can drop the mpcium-bootstrap container entirely, or run mpcium-cli register-peers --peers peers.json once. The manual consul kv put mpc_peers/… still works; it's just no longer required.)
Trap #1 — the distroless UID
The Mpcium node image is [FROM gcr.io/distroless/base-debian12](https://github.com/GoogleContainerTools/distroless) with USER nonroot:nonroot, which is UID/GID 65532. Your host-mounted node dirs are owned by your user, so the moment a node tries to write its BadgerDB it hits permission-denied and crash-loops. One line fixes it, and you'll forget it exactly once:
chown -R 65532:65532 ./mpc/node0 ./mpc/node1 ./mpc/node2
Trap #2 — working_dir is load-bearing
Mpcium resolves identity/ and peers.json relative to the current working directory, not to a config path. Miss working_dir (or WorkingDirectory in a systemd unit) and the node starts in / and can't find its identity files. Pin it.
Section 4: Node configuration
Each node gets its own config.yaml. Most of it is boring. Two fields will bite you.
nats: { url: "nats://nats:4222" }
consul: { address: "consul:8500" }
mpc_threshold: 1 # tss-lib degree t -> t+1 = 2 signers (this is 2-of-3)
environment: "production"
badger_password: "<32 hex chars = 32-byte AES-256 key, DIFFERENT per node>"
event_initiator_algorithm: "ed25519"
event_initiator_pubkey: "<64 hex, SAME on all nodes + backend>"
chain_code: "<64 hex, SAME on all nodes + backend>"
db_path: "./data" # mpcium appends /nodeN — remember this in Section 8
backup_enabled: true
backup_period_seconds: 300
max_concurrent_keygen: 2
max_concurrent_signing: 10
What the important fields do:
-
mpc_threshold— the availability/security knob;1means 2-of-3 (see the gotcha below). -
db_path— where the shards live; Mpcium appends/nodeNto it, which will matter enormously at restore time (Section 8). -
backup_enabled/backup_period_seconds— Mpcium's own incremental encrypted backup, every 300 s. -
max_concurrent_keygen/max_concurrent_signing— per-node concurrency caps. - the three identity fields (
badger_password,event_initiator_pubkey,chain_code) have a shared-vs-per-node rule of their own, right below.
The threshold gotcha (read this twice)
In tss-lib,
mpc_thresholdis the polynomial degreet, and the number of signers required ist + 1. So a 2-of-3 cluster usesmpc_threshold: 1, not2.
And here's the part that makes it dangerous: the Mpcium dev config.yaml.template ships with mpc_threshold: 2 by default (the config.prod.yaml.template correctly ships 1 — but the dev one is what most people copy first). Copy it unthinkingly and you've built a 3-of-3 cluster, which throws away the entire availability benefit of MPC, because now every node must be online to sign. Set it to 1.
One more thing: tss-lib only enforces that the degree is at least 1 and no larger than the node count minus one. For three nodes that leaves 2-of-3 (mpc_threshold: 1) or 3-of-3 (mpc_threshold: 2) — a 1-of-3 where a single node could sign alone is degenerate and rejected. (GG18 tolerates a dishonest majority, so the code allows any 1 ≤ t ≤ n−1; Mpcium's README still recommends an honest-majority t ≥ ⌊n/2⌋+1, and for three nodes there's no sensible quorum below 2-of-3 anyway.)
Shared vs. per-node
This catches people, so let me be blunt:
-
Different on every node:
badger_password(each node encrypts its own BadgerDB with its own key). -
Identical on every node and the backend:
chain_code,event_initiator_pubkey.
event_initiator_pubkey pins who's allowed to ask the cluster to sign, the Ed25519 public key of your backend's initiator identity. chain_code is what makes deterministic derivation work (Section 6).
Identities and peers
Each node dir has an identity/ folder with its own private key (node0_private.key, mode 0600) plus every node's public identity (nodeX_identity.json, mode 0644). And every node carries the same peers.json, whose UUIDs must exactly match what mpcium-bootstrap seeded into Consul:
{
"node0": "00000000-0000-0000-0000-000000000000",
"node1": "11111111-1111-1111-1111-111111111111",
"node2": "22222222-2222-2222-2222-222222222222"
}
Section 5: The initialization ceremony (DKG)
This is the ceremony that generates the master key without ever assembling it. It runs once, on a fresh cluster, with the backend stopped.
The one rule that governs everything: startup order
The single most important operational rule in this whole deployment:
The MPC nodes must create the JetStream streams before anything else connects to NATS.
Here's why. The Go nodes create their mpc-keygen and mpc-signing streams with JetStream's default (limits) retention, but the TypeScript client (@fystack/mpcium-ts) creates those same streams with workqueue retention (and the mpc result stream the other way round — interest on the client, workqueue on the nodes). If the client wins the race and creates either one first, the nodes come up, refuse to reconcile the retention, and immediately crash-loop with:
err_code=10052 ... can not change retention policy to/from workqueue
And JetStream won't let you add or remove workqueue retention on an existing stream, so no config change or API call fixes it in place. The simplest remedy is to recreate the NATS container (or drop just the two offending streams and let the nodes recreate them), either way rebuilding JetStream state you never planned to touch. So the sequence is non-negotiable:

Get this order wrong and JetStream's fixed workqueue retention forces you to recreate NATS from scratch.
# 1. Bring up infra + the three nodes (NOT the app yet)
docker compose up -d postgres redis nats consul \
mpcium-bootstrap mpcium-node0 mpcium-node1 mpcium-node2
# 2. Wait until each node logs that it owns its streams, e.g.:
# "SigningConsumer: Subscribed to signing events"
# "Subscription created successfully"
docker compose logs -f mpcium-node0 # watch for the stream-subscription logs above
# 3. ONLY NOW run the init ceremony / start the app
There's a sneakier version of this trap too: a running backend steals the DKG result. If the app is up during init, its consumer grabs the keygen-result event off the queue before the init script can read it, and init hangs forever on "waiting for keygen result." Stop the app before you initialize.
What the init script actually does
The init is a one-time bootstrap script. Don't underestimate it, ours is 1,160 lines, and that's normal. A one-time bootstrap for a distributed system is never a ten-liner. Four phases:
-
Validate — the compose file, every node config, the initiator key, the backend
.env, and that all three nodes plus NATS and Consul are actually running. -
DKG the master wallet — connect to NATS, create the Mpcium client, call
createWallet(masterWalletId), wait for the keygen-result event (60-second timeout). Take the ECDSA public key, compressed to secp256k1. -
Write env + back up — persist
MPC_MASTER_WALLET_ID,MPC_MASTER_PUBLIC_KEY, andMPC_CHAIN_CODEto the backend's.env; then copy each node's data dir, export the Consul KV, write ametadata.json(nodeCount: 3,threshold: "2-of-3"). Move this backup offline immediately, it's hot-key-equivalent material. -
Verify — derive addresses at indexes 0 and 1, sign a test 32-byte digest at each (30-second timeout, up to four attempts on transient errors), print
=== INITIALIZATION COMPLETE ===.
Timeouts and the slow part of boot
Keygen needs all three nodes at once, so give it the most generous timeout (we use 60 s for keygen, 30 s for signing) — a transient hiccup shouldn't abort a ceremony. The one real cost at boot is Paillier preparameter generation, the bulk of a cold start. Given enough cores the three nodes generate their preparams in parallel; on a smaller host they contend for CPU, so start them serially to avoid starving each other. Either way you pay it once: the preparams are persisted to BadgerDB and reused on every restart, so restarts are cheap.
Section 6: One master wallet, infinite users
Give every user their own MPC wallet the obvious way, and every signup runs its own DKG across all three nodes — and your backups grow without bound: ten thousand users is ten thousand ceremonies and ten thousand more sets of shards you can never lose.
We don't do that. We run DKG exactly once, for a single master wallet, and derive every user wallet from it with BIP-32 child key derivation (CKD):
- Each user gets an integer
wallet_index, stored as a column in Postgres - Their address is derived at path
m/44/60/0/0/{wallet_index}(all levels non-hardened — hardened derivation needs the private key, which never exists here) — pure math, zero MPC nodes required online - To sign, each node applies its own derivation delta to its master share; the child key, like the master, is never assembled
This is what makes the operational model sane, and it's the reason the backup story in the next section is so simple. One derivation path supports about 2.1 billion users (indexes 0 through 2³¹ − 1), so you'll run out of users long before you run out of key space. I covered CKD's security equivalence in the previous article; the short version is that it's identical to per-user DKG under every node-compromise scenario.
Section 7: The backup model — back up three things, not one
The most common way to lose an MPC cluster is to back up the key shares, feel safe, and find out mid-restore that shares alone are useless. You have to back up three things:

Restore only BadgerDB and the cluster boots but can't find any wallet — signing fails with ERROR_KEY_NOT_FOUND.
-
1 — Item: BadgerDB shards · Where: each node's
data/dir · Why it's mandatory: the encrypted key-share state -
2 — Item: Consul KV (
threshold_keyinfo/*,mpc_peers/*) · Where: Consul export · Why it's mandatory: wallet routing; without it every sign failsERROR_KEY_NOT_FOUND -
3 — Item: Metadata (
chain_code, master wallet ID, master pubkey, node configs, identity keys) · Where: configs + backend.env· Why it's mandatory: needed to derive addresses and boot the nodes
One backup, forever
Here's the quiet superpower of CKD. The master shares, the threshold_keyinfo/* routing entries, and the peer metadata never change after the initial DKG. New users are just new rows in Postgres. So your MPC backup strategy is: take one full MPC backup right after init, then back up Postgres normally from then on. That's it. No incremental shard sync, no window where a user created between backups is unrecoverable. Per-user DKG would need a fresh MPC backup after every signup.
And this is the whole disaster-recovery story in one sentence: lose the entire cluster and you still lose nothing. Stand up fresh nodes, restore that single encrypted master backup, and every user wallet comes back, because no user wallet was ever stored on its own. Each one is a deterministic derivation of the master, and its wallet_index lives safely in your Postgres rows. The master backup plus your normal Postgres backups reconstruct every address and every signing key by index, on brand-new hardware, with nothing lost.
The copy-size trap
How you copy BadgerDB matters more than you'd think. A running node's BadgerDB is ~2.2 GB — SSTables, value logs, and the WAL, mostly churn. What you actually ship is Mpcium's own cold backup: with backup_enabled: true, each node writes a pruned, encrypted export to backup_dir (the backup-*.enc files), a few tens of KB apiece. In the drill the full age-wrapped bundle — all three nodes' exports plus every config, identity, and the Consul KV dump — came to 164 KB, four orders of magnitude below the live directory. Two consequences: never copy a live BadgerDB (and never stage a hot copy in a small tmpfs — it can exceed 6 GB and blow out a ~2 GB /tmp); ship the backup_dir export instead.
Restore is a specific sequence
# 1. Start fresh nodes, then STOP them
docker compose up -d && docker compose stop mpcium-node0 mpcium-node1 mpcium-node2
# 2. For each node, decrypt + restore its export INTO the exact db_path/nodeN dir:
# mpcium-cli recover --backup-dir <node's backup-*.enc> \
# --recovery-path <db_path>/nodeN --force # the nodeN subdir is load-bearing — Section 8
# (recover prompts for the badger_password; drive it with `expect` in automation)
# 3. Import the Consul KV dump (skip this and signing fails ERROR_KEY_NOT_FOUND)
consul kv import @backup/consul-kv.json
# 4. Fix ownership, delete stale BadgerDB LOCK files
chown -R 65532:65532 ./mpc && find ./mpc -name LOCK -delete
# 5. Start the nodes, wait for all three to log [READY]
docker compose start mpcium-node0 mpcium-node1 mpcium-node2
# 6. Verify with a signing smoke test — do NOT trust "[READY]" alone (Section 8)
Section 8: The disaster-recovery drill
You don't have a backup until you've restored it. So before go-live we ran the full thing: destroy the cluster, rebuild it from backups on fresh hardware, and prove the recovered cluster signs with the same master key. The pass criterion was strict, recover a signature and confirm via ecrecover that its master public key was byte-for-byte identical to the original.
It passed. It also surfaced eleven runbook and config bugs in a single session. A clean re-run took 45–60 minutes; the first pass took about two hours, precisely because of what it caught. Three of them are worth the price of this whole article.
The silent empty-database restore (the one from the opening)
This is the bug that would have quietly destroyed every user wallet in a real emergency.
Mpcium reads BadgerDB from ${db_path}/nodeN/, it auto-appends the nodeN subdirectory. Drop your restore into .../data instead of .../data/nodeN and the files land one level too high. Mpcium then looks in .../data/nodeN/, finds it empty, and, this is the dangerous part, silently creates a brand-new, empty database.
The cluster boots. All three nodes report [READY]. Health checks are green. But the shares are gone — every existing wallet is orphaned, and the first real signing request fails with ERROR_KEY_NOT_FOUND (newer v0.3.5 builds report the same thing as ERROR_NOT_MAJORITY, "not enough peers to process signing request"). Worse, the empty cluster will happily run a fresh DKG and mint a brand-new master key that has nothing to do with your users' funds. You find out on that first signature, long after the green health check lied to you.

"[READY]" is not "recovered." Restore into the exact db_path/nodeN subdirectory, then verify by signing, never by status alone.
The drill burned about 30 minutes on exactly this. The fix is trivial once you know it, restore into the explicit .../data/nodeN subdir and chown afterward, but you'd never guess it from a green health check. The lesson generalizes: [READY] means "a database is present," not "the right database is present." Verify recovery by signing and checking the recovered public key, never by status.
The documented command that didn't exist
The first-draft runbook confidently referenced an mpcium-cli init-dkg command that doesn't exist in the shipped binary, with a flag set to match. There is no keygen subcommand at all: DKG is triggered by publishing a signed keygen-request message onto NATS (subject mpc.keygen_request.*). Recovery really uses --backup-dir, --recovery-path, and --force, and the binary prompts for its password interactively via term.ReadPassword, so you drive it with expect in automation. Dry-run every command in your runbook against the actual binary before you trust it in an emergency.
openssl rand will hand you an invalid key
BadgerDB's encryption key must be exactly 16, 24, or 32 bytes. The instinctive command produces neither:
openssl rand -base64 32 # WRONG: 44 ASCII chars -> node fatals opening the encrypted BadgerDB:
# the key must be exactly 16, 24, or 32 bytes.
openssl rand -hex 16 # RIGHT: 32 hex characters = 32 bytes = AES-256
-base64 32 turns 32 random bytes into a 44-character string, and the password is used as raw key bytes — so that's a 44-byte key, not a valid AES length. -hex 16 gives 32 characters = 32 bytes: an AES-256-length key carrying 128 bits of entropy, ample for a local DB key.
Five more the drill taught us
-
Never regenerate
badger_passwordafter first start. The KEYREGISTRY is encrypted with the original password; change it and the database is permanently unreadable. -
A wrong or regenerated
chain_codesilently orphans every wallet. The cluster boots and signs perfectly happily, but it now derives a completely different address space, so none of your users' existing addresses come back. Like the empty-DB bug, it sails past[READY]and only a sign-and-compare catches it. Treatchain_codeas recovery-critical metadata, not a rotatable secret. -
Regenerating node identity keys or peer UUIDs is a re-key, not a recovery. Fresh identities make the restored shares reject each other, forcing a brand-new DKG, which produces a new master public key and abandons every address derived from the old one. Same limit if a node is permanently lost: you can't just drop in a replacement, because a reshare needs the surviving shares as input. Mpcium ships that as
Resharing()(Section 11) — it rebuilds the replacement from the survivors while keeping the master key. A recovered 2-of-3 cluster missing its third node has zero remaining fault tolerance until you re-share, so plan node replacement rather than improvising it. -
Consul KV doesn't contain the master public key. It stores only
{participant_peer_ids, threshold, version}. So you can't verify a restore by reading KV, you sign andecrecover. -
Your cold bundle must include the supporting config (
nats.conf, CA templates), or the restored NATS/services crash-loop and you never reach the signing test.
Section 9: Deploying across separate hosts
Everything so far ran from one Docker Compose file — one host, one kernel, one root. That's the right way to learn the stack, and fine while you're testing, but it isn't 2-of-3: own the box and you own all three shares. Production means each node on its own machine, so a single host compromise costs the attacker exactly one share. None of the cryptography changes — same three nodes, same DKG, same backups. What changes is the wiring: instead of a Compose network, the nodes live on separate hosts joined by a private encrypted overlay, and they reach the bus over that overlay instead of a Docker bridge.
The shape
-
Three node hosts — one Mpcium node each. Ideally memory-encrypted, attested VMs (AMD SEV-SNP, AWS Nitro), each with no public IP and a scoped machine identity. Run the node as its own locked-down user with a
0700data dir, so a compromised node process still can't read another node's BadgerDB. - One control host — NATS + Consul (and a small internal CA, below). The bus and the KV store live here; every node and the backend point at it.
- The backend — sends digests, holds zero keys, also on the overlay.
The nodes never talk to each other directly — all protocol traffic goes through NATS. So the only reachability requirement is: every node and the backend can reach the control host's NATS and Consul over the private network. That keeps the network model a simple star, not a full mesh.
The private overlay (WireGuard)
Put every host on one WireGuard network so nothing MPC-related ever touches a public interface. On each node host, one peer pointing at the control host:
# /etc/wireguard/wg0.conf (a node host)
[Interface]
Address = 10.10.0.11/24
PrivateKey = <this node's private key>
ListenPort = 51820
MTU = 1380 # provider MTU minus WireGuard's 80-byte header
[Peer] # control host (NATS + Consul)
PublicKey = <control host public key>
AllowedIPs = 10.10.0.1/32 # just the control host — nodes reach each other via NATS, not directly
Endpoint = <control host public ip>:51820
PersistentKeepalive = 25
The control host runs the mirror image — one [Peer] block per node, each with that node's /32 — and forwards nothing else. Only public keys cross the wire; private keys never leave their host. Then lock the provider firewall so UDP/51820 is the only inbound port and only from the overlay's peers, deny the rest, and bind SSH to the WG address instead of the public one. On a cloud MPC VM that means launching it with no external IP at all and reaching it purely over WG.
Per-node config
Each node's config.yaml is identical to the single-host one, except the addresses now point at the control host over WG and the shared secrets stay shared:
nats: { url: "nats://10.10.0.1:4222" } # control host, over WG
consul: { address: "10.10.0.1:8500" }
mpc_threshold: 1 # still 2-of-3
badger_password: "<per-node, 32 hex>" # distinct per node
chain_code: "<same on all nodes + backend>"
event_initiator_pubkey: "<same on all nodes + backend>"
db_path: "/var/lib/mpcium/data"
Lock the bus itself down with mTLS: issue a client certificate per node from a small internal CA (step-ca is plenty), so only your nodes and backend can speak to NATS — a stray connection that lands on the overlay still can't.
Bringing it up
The startup order from Section 5 still holds, just spread across hosts: bring up NATS + Consul on the control host, register-peers (or start the nodes with --peers) so the UUIDs land in Consul, start the three nodes on their own hosts (streams first — the retention trap doesn't care that the nodes are remote now), wait for [READY] on all three, then run the DKG exactly as before. The backend connects last.
This is the topology every backup, restore, and reshare in this article already assumes — the wiring is all that stood between you and it.
Section 10: Production hardening
Once the cluster works, the remaining work is isolation and discipline.
- Split the nodes across hosts — the from-scratch recipe is Section 9, migrating an existing single-box cluster is Section 11. Three separate machines mean a single host compromise yields at most one share; anything less is theater. Can't split yet? Fall back to a memory-encrypted attested VM (AMD SEV-SNP, AWS Nitro) — the Section 1 note covers exactly what that buys you and what it doesn't. One caveat that lands here: SEV-SNP VMs generally can't live-migrate, so an abrupt host termination becomes your disaster-recovery trigger — exactly why the drill in Section 8 isn't optional.
- Private encrypted network. Put inter-node and node-to-backend traffic on a private mesh (WireGuard). The message bus and KV store should never be exposed publicly.
- Egress firewall. MPC nodes have no business making arbitrary outbound connections. Lock egress to what the protocol needs.
-
Encrypt every backup with
[age](https://github.com/FiloSottile/age) before it leaves the host, and store it on physically separate, offline media. A stolen backup is all shares plus metadata, equivalent to stealing the funds. - Plan for the memory profile. Each node is lightweight at rest, on the order of tens of MB idle, but spikes during Paillier preparameter generation, so provision for the spike, not the idle state, and remember the serial-start caveat on smaller hosts. For reference, the entire dev stack (backend, three nodes, Postgres, Consul, NATS, Redis) fits comfortably in 3.8 GiB.
- Make init lazy, not fatal. The backend shouldn't crash if MPC is briefly unavailable at boot; log a warning and retry on first real use. And handle the result-before-request race: register the pending request before you check for an early result, because a fast cluster can return a signature before your handler is registered.
-
Retry the transient errors. Treat these as retryable with a short backoff (
[0, 500, 2000] ms):symmetrickey key not found,failed to decrypt message,encrypt tss message error,not enough peers to process signing request,timed out waiting for signing result. They're normal under load, not failures.
Section 11: Scaling the cluster — splitting hosts, and adding nodes
Two operations get conflated here, and they cost wildly different amounts. Moving your three nodes onto separate hosts keeps the same key and costs only a maintenance window. Adding a node changes the committee and needs a resharing ceremony — which Mpcium does ship, just not as a one-command move. Know which one you're doing before you touch production.
Splitting one host into three (same key, just downtime)
This is the common "started on one box, now you want real 2-of-3" move, and it is not a re-key. The identities, the shares, the chain_code, and the peer UUIDs all stay byte-for-byte the same — you're only changing where each node runs. Because none of the key material changes, the master public key and every derived address survive untouched. Mechanically it's the Section 8 restore, aimed at three hosts instead of one:
- Stand up the private network first — a WireGuard mesh across the three hosts and the backend. Nodes talk through NATS, not node-to-node, so what actually has to be reachable is one NATS and one Consul over that private network.
- Stop the cluster. Your downtime starts here, and it's a hard stop: nodes down, no signing.
- Move each node's data dir, its
identity/folder, and itsconfig.yamlonto its own host. Keep the peer UUIDs and identity keys identical — regenerating them is a re-key, not a move (Section 8), and it abandons every address. - Point each node's config at the relocated
nats.urlandconsul.address, re-import the Consul KV dump,chown -R 65532:65532, and delete staleLOCKfiles. - Start the nodes, wait for three
[READY], then verify by signing andecrecoverthat the master public key is byte-identical to before. Never trust[READY]alone — that's the whole lesson of Section 8.
Downtime is the stop-to-verified-restart window — minutes if you've rehearsed it. Rehearse it exactly like the DR drill, because mechanically it is one.
Adding a node (2-of-3 → 2-of-4, or 3-of-5): resharing
Reshaping the committee — a fourth node, a higher threshold — can't be a plain docker run: a new node has no share, and the existing shares were generated for a three-party polynomial. The primitive that fixes this is key resharing — the current shareholders jointly re-distribute the secret to a new set of parties, producing fresh shares for the new committee while keeping the same master public key, so every user address is preserved and nothing moves on-chain.
Mpcium ships resharing as a first-class, signed operation (it landed in v0.3.0 and was hardened for new-node joins in v0.3.5). You publish a ResharingMessage and wait for the result:
mpcClient.Resharing({
sessionID, walletID, keyType,
nodeIDs: newCommitteePeerIDs, // include a brand-new peer ID to add a node
newThreshold, // change t here
});
mpcClient.OnResharingResult((r) => { /* new shares are written */ });
Results arrive on mpc.mpc_reshare_result.*; there's a runnable examples/reshare/main.go and it's e2e-tested for both ECDSA and EdDSA. The redistribution stores the new shares under the same walletID with only an internal share-version bump, so the master public key — and every CKD-derived address — survives untouched. Add a node by including its peer ID in nodeIDs; remove one or raise the threshold by editing nodeIDs / newThreshold.
Two honest caveats, because this is where "read the code, not the pitch" earns its keep:
-
It's a two-part move, not one command. There's no
mpcium-cli reshare. A full membership change is: register the new peer first (generate itsidentity.json, thenmpcium-cli register-peers), then theResharing()call that actually hands it a share. Issue #84 is still open — but for a turnkey add/remove-node CLI and end-to-end docs, not because the crypto is missing. -
A reshare rotates the master shares (new version, new committee in
threshold_keyinfo), so it breaks Section 7's "one backup, forever": re-take the MPC backup after every reshare.
Still, pick your n and threshold before the master DKG when you can — a reshare is real coordinated work, and the fewer you run against a cluster holding real funds, the better.
Section 12: Troubleshooting cheat sheet
The failure modes you're most likely to hit, and the one-line fix for each:
-
err_code=10052 ... retention policycrash-loop — Cause: App/client created NATS streams before the nodes (wrong retention) · Fix: Recreate NATS; always start nodes first -
Node permission-denied on
data/— Cause: Distroless UID 65532 vs. host ownership · Fix:chown -R 65532:65532the node dirs -
Signing fails
ERROR_KEY_NOT_FOUND(Key info not found) — Cause: Consul KV (threshold_keyinfo/*) not restored · Fix:consul kv importthe KV dump -
Cluster
[READY]but signing failsERROR_KEY_NOT_FOUND(ERROR_NOT_MAJORITYon v0.3.5) — Cause: Backup restored one dir too high → empty DB, shares gone · Fix: Restore into the exact.../data/nodeNsubdir; verify by signing - DKG hangs on "waiting for keygen result" — Cause: The backend consumed the result event · Fix: Stop the app before running init
-
DKG times out — Cause: Not all three nodes were ready · Fix: Keygen needs all nodes; wait for three
[READY] -
Node fatals opening the encrypted BadgerDB — Cause:
badger_passwordis not 16/24/32 bytes · Fix: Generate withopenssl rand -hex 16 -
Nodes misbehave after re-running
docker compose up— Cause: An override recreated a stateful service (NATS/Consul), wiping its state · Fix: Restart the nodes after any infra bring-up; treat NATS/Consul as stateful
Look at the pattern. Not one of these is a cryptography failure. They're orchestration, filesystem, and config failures. That's the real lesson of running MPC in production: the math is the part that works, and the operations are where you earn your keep.
Section 13: Build it so you don't need the luck
A self-hosted 2-of-3 MPC signing cluster isn't exotic infrastructure. It's a handful of open-source components wired together with a few load-bearing rules: get the startup order right, get the threshold semantics right, back up three things instead of one, and drill your disaster recovery before you need it. Because the failure that actually hurts you isn't a dramatic one. It's a cluster that looks perfectly healthy while it has silently lost every key.
So if you take one thing from all of this, take the opening. Run the drill. A cluster that boots [READY] with every key silently gone isn't a hypothetical, it's the first thing that goes wrong the day you finally need your backup, and it stays invisible until you sign and check. We found ours with half an hour to spare and a lot of luck. Build the thing so you don't need the luck.
If this was useful, I write about backend, crypto infrastructure, and shipping distributed systems to production.
Follow me:
- LinkedIn: https://www.linkedin.com/in/anatolii-zhadan/
- Hackernoon: https://hackernoon.com/u/zhadan
- Github: https://github.com/ImmuneFOMO
Tags: MPC · Multi-Party Computation · DevOps · Docker · Self-Hosted · Threshold Signatures · Disaster Recovery · Backend · Web3
Top comments (0)