DEV Community: ImmuneFOMO

Postgres not using your index? It's counting pages, not rows

ImmuneFOMO — Fri, 13 Nov 2026 14:07:46 +0000

Slow Postgres, part 1 of 8. The SQL and raw numbers for every test are on GitHub.

A Seq Scan reads each page once, in order. An index lookup walks the B-tree and then reads one heap page per TID.

You add an index on status, run EXPLAIN, and Postgres reads the whole table anyway. ANALYZE doesn't change the plan. Someone suggests SET enable_seqscan = off, the plan finally shows your index, and the query gets slower.

Usually the planner is right. It prices a plan by the 8 KB pages it has to read and by whether it reads them in order or jumps around between them, and once you count pages the same way, you can answer most "why is it ignoring my index" tickets yourself.

This series started as my notes from an open lesson on slow Postgres queries by balun.courses. The tests and numbers are my own.

One note before the numbers: the AlphaVPS link at the end is a referral link, so I may get a commission at no extra cost to you, and the netcup vouchers come from my own account.

The short version:

Postgres prices a plan in pages, and in my test table a status with 0.9% of the rows sits on 94% of them, so its index saves 12 page reads out of 221.
On 5 million rows in RAM, the index beat a full scan up to 10% of the rows when the matches were scattered, and at every share I tried (up to 50%) when they were stored together.
Read cold from disk, the scattered lookups at 0.5% and 1% of rows ran 2 to 5 times slower than a full scan, and Postgres still picked them.
A 200-row table gets a Seq Scan even when 2 rows match, so a plan from a tiny test copy says nothing about production.
Keep enable_seqscan out of app code, read pg_stats before CREATE INDEX, and if you only query the rare values, index only those rows.

What a B-tree index stores

A regular B-tree index keeps only keys, the values it is sorted by, and next to each key a TID: the row's address in the heap, which is the table's main storage.

A TID is a page number plus a slot on that page. You can see it as the hidden ctid column:

SELECT ctid, id, status FROM orders WHERE status = 'chargeback' ORDER BY id LIMIT 3;

  ctid   |  id  |   status
---------+------+------------
 (7,48)  | 1000 | chargeback
 (14,96) | 2000 | chargeback
 (22,8)  | 3000 | chargeback

Row 1000 is slot 48 on page 7. An index lookup for 'chargeback' returns a list of addresses like these, and Postgres then visits each of those heap pages to fetch the row and check that your transaction can see it.

The second step is the one people forget. To fetch one row, Postgres has to read the whole 8 KB page it sits on.

The lab: 30,000 orders on 221 pages

The lab runs on PostgreSQL 17.11 in Docker with default settings. My numbers come from a 4-vCPU, 8 GB cloud VM, but any small VPS or a laptop will do.

CREATE TABLE orders (
  id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  created_at   timestamptz NOT NULL,
  amount_cents int NOT NULL,
  status       text NOT NULL
);

INSERT INTO orders (created_at, amount_cents, status)
SELECT now() - (30000 - i) * interval '1 minute',
       (random() * 10000)::int,
       CASE
         WHEN i % 1000 = 0 THEN 'chargeback'  -- 30 rows, 0.1%
         WHEN i % 100  = 0 THEN 'failed'      -- 270 rows, 0.9%
         WHEN i % 10   = 0 THEN 'created'     -- 2,700 rows, 9%
         ELSE 'paid'                          -- 27,000 rows, 90%
       END
FROM generate_series(1, 30000) AS i;

CREATE INDEX orders_status_idx ON orders (status);
ANALYZE orders;

SELECT relpages, reltuples FROM pg_class WHERE relname = 'orders';

 relpages | reltuples
----------+-----------
      221 |     30000

136 rows fit on a page, so 30,000 rows take 221 pages. I picked 30,000 because ANALYZE samples 300 × default_statistics_target rows, which is 30,000 at the default. On this table it reads every row, so the statistics are exact and your plans should come out the same as mine.

What a Seq Scan costs

Ask for the most common value and the index goes unused:

EXPLAIN SELECT * FROM orders WHERE status = 'paid';

Seq Scan on orders  (cost=0.00..596.00 rows=27000 width=25)
  Filter: (status = 'paid'::text)

221 pages  × seq_page_cost     1.0    = 221
30000 rows × cpu_tuple_cost    0.01   = 300
30000 rows × cpu_operator_cost 0.0025 =  75
                                        ---
                                        596

Cost has no unit you can convert to milliseconds. It only means something next to another plan's cost.

The value you filter on appears nowhere in that formula. Ask for 'chargeback' with the index paths switched off and the Seq Scan still costs 596. Only the index path's price moves with the value, so the planner is choosing between a fixed price and a sliding one.

For 'paid', every one of the 221 pages holds paid rows, so the index path would read the whole heap plus the index. Force it with enable_seqscan = off and you get a Bitmap Heap Scan at cost 872.

How many pages each status sits on

Selectivity is the share of rows that pass the filter. The usual rule of thumb goes: one row, use the index; 1%, usually; 20%, it depends; 70%, Seq Scan. Here is the same table counted by pages as well as by rows:

SELECT status,
       count(*)                               AS rows,
       count(DISTINCT (ctid::text::point)[0]) AS pages
FROM orders
GROUP BY status
ORDER BY rows DESC;

Share of rows and share of pages for each status in the 30,000-row table. failed is 0.9% of the rows but sits on 94% of the pages.

The rule of thumb says index failed, and Postgres agrees: Index Scan at cost 235 against 596. Run both with EXPLAIN (ANALYZE, BUFFERS), though, and the index path touches 209 buffers to the Seq Scan's 221, a saving of twelve page reads.

It still finished faster, 0.19 ms versus 0.93, because checking 270 rows is less CPU work than filtering 30,000. For 'chargeback', 30 rows on 30 pages, the index path needs 32 buffers instead of 221.

Two things decide how rows land on pages. Row width sets how many fit on one, and at 136 rows per page any value much more common than 1 in 136 ends up on nearly every page. Physical order is the other. I spread failed evenly on purpose. If your failed orders came from one bad deploy on one afternoon, they'd sit on a few neighbouring pages and the index would look great.

Timing it on 5 million rows in RAM

On 221 pages everything finishes in a couple of milliseconds, so for timings I built a bigger table: 5 million rows, 326 MB, about 120 rows per page.

A bucket column holds a random number from 0 to 999,999, so WHERE bucket < 10000 matches 1% of the rows. I loaded the same rows twice, once in random order and once sorted by bucket, and forced each plan with the enable_* settings at shares from 0.01% to 50%. shared_buffers was 2 GB, which kept the whole table in memory.

Median of 5 warm runs on 5 million rows, one process, JIT off. Dots mark the plan Postgres chose on its own. Grey dotted line: Seq Scan on the clustered copy.

For scattered rows Postgres picked a bitmap scan, which collects all the TIDs first and then reads each heap page once, in page order. Up to 10% of the rows it beat the Seq Scan, 98 ms to 124. At 25% it lost, 174 against 139, and the planner still chose it. A plain Index Scan, one heap visit per row, lost from 5% on.

By 10% the scattered matches sit on every single page, and the bitmap scan still won. It reads all 41,667 pages but checks only the 499,000 rows the bitmap points to, while the Seq Scan checks all 5 million. With the table in RAM a page read costs next to nothing, and the time goes into checking rows.

For clustered rows the index won at every share I tried, even 50%: 123 ms against 142.

The same queries with a cold cache

RAM is the friendly case. For the cold runs I restarted Postgres and dropped the OS page cache before every run, so every page came off the VM's disk.

Same tables and queries read cold from disk, median of 5. Dots mark the plan Postgres chose. Grey dotted line: Seq Scan on the clustered copy.

The Seq Scan got about twice as slow, 250 to 350 ms at every share. The clustered index won everywhere again, 23 ms versus 263 at 5%.

The scattered bitmap scan took 748 ms at 0.5% of rows, to the Seq Scan's 349. At 1% it took 1.27 seconds against 255 ms, five times slower, and that's the plan Postgres picks.

The bitmap scan at 2% read more heap pages than at 1%, 37,939 versus 29,143, and still finished more than six times faster, 190 ms against 1,275. From 2% to 10% it beat the Seq Scan again.

The 2% run touched 91% of the pages, so it was close to one long sequential read. The 1% run touched 70%, with holes in between. My best guess is OS readahead, which prefetches reads that look sequential and gives up when they're full of gaps. I haven't confirmed it yet.

The planner has no way to see that difference. All it has is random_page_cost = 4, a single guess at how much a jump costs next to a sequential read, and it used the same guess for both runs.

Seq Scans on a 200-row table

A 200-row copy shows the other end of the scale, where the table is too small for any index to win.

CREATE TABLE orders_small AS SELECT * FROM orders WHERE id <= 200;
CREATE INDEX ON orders_small (status);
ANALYZE orders_small;

EXPLAIN SELECT * FROM orders_small WHERE status = 'failed';

Seq Scan on orders_small  (cost=0.00..4.50 rows=2 width=25)
  Filter: (status = 'failed'::text)

Only 2 of the 200 rows match, but reading two pages and checking 200 rows costs just 2 + 2 + 0.5 = 4.50. Turn off enable_seqscan and the cheapest index plan costs 6.19, because its one index page alone is priced as a random read at random_page_cost = 4. If you check plans on a 200-row test fixture, this is the arithmetic you're looking at, and it tells you nothing about the same query on 30 million rows in production.

"We're on SSDs, set it to 1.1." Fair, and on this table that flips the plan to an Index Scan at cost 2.77. Both versions finish in 10 microseconds or less, so on a table this size it makes no difference.

A partial index instead of enable_seqscan

The one place for enable_seqscan = off is a scratch session, where it shows you what the other plan would cost. It applies to every query in the session, so in app code it also overrides the plans that were right, like 596 against 872 for 'paid'.

If the queries you care about only ask for the rare values, say a failed-payments dashboard or a chargeback queue, index only those rows:

CREATE INDEX orders_status_not_paid_idx ON orders (status)
  WHERE status <> 'paid';

It holds 3,000 entries instead of 30,000 and takes 5 pages, where the full index takes 28. Paid orders, 90% of the inserts here, never touch it.

Postgres only uses it when it can prove your filter implies the index's WHERE, and a literal status = 'chargeback' does. With only the partial index in place, a prepared status = $1 under a generic plan falls back to a Seq Scan, because the planner can't prove anything about a value it hasn't seen. Generic plans get their own part later in the series.

The planner's statistics in pg_stats

Before adding an index, look at the column in pg_stats:

SELECT null_frac, n_distinct, most_common_vals, most_common_freqs
FROM pg_stats
WHERE schemaname = 'public' AND tablename = 'orders' AND attname = 'status';

 null_frac | n_distinct |         most_common_vals         |   most_common_freqs
-----------+------------+----------------------------------+------------------------
         0 |          4 | {paid,created,failed,chargeback} | {0.9,0.09,0.009,0.001}

That's what the planner believes about status as of the last ANALYZE. If most of your queries filter on paid, at 0.9, they get a Seq Scan, and no extra index will change that.

A negative n_distinct means a fraction of the row count. On tables bigger than the 30,000-row sample these numbers are estimates; you can raise the sample with ALTER TABLE ... ALTER COLUMN ... SET STATISTICS.

The same view has a correlation column, the planner's idea of physical order. On my 5-million-row tables it was 0.01 for the scattered copy and 1 for the clustered one, and that difference in physical order is what separates the scattered and clustered lines in both charts.

Before CREATE INDEX, run the pg_stats query, relpages and the rows-versus-pages count on your own table. If the value you care about sits on most of the pages and the table doesn't stay in memory, the index won't save you reads.

Run it on your own server

All the numbers above come from one 4-vCPU AMD EPYC cloud VM with 8 GB of RAM, so yours will differ. The SQL, the scripts, the raw output and the chart code are on GitHub. Run them on your own server, and if you get a different picture or think I got something wrong, open an issue or a PR.

My own projects run on netcup (VPS from €2.21 a month) and AlphaVPS (VPS from €2.54 a month). If you need a box for this, either works.

For netcup I have a few €5 vouchers for new customers. Each one works once, so take one and leave the rest for the next person:

36nc179418264998
36nc179418264997
36nc179418264996
36nc179418264995
36nc179418264994
36nc179418264993

Paste it into the "Redeem voucher" field at the bottom of the netcup cart. If they're all gone, message me on LinkedIn and I'll send you a new one.

Good luck with your plans. Part 2 is next: how to read Postgres EXPLAIN ANALYZE.

Finish

I hope you understood the instructions correctly. If you have any edits to this article, I will be waiting for your comments. If I really helped you, I kindly ask you to subscribe to my LinkedIn and provide feedback on this article.

Link to follow me:

All my articles and links: fomo.moe

How to Deploy a Self-Hosted 2-of-3 MPC Signing Cluster (Without Losing Everyone's Keys)

ImmuneFOMO — Tue, 14 Jul 2026 10:38:28 +0000

Three independent MPC nodes, each holding one key share. The backend never holds a key. It sends a 32-byte digest and gets a signature back.

A restored MPC cluster can boot up, report [READY] on all three nodes, pass every health check — and have silently lost every private key it was supposed to hold. Ours did exactly that, once, during a disaster-recovery drill. A signature check caught it, with about thirty minutes to spare. In a real emergency, on a bad night, we might not have.

That's the whole difference between a deploy guide and a production playbook, and it's why this one spends as much time restoring the cluster as standing it up.

This is the hands-on half of my previous article, *How to Store Your Users' Crypto Wallets So Nobody Can Steal Them. There I argued *why the private key should never exist as a whole. Here we build the thing: a self-hosted 2-of-3 MPC signing cluster you can run on your own infrastructure, initialize with a distributed key-generation ceremony, back up, and recover from a real disaster.

Everything below is from an actual deployment. The tools are open source, the numbers are measured, and the gotchas cost real hours. I've kept the architecture generic, a custodial Web3 backend that generates wallets and signs transactions for users, so you can map it onto your own stack.

If you've deployed distributed systems before and just want the landmines, jump to Section 5 (the startup-order trap that can force you to rebuild your message bus) and Section 8 (the silent restore bug from the opening). Everything else builds up to those two.

Section 1: Architecture overview

The whole point is one structural property: no single machine can produce a signature alone, and no machine ever holds the whole private key.

The shape that delivers it:

Three MPC nodes, each on its own isolated host, each holding one key share
A 2-of-3 threshold: any two nodes can sign, one can be down or compromised without breaking anything
A message bus for the nodes to exchange protocol messages
A service-discovery / KV store for peer routing and wallet metadata
A backend that holds zero keys and only ever sends a 32-byte digest

The trust boundary is the security model. The recommended topology puts the three nodes on three separate hosts over a private encrypted network, so they fail independently. A local Docker Compose file mirrors the same wiring for development — same services, same connections — and the local layout maps straight onto the multi-host production one in Section 9.

Why not one box with three containers? Because the isolation is the security. Three containers share one kernel and one root, so host compromise = all three shares = the master key reconstructed. Co-located nodes give you the security of one machine, not true 2-of-3. There's an honest middle ground, because you won't always get three hosts on day one: run the nodes as separate users on a single memory-encrypted, attested VM, a Confidential VM (AMD SEV-SNP) or an enclave (AWS Nitro). Even host root can't read the nodes' memory there, and attestation proves the running image wasn't tampered with. It's a legitimate compensating control for a soft launch. Just be honest about what it is: it hardens the one box, it does not give you independent-failure 2-of-3. Treat multi-host as the target you graduate to.

Section 2: The stack

I didn't want to build an MPC server from scratch. The cryptography is the easy part now; the hard part is the server around it, networking, retries, encrypted storage, backups, health checks, a client SDK. That's easily a quarter of an engineering budget before you sign your first transaction.

So the stack sits on **Mpcium** (fystack), the most turnkey open-source MPC server there is, Apache-2.0, written in Go. It hands you a full t-of-n cluster out of the box:

Mpcium — What it is: Open-source MPC server (Go) · Role here: Runs the nodes, the signing/keygen lifecycle, the SDK
tss-lib** (fystack/tss-lib v2.0.3)** — What it is: Threshold-signature library (GG18/GG20) · Role here: The cryptographic core, backing Mpcium
BadgerDB — What it is: Embedded KV store, AES-256 encrypted · Role here: Each node's local key-share storage
NATS JetStream — What it is: Message bus · Role here: Carries keygen/signing request + result events
Consul KV — What it is: Service discovery + KV · Role here: Peer routing and wallet metadata
PostgreSQL — What it is: Relational DB · Role here: User rows, including each user's derivation index

Out of the box you also get ECDSA (secp256k1) + EdDSA (Ed25519), Ed25519 mutual auth on every message, ECDH-encrypted peer channels, incremental encrypted backups, and a TypeScript client ([@fystack/mpcium-ts](https://www.npmjs.com/package/@fystack/mpcium-ts)) that drops into a NestJS backend.

Two things it deliberately doesn't have, and how we deal with them:

No trusted-dealer key import. You can't import an existing private key. Section 6 makes that a non-issue by deriving all user wallets from one generated master instead of importing anything.
No REST/gRPC API, it's NATS pub/sub + SDK only. Fine for a backend already on a message bus; wrap it if you need HTTP.

One thing worth doing before you trust any of this. Mpcium runs GG18/GG20, the family with the *BitForge* vulnerability class from the last article. It's acceptable here for two reasons. BitForge needs a malicious signing party, which is irrelevant when you run all three nodes. And more importantly, I checked the patch is actually in the code path. The tss-lib v2 line carries the two hardening proofs, Paillier-Blum in crypto/modproof/ and no-small-factor in crypto/facproof/, generated in GG18 keygen round 2 and verified in round 3. Mpcium pins the fystack/tss-lib v2.0.3 fork through a go.mod replace, so I grepped its whole tree for the flags that would disable those proofs (SetNoProofMod, SetNoProofFac) and found them called nowhere in the keygen path, only in the library's own tests. That's the level of due diligence a signing stack earns: don't trust the protocol name, read the code path.

(Starting fresh and want the newest protocol? [cggmp24](https://github.com/LFDT-Lockness/cggmp21) (the cggmp21 crate, which now implements CGGMP24 — patched, v0.7.0-alpha.2+) and Coinbase's Cure53-audited [cb-mpc](https://github.com/coinbase/cb-mpc) are the forward-looking cores. But they're libraries, not servers, so you're back to building the server yourself.)

Section 3: Docker Compose — nine services

The full stack is nine services, eight long-running plus a one-shot bootstrap. Here's how they fit, and the two traps that will bite you.

**app** — Image / build: your backend (NestJS) · Role: Custodial backend + MPC client. Starts last.
**postgres** — Image / build: postgres:15-alpine · Role: User rows, including each user's derivation index
**redis** — Image / build: redis:7-alpine · Role: Cache (not part of MPC state)
**nats** — Image / build: nats:2.10-alpine (-js) · Role: Message bus (JetStream enabled)
**consul** — Image / build: hashicorp/consul:1.15 · Role: Service discovery + routing KV
**mpcium-bootstrap** — Image / build: hashicorp/consul:1.15 · Role: One-shot: seeds peer IDs into Consul, then exits
**mpcium-node0/1/2** — Image / build: your Mpcium node image · Role: The three MPC nodes

depends_on enforces the order: Consul healthy → the one-shot bootstrap completes → nodes start → the backend connects last.

The bootstrap pattern

The nodes need to know about each other before they start. Rather than bake that in, a one-shot mpcium-bootstrap container writes the three peer UUIDs into Consul and exits 0. Every node then depends on it with service_completed_successfully:

services:
  mpcium-bootstrap:                 # one-shot: seed peer UUIDs into Consul, then exit 0
    image: hashicorp/consul:1.15
    environment: [CONSUL_HTTP_ADDR=consul:8500]
    command:
      - sh
      - -c
      - |
        consul kv put mpc_peers/node0 <node0-uuid> &&
        consul kv put mpc_peers/node1 <node1-uuid> &&
        consul kv put mpc_peers/node2 <node2-uuid>
    depends_on: { consul: { condition: service_healthy } }
    restart: "no"

  mpcium-node0:
    build: ./mpc/node                # distroless image, runs as UID 65532
    command: ["start", "-n", "node0"]
    working_dir: /cluster/node0      # mpcium reads identity/ + peers.json CWD-relative
    volumes:
      - ./mpc/node0:/cluster/node0                              # host dir: chown 65532:65532
      - ./mpc/config/node0.yaml:/cluster/node0/config.yaml:ro
    depends_on:
      mpcium-bootstrap: { condition: service_completed_successfully }
      nats:   { condition: service_started }
      consul: { condition: service_healthy }
    restart: unless-stopped
  # node1, node2 are identical. The `app` service depends on all three and starts LAST.

A few load-bearing lines in there:

restart: "no" on mpcium-bootstrap — it's a one-shot: seed the peer IDs, exit 0, done. Restart it and it re-seeds on every boot.
condition: service_completed_successfully — the nodes block until that clean exit, so peers always exist before a node reads them.
the :ro config mount — each node's config.yaml is read-only; nothing at runtime should mutate it.
working_dir and the volume's owner are both load-bearing enough to be their own traps, below.

(As of v0.3.5 there's a lighter option: pass --peers=peers.json to each node and it syncs any new peers into Consul on startup, skipping ones already there — so you can drop the mpcium-bootstrap container entirely, or run mpcium-cli register-peers --peers peers.json once. The manual consul kv put mpc_peers/… still works; it's just no longer required.)

Trap #1 — the distroless UID

The Mpcium node image is [FROM gcr.io/distroless/base-debian12](https://github.com/GoogleContainerTools/distroless) with USER nonroot:nonroot, which is UID/GID 65532. Your host-mounted node dirs are owned by your user, so the moment a node tries to write its BadgerDB it hits permission-denied and crash-loops. One line fixes it, and you'll forget it exactly once:

chown -R 65532:65532 ./mpc/node0 ./mpc/node1 ./mpc/node2

Trap #2 — `working_dir` is load-bearing

Mpcium resolves identity/ and peers.json relative to the current working directory, not to a config path. Miss working_dir (or WorkingDirectory in a systemd unit) and the node starts in / and can't find its identity files. Pin it.

Section 4: Node configuration

Each node gets its own config.yaml. Most of it is boring. Two fields will bite you.

nats:   { url: "nats://nats:4222" }
consul: { address: "consul:8500" }

mpc_threshold: 1                   # tss-lib degree t -> t+1 = 2 signers (this is 2-of-3)
environment: "production"

badger_password: "<32 hex chars = 32-byte AES-256 key, DIFFERENT per node>"
event_initiator_algorithm: "ed25519"
event_initiator_pubkey:  "<64 hex, SAME on all nodes + backend>"
chain_code:              "<64 hex, SAME on all nodes + backend>"

db_path: "./data"                  # mpcium appends /nodeN — remember this in Section 8
backup_enabled: true
backup_period_seconds: 300
max_concurrent_keygen: 2
max_concurrent_signing: 10

What the important fields do:

mpc_threshold — the availability/security knob; 1 means 2-of-3 (see the gotcha below).
db_path — where the shards live; Mpcium appends /nodeN to it, which will matter enormously at restore time (Section 8).
backup_enabled / backup_period_seconds — Mpcium's own incremental encrypted backup, every 300 s.
max_concurrent_keygen / max_concurrent_signing — per-node concurrency caps.
the three identity fields (badger_password, event_initiator_pubkey, chain_code) have a shared-vs-per-node rule of their own, right below.

The threshold gotcha (read this twice)

In tss-lib, mpc_threshold is the polynomial degree t, and the number of signers required is t + 1. So a 2-of-3 cluster uses mpc_threshold: 1, not 2.

And here's the part that makes it dangerous: the Mpcium dev config.yaml.template ships with mpc_threshold: 2 by default (the config.prod.yaml.template correctly ships 1 — but the dev one is what most people copy first). Copy it unthinkingly and you've built a 3-of-3 cluster, which throws away the entire availability benefit of MPC, because now every node must be online to sign. Set it to 1.

One more thing: tss-lib only enforces that the degree is at least 1 and no larger than the node count minus one. For three nodes that leaves 2-of-3 (mpc_threshold: 1) or 3-of-3 (mpc_threshold: 2) — a 1-of-3 where a single node could sign alone is degenerate and rejected. (GG18 tolerates a dishonest majority, so the code allows any 1 ≤ t ≤ n−1; Mpcium's README still recommends an honest-majority t ≥ ⌊n/2⌋+1, and for three nodes there's no sensible quorum below 2-of-3 anyway.)

Shared vs. per-node

This catches people, so let me be blunt:

Different on every node: badger_password (each node encrypts its own BadgerDB with its own key).
Identical on every node and the backend: chain_code, event_initiator_pubkey.

event_initiator_pubkey pins who's allowed to ask the cluster to sign, the Ed25519 public key of your backend's initiator identity. chain_code is what makes deterministic derivation work (Section 6).

Identities and peers

Each node dir has an identity/ folder with its own private key (node0_private.key, mode 0600) plus every node's public identity (nodeX_identity.json, mode 0644). And every node carries the same peers.json, whose UUIDs must exactly match what mpcium-bootstrap seeded into Consul:

{
  "node0": "00000000-0000-0000-0000-000000000000",
  "node1": "11111111-1111-1111-1111-111111111111",
  "node2": "22222222-2222-2222-2222-222222222222"
}

Section 5: The initialization ceremony (DKG)

This is the ceremony that generates the master key without ever assembling it. It runs once, on a fresh cluster, with the backend stopped.

The one rule that governs everything: startup order

The single most important operational rule in this whole deployment:

The MPC nodes must create the JetStream streams before anything else connects to NATS.

Here's why. The Go nodes create their mpc-keygen and mpc-signing streams with JetStream's default (limits) retention, but the TypeScript client (@fystack/mpcium-ts) creates those same streams with workqueue retention (and the mpc result stream the other way round — interest on the client, workqueue on the nodes). If the client wins the race and creates either one first, the nodes come up, refuse to reconcile the retention, and immediately crash-loop with:

err_code=10052 ... can not change retention policy to/from workqueue

And JetStream won't let you add or remove workqueue retention on an existing stream, so no config change or API call fixes it in place. The simplest remedy is to recreate the NATS container (or drop just the two offending streams and let the nodes recreate them), either way rebuilding JetStream state you never planned to touch. So the sequence is non-negotiable:

Get this order wrong and JetStream's fixed workqueue retention forces you to recreate NATS from scratch.

# 1. Bring up infra + the three nodes (NOT the app yet)
docker compose up -d postgres redis nats consul \
  mpcium-bootstrap mpcium-node0 mpcium-node1 mpcium-node2

# 2. Wait until each node logs that it owns its streams, e.g.:
#    "SigningConsumer: Subscribed to signing events"
#    "Subscription created successfully"
docker compose logs -f mpcium-node0   # watch for the stream-subscription logs above

# 3. ONLY NOW run the init ceremony / start the app

There's a sneakier version of this trap too: a running backend steals the DKG result. If the app is up during init, its consumer grabs the keygen-result event off the queue before the init script can read it, and init hangs forever on "waiting for keygen result." Stop the app before you initialize.

What the init script actually does

The init is a one-time bootstrap script. Don't underestimate it, ours is 1,160 lines, and that's normal. A one-time bootstrap for a distributed system is never a ten-liner. Four phases:

Validate — the compose file, every node config, the initiator key, the backend .env, and that all three nodes plus NATS and Consul are actually running.
DKG the master wallet — connect to NATS, create the Mpcium client, call createWallet(masterWalletId), wait for the keygen-result event (60-second timeout). Take the ECDSA public key, compressed to secp256k1.
Write env + back up — persist MPC_MASTER_WALLET_ID, MPC_MASTER_PUBLIC_KEY, and MPC_CHAIN_CODE to the backend's .env; then copy each node's data dir, export the Consul KV, write a metadata.json (nodeCount: 3, threshold: "2-of-3"). Move this backup offline immediately, it's hot-key-equivalent material.
Verify — derive addresses at indexes 0 and 1, sign a test 32-byte digest at each (30-second timeout, up to four attempts on transient errors), print === INITIALIZATION COMPLETE ===.

Timeouts and the slow part of boot

Keygen needs all three nodes at once, so give it the most generous timeout (we use 60 s for keygen, 30 s for signing) — a transient hiccup shouldn't abort a ceremony. The one real cost at boot is Paillier preparameter generation, the bulk of a cold start. Given enough cores the three nodes generate their preparams in parallel; on a smaller host they contend for CPU, so start them serially to avoid starving each other. Either way you pay it once: the preparams are persisted to BadgerDB and reused on every restart, so restarts are cheap.

Section 6: One master wallet, infinite users

Give every user their own MPC wallet the obvious way, and every signup runs its own DKG across all three nodes — and your backups grow without bound: ten thousand users is ten thousand ceremonies and ten thousand more sets of shards you can never lose.

We don't do that. We run DKG exactly once, for a single master wallet, and derive every user wallet from it with BIP-32 child key derivation (CKD):

Each user gets an integer wallet_index, stored as a column in Postgres
Their address is derived at path m/44/60/0/0/{wallet_index} (all levels non-hardened — hardened derivation needs the private key, which never exists here) — pure math, zero MPC nodes required online
To sign, each node applies its own derivation delta to its master share; the child key, like the master, is never assembled

This is what makes the operational model sane, and it's the reason the backup story in the next section is so simple. One derivation path supports about 2.1 billion users (indexes 0 through 2³¹ − 1), so you'll run out of users long before you run out of key space. I covered CKD's security equivalence in the previous article; the short version is that it's identical to per-user DKG under every node-compromise scenario.

Section 7: The backup model — back up three things, not one

The most common way to lose an MPC cluster is to back up the key shares, feel safe, and find out mid-restore that shares alone are useless. You have to back up three things:

Restore only BadgerDB and the cluster boots but can't find any wallet — signing fails with ERROR_KEY_NOT_FOUND.

1 — Item: BadgerDB shards · Where: each node's data/ dir · Why it's mandatory: the encrypted key-share state
2 — Item: Consul KV (threshold_keyinfo/*, mpc_peers/*) · Where: Consul export · Why it's mandatory: wallet routing; without it every sign fails ERROR_KEY_NOT_FOUND
3 — Item: Metadata (chain_code, master wallet ID, master pubkey, node configs, identity keys) · Where: configs + backend .env · Why it's mandatory: needed to derive addresses and boot the nodes

One backup, forever

Here's the quiet superpower of CKD. The master shares, the threshold_keyinfo/* routing entries, and the peer metadata never change after the initial DKG. New users are just new rows in Postgres. So your MPC backup strategy is: take one full MPC backup right after init, then back up Postgres normally from then on. That's it. No incremental shard sync, no window where a user created between backups is unrecoverable. Per-user DKG would need a fresh MPC backup after every signup.

And this is the whole disaster-recovery story in one sentence: lose the entire cluster and you still lose nothing. Stand up fresh nodes, restore that single encrypted master backup, and every user wallet comes back, because no user wallet was ever stored on its own. Each one is a deterministic derivation of the master, and its wallet_index lives safely in your Postgres rows. The master backup plus your normal Postgres backups reconstruct every address and every signing key by index, on brand-new hardware, with nothing lost.

The copy-size trap

How you copy BadgerDB matters more than you'd think. A running node's BadgerDB is ~2.2 GB — SSTables, value logs, and the WAL, mostly churn. What you actually ship is Mpcium's own cold backup: with backup_enabled: true, each node writes a pruned, encrypted export to backup_dir (the backup-*.enc files), a few tens of KB apiece. In the drill the full age-wrapped bundle — all three nodes' exports plus every config, identity, and the Consul KV dump — came to 164 KB, four orders of magnitude below the live directory. Two consequences: never copy a live BadgerDB (and never stage a hot copy in a small tmpfs — it can exceed 6 GB and blow out a ~2 GB /tmp); ship the backup_dir export instead.

Restore is a specific sequence

# 1. Start fresh nodes, then STOP them
docker compose up -d && docker compose stop mpcium-node0 mpcium-node1 mpcium-node2
# 2. For each node, decrypt + restore its export INTO the exact db_path/nodeN dir:
#    mpcium-cli recover --backup-dir <node's backup-*.enc> \
#      --recovery-path <db_path>/nodeN --force   # the nodeN subdir is load-bearing — Section 8
#    (recover prompts for the badger_password; drive it with `expect` in automation)
# 3. Import the Consul KV dump  (skip this and signing fails ERROR_KEY_NOT_FOUND)
consul kv import @backup/consul-kv.json
# 4. Fix ownership, delete stale BadgerDB LOCK files
chown -R 65532:65532 ./mpc && find ./mpc -name LOCK -delete
# 5. Start the nodes, wait for all three to log [READY]
docker compose start mpcium-node0 mpcium-node1 mpcium-node2
# 6. Verify with a signing smoke test — do NOT trust "[READY]" alone (Section 8)

Section 8: The disaster-recovery drill

You don't have a backup until you've restored it. So before go-live we ran the full thing: destroy the cluster, rebuild it from backups on fresh hardware, and prove the recovered cluster signs with the same master key. The pass criterion was strict, recover a signature and confirm via ecrecover that its master public key was byte-for-byte identical to the original.

It passed. It also surfaced eleven runbook and config bugs in a single session. A clean re-run took 45–60 minutes; the first pass took about two hours, precisely because of what it caught. Three of them are worth the price of this whole article.

The silent empty-database restore (the one from the opening)

This is the bug that would have quietly destroyed every user wallet in a real emergency.

Mpcium reads BadgerDB from ${db_path}/nodeN/, it auto-appends the nodeN subdirectory. Drop your restore into .../data instead of .../data/nodeN and the files land one level too high. Mpcium then looks in .../data/nodeN/, finds it empty, and, this is the dangerous part, silently creates a brand-new, empty database.

The cluster boots. All three nodes report [READY]. Health checks are green. But the shares are gone — every existing wallet is orphaned, and the first real signing request fails with ERROR_KEY_NOT_FOUND (newer v0.3.5 builds report the same thing as ERROR_NOT_MAJORITY, "not enough peers to process signing request"). Worse, the empty cluster will happily run a fresh DKG and mint a brand-new master key that has nothing to do with your users' funds. You find out on that first signature, long after the green health check lied to you.

"[READY]" is not "recovered." Restore into the exact db_path/nodeN subdirectory, then verify by signing, never by status alone.

The drill burned about 30 minutes on exactly this. The fix is trivial once you know it, restore into the explicit .../data/nodeN subdir and chown afterward, but you'd never guess it from a green health check. The lesson generalizes: [READY] means "a database is present," not "the right database is present." Verify recovery by signing and checking the recovered public key, never by status.

The documented command that didn't exist

The first-draft runbook confidently referenced an mpcium-cli init-dkg command that doesn't exist in the shipped binary, with a flag set to match. There is no keygen subcommand at all: DKG is triggered by publishing a signed keygen-request message onto NATS (subject mpc.keygen_request.*). Recovery really uses --backup-dir, --recovery-path, and --force, and the binary prompts for its password interactively via term.ReadPassword, so you drive it with expect in automation. Dry-run every command in your runbook against the actual binary before you trust it in an emergency.

`openssl rand` will hand you an invalid key

BadgerDB's encryption key must be exactly 16, 24, or 32 bytes. The instinctive command produces neither:

openssl rand -base64 32     # WRONG: 44 ASCII chars -> node fatals opening the encrypted BadgerDB:
                            # the key must be exactly 16, 24, or 32 bytes.
openssl rand -hex 16        # RIGHT: 32 hex characters = 32 bytes = AES-256

-base64 32 turns 32 random bytes into a 44-character string, and the password is used as raw key bytes — so that's a 44-byte key, not a valid AES length. -hex 16 gives 32 characters = 32 bytes: an AES-256-length key carrying 128 bits of entropy, ample for a local DB key.

Five more the drill taught us

Never regenerate badger_password after first start. The KEYREGISTRY is encrypted with the original password; change it and the database is permanently unreadable.
A wrong or regenerated chain_code silently orphans every wallet. The cluster boots and signs perfectly happily, but it now derives a completely different address space, so none of your users' existing addresses come back. Like the empty-DB bug, it sails past [READY] and only a sign-and-compare catches it. Treat chain_code as recovery-critical metadata, not a rotatable secret.
Regenerating node identity keys or peer UUIDs is a re-key, not a recovery. Fresh identities make the restored shares reject each other, forcing a brand-new DKG, which produces a new master public key and abandons every address derived from the old one. Same limit if a node is permanently lost: you can't just drop in a replacement, because a reshare needs the surviving shares as input. Mpcium ships that as Resharing() (Section 11) — it rebuilds the replacement from the survivors while keeping the master key. A recovered 2-of-3 cluster missing its third node has zero remaining fault tolerance until you re-share, so plan node replacement rather than improvising it.
Consul KV doesn't contain the master public key. It stores only {participant_peer_ids, threshold, version}. So you can't verify a restore by reading KV, you sign and ecrecover.
Your cold bundle must include the supporting config (nats.conf, CA templates), or the restored NATS/services crash-loop and you never reach the signing test.

Section 9: Deploying across separate hosts

Everything so far ran from one Docker Compose file — one host, one kernel, one root. That's the right way to learn the stack, and fine while you're testing, but it isn't 2-of-3: own the box and you own all three shares. Production means each node on its own machine, so a single host compromise costs the attacker exactly one share. None of the cryptography changes — same three nodes, same DKG, same backups. What changes is the wiring: instead of a Compose network, the nodes live on separate hosts joined by a private encrypted overlay, and they reach the bus over that overlay instead of a Docker bridge.

The shape

Three node hosts — one Mpcium node each. Ideally memory-encrypted, attested VMs (AMD SEV-SNP, AWS Nitro), each with no public IP and a scoped machine identity. Run the node as its own locked-down user with a 0700 data dir, so a compromised node process still can't read another node's BadgerDB.
One control host — NATS + Consul (and a small internal CA, below). The bus and the KV store live here; every node and the backend point at it.
The backend — sends digests, holds zero keys, also on the overlay.

The nodes never talk to each other directly — all protocol traffic goes through NATS. So the only reachability requirement is: every node and the backend can reach the control host's NATS and Consul over the private network. That keeps the network model a simple star, not a full mesh.

The private overlay (WireGuard)

Put every host on one WireGuard network so nothing MPC-related ever touches a public interface. On each node host, one peer pointing at the control host:

# /etc/wireguard/wg0.conf  (a node host)
[Interface]
Address    = 10.10.0.11/24
PrivateKey = <this node's private key>
ListenPort = 51820
MTU        = 1380          # provider MTU minus WireGuard's 80-byte header

[Peer]                     # control host (NATS + Consul)
PublicKey           = <control host public key>
AllowedIPs          = 10.10.0.1/32          # just the control host — nodes reach each other via NATS, not directly
Endpoint            = <control host public ip>:51820
PersistentKeepalive = 25

The control host runs the mirror image — one [Peer] block per node, each with that node's /32 — and forwards nothing else. Only public keys cross the wire; private keys never leave their host. Then lock the provider firewall so UDP/51820 is the only inbound port and only from the overlay's peers, deny the rest, and bind SSH to the WG address instead of the public one. On a cloud MPC VM that means launching it with no external IP at all and reaching it purely over WG.

Per-node config

Each node's config.yaml is identical to the single-host one, except the addresses now point at the control host over WG and the shared secrets stay shared:

nats:   { url: "nats://10.10.0.1:4222" }   # control host, over WG
consul: { address: "10.10.0.1:8500" }
mpc_threshold: 1                            # still 2-of-3
badger_password: "<per-node, 32 hex>"       # distinct per node
chain_code:             "<same on all nodes + backend>"
event_initiator_pubkey: "<same on all nodes + backend>"
db_path: "/var/lib/mpcium/data"

Lock the bus itself down with mTLS: issue a client certificate per node from a small internal CA (step-ca is plenty), so only your nodes and backend can speak to NATS — a stray connection that lands on the overlay still can't.

Bringing it up

The startup order from Section 5 still holds, just spread across hosts: bring up NATS + Consul on the control host, register-peers (or start the nodes with --peers) so the UUIDs land in Consul, start the three nodes on their own hosts (streams first — the retention trap doesn't care that the nodes are remote now), wait for [READY] on all three, then run the DKG exactly as before. The backend connects last.

This is the topology every backup, restore, and reshare in this article already assumes — the wiring is all that stood between you and it.

Section 10: Production hardening

Once the cluster works, the remaining work is isolation and discipline.

Split the nodes across hosts — the from-scratch recipe is Section 9, migrating an existing single-box cluster is Section 11. Three separate machines mean a single host compromise yields at most one share; anything less is theater. Can't split yet? Fall back to a memory-encrypted attested VM (AMD SEV-SNP, AWS Nitro) — the Section 1 note covers exactly what that buys you and what it doesn't. One caveat that lands here: SEV-SNP VMs generally can't live-migrate, so an abrupt host termination becomes your disaster-recovery trigger — exactly why the drill in Section 8 isn't optional.
Private encrypted network. Put inter-node and node-to-backend traffic on a private mesh (WireGuard). The message bus and KV store should never be exposed publicly.
Egress firewall. MPC nodes have no business making arbitrary outbound connections. Lock egress to what the protocol needs.
Encrypt every backup with [age](https://github.com/FiloSottile/age) before it leaves the host, and store it on physically separate, offline media. A stolen backup is all shares plus metadata, equivalent to stealing the funds.
Plan for the memory profile. Each node is lightweight at rest, on the order of tens of MB idle, but spikes during Paillier preparameter generation, so provision for the spike, not the idle state, and remember the serial-start caveat on smaller hosts. For reference, the entire dev stack (backend, three nodes, Postgres, Consul, NATS, Redis) fits comfortably in 3.8 GiB.
Make init lazy, not fatal. The backend shouldn't crash if MPC is briefly unavailable at boot; log a warning and retry on first real use. And handle the result-before-request race: register the pending request before you check for an early result, because a fast cluster can return a signature before your handler is registered.
Retry the transient errors. Treat these as retryable with a short backoff ([0, 500, 2000] ms): symmetrickey key not found, failed to decrypt message, encrypt tss message error, not enough peers to process signing request, timed out waiting for signing result. They're normal under load, not failures.

Section 11: Scaling the cluster — splitting hosts, and adding nodes

Two operations get conflated here, and they cost wildly different amounts. Moving your three nodes onto separate hosts keeps the same key and costs only a maintenance window. Adding a node changes the committee and needs a resharing ceremony — which Mpcium does ship, just not as a one-command move. Know which one you're doing before you touch production.

Splitting one host into three (same key, just downtime)

This is the common "started on one box, now you want real 2-of-3" move, and it is not a re-key. The identities, the shares, the chain_code, and the peer UUIDs all stay byte-for-byte the same — you're only changing where each node runs. Because none of the key material changes, the master public key and every derived address survive untouched. Mechanically it's the Section 8 restore, aimed at three hosts instead of one:

Stand up the private network first — a WireGuard mesh across the three hosts and the backend. Nodes talk through NATS, not node-to-node, so what actually has to be reachable is one NATS and one Consul over that private network.
Stop the cluster. Your downtime starts here, and it's a hard stop: nodes down, no signing.
Move each node's data dir, its identity/ folder, and its config.yaml onto its own host. Keep the peer UUIDs and identity keys identical — regenerating them is a re-key, not a move (Section 8), and it abandons every address.
Point each node's config at the relocated nats.url and consul.address, re-import the Consul KV dump, chown -R 65532:65532, and delete stale LOCK files.
Start the nodes, wait for three [READY], then verify by signing and ecrecover that the master public key is byte-identical to before. Never trust [READY] alone — that's the whole lesson of Section 8.

Downtime is the stop-to-verified-restart window — minutes if you've rehearsed it. Rehearse it exactly like the DR drill, because mechanically it is one.

Adding a node (2-of-3 → 2-of-4, or 3-of-5): resharing

Reshaping the committee — a fourth node, a higher threshold — can't be a plain docker run: a new node has no share, and the existing shares were generated for a three-party polynomial. The primitive that fixes this is key resharing — the current shareholders jointly re-distribute the secret to a new set of parties, producing fresh shares for the new committee while keeping the same master public key, so every user address is preserved and nothing moves on-chain.

Mpcium ships resharing as a first-class, signed operation (it landed in v0.3.0 and was hardened for new-node joins in v0.3.5). You publish a ResharingMessage and wait for the result:

mpcClient.Resharing({
  sessionID, walletID, keyType,
  nodeIDs:      newCommitteePeerIDs,  // include a brand-new peer ID to add a node
  newThreshold,                       // change t here
});
mpcClient.OnResharingResult((r) => { /* new shares are written */ });

Results arrive on mpc.mpc_reshare_result.*; there's a runnable examples/reshare/main.go and it's e2e-tested for both ECDSA and EdDSA. The redistribution stores the new shares under the same walletID with only an internal share-version bump, so the master public key — and every CKD-derived address — survives untouched. Add a node by including its peer ID in nodeIDs; remove one or raise the threshold by editing nodeIDs / newThreshold.

Two honest caveats, because this is where "read the code, not the pitch" earns its keep:

It's a two-part move, not one command. There's no mpcium-cli reshare. A full membership change is: register the new peer first (generate its identity.json, then mpcium-cli register-peers), then the Resharing() call that actually hands it a share. Issue #84 is still open — but for a turnkey add/remove-node CLI and end-to-end docs, not because the crypto is missing.
A reshare rotates the master shares (new version, new committee in threshold_keyinfo), so it breaks Section 7's "one backup, forever": re-take the MPC backup after every reshare.

Still, pick your n and threshold before the master DKG when you can — a reshare is real coordinated work, and the fewer you run against a cluster holding real funds, the better.

Section 12: Troubleshooting cheat sheet

The failure modes you're most likely to hit, and the one-line fix for each:

err_code=10052 ... retention policy crash-loop — Cause: App/client created NATS streams before the nodes (wrong retention) · Fix: Recreate NATS; always start nodes first
Node permission-denied on data/ — Cause: Distroless UID 65532 vs. host ownership · Fix: chown -R 65532:65532 the node dirs
Signing fails ERROR_KEY_NOT_FOUND (Key info not found) — Cause: Consul KV (threshold_keyinfo/*) not restored · Fix: consul kv import the KV dump
Cluster [READY] but signing fails ERROR_KEY_NOT_FOUND (ERROR_NOT_MAJORITY on v0.3.5) — Cause: Backup restored one dir too high → empty DB, shares gone · Fix: Restore into the exact .../data/nodeN subdir; verify by signing
DKG hangs on "waiting for keygen result" — Cause: The backend consumed the result event · Fix: Stop the app before running init
DKG times out — Cause: Not all three nodes were ready · Fix: Keygen needs all nodes; wait for three [READY]
Node fatals opening the encrypted BadgerDB — Cause: badger_password is not 16/24/32 bytes · Fix: Generate with openssl rand -hex 16
Nodes misbehave after re-running docker compose up — Cause: An override recreated a stateful service (NATS/Consul), wiping its state · Fix: Restart the nodes after any infra bring-up; treat NATS/Consul as stateful

Look at the pattern. Not one of these is a cryptography failure. They're orchestration, filesystem, and config failures. That's the real lesson of running MPC in production: the math is the part that works, and the operations are where you earn your keep.

Section 13: Build it so you don't need the luck

A self-hosted 2-of-3 MPC signing cluster isn't exotic infrastructure. It's a handful of open-source components wired together with a few load-bearing rules: get the startup order right, get the threshold semantics right, back up three things instead of one, and drill your disaster recovery before you need it. Because the failure that actually hurts you isn't a dramatic one. It's a cluster that looks perfectly healthy while it has silently lost every key.

So if you take one thing from all of this, take the opening. Run the drill. A cluster that boots [READY] with every key silently gone isn't a hypothetical, it's the first thing that goes wrong the day you finally need your backup, and it stays invisible until you sign and check. We found ours with half an hour to spare and a lot of luck. Build the thing so you don't need the luck.

If this was useful, I write about backend, crypto infrastructure, and shipping distributed systems to production.

Follow me:

LinkedIn: https://www.linkedin.com/in/anatolii-zhadan/
Hackernoon: https://hackernoon.com/u/zhadan
Github: https://github.com/ImmuneFOMO

Tags: MPC · Multi-Party Computation · DevOps · Docker · Self-Hosted · Threshold Signatures · Disaster Recovery · Backend · Web3

How to Store Your Users' Crypto Wallets So Nobody Can Steal Them

ImmuneFOMO — Tue, 07 Jul 2026 09:00:00 +0000

Encrypting the private key still leaves one decryptable key behind one server — the honeypot that MPC later removes.

Your backend can decrypt every user's private key. So can your ENCRYPTION_KEY. So can whoever gets far enough into your servers: a leaked cloud token, a poisoned CI job, one unlucky RCE.

You encrypted the keys. You feel safe. You've quietly built the biggest honeypot in the company.

I know, because that's how we started: Wallet.createRandom(), AES-256-GCM, an encrypted blob in Postgres, one ENCRYPTION_KEY in the env. It ships fast and it passes the demo. It also means one stolen secret unlocks every user at once. That's the exact failure that drained Ronin (~$625M) and Harmony (~$100M). Nobody cracked the cryptography in either case. They found the one place that could.

This article is the fix I wish I'd read first: MPC-based threshold signing, where the private key is never assembled anywhere. We'll cover what it actually changes, what it does not solve, which protocols and libraries are safe to build on in 2026, and how to migrate an existing "encrypted key in a database" system without a big-bang cutover.

If you already run custodial Web3 infra and know how MPC-TSS works, skip to Section 4 (the protocol reality check) and Section 7 (one master wallet for a million users). That's where the real decisions live.

Section 1: The real problem — not encryption, but centralization

Here's the custodial flow almost every team ships first. It works, and that's the trap:

Generate a wallet, a secp256k1 private key
Encrypt it with AES-256-GCM
Store the encrypted blob in PostgreSQL
Decrypt it in the backend whenever you need to sign

This protects you against exactly one thing: a database-only leak. Steal the Postgres dump and nothing else, and all you have is ciphertext.

But your backend has to decrypt to function. So an attacker who gets enough control of it, through RCE, stolen cloud credentials, a CI/CD compromise, or a bad insider, doesn't need to break any cryptography. They use your own signing path. The AES does nothing, because your server was always going to decrypt.

This keeps draining real money, and it always looks the same:

Ronin Bridge (2022, ~$625M): attackers got 5 of 9 validator keys. Control was too concentrated; a Lazarus phishing campaign was enough to reach them.
Harmony Horizon Bridge (2022, ~$100M): a 2-of-5 multisig where the attacker collected just 2 keys and drained it.

Nobody cracked ECDSA. They collected enough key material from too few places. So the goal isn't "encrypt the private key better." It's this:

Remove the possibility that any single machine, process, or operator can unilaterally use the key.

That's what multi-party computation gives you.

Section 2: What MPC-TSS actually changes

In threshold MPC signing (MPC-TSS), the private key is never assembled in one place. Not in memory, not on disk, not for a microsecond.

The key is split into shares, each share lives on an independent node, and a threshold of nodes cooperates to produce a signature. The blockchain sees a completely normal ECDSA signature and can't tell the difference.

The canonical setup is three nodes with a 2-of-3 threshold:

Any two of three nodes can jointly sign. One node, or the backend, is never enough.

Walk the attack scenarios and the difference is structural, not cosmetic:

The database dump — Traditional (encrypted key): Ciphertext (safe, for now) · MPC 2-of-3: Nothing useful
Database + encryption key — Traditional (encrypted key): All keys stolen · MPC 2-of-3: Nothing useful
Full backend / RCE — Traditional (encrypted key): All keys stolen · MPC 2-of-3: No key exists to steal
One MPC node — Traditional (encrypted key): — · MPC 2-of-3: Not enough to sign
One disk snapshot of one node — Traditional (encrypted key): — · MPC 2-of-3: Not enough to sign

One honest caveat on that last row: an attacker who owns the backend can't exfiltrate a key, because there isn't one. But they can still ask the cluster to sign while they hold the box. MPC removes key theft, not a hijacked signing path. We come back to that in Section 10.

The backend holds zero private keys. When it needs a signature, it hashes the payload locally and sends a 32-byte digest to the cluster, which returns (r, s, v). Own the application server completely, and you get nothing to steal, because the thing you came for was never there.

Section 3: MPC in 60 seconds (for engineers)

Two different things get called "MPC," and mixing them up causes half the confusion in this space:

General MPC / SMPC — parties jointly compute a function over private inputs without revealing them. Banks scoring fraud together, hospitals running joint analytics, private ML.
Threshold signature MPC (TSS) — the specialized case for key management and signing.

Wallet infra lives in the second one: threshold ECDSA (Ethereum, Bitcoin, most EVM/L2 chains) or threshold EdDSA (Solana and friends).

There are only two ceremonies to understand:

DKG (distributed key generation). The nodes jointly generate key shares. No node ever sees the whole private key; each ends up holding its own share plus the shared public key. Keygen needs every participating node online.

Signing. Your backend hashes the message and sends the 32-byte digest. A threshold of nodes runs the multi-round protocol and returns a standard (r, s, v), valid on-chain.

DKG runs once. After that, signing only needs a threshold of nodes, not all of them.

One detail trips up everyone on day one:

In tss-lib-style implementations, threshold is the polynomial degree t, and the number of signers required is t + 1. So a 2-of-3 wallet is configured with threshold: 1 (degree 1 → 2 signers), not threshold: 2.

Get that wrong and you've mis-provisioned your security by one. And remember the asymmetry: keygen needs every node, signing needs only the threshold. A node can be down and you can still sign, which is exactly the availability you want.

Section 4: The protocol reality check (2026)

This is where naive advice goes wrong. People pick a protocol by name and stop. A protocol name is not a security guarantee. You have to evaluate implementation, version, and threat model together.

The same protocol family can be safe or unusable depending on the exact library and version. The name alone tells you nothing.

GG18 / GG20

The most widely implemented family, and the basis of most turnkey servers. It also had serious holes:

**BitForge (CVE-2023-33241)*, disclosed August 2023, CVSS 9.1–9.6. A malicious signer crafts a bad Paillier modulus and, in some implementations, extracts another party's key share in as few as 16 signing attempts (the real range runs from 16 up to ~a billion, depending on the implementation). The catch: it needs a malicious *participating party. If you control all your own nodes, an outside attacker can't mount it.
**TSSHOCK** (Verichains, Black Hat USA 2023), a related class against GG18/GG20/CGGMP21 that can "leave no trace and appear innocent to the other parties."

The fix, Paillier-Blum plus no-small-factor proofs, is in tss-lib v2.x and maintained forks. So GG18/GG20 on a patched implementation, with all nodes under your control, is a defensible self-hosted choice. On an unpatched one, it isn't.

CGGMP21

Newer, academically stronger, and carrying a live problem in Rust. **CVE-2025-66017 / RUSTSEC-2025-0127**: signature forgery through altered presignatures, CVSS 8.2. The advisory is blunt: "Patched: no patched versions" for the cggmp21 crate. There's a second twist that lands right on our path here: presignatures plus HD/BIP-32 derivation drop effective security to 85 bits, and this whole article is about to steer you toward HD derivation (Section 7).

Every serious writeup says the same thing: don't start a new system on raw cggmp21.

CGGMP24

The safer forward path. The same class of issue is patched in [v0.7.0-alpha.2](https://github.com/LFDT-Lockness/cggmp21), and the API hardening is real. If you're choosing a modern protocol today, this is the direction.

DKLS-family (DKLS23)

Strong research, good performance in specific stacks. Watch the license (Section 6). Implementation quality and licensing vary wildly by vendor.

GG18 / GG20 — Status 2026: Mature, patched forks exist · New self-hosted system?: Yes, if patched and all nodes yours
CGGMP21 — Status 2026: Unpatched forgery CVE · New self-hosted system?: No, don't start here
CGGMP24 — Status 2026: Patched (v0.7.0-alpha.2) · New self-hosted system?: Yes, modern forward path
DKLS23 — Status 2026: Strong, license-dependent · New self-hosted system?: Only after reading the LICENSE

Never trust a "we use protocol X" claim. Check the exact library, the exact version, whether the CVE patch is actually in the code path you call, and whether your threat model even includes a malicious party.

Section 5: Build your own vs. a turnkey server

If you want self-hosted MPC today, you have two real options.

Option A — build on a library

You take a cryptographic library and build the server, networking, orchestration, storage, backups, and observability yourself.

[**bnb-chain/tss-lib](https://github.com/bnb-chain/tss-lib)** — Language: Go · License: MIT · Audit: Kudelski (Oct 2019) · Notes: The de-facto GG18/GG20 lib
[**coinbase/cb-mpc](https://github.com/coinbase/cb-mpc)** — Language: C++ · License: MIT · Audit: Cure53 (Q4 2024) · Notes: High-quality, library only — no server/storage/API
LFDT-Lockness [cggmp24](https://github.com/LFDT-Lockness/cggmp21) — Language: Rust · License: MIT/Apache-2.0 · Audit: Kudelski · Notes: Modern protocol, patched

You get total control and protocol-level flexibility. You also now own networking, retries, key storage, backups, disaster recovery, monitoring, and the entire incident-response surface. That's weeks to months of work before you sign your first production transaction.

Option B — an open-source MPC server

The most turnkey route right now is **Mpcium**: a Go server backed by tss-lib, shipping a full cluster (nodes, message bus, service discovery, encrypted storage) plus client SDKs, under Apache-2.0.

You get a real cluster out of the box and a dramatically lower starting cost. In exchange you accept an opinionated architecture, feature gaps you'll fork around (trusted-dealer key import, a REST façade over its NATS pub/sub), and a DevOps burden that's still yours.

The heuristic I keep coming back to: weeks of building networking, storage, and DR from scratch, versus adding one feature to a server that already works. Unless you have a hard protocol requirement, extend the existing server.

A quick note on the rest of the field, so you don't lose a week evaluating them:

ZenGo — the [multi-party-ecdsa](https://github.com/ZenGo-X/multi-party-ecdsa) library is explicitly abandoned; [gotham-city](https://github.com/ZenGo-X/gotham-city) is 2-of-2 only, demo-grade, and effectively dormant (last commit 2024).
**Safeheron** — open crypto libraries, proprietary server.
**Sodot** — self-hostable, but proprietary: a closed-source SDK you can't inspect or fork.
**Lit Protocol** — a decentralized node network you consume, not a private cluster you run yourself.

Section 6: Licensing traps

This one surprises people in legal review, not code review.

A repo can be source-available or non-commercial while looking open at a glance. The protocol paper can be fully open while the implementation's license is restrictive.

The cleanest example: one popular dkls23 implementation (Silence Labs) ships under a non-commercial license (SLL), and commercial use needs a separate paid license. A different, independent DKLs23 implementation (0xCarbon) is dual MIT/Apache-2.0. Same protocol, completely different rights.

Open the actual LICENSE file, in the exact repo and version you plan to ship. Not the README, not the paper. The license, pinned to your version.

Section 7: One master wallet for a million users (CKD)

Give every user their own wallet the obvious way and each one gets its own DKG across your nodes. That's Mpcium's default, and it works fine. The catch isn't performance, it's what it does to your backups and your operations.

Every per-user DKG needs all three nodes online, and every wallet it creates is another independent set of shards on every node. So the MPC state you have to protect grows with your user base: every signup is one more ceremony to run and one more set of shards you can never lose. At a handful of users that's nothing. At a few million, provisioning and backup are the whole problem.

CKD collapses all of it into one ceremony. BIP-32 hierarchical deterministic wallets, the trick Bitcoin gave us in 2012, derive an effectively infinite tree of child keys from one master. Run the DKG once, and every user after that is a pure derivation off the master — no ceremony, no nodes online, and a single backup covers all of them. That's CKD (child key derivation):

Run one DKG at init → a single master MPC wallet (still split into 3 shards, still never assembled)
For each user, derive child = derive(masterPubKey, chainCode, userIndex). Pure math, no ceremony, no nodes required online
To sign, each node applies the derivation offset to its own share and signs. The child key, like the master, is never assembled

Per-user DKG on the left runs a full ceremony per user. CKD on the right derives every user from one master, with no per-user ceremony.

The operational gap isn't incremental, it's categorical:

Creating a user — Per-user DKG: a key ceremony, all nodes · CKD (one master): a derivation, no ceremony
10,000 users — Per-user DKG: 10k ceremonies, 30k shards · CKD (one master): 1 ceremony, 3 shards
Backup cadence — Per-user DKG: after every new user · CKD (one master): once, at init
MPC node down — Per-user DKG: new users wait · CKD (one master): new users still created
Disaster recovery — Per-user DKG: restore everything · CKD (one master): restore master; users are DB rows

"Is it as secure?" is the fair question, and the answer is yes, under the same threat model:

1 node compromised — Per-user DKG: Safe · CKD: Safe
2 of 3 nodes compromised — Per-user DKG: Compromised · CKD: Compromised
Backup stolen — Per-user DKG: Compromised · CKD: Compromised
2 nodes in a secure enclave — Per-user DKG: Protected · CKD: Protected

That equivalence is about node compromise. Be clear-eyed about what CKD gives up, because this is the part the marketing pages skip:

You can't rotate or revoke a single user's key in isolation. Everyone derives from the same master; there's no per-user share to roll.
You can't prove a user's key is cryptographically independent of the others, which some regulatory or audit regimes want.
Anyone holding the master public key and chain code can enumerate every user address. For an opt-in custodial service that's a non-issue; for a privacy-sensitive product it's a real deanonymization vector.

If none of that bites your product, CKD is a clean way to scale. If one does — and for many products one will — per-user DKG is the safer default, and that's when it earns its cost.

The BIP-32 caveat, stated honestly: with the parent public key, the chain code, and a full child private key, you can recover the parent key. In MPC that's defused, because the child private key never exists as a whole; each node holds only a share delta. The chain_code isn't secret-shared either (every node has a full copy), but that doesn't change the picture: recovering the parent still needs a full child private key, and that needs a threshold of nodes, which is total compromise already.

I didn't take any of this on faith. An end-to-end recovery test ran the full flow, create the master, derive several users, sign with each, back up, destroy the cluster, restore, verify, and passed 12 of 12 checks: every derived address matched, every signature still verified, and a brand-new user created after the restore worked too. A separate cross-machine drill, destroy the cluster and rebuild on fresh hardware, is what flushed out eleven runbook bugs, one of which would have quietly lost every key in a real emergency. The next article takes that drill apart in full.

Section 8: Who actually runs MPC in production

MPC is no longer a crypto-startup curiosity. The clearest sign of how seriously the industry takes it: Fireblocks, one of the largest custody providers, is the team that discovered and disclosed the BitForge vulnerability class from Section 4. When your custody vendor is the one running the offensive research that finds MPC's own weak spots, the primitive has left the lab.

Who else is standing on it:

**IBM Digital Asset Haven** (2025), developed with Dfns, combines Dfns's threshold signing with IBM's HSM and offline-signing models for institutional custody. MPC under an IBM-branded product is about as mainstream as it gets.
Dfns runs its KU23 protocol on its key-management network — by its own figures 10M+ keys and ~$1B signed per month — and plans to open-source it and move governance to the Linux Foundation's LF Decentralized Trust (the Lockness project).
Coinbase open-sourced its own MPC library, [cb-mpc](https://github.com/coinbase/cb-mpc) (MIT, Cure53-audited). You don't open-source your custody crypto unless it's core to how you operate.
Fordefi, BitGo, and Copper ship MPC/TSS custody to institutional desks and DeFi.

Adopting MPC isn't an exotic bet. It's the same primitive regulated, multi-billion-dollar custodians standardized on.

Section 9: Migrating from encrypted keys to MPC

You rarely start clean. Here's a rollout that doesn't need a big-bang cutover.

The safe path is a signer abstraction: your service code never changes, only what sits behind the interface does.

Inventory every signing call site. Every place a key is created, decrypted, or used, across services, jobs, and scripts.
Introduce a signer abstraction. Replace direct wallet creation with a single signing interface so the rest of your code stops caring where the key lives.

   // Before: the key exists in your process
   const wallet = new ethers.Wallet(decrypt(row.encryptedKey));
   const sig = await wallet.signTransaction(tx);

   // After: the key never exists here — you send a digest out to sign
   const signer = mpc.getSigner(user.derivationPath);
   const sig = await signer.signTransaction(tx); // digest -> MPC -> (r, s, v)

Add MPC signer adapters. One thin adapter for ethers, one for viem, both just forwarding the digest to the cluster.
Dual-run behind a flag. MPC_ENABLED, rolled out to a controlled cohort first.
Cut over new users first. New wallets go through MPC; the legacy path becomes read-only.
Migrate or rotate legacy keys. Import if your server supports trusted-dealer import; otherwise move funds to MPC-managed wallets and retire the old ones.
Delete the decryption path. Rip out every getPrivateKey()-style helper. If the code can't decrypt a key, neither can an attacker who owns the process.
Test disaster recovery before launch. Restore drills are mandatory, not optional. (The next article is one long argument for why.)

Section 10: The mistakes teams make

After shipping this to production, the most useful thing I learned is that the cryptography almost never breaks. The orchestration does. The mistakes repeat:

Misreading threshold semantics, the t vs t + 1 count from Section 3.
Underestimating the message bus and metadata store. The signing protocol is chatty and stateful, and your NATS/Consul/BadgerDB layer is now load-bearing.
Assuming one datastore backup is enough. It isn't; you need the key shares and the routing metadata.
Running keygen while application consumers race for the result events, silently stealing DKG output from your init flow.
Treating MPC as "security complete" and skipping the policy layer, which brings us to the most important caveat here.

MPC is not magic

MPC dramatically reduces key-theft risk. On its own it does nothing about:

signing the wrong payload, if your backend is tricked into it
compromised approval or policy logic
frontend / UI supply-chain attacks that fool an operator into approving a bad action
broad application-layer abuse after a backend compromise

The cautionary tale is **Bybit (February 2025, ~$1.5B)*, the largest crypto theft to date. Bybit *had multisig (Safe). The attackers didn't break the keys; they compromised the UI so operators approved a malicious transaction. MPC wouldn't have stopped it, because the failure was in intent, not custody.

So treat MPC as a very strong primitive inside a layered system. Policy engine, transaction simulation, allowlists, anomaly detection, segregated duties, hardened ops: all still matter.

Section 11: One strong primitive, not the whole system

If your backend can decrypt every user's private key today, you don't have a cryptography problem. You have a system-design problem: one key, one server, one blast radius.

MPC-TSS is the most practical way I know to redesign that system so key theft becomes structurally hard and the blast radius of a compromise shrinks. Not impossible. Materially harder. In security architecture, that difference is the whole game.

One warning before you run off and build it, though. The day I moved our first wallet onto MPC and deleted the old getPrivateKey() helper felt great, right up until I realized MPC had solved the theft problem and none of the others. It will faithfully sign the wrong transaction if something upstream tells it to. So ship it as one strong primitive in a layered system, not as the thing that lets you stop worrying.

Next up, the hands-on half: deploying a self-hosted 2-of-3 MPC signing cluster with Docker. The DKG ceremony, the backup model, and the disaster-recovery drill that caught a silent, cluster-killing bug — the one I mentioned above, in full.