I take down a healthy primary on purpose. You should too.

At one company I built an autoscaling pipeline that replaced MongoDB cluster nodes automatically. CloudWatch caught the CPU threshold, and the automation rolled the cluster node by node. Secondary first, second secondary next, primary last. Twelve minutes, no human, no application impact.

The order is the interesting part. Rolling the primary last means the cluster experiences exactly one election per cycle, at a moment the automation chooses. Which only works if you trust elections. And you only get to trust elections by triggering them yourself, regularly, and watching what your application does.

What actually happens in an election

The mechanics, compressed.

1. Every node heartbeats every 2 seconds. A node that misses heartbeats for electionTimeoutMillis (default 10 seconds) is presumed gone.
2. An electable secondary calls an election. Voting is not seniority. A node only wins if its oplog is at least as recent as each voter's. Freshness outranks priority.
3. priority settings bias who stands for election, not who deserves to win. A priority 10 node with a stale oplog loses to a priority 1 node that's caught up.
4. Majority vote, new primary, secondaries sync from it.

Wall clock cost is typically a few seconds for the election itself, bounded by that 10 second detection window on the front. Call it 5 to 15 seconds of no primary, during which writes cannot be acknowledged.

The part that pages you isn't the election

It's what your application does during those seconds. Three behaviours, in ascending order of maturity.

• Errors bubble to users. Driver throws NotWritablePrimary, nobody catches it, checkout fails.
• Blind retry loops. Sometimes double applies the write. Worse than failing.
• retryWrites=true plus a sane serverSelectionTimeoutMS. The driver holds the write, discovers the new primary from the topology update, retries exactly once. To the user, one slow request.

You do not find out which of these you are during a real incident. That's the wrong time.

The drill

Quarterly. In staging first, then production during a low traffic window. Yes, production. Staging tells you the drill works, production tells you the truth.

// 1. Note current topology
rs.status().members.map(m => ({ name: m.name, state: m.stateStr }))

// 2. Ask the primary to step down gracefully
rs.stepDown(60)   // refuses re-election for 60s

// 3. Watch the election from a secondary
rs.status()

While it runs, the numbers I capture:

detection to new primary elected : ___ s
application error count          : ___
p99 latency during window        : ___ ms
first write after failover       : double applied? (check by unique key)

Those blanks are the deliverable. Fill them from your own drill.

rs.stepDown() is the polite version. The primary flushes and closes cleanly. Once that's boring, graduate to the impolite version. kill -9 the primary's mongod, or drop its network. That's the one that resembles an actual cloud incident, and it exercises the full 10 second detection window instead of skipping it.

Two configuration notes from doing this at scale

Even numbered voting members are a bug. Four voters can split 2 to 2 and elect nobody. Keep voting members odd. Use an arbiter only if you truly can't afford a third data node, and know that arbiters don't help w:"majority" acknowledge anything.

Priorities encode your topology intent. In multi region clusters we set higher priorities in the primary region so failback happens automatically once nodes recover. But again, priority only nominates. The oplog decides.

An election you've rehearsed is an operational event. An election you haven't is an outage with paperwork.