DEV Community: Twinhull

How long does a Patroni failover take? We measured it

Twinhull — Mon, 05 Oct 2026 13:00:00 +0000

Tested on: Patroni 4.1.5, etcd 3.6.15, HAProxy 2.8, PostgreSQL 16

“A few seconds” is the usual answer. It's true for one kind of failover and wrong for the one that matters most. Here are real numbers from a 3-node cluster, what drives them, and what tuning actually buys you.

The short answer

What happened to the leader	Writes unavailable	Why
Planned switchover (`patronictl switchover`)	4.7–5.0 s	Leader steps down cleanly and hands over the lock
Clean stop (`systemctl stop patroni`, reboot)	4.7–7.0 s	Patroni releases the lock as it shuts down
Hard crash (power loss, kernel panic, `kill -9`)	24–30 s	Nobody releases the lock; it has to expire

Default timings (ttl 30 s, loop_wait 10 s, retry_timeout 10 s). Asynchronous replication, HAProxy routing on the Patroni REST API. Ranges cover repeated runs. Zero committed transactions were lost in every run.

How we measured

A writer opened a new connection through HAProxy's read-write port and committed one row every 200 ms, the way an application with a retrying connection pool behaves. We then took the leader away and recorded:

Write downtime: from the first failed write to the first successful write on the new leader. This is what your users feel.
Lost transactions: every row the client saw as committed, checked against the database afterwards.
Rejoin time: until the old leader streamed again as a replica, which shows that pg_rewind worked.

We measured the write path end to end, not just Patroni's log, because what matters is when writes succeed again. That includes Patroni's election, PostgreSQL's promotion, and HAProxy noticing the new leader.

Why a crash takes ~30 seconds

Patroni's leader holds a lock (a key with a time-to-live) in etcd and renews it every loop_wait seconds. When the leader is stopped or switched over, Patroni deletes the key, so a replica can take over on its next loop.

When the leader crashes, nothing deletes the key. The replicas must assume the leader might still be alive, perhaps just slow, until the lock expires after ttl seconds. That waiting is deliberate: it's what stops two nodes believing they're both the primary. So crash failover time is roughly:

ttl  (time since the last renewal, up to 30 s)
+ promotion of the best replica      (~1 s)
+ HAProxy health check notices it    (1–3 s)

With the default ttl: 30, we measured 24 to 30 seconds. The spread comes from when in the renewal cycle the crash happens.

What tuning buys you

The three settings live in Patroni's cluster configuration (patronictl edit-config) and must satisfy ttl ≥ loop_wait + 2 × retry_timeout:

ttl / loop_wait / retry_timeout	Crash downtime measured
30 / 10 / 10 (default)	24–30 s
20 / 5 / 5	19.9–20.8 s

Cutting the timings by a third saved us about 5–8 seconds. It comes at a cost: retry_timeout is also how long Patroni tolerates a slow etcd or a network blip before it gives up. Set it too low, and a 6-second hiccup on a busy network demotes a healthy primary. You'd trade a rare 30-second outage for more frequent, unnecessary 5-second ones. For most clusters on a normal LAN, the defaults are right.

Two settings that matter more than people expect:

failsafe_mode: true. Without it, losing etcd quorum demotes the primary even though PostgreSQL is fine. In our test, with 2 of 3 etcd members down and failsafe mode on, the primary kept accepting writes for the whole test. Only automatic failover was lost.
HAProxy's check interval. inter 2s fastinter 500ms fall 2 rise 1 detects the change within a couple of seconds. The common tutorial value inter 3s fall 3 rise 2 can add 6–9 seconds on its own.

Does synchronous replication slow failover down?

Not measurably. With synchronous_mode: true a crash took 23.2 seconds, within the async range. What synchronous mode changes is what you can lose: it guarantees zero lost committed transactions, at the cost of one network round trip on every commit. Async lost nothing in our runs too, but it can't promise that.

Why a switchover still takes ~5 seconds

Even a planned switchover interrupts writes: the old leader must stop accepting writes, flush its last WAL to the candidate, and restart as a replica, while the candidate is promoted. Existing connections are cut. Your application needs to reconnect. That's why a connection pool with automatic reconnect matters as much as anything in the database layer.

Measure your own cluster

Your network, disks and HAProxy placement will produce different numbers. Measure before you promise an RTO to anyone:

Build a staging cluster that matches production.
Run a writer through the same connection path your applications use.
Test all three: switchover, clean stop, and a real crash (SIGKILL to Patroni and PostgreSQL).
Check lost rows, not just downtime. Confirm the old leader rejoins by itself.
Repeat each test several times and report the range, not the best run.

Originally published at twinhullhq.com. The health-check script used in these tests is free and open source: github.com/twinhullhq/ha-check. If you run Patroni in production, the full kit (templates, failover drills, DR runbook, 3-node lab) is at twinhullhq.com.

Patroni: pg_rewind “password authentication failed” and the PGPASSWORD trap

Twinhull — Sat, 03 Oct 2026 13:00:00 +0000

Tested on: Patroni 4.1, PostgreSQL 16

Your cluster passes every switchover test. Then the leader crashes for real, a replica takes over as it should, and the old leader never comes back: it sits in start failed. We hit this in our own test lab. This article covers the cause, a one-line check, and the fix.

The symptom

patronictl list shows the crashed node stuck:

| pg2    | 10.0.0.12:5432 | Replica | start failed |    |     unknown |     |    unknown |     |

PostgreSQL's log on that node repeats every few seconds:

FATAL:  requested timeline 3 is not a child of this server's history
DETAIL:  Latest checkpoint is at 0/8000028 on timeline 2, but in the history of the
         requested timeline, the server forked off from that timeline at 0/7008BA0.

That message only says the old leader wrote WAL the new leader never saw, so it can't follow the new timeline. Patroni is supposed to fix exactly this with pg_rewind. The Patroni log shows why it didn't:

INFO: running pg_rewind from pg3
INFO: running pg_rewind from dbname=postgres user=rewind_user host=10.0.0.13 port=5432 ...
INFO: pg_rewind exit code=1
INFO:  stderr=pg_rewind: error: connection to server at "10.0.0.13", port 5432 failed:
       FATAL:  password authentication failed for user "rewind_user"
ERROR: Failed to rewind from healthy primary: pg3
INFO: starting as a secondary

The cause: PGPASSWORD beats .pgpass

Patroni doesn't pass passwords on the command line. It writes them to a .pgpass file (the postgresql.pgpass setting) and points pg_rewind and pg_basebackup at it. But libpq, the library under every PostgreSQL client, looks for a password in this order:

a password in the connection string itself,
the PGPASSWORD environment variable,
the password file.

So if Patroni was started with PGPASSWORD in its environment, typically the superuser's password left over from someone's shell, pg_rewind sends that password as rewind_user. Authentication fails, and the rewind is skipped.

Why your tests didn't catch it

The variable only hurts the Patroni process that inherited it. Nodes started cleanly by systemd weeks ago keep replicating happily. The dangerous moment is the restart after an incident: the leader crashed, someone logs in, runs a few psql commands with PGPASSWORD exported, and starts Patroni from that same shell, or a recovery script does it for them. That process is the one that has to run pg_rewind, and it's the one carrying the wrong password.

It gets worse: Patroni points replication at the same .pgpass file (primary_conninfo uses passfile=…, not a password), so even after a manual rebuild, a node running with that environment can't stream from the leader either. Planned switchovers in a clean test environment never show any of this. In our lab, switchovers passed every time, while every crash test failed until we found the variable in the restart path.

How the variable gets there

Starting Patroni by hand after an incident, from a shell where you exported PGPASSWORD to run psql, e.g. nohup patroni patroni.yml &.
Wrapper and deploy scripts that source an environment file with set -a, which exports every variable in it, and then start services.
systemd units with Environment=PGPASSWORD=… or an EnvironmentFile= shared with other tools.
Container images that set PGPASSWORD for convenience.

Check in 10 seconds

Look at the environment of the running Patroni process on every node:

sudo tr '\0' '\n' < /proc/$(pgrep -f 'bin/patroni' | head -1)/environ | grep -E '^PG|^PATRONI_'

Any PGPASSWORD, PGUSER, PGHOST or PGSERVICE line is a problem. So is any unexpected PATRONI_* line: Patroni reads every PATRONI_-prefixed variable as configuration, and it overrides patroni.yml. That's a related trap we also hit, with a PATRONI_LOG_DIR from a script sending logs somewhere unexpected.

The fix

Start Patroni only through systemd, with a clean environment:

    [Service]
    User=postgres
    Environment=PATH=/opt/patroni/bin:/usr/local/bin:/usr/bin:/bin
    ExecStart=/opt/patroni/bin/patroni /etc/patroni/patroni.yml
    KillMode=process

Remove any PGPASSWORD from Environment= and EnvironmentFile= lines, then systemctl daemon-reload && systemctl restart patroni. PostgreSQL keeps running while Patroni restarts.
On the stuck node, restarting Patroni is usually enough: it retries pg_rewind with the right password. If the rewind can't work any more (the WAL it needs has been recycled), rebuild it: patronictl reinit <cluster> <node>.

Prove it's fixed

A switchover won't prove anything here; only a crash will. In a test environment, kill the leader's Patroni and PostgreSQL with SIGKILL while writing, wait for the new leader, restart the old node, and confirm it comes back as a streaming replica. That's exactly what our failover-drill --mode crash automates. After the fix, our lab's crashed leaders rejoined in about 34 seconds, every time.

This trap is handled in the Twinhull HA Kit

The kit's systemd units start Patroni with a clean environment, preflight warns about exported PG* variables, and the config generator refuses PATRONI_* names. The crash drill proves the rewind works before production needs it.

See the kit

Patroni “Failed to get list of machines… 'int' object has no attribute 'get'” with etcd on Ubuntu 24.04

Twinhull — Thu, 01 Oct 2026 13:00:00 +0000

Tested on: Patroni 4.1.5, etcd 3.4.30 → 3.6.15

You installed etcd with apt, started a healthy 3-member cluster, and pointed Patroni at it. Patroni never starts PostgreSQL. It just logs the same error every five seconds. The problem is the etcd version, not your configuration.

The symptom

ERROR: Failed to get list of machines from http://127.0.0.1:2379/v3: AttributeError("'int' object has no attribute 'get'")
ERROR: Failed to get list of machines from http://127.0.0.1:2380/v3: AttributeError("'int' object has no attribute 'get'")
INFO: waiting on etcd

Meanwhile etcd itself looks perfectly healthy, which is what makes this confusing:

$ etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.1ms

The cause

Patroni's etcd3 backend doesn't use gRPC. It talks to etcd's JSON gateway over plain HTTP, the /v3/… endpoints. Ubuntu 24.04's etcd-server package is etcd 3.4.30. On that build, the endpoint Patroni calls first to discover cluster members doesn't answer the way Patroni expects:

$ curl -s http://127.0.0.1:2379/version
{"etcdserver":"3.4.30","etcdcluster":"3.4.0"}

$ curl -s -X POST http://127.0.0.1:2379/v3/cluster/member/list -d '{}'
404 page not found

Patroni gets back something that isn't the JSON object it expects, fails while parsing it, and retries forever. The error names a Python type, not etcd, which sends people looking in the wrong place.

The same request against etcd 3.6 returns the member list, and Patroni bootstraps immediately. We confirmed both on the same machine while testing the Twinhull kit.

Confirm it on your system

etcd --version | head -1                 # 3.4.x → this is your problem
curl -s -X POST http://<etcd-ip>:2379/v3/cluster/member/list -d '{}' | head -c 200

If the second command prints 404 page not found instead of JSON with a members array, upgrade etcd.

The fix: install etcd 3.5 or newer from the official release

Distribution packages lag behind. Install the upstream release binaries, which are statically linked and have no dependencies (current as of September 2026: 3.6.15; check github.com/etcd-io/etcd/releases):

sudo systemctl disable --now etcd                    # stop the distro etcd
sudo apt remove -y etcd-server etcd-client

ETCD_VER=v3.6.15
curl -fsSL -o /tmp/etcd.tgz \
  https://github.com/etcd-io/etcd/releases/download/$ETCD_VER/etcd-$ETCD_VER-linux-amd64.tar.gz
tar xzf /tmp/etcd.tgz -C /tmp
sudo install -m 755 /tmp/etcd-$ETCD_VER-linux-amd64/{etcd,etcdctl,etcdutl} /usr/local/bin/
etcd --version | head -1                             # etcd Version: 3.6.15

Then run etcd from /usr/local/bin/etcd with your own systemd unit and a config file (--config-file /etc/etcd/etcd.conf.yml).

Upgrading an existing 3.4 cluster? etcd supports upgrading one minor version at a time: 3.4 → 3.5 → 3.6, member by member. For a new Patroni cluster that has never stored anything, it's simpler to stop all members, wipe their data directories, and start fresh on 3.6.

Two more things while you're here

Don't switch Patroni to the old etcd (v2) backend to work around this. The v2 API is deprecated, and etcd 3.6 no longer serves it.
Use etcdctl without a proxy. If your servers export HTTPS_PROXY, add the etcd addresses to NO_PROXY, or health checks may go through the proxy and time out.

Caught automatically in the Twinhull HA Kit

The kit's preflight check fails on etcd versions older than 3.5 before you install anything, and the guide installs the official 3.6 release with a hardened systemd unit and optional mutual TLS.

See the kit