DEV Community: Twinhull

Patroni: pg_rewind “password authentication failed” and the PGPASSWORD trap

Twinhull — Sat, 03 Oct 2026 13:00:00 +0000

Tested on: Patroni 4.1, PostgreSQL 16

Your cluster passes every switchover test. Then the leader crashes for real, a replica takes over as it should, and the old leader never comes back: it sits in start failed. We hit this in our own test lab. This article covers the cause, a one-line check, and the fix.

The symptom

patronictl list shows the crashed node stuck:

| pg2    | 10.0.0.12:5432 | Replica | start failed |    |     unknown |     |    unknown |     |

PostgreSQL's log on that node repeats every few seconds:

FATAL:  requested timeline 3 is not a child of this server's history
DETAIL:  Latest checkpoint is at 0/8000028 on timeline 2, but in the history of the
         requested timeline, the server forked off from that timeline at 0/7008BA0.

That message only says the old leader wrote WAL the new leader never saw, so it can't follow the new timeline. Patroni is supposed to fix exactly this with pg_rewind. The Patroni log shows why it didn't:

INFO: running pg_rewind from pg3
INFO: running pg_rewind from dbname=postgres user=rewind_user host=10.0.0.13 port=5432 ...
INFO: pg_rewind exit code=1
INFO:  stderr=pg_rewind: error: connection to server at "10.0.0.13", port 5432 failed:
       FATAL:  password authentication failed for user "rewind_user"
ERROR: Failed to rewind from healthy primary: pg3
INFO: starting as a secondary

The cause: PGPASSWORD beats .pgpass

Patroni doesn't pass passwords on the command line. It writes them to a .pgpass file (the postgresql.pgpass setting) and points pg_rewind and pg_basebackup at it. But libpq, the library under every PostgreSQL client, looks for a password in this order:

a password in the connection string itself,
the PGPASSWORD environment variable,
the password file.

So if Patroni was started with PGPASSWORD in its environment, typically the superuser's password left over from someone's shell, pg_rewind sends that password as rewind_user. Authentication fails, and the rewind is skipped.

Why your tests didn't catch it

The variable only hurts the Patroni process that inherited it. Nodes started cleanly by systemd weeks ago keep replicating happily. The dangerous moment is the restart after an incident: the leader crashed, someone logs in, runs a few psql commands with PGPASSWORD exported, and starts Patroni from that same shell, or a recovery script does it for them. That process is the one that has to run pg_rewind, and it's the one carrying the wrong password.

It gets worse: Patroni points replication at the same .pgpass file (primary_conninfo uses passfile=…, not a password), so even after a manual rebuild, a node running with that environment can't stream from the leader either. Planned switchovers in a clean test environment never show any of this. In our lab, switchovers passed every time, while every crash test failed until we found the variable in the restart path.

How the variable gets there

Starting Patroni by hand after an incident, from a shell where you exported PGPASSWORD to run psql, e.g. nohup patroni patroni.yml &.
Wrapper and deploy scripts that source an environment file with set -a, which exports every variable in it, and then start services.
systemd units with Environment=PGPASSWORD=… or an EnvironmentFile= shared with other tools.
Container images that set PGPASSWORD for convenience.

Check in 10 seconds

Look at the environment of the running Patroni process on every node:

sudo tr '\0' '\n' < /proc/$(pgrep -f 'bin/patroni' | head -1)/environ | grep -E '^PG|^PATRONI_'

Any PGPASSWORD, PGUSER, PGHOST or PGSERVICE line is a problem. So is any unexpected PATRONI_* line: Patroni reads every PATRONI_-prefixed variable as configuration, and it overrides patroni.yml. That's a related trap we also hit, with a PATRONI_LOG_DIR from a script sending logs somewhere unexpected.

The fix

Start Patroni only through systemd, with a clean environment:

    [Service]
    User=postgres
    Environment=PATH=/opt/patroni/bin:/usr/local/bin:/usr/bin:/bin
    ExecStart=/opt/patroni/bin/patroni /etc/patroni/patroni.yml
    KillMode=process

Remove any PGPASSWORD from Environment= and EnvironmentFile= lines, then systemctl daemon-reload && systemctl restart patroni. PostgreSQL keeps running while Patroni restarts.
On the stuck node, restarting Patroni is usually enough: it retries pg_rewind with the right password. If the rewind can't work any more (the WAL it needs has been recycled), rebuild it: patronictl reinit <cluster> <node>.

Prove it's fixed

A switchover won't prove anything here; only a crash will. In a test environment, kill the leader's Patroni and PostgreSQL with SIGKILL while writing, wait for the new leader, restart the old node, and confirm it comes back as a streaming replica. That's exactly what our failover-drill --mode crash automates. After the fix, our lab's crashed leaders rejoined in about 34 seconds, every time.

This trap is handled in the Twinhull HA Kit

The kit's systemd units start Patroni with a clean environment, preflight warns about exported PG* variables, and the config generator refuses PATRONI_* names. The crash drill proves the rewind works before production needs it.

See the kit

Originally published at twinhullhq.com. The health-check script used in these tests is free and open source: github.com/twinhullhq/ha-check. If you run Patroni in production, the full kit (templates, failover drills, DR runbook, 3-node lab) is at twinhullhq.com.

Patroni “Failed to get list of machines… 'int' object has no attribute 'get'” with etcd on Ubuntu 24.04

Twinhull — Thu, 01 Oct 2026 13:00:00 +0000

Tested on: Patroni 4.1.5, etcd 3.4.30 → 3.6.15

You installed etcd with apt, started a healthy 3-member cluster, and pointed Patroni at it. Patroni never starts PostgreSQL. It just logs the same error every five seconds. The problem is the etcd version, not your configuration.

The symptom

ERROR: Failed to get list of machines from http://127.0.0.1:2379/v3: AttributeError("'int' object has no attribute 'get'")
ERROR: Failed to get list of machines from http://127.0.0.1:2380/v3: AttributeError("'int' object has no attribute 'get'")
INFO: waiting on etcd

Meanwhile etcd itself looks perfectly healthy, which is what makes this confusing:

$ etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.1ms

The cause

Patroni's etcd3 backend doesn't use gRPC. It talks to etcd's JSON gateway over plain HTTP, the /v3/… endpoints. Ubuntu 24.04's etcd-server package is etcd 3.4.30. On that build, the endpoint Patroni calls first to discover cluster members doesn't answer the way Patroni expects:

$ curl -s http://127.0.0.1:2379/version
{"etcdserver":"3.4.30","etcdcluster":"3.4.0"}

$ curl -s -X POST http://127.0.0.1:2379/v3/cluster/member/list -d '{}'
404 page not found

Patroni gets back something that isn't the JSON object it expects, fails while parsing it, and retries forever. The error names a Python type, not etcd, which sends people looking in the wrong place.

The same request against etcd 3.6 returns the member list, and Patroni bootstraps immediately. We confirmed both on the same machine while testing the Twinhull kit.

Confirm it on your system

etcd --version | head -1                 # 3.4.x → this is your problem
curl -s -X POST http://<etcd-ip>:2379/v3/cluster/member/list -d '{}' | head -c 200

If the second command prints 404 page not found instead of JSON with a members array, upgrade etcd.

The fix: install etcd 3.5 or newer from the official release

Distribution packages lag behind. Install the upstream release binaries, which are statically linked and have no dependencies (current as of September 2026: 3.6.15; check github.com/etcd-io/etcd/releases):

sudo systemctl disable --now etcd                    # stop the distro etcd
sudo apt remove -y etcd-server etcd-client

ETCD_VER=v3.6.15
curl -fsSL -o /tmp/etcd.tgz \
  https://github.com/etcd-io/etcd/releases/download/$ETCD_VER/etcd-$ETCD_VER-linux-amd64.tar.gz
tar xzf /tmp/etcd.tgz -C /tmp
sudo install -m 755 /tmp/etcd-$ETCD_VER-linux-amd64/{etcd,etcdctl,etcdutl} /usr/local/bin/
etcd --version | head -1                             # etcd Version: 3.6.15

Then run etcd from /usr/local/bin/etcd with your own systemd unit and a config file (--config-file /etc/etcd/etcd.conf.yml).

Upgrading an existing 3.4 cluster? etcd supports upgrading one minor version at a time: 3.4 → 3.5 → 3.6, member by member. For a new Patroni cluster that has never stored anything, it's simpler to stop all members, wipe their data directories, and start fresh on 3.6.

Two more things while you're here

Don't switch Patroni to the old etcd (v2) backend to work around this. The v2 API is deprecated, and etcd 3.6 no longer serves it.
Use etcdctl without a proxy. If your servers export HTTPS_PROXY, add the etcd addresses to NO_PROXY, or health checks may go through the proxy and time out.

Caught automatically in the Twinhull HA Kit

The kit's preflight check fails on etcd versions older than 3.5 before you install anything, and the guide installs the official 3.6 release with a hardened systemd unit and optional mutual TLS.

See the kit