DEV Community

Diego Gil Serrano
Diego Gil Serrano

Posted on Originally published at highavailabilitydata.com Fully Autonomous

Why Patroni failover takes 26–49 seconds with default settings (and how to make it faster)

#postgres #database #devops #sre

You build a Patroni + etcd + HAProxy cluster, kill the primary in a test, and your write loop stops for half a minute or more. Nothing is broken. That is what the default settings are designed to do. This post explains where those seconds go and what you can change.

The three settings that matter

All three live in Patroni's dynamic configuration (stored in the DCS, changed with patronictl edit-config). From the official documentation:

Setting What it is Default Minimum
loop_wait "the number of seconds the loop will sleep" between HA cycles 10 1
ttl the TTL of the leader lock; "think of it as the length of time before initiation of the automatic failover process" 30 20
retry_timeout "timeout for DCS and PostgreSQL operation retries"; "DCS or network issues shorter than this will not cause Patroni to demote the leader" 10 3

The documentation also sets a hard rule when you change them:

loop_wait + 2 * retry_timeout <= ttl
Enter fullscreen mode Exit fullscreen mode

With defaults: 10 + 2 × 10 = 30 ≤ 30. The defaults sit exactly on the limit.

What happens when the primary dies

  1. While healthy, the primary renews the leader key in etcd on every HA loop, roughly every loop_wait (10 s).
  2. The primary crashes (power loss, kernel panic, hard network isolation). Nobody renews the key.
  3. The key lives until ttl (30 s) after its last renewal. Since renewals happen every ~10 s, the failure can land anywhere in that cycle: the key expires between ~20 s and ~30 s after the failure.
  4. A replica notices the key is gone on its next HA cycle (between 0 and loop_wait seconds later), checks it is healthy enough (for example within maximum_lag_on_failover), takes the key and promotes PostgreSQL.
  5. HAProxy, which polls each node's GET /primary (HTTP 200 only on the node "running as the primary with leader lock"), needs one or more successful checks on the new primary before it routes writes there. With checks every 2 s and a rise greater than 1, that adds a few more seconds.

Add up the pieces: 20–30 s for the key to expire, 0–10 s for a replica to notice, a second or two to promote, and a few seconds of health checks. You get the 26–49 s range you see with plain Patroni + etcd defaults and HAProxy checking every 2 s. The exact figure depends on where in the renewal cycle the failure happens, on how fast promotion completes, and on your health check settings.

Two important exceptions

  • Clean shutdown is fast. If you stop Patroni normally (systemctl stop patroni, a planned reboot) it releases the leader key on the way out, so a replica can take over without waiting for the TTL. That is why "I rebooted the primary and failover took 5 seconds" and "I pulled the power cable and it took 40" are both true.
  • A frozen primary is the slow case. A node that hangs (stuck I/O, process stopped) may keep its key until it expires, and it is still in HAProxy's backend until checks fail. This is the scenario to test, because it is the one that hurts most.

How to make it faster

1. Shrink ttl, loop_wait and retry_timeout together

For example:

# patronictl -c /etc/patroni/patroni.yml edit-config
loop_wait: 5
retry_timeout: 5
ttl: 20
Enter fullscreen mode Exit fullscreen mode

5 + 2 × 5 = 15 ≤ 20, so the rule holds, and the worst-case key expiry drops from ~30 s to ~20 s after the last renewal. ttl cannot go below 20.

The trade-off is real:

  • A shorter retry_timeout means a shorter etcd or network hiccup is enough to demote a healthy primary. If your etcd runs on slow disks or across a flaky link, you will trade slow failovers for unnecessary ones.
  • A shorter loop_wait means more frequent DCS writes and more work per node.

Measure your etcd latency (etcdctl check perf, fsync metrics) before going aggressive.

2. Tune the HAProxy health checks

The usual Patroni HAProxy backend looks like this:

backend primary
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 2s fall 3 rise 2 on-marked-down shutdown-sessions
    server db1 10.0.0.11:5432 check port 8008
    server db2 10.0.0.12:5432 check port 8008
Enter fullscreen mode Exit fullscreen mode

inter × rise is the minimum time before the new primary receives traffic, and inter × fall the time before the old one is taken out. on-marked-down shutdown-sessions closes existing connections to a node marked down, so clients do not keep writing into a dead socket. Lowering inter to 1 s is cheap; it only adds HTTP requests to the Patroni REST API.

3. Make clients reconnect quickly

Database failover time is not application recovery time. Check your driver's connect timeout, TCP keepalives and retry logic. A pool that waits 2 minutes on a half-open TCP connection will hide every second you saved in Patroni.

4. Handle the frozen primary explicitly

Lower TTLs help, but a hung node is a detection problem, not just a timer problem. Options include Patroni's watchdog support (so a node that stops running its HA loop is reset) and external fencing that removes the old primary from every load balancer as soon as a failover is decided.

5. Measure, do not assume

Run a write loop through the load balancer and look at the gap:

while true; do
  psql -h VIRTUAL_IP -p 5000 -U app -d app -qAt \
    -c "INSERT INTO ha_test DEFAULT VALUES RETURNING now()" 2>&1
  sleep 0.5
done
Enter fullscreen mode Exit fullscreen mode

Then try a switchover, a clean stop, a kill -9, and a SIGSTOP of Patroni and PostgreSQL on the primary (test environments only). Each one exercises a different path.

Summary

  • With defaults, the leader key expires 20–30 s after a hard failure; add detection, promotion and health checks and you land in 26–49 s.
  • Any change must respect loop_wait + 2 * retry_timeout <= ttl, and ttl cannot go below 20.
  • Faster settings increase the risk of demoting a healthy primary on short DCS hiccups.
  • HAProxy check intervals and client reconnect behaviour matter as much as Patroni's timers.

Disclosure: I work on HighAvailabilityData, a free on-premise console from NUTRADELIS S.L. for Patroni clusters. Instead of only waiting for the TTL, it lets the HAProxy instances and the console vote: if a strict majority sees the primary down for lb_failover_after (10 s by default), it promotes the healthy replica with the most WAL applied through Patroni's POST /failover, releases the stale leader key with a compare-and-delete and puts the old primary in maintenance on every HAProxy. In our tests writes came back in about 15–20 s; measure in your own environment. The original article includes a step-by-step test plan.

Sources

Top comments (0)