Symptom
$ patronictl -c /etc/patroni/patroni.yml list
+ Cluster: pg-prod ------+---------+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------+---------------+---------+-----------+----+-----------+
| pg1 | 10.0.1.11 | Replica | running | 7 | 0 |
| pg2 | 10.0.1.12 | Replica | running | 7 | 0 |
| pg3 | 10.0.1.13 | Replica | stopped | | unknown |
+--------+---------------+---------+-----------+----+-----------+
No Leader row. HAProxy health checks (/primary) fail on all nodes, so writes stop.
Step 1: is the DCS healthy?
Patroni needs the DCS (etcd/Consul) to hold the leader key. Without a DCS quorum, Patroni demotes the primary so a split-brain can't happen.
etcdctl --endpoints=http://10.0.1.11:2379,http://10.0.1.12:2379,http://10.0.1.13:2379 endpoint health
etcdctl --endpoints=... endpoint status -w table
journalctl -u patroni --since "30 min ago" | grep -iE "dcs|etcd|leader|lock"
| Finding | Meaning |
|---|---|
| Fewer than a majority of etcd members healthy | No quorum: fix etcd first (Step 2) |
| etcd healthy, Patroni logs "failed to acquire lock" / no candidates | Failover rules block every node (Step 3) |
Maintenance mode: on in patronictl list | Cluster paused (Step 4) |
Step 2: restore etcd quorum
- Start stopped etcd members:
systemctl start etcd. - Check disk space and time sync on etcd nodes (
chronyc tracking). etcd is sensitive to slow disks and clock problems. - If a member is permanently lost and quorum can't be reached, follow the etcd disaster-recovery procedure (restore a snapshot /
--force-new-clusteron one member). That's a significant step: do it on purpose, following etcd's docs for your version.
Once etcd has quorum, Patroni normally elects a leader within one or two loop_wait cycles. Run patronictl list again.
Step 3: no candidate qualifies
Patroni won't promote a replica that breaks its rules:
| Rule | Check |
|---|---|
maximum_lag_on_failover | Replica lag above the limit (data loss protection) |
Synchronous mode (synchronous_mode: true / synchronous_mode_strict) | Only the last known sync standby may be promoted |
nofailover: true tag | Node excluded from becoming leader |
| Timeline / WAL position | Replica isn't healthy or is behind |
patronictl -c /etc/patroni/patroni.yml show-config
patronictl -c /etc/patroni/patroni.yml history
If the best node is blocked only by a lag limit and you accept the possible data loss, promote it manually:
patronictl -c /etc/patroni/patroni.yml failover pg-prod --candidate pg1 --force
(Older versions take the cluster name as a positional argument; check patronictl failover --help.)
Caution
Forcing a candidate that is behind loses the transactions it doesn't have. Check lag (
Lag in MB) and record the decision.
Step 4: maintenance mode
patronictl -c /etc/patroni/patroni.yml resume pg-prod
In paused mode Patroni doesn't manage PostgreSQL. If someone stopped the primary while paused, start PostgreSQL on it manually or resume and let Patroni decide.
Step 5: confirm
patronictl -c /etc/patroni/patroni.yml list # one Leader, replicas streaming
curl -s http://10.0.1.11:8008/primary -o /dev/null -w "%{http_code}\n" # 200 on the leader
Test a write through HAProxy / the application endpoint.
Prevent it next time
- Three or five etcd members on separate hosts/zones, not on the same disk as PostgreSQL WAL.
- Monitoring on etcd health and Patroni REST endpoints, not just PostgreSQL.
- Review
maximum_lag_on_failoverand synchronous mode with the business: zero data loss and automatic availability pull in opposite directions. - Practise a leader loss in staging every quarter.