Sev1: Redis master is down and Sentinel won't fail over

#Symptom

• Clients: Connection refused, timeouts, or READONLY You can't write against a read only replica.
• redis-cli -h <master> ping fails.
• Sentinel still reports the old master.

#Step 1: ask Sentinel what it sees

redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
redis-cli -p 26379 SENTINEL master mymaster        # flags, num-other-sentinels, quorum
redis-cli -p 26379 SENTINEL replicas mymaster       # replica states and flags
redis-cli -p 26379 SENTINEL sentinels mymaster      # other sentinels it knows about
redis-cli -p 26379 SENTINEL ckquorum mymaster

Check the Sentinel log (/var/log/redis/sentinel.log) for +sdown, +odown, +try-failover, -failover-abort-no-good-slave.

#Step 2: match the cause

#Replica eligibility

A replica is skipped if:

• replica-priority 0 (never promote),
• it's been disconnected from the master too long (down-after-milliseconds × 10 plus time since the master was marked down),
• it's flagged s_down or disconnected itself.

redis-cli -h replica1 INFO replication     # master_link_status, master_last_io_seconds_ago
redis-cli -h replica1 CONFIG GET replica-priority

#Step 3: force a failover

If sentinels are healthy but you need to move now (or they elected nothing because the master is "up" but broken):

redis-cli -p 26379 SENTINEL failover mymaster

This forces a failover without needing agreement from other sentinels. Then:

redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster   # new address
redis-cli -h <new-master> INFO replication                       # role:master

#Step 4: no sentinel majority at all

Promote a replica by hand and repoint the others:

redis-cli -h replica1 REPLICAOF NO ONE
redis-cli -h replica2 REPLICAOF replica1 6379

Then fix Sentinel: stop all sentinels, correct the sentinel monitor mymaster <replica1-ip> 6379 2 line in each sentinel.conf (Sentinel rewrites this file, so edit it with Sentinel stopped), and start them again.

Warning

If the old master comes back, it might still think it's the master and accept writes from clients connected directly to it. Sentinel will reconfigure it as a replica when it sees it. If sentinels are broken, stop it yourself or set it as a replica before re-enabling traffic.

#Step 5: clients

The most common "Sentinel didn't work" problem is clients that connect to a fixed master IP instead of asking Sentinel:

• Use a Sentinel-aware client config (list of sentinel addresses + master name), e.g. redis-py Sentinel([...]).master_for('mymaster'), Lettuce/Jedis sentinel pools.
• Clients must reconnect on READONLY errors. Many libraries need that enabled.

#Prevent it next time

• At least 3 sentinels on separate hosts/zones; quorum 2.
• down-after-milliseconds tuned (e.g. 5000) so failures are detected quickly but not on short blips.
• Replicas with sensible replica-priority; at least one replica per zone.
• min-replicas-to-write / min-replicas-max-lag to limit writes lost in a partition.
• Quarterly failover test with SENTINEL failover in staging.